@kingston2 Blog

After MCS Image Update, ‘Delete Virtual Disk’ Tasks Repeatedly Fail

There appears to be an interesting behavior in XenDesktop 7.x that only occurs after an MCS Image Update process has completed where, repeatedly, a task to delete a non-existent virtual disk fails. The user account that XenDesktop uses to connect to VMware vCenter Server is shown as the initiator of the action, so there is little doubt what service causes the error.

This action, and subsequent failure, will quickly fill up the 'Recent Tasks' pane.

This action, and subsequent failure, will quickly fill up the ‘Recent Tasks’ pane.

This … consistently occurs after every image update.

Unfortunately, there is no way to control this task from Citrix Studio and PowerShell must be used. Fortunately for me, I sat in Adam Platt‘s PowerShell Masterclass and immediately got to work! The hardest part was figuring out which cmdlet was used for MCS Tasks. After some digging through PowerShell ISE, I found the ProvTask cmdlet, but initially dismissed it because I thought, “This is MCS, not Provisioning Services!”. After some quick googling, I was surprised to find out that I was wrong and MCS uses that cmdlet.

The following two commands will first stop the task from running, and second delete the task. If you so desire, you can save this as a PowerShell script, but don’t forget to add the Citrix Snap-in first.

  • Get-ProvTask | where {$_.Type -eq “DisusedImageCleanup” -and $_.Status -ne “Finished”} | Stop-ProvTask
  • Get-ProvTask | where {$_.Type -eq “DisusedImageCleanup” -and $_.WorkflowStatus -eq “Terminated”} | Remove-ProvTask

This behavior consistently occurs after every image update. Hopefully this will be resolved in a future version.

Debugging AppSense Environment Manager Logon Performance

More often then not, AppSense Environment Manager (EM) is blamed for poor logon performance. But with such limited logging, how can you tell if there is any merit to this accusation? Luckily, AppSense includes an application for creating and managing EM logs. Before you can analyze any logs, you must first install EM Tools which can be found in the DesktopNow installation media:

<root>\Software\Products\EnvironmentManagerTools32[64].msi.

Once installed, AppSense EM Debug Setup Tool must be configured.

AppSenseEMDebugTool

AppSense Environment Manager Debug Setup Tool must be configured before any logs are generated.

Once you have clicked the monitor icon to enable logging, then specified a path for the logs and enabled any settings you require, click ‘OK’.  You will be prompted that the User Virtualization service will be restarted, so click ‘Yes’. Here are some considerations when enabling debug logging:

  • The slider for Log Detail is defaulted to ’10’, which means the logs will be extremely verbose. Depending upon the level of detail required, ’10’ may not be necessary.
  • If you did not change the detail level, expect these logs to grow quite big – particularly if the endpoint is a shared hosted server with multiple tenants.
  • In the case of pooled or streamed endpoints, a local path will result in the write cache/differencing disk as the location for the logs and, thus, the logs will not persist between reboots.
  • In the case of pooled or streamed endpoints, consider using a network path for the logs. If logging is not required, make sure to disable logging on the master image prior to deploying it into production.
  • If the endpoint is a virtual desktop with PvD, the logs will be committed to the personal vDisk application VHD and, by nature of the PvD, will persist. Depending upon the level of detail selected, this may be a nightmare to manage.

EmUser.exe

The EM logon process is handled by EmUser.exe, so when analyzing EM logon performance you must use the log created with that name. At first, you may notice multiple logs formatted EmUser-x64_Session-2_Pid-nnnn, where nnnn is the session Process ID. Once you have determined which EmUser log is relevant to your debugging, make a copy of it for analyzing (But don’t open it – the format will probably make very little sense).

EM Debug Log Parser

The best tool to review this log is not included in the installation of EM Tools, however it can be downloaded from AppSense Exchange. If you do not have an account with that site yet, do it now! After downloading and opening EM Log Parser, drag the copy of the EmUser log into the EM Log Parser window. Depending upon the size of the EmUser log, the total duration of importing and formatting will vary.

I find EM Debug Log Parser a great way to show when AppSense is not the reason for poor logon performance.

Once the import is complete, click View > Timeline View. Right away, you can see which conditions/actions/triggers are responsible for causing performance issues. Moreover, I find it a great way to show when AppSense is not the reason for poor logon performance. In any case, the documentation included with the download provides great detail on how to use EM Log Parser.

Duration of each step, and the entirety of the logon process, is easily represented in the Timeline View.

Duration of each step, and the entirety of the logon process, is easily represented in the Timeline View.

Desktop Director and Disjoint Namespace

Desktop Director 2.x, included with Citrix XenDesktop 5.x, is an incredibly useful tool for Operations staff. As a matter of fact, it is even more useful than Desktop Studio, and that is because it not only includes information provided by the VDA, it also includes information from the Windows Remote Management (WinRM) service.

Troubleshooting WinRM

When WinRM is not functional, Desktop Director is unable to provide information that is extremely useful to Operations staff, including metrics for:

  • vCPU,
  • Memory,
  • Hard disk,
  • personal vDisk profile size,
  • personal vDisk apps size,
  • ICA latency,
  • Citrix policies, and
  • Overall Activity.

Next to these measurements you will see WinRM_fail or redX, indicating some failure to connect to the WinRM service. To troubleshoot these issues, I ask the following questions through commands:

Is WinRM configured?

  • winrm get winrm/config

Is WinRM listening for connections?

  • netstat -ano | findstr 5985

Is the WinRM service running?

  • Get-Service WinRM

… and of course I check Windows Firewall with Advanced Security Inbound Rules for all Windows Remote Management (HTTP-In) being enabled.

But for one particular client, those troubleshooting steps were not enough and Desktop Director was failing to connect to every single virtual desktop’s WinRM service in the Site. All the commands yielded information indicating that the WinRM service was functional. Specifically, this is the error that Desktop Director reported:

Failed to retrieve data: Machine unresponsive or reported an error (error code 105). View server event logs for further information.

Root Cause

CTX131197 documents this issue, however, I felt it was extremely unlikely that every virtual desktop had a corrupt WinRM listener. I was fairly certain the issue was not a Citrix issue, so I started from the beginning and asked myself, “How does a server establish a remote connection with an endpoint using WinRM?” Before any remote connection can be established, the server must know how to resolve the endpoint’s address. In the case of WinRM, the Service Principal Name (SPN) is what is used to resolved the endpoint’s address through Kerberos name-suffix routing.

A disjoint suffix used by domain members matches an Active Directory domain name in this or another forest. This breaks Kerberos name-suffix routing.

How to check if the WinRM SPN is configured properly:

  • setspn -L hostname

I looked for two entries starting with WSMAN/… and found that both entries were there. However, I noticed that the FQDN entry had a different domain name than the Forest that the virtual desktop was joined to. All the virtual desktops in the XenDesktop Site had this mismatch, which led me to this Microsoft article. While Disjoint Namespaces are supported by Microsoft technologies (like WinRM), it is only supported under specific situations. Unfortunately, the DNS Namespace these virtual desktops were assigned to was a shared DNS namespace with desktops in a different Forest.

Once I determined the root cause, and realized the problem was 100% due to architecture, I could only submit my findings and propose a change be made to the architecture. I can only wonder why that architecture decision was made in the first place, but I have a feeling I will come across it again!

 

Upgrading to Nutanix NOS 4.0.1

After setting up my first Nutanix block for a customer, I was excited to view the dashboard that Controller VMs provided. This particular Nutanix block shipped with NOS 3.1.3. While I found the real-time information presented to be useful, it was quite limited. I held off on upgrading because I was more focused with migrating the existing environment to Nutanix. Then the client’s busy season hit full force, so I postponed the upgrade.

The Upgrade Process

In the beginning of July, Nutanix announced NOS 4.0.1 was available for download and installation. I had been eager to upgrade the client’s Nutanix block and things were finally starting to slow down enough for the client that I could schedule the upgrade.

[To create] an account in the Support Portal, your e-mail domain must match the organization the serial number is registered to.

Nutanix offers some of the best documentation I have ever used, so I was expecting a very pain-free upgrade process. The upgrade guide did not disappoint, and within a half hour all nodes were updated to NOS 4.0.1 via the rolling upgrade process. While I am not a fan of performing all upgrade steps from a command line, the documentation was clear enough to make it not feel like a chore.

The only hiccup came when attempting to generate a license file for the cluster. The documentation provided steps that directed the administrator to use the Nutanix Support Portal. As a partner, I only have an account with the Nutanix Partner Portal and attempts to create an account with the Support Portal failed. I created an incident and Nutanix support was quick to get back to me. Apparently, when creating an account in the Support Portal, your e-mail domain must match the organization that the block’s serial number is registered to.

Unfortunately, not having an e-mail address with the client’s e-mail domain meant not being able to generate a license file. Fortunately, an operations member of my team did have an e-mail address with the client. We were finally able to create a support account and generate the license file. Nutanix support informed me license generation would be coming to the Partner Portal soon.

After the Upgrade

As recommended by Steven Poitras (@StevenPoitras), I upgraded the Controller VMs to 32 GB of memory and enabled Shadow Clones. The next day, I noticed a 2-3 second improvement in logon times in the Citrix XenDesktop 7.x environment. Everything felt noticeably faster, particularly during the MCS image update process. If you are currently using MCS to manage your application/desktop deployment and have Nutanix, following Steven’s recommendations will make managing the image update process a much less time consuming.

SPConvo

Coming from Nutanix 3.1.3, Prism Central is a huge upgrade for detail-oriented administrators like me. The landing page presents the administrator with a view of the overall health of the cluster. Other section include real-time dashboards, tables and sometimes diagrams that can be manipulated to identify potential bottlenecks in performance. Read all about it here.

AppSense DesktopNow Native Configuration File Warning

AppSense DesktopNow 8.5 introduced an additional configuration setting for deploying packages through Management Center called “Native Configuration file”. When a location is specified, packages are saved to this folder based on the configuration schedule. This new method is preferable for non-persistent images as changes to a configuration *no longer require an update to the master image.

When this option is selected, a warning will appear in the top of the screen:

Some assigned agents/configurations do not support native configurations.

I am not sure why AppSense’s warning message is so cryptic here as only one package will not be deployed as a native configuration, the Performance Manager configuration. Also, all assigned agents are installed regardless of the configuration setting chosen.

AppSenseDesktopNowPMNativeConfFile

I am certain there is a better way of presenting this information to the administrator than how it is currently. That said, the functionality of using ‘Native Configuration file’ is solid with one exception: Once this option is chosen and packages have been deployed, do not change the setting to ‘Windows Installer MSI file’. When the CCA reports on the status of the deployed packages to Management Center, the location will not be in the Add/Remove Programs list. In some cases, Management Center will display the Deployed % as less than 100% and result in a frustrating experience!

*If a change is made to the Computer Startup trigger (Environment Manager), then it is best to update the master image with the new configuration package.

Citrix AppDNA Integrated Login

I am mostly posting this blog post out of frustration that AppDNA works this way…

Single Sign-On (SSO) with AppDNA 7.5 through Active Directory is possible through importing, then linking the AD Account and having the user select ‘Integrated Login’ from the client application. This is ideal because an AppDNA system administrator does not need to maintain user credentials.

The “Production” configuration of AppDNA requires the IIS Role to be installed, so one would think that the web client would have the ability to support SSO as well. Not the case, unfortunately, as there is no option for Integrated Login, nor is there support for manually entering AD account credentials.

No Other Choice

I ended up clicking ‘Import from AD’ from the User Management window to get the user account, then un-checking the ‘AD Linked’ check box to allow the user access to the web client.

To Citrix’s credit, the web client does state that it is “Version 1.0”.

Migrate MCS Master Image to PVS

I am currently working on a project that involves moving an Machine Creation Services (MCS) master image to Provisioning Services (PVS). While I found Citrix’s White Paper to be particularly useful in getting started, the process may prove to be a little more complicated in conjunction with personal vDisk (PvD).

Background:

Here is a description of the technology used for this particular environment:

  • VMware vSphere 5.1 Infrastructure,
  • Citrix XenDesktop 7.1 Site and
  • Windows 7 Professional Virtual Desktops with PvD.

Importance of User Profile Management

Now this part is particularly important to note – if you are not using a profile management solution like Citrix User Profile Manager (UPM) or AppSense Personalization Server, and simply keeping the user’s profile data stored on the PvD, then you should rethink your migration. Keeping user profile data on the PvD, which is a static mapping to the virtual desktop, makes this a persistent virtual desktop and not something I would recommend after dealing with many PvD corruptions.

As PvD is not mentioned in Citrix’s White Paper, I tried very hard to preserve the PvD in the migration process. Unfortunately, I was unable to as the XenDesktop Wizard in PVS wanted to create its own PvD to manage.

It goes without saying that any User Installed Applications (UIA) were lost in the process. However, thanks to having a profile management solution in place, the PvD became mostly disposable and made for a mostly non-persistent virtual desktop. And once you have a profile management solution in place, you should check out Nick Rintalan’s post on modifying the split in PvD.

Migration Process

Prior to performing any migration, consider the following points that can result in either poor performance or will break the image altogether.

  • VMXNET3 vNIC should be installed on the master image and any references to a different vNIC should be removed in Device Manager.
  • If you are going to create a wC overflow volume on the master image, pay special attention to the SCSI locations!
  • 256 MB RAM Cache for the wC is based off PVS 7.1 recommendations from Citrix.
  • Base virtual desktop memory is only 1 vCPU and 2 GB RAM because I am using AppSense Performance Manager to get more performance with less hardware.

ConversionMCStoPVS

 

Citrix PVS 7.1 Write Cache Enhancement Alone Worth the Upgrade

However, if you are going to attempt to implement PVS 7.1 to your existing XenDesktop 5.6 Site, think again…

Your upgrade path may not be as smooth as you might think depending upon the size of your deployment. Here is a quick road-map of the upgrade path I put together for a client. 

Roadmap

The performance improvement that I saw from the upgrade in my lab environment was noticeable, however until I run it at the client I don’t want to post any numbers. 

Stay tuned for my follow up blog post where I detail my experience migrating the master image from MCS to PVS with PvD, which is not as smooth as you might hope.

Want to learn more about the wC enhancement? Watch this great presentation I linked to about a month ago.