There appears to be an interesting behavior in XenDesktop 7.x that only occurs after an MCS Image Update process has completed where, repeatedly, a task to delete a non-existent virtual disk fails. The user account that XenDesktop uses to connect to VMware vCenter Server is shown as the initiator of the action, so there is little doubt what service causes the error.
This … consistently occurs after every image update.
Unfortunately, there is no way to control this task from Citrix Studio and PowerShell must be used. Fortunately for me, I sat in Adam Platt‘s PowerShell Masterclass and immediately got to work! The hardest part was figuring out which cmdlet was used for MCS Tasks. After some digging through PowerShell ISE, I found the ProvTask cmdlet, but initially dismissed it because I thought, “This is MCS, not Provisioning Services!”. After some quick googling, I was surprised to find out that I was wrong and MCS uses that cmdlet.
The following two commands will first stop the task from running, and second delete the task. If you so desire, you can save this as a PowerShell script, but don’t forget to add the Citrix Snap-in first.
This behavior consistently occurs after every image update. Hopefully this will be resolved in a future version.
More often then not, AppSense Environment Manager (EM) is blamed for poor logon performance. But with such limited logging, how can you tell if there is any merit to this accusation? Luckily, AppSense includes an application for creating and managing EM logs. Before you can analyze any logs, you must first install EM Tools which can be found in the DesktopNow installation media:
Once installed, AppSense EM Debug Setup Tool must be configured.
Once you have clicked the monitor icon to enable logging, then specified a path for the logs and enabled any settings you require, click ‘OK’. You will be prompted that the User Virtualization service will be restarted, so click ‘Yes’. Here are some considerations when enabling debug logging:
The EM logon process is handled by EmUser.exe, so when analyzing EM logon performance you must use the log created with that name. At first, you may notice multiple logs formatted EmUser-x64_Session-2_Pid-nnnn, where nnnn is the session Process ID. Once you have determined which EmUser log is relevant to your debugging, make a copy of it for analyzing (But don’t open it – the format will probably make very little sense).
EM Debug Log Parser
The best tool to review this log is not included in the installation of EM Tools, however it can be downloaded from AppSense Exchange. If you do not have an account with that site yet, do it now! After downloading and opening EM Log Parser, drag the copy of the EmUser log into the EM Log Parser window. Depending upon the size of the EmUser log, the total duration of importing and formatting will vary.
I find EM Debug Log Parser a great way to show when AppSense is not the reason for poor logon performance.
Once the import is complete, click View > Timeline View. Right away, you can see which conditions/actions/triggers are responsible for causing performance issues. Moreover, I find it a great way to show when AppSense is not the reason for poor logon performance. In any case, the documentation included with the download provides great detail on how to use EM Log Parser.
Desktop Director 2.x, included with Citrix XenDesktop 5.x, is an incredibly useful tool for Operations staff. As a matter of fact, it is even more useful than Desktop Studio, and that is because it not only includes information provided by the VDA, it also includes information from the Windows Remote Management (WinRM) service.
When WinRM is not functional, Desktop Director is unable to provide information that is extremely useful to Operations staff, including metrics for:
Is WinRM configured?
Is WinRM listening for connections?
Is the WinRM service running?
… and of course I check Windows Firewall with Advanced Security Inbound Rules for all Windows Remote Management (HTTP-In) being enabled.
But for one particular client, those troubleshooting steps were not enough and Desktop Director was failing to connect to every single virtual desktop’s WinRM service in the Site. All the commands yielded information indicating that the WinRM service was functional. Specifically, this is the error that Desktop Director reported:
Failed to retrieve data: Machine unresponsive or reported an error (error code 105). View server event logs for further information.
CTX131197 documents this issue, however, I felt it was extremely unlikely that every virtual desktop had a corrupt WinRM listener. I was fairly certain the issue was not a Citrix issue, so I started from the beginning and asked myself, “How does a server establish a remote connection with an endpoint using WinRM?” Before any remote connection can be established, the server must know how to resolve the endpoint’s address. In the case of WinRM, the Service Principal Name (SPN) is what is used to resolved the endpoint’s address through Kerberos name-suffix routing.
A disjoint suffix used by domain members matches an Active Directory domain name in this or another forest. This breaks Kerberos name-suffix routing.
How to check if the WinRM SPN is configured properly:
I looked for two entries starting with WSMAN/… and found that both entries were there. However, I noticed that the FQDN entry had a different domain name than the Forest that the virtual desktop was joined to. All the virtual desktops in the XenDesktop Site had this mismatch, which led me to this Microsoft article. While Disjoint Namespaces are supported by Microsoft technologies (like WinRM), it is only supported under specific situations. Unfortunately, the DNS Namespace these virtual desktops were assigned to was a shared DNS namespace with desktops in a different Forest.
Once I determined the root cause, and realized the problem was 100% due to architecture, I could only submit my findings and propose a change be made to the architecture. I can only wonder why that architecture decision was made in the first place, but I have a feeling I will come across it again!
After setting up my first Nutanix block for a customer, I was excited to view the dashboard that Controller VMs provided. This particular Nutanix block shipped with NOS 3.1.3. While I found the real-time information presented to be useful, it was quite limited. I held off on upgrading because I was more focused with migrating the existing environment to Nutanix. Then the client’s busy season hit full force, so I postponed the upgrade.
The Upgrade Process
In the beginning of July, Nutanix announced NOS 4.0.1 was available for download and installation. I had been eager to upgrade the client’s Nutanix block and things were finally starting to slow down enough for the client that I could schedule the upgrade.
[To create] an account in the Support Portal, your e-mail domain must match the organization the serial number is registered to.
Nutanix offers some of the best documentation I have ever used, so I was expecting a very pain-free upgrade process. The upgrade guide did not disappoint, and within a half hour all nodes were updated to NOS 4.0.1 via the rolling upgrade process. While I am not a fan of performing all upgrade steps from a command line, the documentation was clear enough to make it not feel like a chore.
The only hiccup came when attempting to generate a license file for the cluster. The documentation provided steps that directed the administrator to use the Nutanix Support Portal. As a partner, I only have an account with the Nutanix Partner Portal and attempts to create an account with the Support Portal failed. I created an incident and Nutanix support was quick to get back to me. Apparently, when creating an account in the Support Portal, your e-mail domain must match the organization that the block’s serial number is registered to.
Unfortunately, not having an e-mail address with the client’s e-mail domain meant not being able to generate a license file. Fortunately, an operations member of my team did have an e-mail address with the client. We were finally able to create a support account and generate the license file. Nutanix support informed me license generation would be coming to the Partner Portal soon.
After the Upgrade
As recommended by Steven Poitras (@StevenPoitras), I upgraded the Controller VMs to 32 GB of memory and enabled Shadow Clones. The next day, I noticed a 2-3 second improvement in logon times in the Citrix XenDesktop 7.x environment. Everything felt noticeably faster, particularly during the MCS image update process. If you are currently using MCS to manage your application/desktop deployment and have Nutanix, following Steven’s recommendations will make managing the image update process a much less time consuming.
Coming from Nutanix 3.1.3, Prism Central is a huge upgrade for detail-oriented administrators like me. The landing page presents the administrator with a view of the overall health of the cluster. Other section include real-time dashboards, tables and sometimes diagrams that can be manipulated to identify potential bottlenecks in performance. Read all about it here.
AppSense DesktopNow 8.5 introduced an additional configuration setting for deploying packages through Management Center called “Native Configuration file”. When a location is specified, packages are saved to this folder based on the configuration schedule. This new method is preferable for non-persistent images as changes to a configuration *no longer require an update to the master image.
When this option is selected, a warning will appear in the top of the screen:
Some assigned agents/configurations do not support native configurations.
I am not sure why AppSense’s warning message is so cryptic here as only one package will not be deployed as a native configuration, the Performance Manager configuration. Also, all assigned agents are installed regardless of the configuration setting chosen.
I am certain there is a better way of presenting this information to the administrator than how it is currently. That said, the functionality of using ‘Native Configuration file’ is solid with one exception: Once this option is chosen and packages have been deployed, do not change the setting to ‘Windows Installer MSI file’. When the CCA reports on the status of the deployed packages to Management Center, the location will not be in the Add/Remove Programs list. In some cases, Management Center will display the Deployed % as less than 100% and result in a frustrating experience!
AppSense Tip: If you select ‘Native Configuration file’ in the Deployment Group Settings, commit to it; otherwise EM Policy may not load!
— Stephen Stetler (@Kingston2) August 4, 2014
*If a change is made to the Computer Startup trigger (Environment Manager), then it is best to update the master image with the new configuration package.
I am mostly posting this blog post out of frustration that AppDNA works this way…
#AppDNA tip: Another consideration re: Integrated Login, even on the latest version there is no Integrated Login for the web client.
— Stephen Stetler (@Kingston2) August 28, 2014
Single Sign-On (SSO) with AppDNA 7.5 through Active Directory is possible through importing, then linking the AD Account and having the user select ‘Integrated Login’ from the client application. This is ideal because an AppDNA system administrator does not need to maintain user credentials.
— Stephen Stetler (@Kingston2) August 28, 2014
The “Production” configuration of AppDNA requires the IIS Role to be installed, so one would think that the web client would have the ability to support SSO as well. Not the case, unfortunately, as there is no option for Integrated Login, nor is there support for manually entering AD account credentials.
No Other Choice
I ended up clicking ‘Import from AD’ from the User Management window to get the user account, then un-checking the ‘AD Linked’ check box to allow the user access to the web client.
To Citrix’s credit, the web client does state that it is “Version 1.0”.
I am currently working on a project that involves moving an Machine Creation Services (MCS) master image to Provisioning Services (PVS). While I found Citrix’s White Paper to be particularly useful in getting started, the process may prove to be a little more complicated in conjunction with personal vDisk (PvD).
Here is a description of the technology used for this particular environment:
Importance of User Profile Management
Now this part is particularly important to note – if you are not using a profile management solution like Citrix User Profile Manager (UPM) or AppSense Personalization Server, and simply keeping the user’s profile data stored on the PvD, then you should rethink your migration. Keeping user profile data on the PvD, which is a static mapping to the virtual desktop, makes this a persistent virtual desktop and not something I would recommend after dealing with many PvD corruptions.
As PvD is not mentioned in Citrix’s White Paper, I tried very hard to preserve the PvD in the migration process. Unfortunately, I was unable to as the XenDesktop Wizard in PVS wanted to create its own PvD to manage.
It goes without saying that any User Installed Applications (UIA) were lost in the process. However, thanks to having a profile management solution in place, the PvD became mostly disposable and made for a mostly non-persistent virtual desktop. And once you have a profile management solution in place, you should check out Nick Rintalan’s post on modifying the split in PvD.
Prior to performing any migration, consider the following points that can result in either poor performance or will break the image altogether.
However, if you are going to attempt to implement PVS 7.1 to your existing XenDesktop 5.6 Site, think again…
“Provisioning Services 7.x only supports XenDesktop 7.x.” http://t.co/MksCUoAnX1
— Stephen Stetler (@Kingston2) August 14, 2014
Your upgrade path may not be as smooth as you might think depending upon the size of your deployment. Here is a quick road-map of the upgrade path I put together for a client.
The performance improvement that I saw from the upgrade in my lab environment was noticeable, however until I run it at the client I don’t want to post any numbers.
Stay tuned for my follow up blog post where I detail my experience migrating the master image from MCS to PVS with PvD, which is not as smooth as you might hope.
Want to learn more about the wC enhancement? Watch this great presentation I linked to about a month ago.
— Stephen Stetler (@Kingston2) August 5, 2014