VMware vCenter 6 Web Console doesn’t load or very slow

Environment

VMware vCenter 6.0.0 Build 4541948 (Update 2a) – could effect other vCenter versions also.

Issue:

After a period of a week or more the vCenter Web Console slows down dramatically to the point where the website doesn’t load anymore and the browser times out.

Looking into the /var/log/vmware/vsphere-client/logs/vsphere-client-virgo.log file you it displays the following Out Of Memory error:

Run: less vsphere-client-virgo.log

“[2017-03-15T13:59:07.853+13:00] [ERROR] ing.timer.TimerFactoryBean#0 o.s.scheduling.timer.MethodInvokingTimerTaskFactoryBean Invocation of method ‘update’ on target class [class com.vmware.vise.vim.cm.healthstatus. HealthStatusUpdater] failed java.lang.OutOfMemoryError: PermGen space”

Solution:

1. (Workaround) Reboot the vCenter, according to VMware Support you should reboot it often to ensure the vCenter doesn’t run out of memory.
2. (Permanent fix) Increase the memory size for vspherewebclientsvc in the service-layout.mfx file from 256 to 512.
   1. Use WinSCP to connect to vCenter Server
   2. Browse to and edit: /etc/vmware/service-layout.mfx
   3. Find the line with the firstboot_name ngc_firstboot
   4. Under the column ‘MaxPermMB’ change the value from 256 to 512
   5. Save service-layout.mfx file
   6. SSH to vCenter Server
   7. Run: /etc/init.d/service vsphere-client restart

Deploy ESXi 6.0 upgrade to Update 2 Fails

Issue:

When attempting to remediate an ESXi host with 6.0 Update 2, the following error displays in Update Manager:

“The host returns esxupdate error code:14. There is an error when resolving dependencies. Check the Update Manager log files and esxupdate log files for more details.”

In /var/log/esxupdate.log

esxupdate: ERROR: DependencyError: VIB VMware_bootbank_vsanhealth_6.0.0-3000000.3.0.2.43.4064824 requires esx-base >= 6.0.0-2.43, but the requirement cannot be satisfied within the ImageProfile.

Solution:

There is a dependency for Update 2 that needs installing first,  weirdly enough the dependency is included in Update 2.

Resolution:

• Enable ESXi Shell and SSH on ESXi host
• Download Update 2 from VMware Patches
• Copy zip file to /vmfs/volumes/datastore (file is too large to copy to /var/tmp)
• Use Putty to connect to ESXi host
• Run: esxcli software vib install -n esx-base -n vsan -n vsanhealth -d /vmfs/volumes/datastore1/update-from-esxi6.0 6.0_update02.zip

-n = Specifies VIBs from a depot, using one of the following forms: name, name:version, vendor:name, or vendor:name:version.

-d = Specifies full remote URLs of the depot index.xml or server file path pointing to an offline bundle .zip file.

• Reboot the ESXi host.
• To confirm the components installed correctly run: esxcli software vib list
• These are the dependencies and versions installed:

• Once the server is back online now use Update Manager to remediate Update 2 (and any additional patches) onto ESXi host.

NetScaler SDX – Unknown Device, not supported by Command Center

Environment:

• Citrix NetScaler SDX appliances
• Citrix Command Centre 5.x (and think it occurs in version 4.x also) on Windows Server 2008 R2

Issue:

While running a NetScaler SDX discovery in Command Centre the discovery fails with the following error message:

“Unknown device ; <sdx_ip_address> ; not supported by command center”

(The discovery of the NetScaler VPX instances discovered fine independently)

– Checked discovery logs and found the below error:

Troubleshooting Steps:

– Checked from the Command Centre server and was able to ping and SSH to SDX Management IP with same credentials used in the Command Centre NetScaler SDX profile.
– In the C:\Program Files (x86)\Citrix\Citrix Command Center\logs\discoveryLogs.txt log file, I could see it was trying to add the NetScaler SDX as “CBSDX” (CloudBridge SDX I assume).

• INVENTORYERR: Device not recognized as CBSDX. SYSID null <sdx_ip_address>
• INVENTORYERR: Exception while login to =<sdx_ip_address>
• Localized : English : nsroot failed to login

Resolution:

Within Command Centre,

• On the Administration tab,
• Under Settings,
• Click SSL Settings and then
• Select the Enable TLSv1 option
• Restart the Command Centre service.

NOTE: After restarting the Command Centre service and its running again, it can take a while before your able to log in again.

Check the service status in the C:\Program Files (x86)\Citrix\Citrix Command Center\logs\wrapper.log file

INFO | jvm 1 | 2016/05/23 14:47:00 | Starting up as Primary Server and correcting the Server information in the Database.
INFO | jvm 1 | 2016/05/23 14:47:00 |
INFO | jvm 1 | 2016/05/23 14:47:01 |
INFO | jvm 1 | 2016/05/23 14:47:01 | Starting Citrix Command Center “Primary” Server Modules, please wait
INFO | jvm 1 | 2016/05/23 14:47:15 |
INFO | jvm 1 | 2016/05/23 14:47:15 |
INFO | jvm 1 | 2016/05/23 14:47:15 |
INFO | jvm 1 | 2016/05/23 14:47:15 | Process : SyslogServerBE [ Started ]
INFO | jvm 1 | 2016/05/23 14:47:16 | Process : ProcessTest [ Started ]
INFO | jvm 1 | 2016/05/23 14:47:17 | Process : ShutdownDatabaseServer [ Started ]
INFO | jvm 1 | 2016/05/23 14:47:19 | Process : CLIRMIService [ Started ]

If you see a lot of [Failed] statues, restart the Command Centre server.

 

“Please wait for User Profile Services” – Logon delay

Environment

• Windows Server 2008 R2
• Citrix XenApp 6.5 RU5

Issue

During users logon to XenApp published applications and published desktops there can be a lengthy delay on “Please wait for the User Profile Services”, it can hang on this for up to 30 secs.

Solution

In the Group Policy Management browse to:

• Computer Configuration
  • Administrative Templates
    • System
      • User Profiles

Modify the policy setting “Set maximum wait time for the network if a user has a roaming user profile or remote home directory” to “Enabled” and set the value Wait for network for maximum (seconds) to 0.

User session stuck on XenApp server

Environment

• Windows Server 2008 R2
• Citrix XenApp 6.5

Issue

User’s session is stuck on XenApp server, logging off from AppCenter doesn’t remove them.

Resolution 

1.Log onto the RDS server and launch Remote Desktop Services Manager or connect via another server.

2.Get the Session ID of the hung session from the Session tab:

• Session: Disconnected
• User: <Blank>
• ID: <number>
• State: Disconnected
• ClientName: <Blank>
• LogOnTime: Unknown

3.Click on Processes tab

4.Click on ID column to sort the processes by ID

5.Find the hung ID number

6.There is normally 3 processes visible

• LogonUI.exe
• Winlogon.exe
• Csrss.exe

7.Right-click on winlogon.exe and End Process, once you end this process the other 2 will disappear.

8.DO NOT end process on csrss.exe, this will crash and reboot the RDS server

9.Now the user has cleared from the RDS server and from AppCenter.

Windows cannot connect to the printer – error 0x00000006

Environment:

• Windows Server 2008 R2 SP1
• Citrix XenApp 6.5 RU5

Issue:

Attempting to add a network printer the following error is displayed

• “Connect to Printer. Windows cannot connect to the printer. No printers were found”
• Operation failed with error 0x00000006

Resolution:

Microsoft’s resolution for this issue is to install KB2778831, however there are instance of this hotfix being applied and the issue still occurring.

Manual Fix:

1. Open Registry Editor
2. Browse to HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Print\Providers
3. Back up then delete the Client Side Rendering Print Providers key
4. Restart the Printer Spooler service
5. Printers should now add successfully

Exchange RPC Timeouts when using Citrix NetScaler

Updated 9 June 2016

Environment

---

Exchange 2010

Citrix NetScaler 10.x/11.x

Issue

---

• Users creating a new email and clicking on To: to browse the Global Address List would receive an Exchange timeout error message.
• Accepting meeting invites puts the appointment in the recpients calendar but not the organisers and no email gets sent to the organiser of the appectance. Takes clicking the Accept button multiple times before it processes the acceptance.

Resolutions

---

Create a new TCP Profile

GUI:

• System
• Profiles
• Add
• Name: RPC_Profile
• MSS: 1460
• Keep-Alive Probes – Enable
• Connection idle time before sending probe (secs) – 30
• Keep-alive probe interval (secs) – 20
• Everything else defaults

Bind this profile to the RPC Load Balanced VIP

• Traffic Management
• Load Balancing
• Virtual Servers
• <Exchange RPC LB VIP Name>
• Profiles > Add
• TCP Profile – Select RPC_Profile
• Ok
• Done

CLI:

add ns tcpProfile RPC_Profile_RPC -mss 1460 -KA ENABLED -KAconnIdleTime 30 -KAprobeInterval 20

add lb vserver <Exchange RPC LB VIP Name> TCP <IP_Address> * -persistenceType SOURCEIP -timeout 150 -cltTimeout 180 -tcpProfileName RPC_Profile

 

Additional Timeout customization recommendations

• Add ‘Thresholds & Timeouts node to the RPC ServiceGroup
  • Configure ‘Client Idle Time-out’ setting to: 28800
• Disable AppFlow on the RPC VIP and ServiceGroup
• Ensure ‘Surge Protection’ is disabled on the RPC ServiceGroup