Guest

Preview Tool

Cisco Bug: CSCva09767 - IMP 10. / 11.x -- HDD & RAM Maxed After VMTools Upgrade (ESXi 5/6)

Last Modified

Aug 11, 2017

Products (1)

  • Cisco Unified Communications Manager IM & Presence Service

Known Affected Releases

10.0(1) 10.5(1) 10.5(2) 11.0(1) 11.5(1)

Description (partial)

=================
WORKAROUND
=================

I. During Outages --

[1] Automated Resolution Method:

a. From IMP CLI, run "utils os secure permissive" to disable selinux, which will prevent the problem from getting worse.

b. If the node is responding very slowly, attempt to restart the node, via "utils system restart" and when it restarts ensure the selinux has been disabled via "cisco utils os secure status" -- this should at least clear up the memory issues.

c. Resolution to this issue is also available via a standalone COP file "ciscocm.IMP_VMwareTools2016c.cop.sgn" published to software.cisco.com. Details on COP file: http://www.cisco.com/web/software/282074312/93622/ciscocm.IMP_VMwareTools2016cReadme.pdf


[2] Where a node rebuild is required (system has become unusable, unstable), see section II. Install 

For ESXi Update / Patch Scenarios Apply this COP file prior to restarting CUCM applications where automatic update could take place during power up or prior to initiating automatic vmware tools update from vi-clients.


II. Install Workaround
If you must Fresh Install / Re-Install IM&P 10.X or 11.X version that does NOT have this fix present on top of ESXi 5.5 or 6.0 which already has VMware Tools 10.X version bundled, you must follow these steps:

[1] After deploying the VM using the respective OVA configuration
[2] Edit VM configuration
[3] Uncheck the option for VMware Tools => "Check and upgrade Tools during power cycling" <== This is checked by default on CUCM 10+ OVA, Unchecked on CUCM 8.X, 9.X OVAs
[4] Proceed with power on the VM and Fresh installation
[5] Post Fresh Install the COP File
[6] Edit VM configuration
[7] Re-Enable the option for VMware Tools => "Check and upgrade Tools during power cycling"
[8] Reboot the VM

==============
NOTES ON COP FILE
==============
This COP file will:
[1] Update the Selinux Policy files so that VMWare tools upgrade completes without failing under various tools update scenarios.
[2] Allow the VMware tools version 10.0+ to operate without causing excessive memory utilization or filling up the active partition with logs.
[3] Installs CLI script so that customers can use the "utils vmtools caf-logs delete" command to recovery disk space after outage situations.

Symptom:
After upgrading utilizing the IMP CLI command "utils vmtools refresh" and instigating install on hypervisor side for VMWare Tools (ESXi 5.5 / 6):

--Logs fill the primary / active partition
--Memory utilization consumes all available 
--Node eventually becomes unusable
--Using "utils os secure permissive" on nodes that have not become entirely unresponsive stops the problem from getting worse.

Conditions:
Problem is seen after Upgrading to latest builds of ESXi 5.5 or 6.0 builds greater than 3248547 which bundles 10240+ (10.0.0+) version of VMware Tools and brings in a new vmware-caf functionality.

The same condition will occur where selinux denials are logged preventing vmware-caf operations after a Fresh install of IMP 10.X or 11.0 on top of ESXi 5.5 or 6.0 builds that bundles 10.X version of VMware Tools.

When CUCM 10.X or 11.X OVA is used the VM Setting named VMware Tools => "Check and update Tools during power cycling" will be enabled by default. This setting being enabled during a Fresh Installation operation allows VMware Tools upgrade to 10.0+ however post installation when selinux is put back in to enforcing mode the denials will start and you will face same running out of virtual memory as well as vmware-caf logs consuming 100% of the active root partition.

Related Community Discussions

Cisco UC アプリケーションの仮想マシンの Vmware Tools アップデートが失敗する
2016 年 10 月 24 日(初版) TAC SR Collection 主な問題 Cisco UC アプリケーション(CUCM、CUC、IM&P、UCCX)バージョン 10.5 以降の仮想マシンにおいて、バージョン 10 以降の Vmware Tools へのアップデートが失敗します。この際に、Vmware Tools のステータスが「実行していません(未インストール)」と表示されます。 Cisco UC アプリケーション側の Syslog に、以下のような SELinux エラーメッセージが表示されます。 --------------------------------------------------- | Jan 14 09:13:23 xxxxxx user 3 setroubleshoot: SELinux is preventing /bin/rm from search access on the directory /var/lock/subsys/vmware-tools. For complete SELinux messages. run sealert -l ...
Latest activity: Oct 24, 2016
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.