Cisco Bug: CSCva09767 - IMP 10. / 11.x -- HDD & RAM Maxed After VMTools Upgrade (ESXi 5/6)
Jun 21, 2017
Cisco Unified Communications Manager IM & Presence Service
Known Affected Releases
10.0(1) 10.5(1) 10.5(2) 11.0(1) 11.5(1)
================= WORKAROUND ================= I. During Outages --  Automated Resolution Method: a. From IMP CLI, run "utils os secure permissive" to disable selinux, which will prevent the problem from getting worse. b. If the node is responding very slowly, attempt to restart the node, via "utils system restart" and when it restarts ensure the selinux has been disabled via "cisco utils os secure status" -- this should at least clear up the memory issues. c. Resolution to this issue is also available via a standalone COP file "ciscocm.IMP_VMwareTools2016c.cop.sgn" published to software.cisco.com. Details on COP file: http://www.cisco.com/web/software/282074312/93622/ciscocm.IMP_VMwareTools2016cReadme.pdf  Where a node rebuild is required (system has become unusable, unstable), see section II. Install For ESXi Update / Patch Scenarios Apply this COP file prior to restarting CUCM applications where automatic update could take place during power up or prior to initiating automatic vmware tools update from vi-clients. II. Install Workaround If you must Fresh Install / Re-Install IM&P 10.X or 11.X version that does NOT have this fix present on top of ESXi 5.5 or 6.0 which already has VMware Tools 10.X version bundled, you must follow these steps:  After deploying the VM using the respective OVA configuration  Edit VM configuration  Uncheck the option for VMware Tools => "Check and upgrade Tools during power cycling" <== This is checked by default on CUCM 10+ OVA, Unchecked on CUCM 8.X, 9.X OVAs  Proceed with power on the VM and Fresh installation  Post Fresh Install the COP File  Edit VM configuration  Re-Enable the option for VMware Tools => "Check and upgrade Tools during power cycling"  Reboot the VM ============== NOTES ON COP FILE ============== This COP file will:  Update the Selinux Policy files so that VMWare tools upgrade completes without failing under various tools update scenarios.  Allow the VMware tools version 10.0+ to operate without causing excessive memory utilization or filling up the active partition with logs.  Installs CLI script so that customers can use the "utils vmtools caf-logs delete" command to recovery disk space after outage situations. Symptom: After upgrading utilizing the IMP CLI command "utils vmtools refresh" and instigating install on hypervisor side for VMWare Tools (ESXi 5.5 / 6): --Logs fill the primary / active partition --Memory utilization consumes all available --Node eventually becomes unusable --Using "utils os secure permissive" on nodes that have not become entirely unresponsive stops the problem from getting worse. Conditions: Problem is seen after Upgrading to latest builds of ESXi 5.5 or 6.0 builds greater than 3248547 which bundles 10240+ (10.0.0+) version of VMware Tools and brings in a new vmware-caf functionality. The same condition will occur where selinux denials are logged preventing vmware-caf operations after a Fresh install of IMP 10.X or 11.0 on top of ESXi 5.5 or 6.0 builds that bundles 10.X version of VMware Tools. When CUCM 10.X or 11.X OVA is used the VM Setting named VMware Tools => "Check and update Tools during power cycling" will be enabled by default. This setting being enabled during a Fresh Installation operation allows VMware Tools upgrade to 10.0+ however post installation when selinux is put back in to enforcing mode the denials will start and you will face same running out of virtual memory as well as vmware-caf logs consuming 100% of the active root partition.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.
Bug Details Include
- Full Description (including symptoms, conditions and workarounds)
- Known Fixed Releases
- Related Community Discussions
- Number of Related Support Cases