Guest

Preview Tool

Cisco Bug: CSCvr97169 - Memory cgroup out of memory, processes within the guestshell are killed

Last Modified

Oct 13, 2020

Products (2)

  • Cisco Nexus 9000 Series Switches
  • Cisco Nexus 9000 Series Switches

Known Affected Releases

7.0(3)I4(4) 7.0(3)I7(7.9)

Description (partial)

Symptom:
Messages similar to the following are being generated which indicates the guesthell has exceeded the maximum amount of memory it is allowed to use:

2019 Oct 11 10:01:01.782 SAF-EXT-DS01 %KERN-3-SYSTEM_MSG: [78831490.588655] Memory cgroup out of memory: Kill process 13246 (libvirt_lxc) score 3 or sacrifice child - kernel

2019 Oct 11 10:01:01.782 SAF-EXT-DS01 %KERN-3-SYSTEM_MSG: [78831490.588660] Killed process 13248 (systemd) total-vm:53736kB, anon-rss:1104kB, file-rss:4kB - kernel

2019 Oct 11 10:01:05.921 SAF-EXT-DS01 %VMAN-2-VIRT_INST_STATE: Virtual Service [guestshell+]::Failure::Unexpecte
d virtual service state::The virtual service has failed and is in an unexpected state::State: Shut Off

2019 Oct 11 10:01:15.082 SAF-EXT-DS01 %VMAN-5-PACKAGE_SIGNING_LEVEL_ON_INSTALL: Package 'guestshell.ova' for service container 'guestshell+' is 'Cisco signed', matches signing level cached on original install, signing level allowed is 'Cisco signed'

2019 Oct 11 10:01:34.363 SAF-EXT-DS01 %VMAN-2-ACTIVATION_STATE: Successfully activated virtual service 'guestshell+'

Conditions:
This can happen on any Nexus switch with the guestshell that is activated.  With systems where the guestshell is activated, but no additional applications have been installed within the guestshell, it tends to take in the range of 900 days of continuous running before this is seen and would typically be another 900 before it is seen again.  On systems where applications have been installed within the guestshell, this condition can be reached much sooner, depending on the amount of logging is happening.

Error message trigger time is often xx:01:01

This trigger time is due to journal logging within the guestshell for cron jobs that are running in the guestshell.
Eventually the journal log files on the /run tmpfs within the guestshell gets large enough that it triggers OOM to kill processes within the guestshell due the the memory cgroup limit being reached.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.