Guest

Preview Tool

Cisco Bug: CSCup37354 - /var/tmp and /var/sysgmr full due to generated cores.

Last Modified

Nov 25, 2018

Products (1)

  • Cisco Unified Computing System

Known Affected Releases

2.0(3c)B

Description (partial)

Symptom:
Linux(debug)# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root             300M  129M  172M  43% /
none                  2.0M  4.0K  2.0M   1% /post
none                  1.5G  988M  513M  66% /isan  
none                  300M  300M     0 100% /var/tmp  <=== Here
none                  500M  496M  4.5M 100% /var/sysmgr  <=== Here
none                   35M   56K   35M   1% /var/sysmgr/ftp
none                   32M     0   32M   0% /callhome
none                  256M  101M  156M  40% /dev/shm
none                   60M  4.0K   60M   1% /volatile
none                  2.0M  8.0K  2.0M   1% /debug
/dev/sda5              76M  4.3M   67M   6% /mnt/cfg/0
/dev/sda6              76M  4.3M   67M   6% /mnt/cfg/1
none                  100M  3.1M   97M   4% /var/sysmgr/startup-cfg
/dev/mtdblock2         55M  1.7M   54M   4% /mnt/plog
/dev/sda4              76M  6.7M   65M  10% /mnt/pss
/dev/sda3             8.5G  1.7G  6.4G  21% /bootflash
/dev/sda7             1.9G   64M  1.8G   4% /opt
/dev/mtdblock3        3.9M  2.0M  1.7M  54% /opt/db/nvram
/dev/sda8             278M   19M  246M   7% /workspace
tmpfs                 1.0M  8.0K 1016K   1% /bootflash/sysdebug/tftpd_logs
none                   50M  4.9M   46M  10% /debug-plugin

BR1NIFFI-A(nxos)# show system inter flash
df: write error: No space left on device
Mount-on                  1K-blocks      Used   Available   Use%  Filesystem

Linux(debug)# cd /var/tmp
Linux(debug)# ls -lS
total 297848
-rw-------  1 root root       19787776 Jun 13 10:40 core.10547
-rw-------  1 root root       19787776 Jun 13 10:49 core.11537
-rw-------  1 root root       19787776 May 17 09:26 core.28302
-rw-------  1 root root       19787776 May 19 14:54 core.28335
-rw-------  1 root root       19787776 May  6 00:45 core.28848
-rw-------  1 root root       19787776 May 16 11:11 core.4007
-rw-------  1 root root       19787776 Apr  4 11:52 core.4548
-rw-------  1 root root       19787776 Apr  4 11:55 core.5032
-rw-------  1 root root       19787776 May 30 01:19 core.5408
-rw-------  1 root root       19787776 May  3 09:06 core.5681
-rw-------  1 root root       19787776 Apr 29 04:14 core.6004
-rw-------  1 root root       19787776 May 27 20:01 core.6247
-rw-------  1 root root       19787776 Jun 13 04:13 core.6507
-rw-------  1 root root       19787776 Jun 10 16:51 core.7091
-rw-------  1 root root       19787776 May 11 13:19 core.7321
-rw-------  1 root root       19787776 Jun 10 16:54 core.7364
-rw-------  1 root root       19787776 May 31 01:32 core.7867
-rw-------  1 root root       19783680 Jun 13 10:52 core.11875
-rw-------  1 root root        4771840 Apr  4 12:25 core.7911
-rw-r--r--  1 root root         165598 Jan 11 14:22 sam_bridge_boot.log
-rw-rw-rw-  1 root root           5003 Jan 11 14:24 first_setup.log
-rw-------  1 root root           4096 Jun  8 15:36 core.29930
... More files

'show logging log' shows a large number of the following entries.

2014 May 16 10:23:37 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-client)" (PID 26711) hasn't caught signal 6 (no core).
2014 May 16 10:26:37 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-server)" (PID 26981) hasn't caught signal 6 (core will be saved).
2014 May 16 10:26:37 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-client)" (PID 26982) hasn't caught signal 6 (no core).
2014 May 16 10:29:37 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-server)" (PID 27261) hasn't caught signal 6 (core will be saved).
2014 May 16 10:29:37 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-client)" (PID 27262) hasn't caught signal 6 (no core).
2014 May 16 10:32:38 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-server)" (PID 27503) hasn't caught signal 6 (core will be saved).
2014 May 16 10:32:38 BR1NIFFI-A %SYSMGR-2-SERVICE_CRASHED: Service "System Manager (core-client)" (PID 27504)

Linux(debug)# pwd
/var/sysmgr
Linux(debug)# ls -l
total 0
drwxrwxrwx   6 root root    180 Jan 11 14:23 ftp
lrwxrwxrwx   1 root root     33 Jan 11 14:23 logs -> /bootflash/sysdebug/coremgmt/logs
drwxrwxrwx   2 root root     40 Jan 11 14:23 remote
drwxrwxrwx   2 root root     40 Jan 11 14:23 rev-gsync
drwxrwxrwx   4 root root   1320 Jun 13 10:17 sam_logs
drwxrwxrwx  11 root root    240 Jan 11 14:21 startup-cfg
drwxrwxrwx   2 root root 416280 Jun 13 11:58 sysmgr-subproc
lrwxrwxrwx   1 root root     37 Jan 11 14:23 tmp_logs -> /bootflash/sysdebug/coremgmt/tmp_logs
drwxrwxrwx   2 root root     40 Jan 11 14:23 vsh
lrwxrwxrwx   1 root root     33 Jan 11 14:23 work -> /bootflash/sysdebug/coremgmt/work
drwxrwxrwt   2 root root     40 Jan 11 14:24 www
drwxrwxrwx   2 root root     40 Jan 11 14:23 xml
Linux(debug)#

# Large Number of files too many to ls 
Linux(debug)# pwd
/var/sysmgr/sysmgr-subproc
Linux(debug)# ls | wc -l
20812
Linux(debug)#

Conditions:
Fabric Interconnect A:
    Running-Kern-Vers: 5.0(3)N2(2.03c)
    Running-Sys-Vers: 5.0(3)N2(2.03c)
    Package-Vers: 2.0(3c)A
    Startup-Kern-Vers:
    Startup-Sys-Vers:
    Act-Kern-Status: Ready
    Act-Sys-Status: Ready
    Bootloader-Vers:   v1.5.0(11/30/10)

Fabric Interconnect B:
    Running-Kern-Vers: 5.0(3)N2(2.03c)
    Running-Sys-Vers: 5.0(3)N2(2.03c)
    Package-Vers: 2.0(3c)A
    Startup-Kern-Vers: 5.0(3)N2(2.03c)
    Startup-Sys-Vers: 5.0(3)N2(2.03c)
    Act-Kern-Status: Ready
    Act-Sys-Status: Ready
    Bootloader-Vers:   v1.5.0(11/30/10)

BR1NIFFI-A(nxos)# show ver
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2012, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained herein are owned by
other third parties and are used and distributed under license.
Some parts of this software are covered under the GNU Public
License. A copy of the license is available at
http://www.gnu.org/licenses/gpl.html.

Software
  BIOS:      version 1.5.0
  loader:    version N/A
  kickstart: version 5.0(3)N2(2.03c)
  system:    version 5.0(3)N2(2.03c)
  power-seq: Module 1: version v1.2
  BIOS compile time:       11/30/10
  kickstart image file is: bootflash:/installables/switch/ucs-6100-k9-kickstart.5.0.3.N2.2.03c.bin
  kickstart compile time:  8/23/2012 18:00:00 [08/23/2012 22:22:26]
  system image file is:    bootflash:/installables/switch/ucs-6100-k9-system.5.0.3.N2.2.03c.bin
  system compile time:     8/23/2012 18:00:00 [08/24/2012 00:04:53]


Hardware
  cisco UCS 6100 Series Fabric Interconnect ("20x10GE/Supervisor")
  Intel(R) Celeron(R) M CPU    with 3633996 kB of memory.
  Processor Board ID JAF1430EMDQ

  Device name: BR1NIFFI-A
  bootflash:          0 kB

Kernel uptime is 152 day(s), 20 hour(s), 57 minute(s), 46 second(s)

Last reset
  Reason: Unknown
  System version: 5.0(3)N2(2.03c)
  Service:

plugin
  Core Plugin, Ethernet Plugin, Fc Plugin, Virtualization Plugin
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.