Guest

Preview Tool

Cisco Bug: CSCvs54562 - memory leak due to 'fed main event'

Last Modified

May 08, 2020

Products (1)

  • Cisco Catalyst 3650 Series Switches

Known Affected Releases

Denali-16.3.7

Description (partial)

Symptom:
The RSS value of 'fed main event' continuous increase

	process name	        date	                     rss	         total
	fed main event	2019/11/20	              1925232	     4183192
		                        2019/11/21	              1938120	     4183192
		                        2019/11/22	              1946064	     4183192
		 	 	            n/a
		 	 	            n/a
         		                2019/11/25	              1981132	     4183192
	        	                2019/11/26	              1991596       4248728
		                        2019/11/27	              2004456	     4248728

Conditions:
Customer one WS-C3650-48TQ-E (ver:16.3.7)switch experienced high memory usage and protocol flap(PIM/HSRP/OSPF) after nearly 25 weeks of operation.

When memory is about to run out, the protocol flapping (neighbor reset), The flapping state will last about 20 minutes.

Automatically recover after free up memory (about 2% free up), When 2% of the memory is exhausted, it takes nearly 1 week, and the problem repeats.

When the memory is exhausted and the protocol is flapping, two types of trackback information will be generated.

BDB analysis shows that hit the iosd and fed memory leaks bug.

- CSCvn30230:  Catalyst 3k/9k: Slow memory leak in linux_iosd-imag
- CSCvg00548:  fed memory leak with multicast  ( This bug is only affected on 16.6.1 )

From the collected information, IOSD does not have a memory leak, but the fed process has high memory(about 1.9G~2G>) usage and continuous memory increase.
Note: Normal switch iosd memory usage is about 600M

The following table is the collection of memory usage under abnormal conditions
The RSS value of 'fed main event' continuous increase
	process name	        date	                     rss	         total
	fed main event	2019/11/20	              1925232	     4183192
		                        2019/11/21	              1938120	     4183192
		                        2019/11/22	              1946064	     4183192
		 	 	            n/a
		 	 	            n/a
         		                2019/11/25	              1981132	     4183192
	        	                2019/11/26	              1991596       4248728
		                        2019/11/27	              2004456	     4248728

process name	              date	                              rss	          total	 heap1	heap2	heap3
linux-iosd-imag	        2019-11-20	               657556	      2003860	   n/a	   n/a	 n/a
	                                2019-11-21	               661604	      2005124	   n/a	   n/a	 n/a
	                                2019-11-22	               662176	      2005888	    80	    4	       212420
					      n/a	
					      n/a	
	                                2019-11-25	               654900	      2009308	    80	4	       215980
	                                2019-11-26	               653132	      2010448	    80	4	       217016
	                                2019-11-27	               646372	      2011712	    80	4	       218192

name	                          date	           used	            free
DRAM	                        2019-11-20	3886480K	72576K
	                                2019-11-21	3898484K	60572K
	                                2019-11-22	3907616K	51440K
		                              n/a
                                              n/a	
	                                2019-11-25	3932988K	26068K
	                                2019-11-26	3937660K	21396K
	                                2019-11-27	3940668K	18388K
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.