Guest

Preview Tool

Cisco Bug: CSCvu42434 - ASA: High CPU due to stuck running SSH sessions / Unable to SSH to ASA

Last Modified

Oct 05, 2020

Products (1)

  • Cisco ASA 5500-X Series Firewalls

Known Affected Releases

9.12.3.12

Description (partial)

Symptom:
ASA may experience a High CPU caused by stuck SSH sessions running despite the fact that originating connections are no longer place.

Depending on the number of SSH sessions attempted, maximum concurrent number of SSH sessions is 5, you may be unable to SSH into the ASA in case you have 5 stuck SSH sessions.

Symptoms seen on affected devices:

1 - ASA showing High CPU:

> On single core platforms:

asa# show cpu usage
CPU utilization for 5 seconds = 97%; 1 minute: 97%; 5 minutes: 97%

> On multi-core platforms, High CPU will be seen on few particular cores:

asa# show cpu detailed 

Break down of per-core data path versus control point cpu usage:
Core         5 sec              1 min              5 min
Core 0       51.0 (1.0 + 50.0)  51.1 (1.0 + 50.1)  51.1 (1.0 + 50.0)      <<<<<<
Core 1        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 2        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 3        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 4        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 5        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 6        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 7        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 8        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 9        1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 10       1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)   1.8 (1.8 +  0.0)
Core 11      51.4 (1.4 + 50.0)  51.0 (1.3 + 49.8)  51.0 (1.2 + 49.8)        <<<<<<


2 - Show processes output displaying concurrent SSH sessions, even if there are not the same number of active ones:

asa# show processes cpu-usage sorted non-zero
ASLR enabled, text region 559d7d5f2000-559d81f75345
PC         Thread       5Sec     1Min     5Min   Process
   -          -        26.8%    27.7%    28.1%   DATAPATH-0-1673
0x0000559d7f891dad   0x00007f858d649b20    16.1%    15.8%    15.7%   ssh
0x0000559d7f891dad   0x00007f858d64c6a0    13.5%    13.3%    13.3%   ssh
0x0000559d7f891dad   0x00007f858d648ca0    11.1%    10.9%    10.9%   ssh
0x0000559d7f891dad   0x00007f858d64a600    11.0%    10.8%    10.8%   ssh
0x0000559d7f891dad   0x00007f858d6476e0    10.7%    10.6%    10.6%   ssh
0x0000559d7fef99a4   0x00007f858d6e0d00     4.0%     4.1%     4.1%   Logger
0x0000559d7e7f5a95   0x00007f858d6d00c0     2.3%     2.4%     2.4%   CP Processing
0x0000559d7e322516   0x00007f858d6ee680     0.9%     0.9%     0.9%   Config History Thread
0x0000559d7ff9f633   0x00007f858d6c2e80     0.1%     0.0%     0.0%   snmp

3 - Show resource usage showing device (or contexts) with high number of concurrent sessions for line "SSH Server":

asa# show resource usage resource ssh
Resource                 Current        Peak      Limit        Denied Context
SSH Server                     1           2          5             0 admin
SSH Server                     0           1          5             0 one
SSH Server                     0           6          5             0 two
SSH Server                     0           1          5             0 three
SSH Server                     5           6          5             0 four        <<<<<<<

4 - Despite command output above, no actual SSH connection is found active on context in question:

asa/four/act# show conn all protocol tcp port 22 detail
1203 in use, 1902 most used

asa/four/act# show ssh sessions
[no sessions present]

Conditions:
Problem first seen on:

> ASA devices upgraded from any code to release 9.12.X
> ASA configured with an empty enable password, which triggers the "enable password" enforcement when attempting SSH to the device
> On next SSH session established after the upgrade, ASA will present the following prompt:

ASA > ena
The enable password is not set.  Please set it now.
Enter  Password: 
Repeat Password: 

> If user fails to enter a matching password on the prompt above, and terminates the SSH session abruptly, the SSH session in question is not grecefully terminated by the ASA and keeps running on the background

> This situation was also seen by customers that have monitoring tools connecting to the ASA and running commands scripts. As those scripts are not expecting to hit such password prompt, their execution fails and the SSH sessions will get stuck as well.



>>> Related feature introduced on 9.12:

https://www.cisco.com/c/en/us/td/docs/security/asa/asa912/release/notes/asarn912.html#reference_u13_j2w_gfb

"enable password change now required on a login

The default enable password is blank. When you try to access privileged EXEC mode on the ASA, you are now required to change the password to a value of 3 characters or longer. You cannot keep it blank. The no enable password command is no longer supported.

At the CLI, you can access privileged EXEC mode using the enable command, the login command (with a user at privilege level 2+), or an SSH or Telnet session when you enable aaa authorization exec auto-enable . All of these methods require you to set the enable password.

This password change requirement is not enforced for ASDM logins. In ASDM, by default you can log in without a username and with the enable password."
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.