Cisco Bug: CSCul06653 - 3.4ISR CDE470:All the 10K recordings are forced to stop with heavy load
Jun 21, 2020
- Cisco Videoscape Distribution Suite for Television
- Cisco TV Streamer Application
- Cisco Visual Quality Experience (VQE) Channel Provisioning Tools
- Cisco Visual Quality Experience Application
Known Affected Releases
Symptom: When recording and streaming exceeds the capacity of the disk channel, the disk channel will get behind doing writes. When the system is more than 75 GB behind in writes, it stops recordings in an effort to reduce the load on the system. This is part of the system design. Because of the way recordings are done, all recordings for the same channel that start at the same time are grouped together in the same recording session. The data is processed one time for each recording in the session / group and then written out to separate GOIDs so that each customer has their personal copy of the data. This is also part of the design. When the system is behind on writes, it selects a recording "session" to end, and all recordings that are part of that session / group get cancelled. Currently it ends one session every 2 seconds until the pending writes fall back below the desired threshold. When all of the recordings are part of the same session / group, or when the majority of the recordings are contained by a very small number of sessions, cancelling a few sessions has the effect of cancelling the majority of the active recordings. Conditions: #1) The recorder's disk channel can't keep up with all the read / write requests and it is getting behind. Note that the disk channel for the CDE470 is designed to handle around 50 Gbps combined read / write traffic and the CDE460 is designed to handle half that load. So it takes a heavy load before the system starts to get behind. If you are operating the recorder at or below the designed level, you should be able to avoid this problem. #2) There must be a low number of recording groups / sessions. For example, the test used to demonstrate this problem had 5 sessions of 2,000 recordings each; so each session cancelation removed 2,000 recordings. If the recording sessions comprise a lot of different channel feeds and different start times, then there will be a large number of sessions and you will be able reduce the load on the system more gradually and avoid cancelling the majority of your recordings. Note also that with a new system, as it initially fills up with data, the first third of the disk space will be written at a faster rate than mentioned above, and the last third will be written at a slower rate than mentioned above. Once the system reaches the desired full point and recordings are archived, the available space will average out and long term the desired disk channel bandwidth will be available. It is just during the initial fill stage that this is an issue.
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.
Bug Details Include
- Full Description (including symptoms, conditions and workarounds)
- Known Fixed Releases
- Related Community Discussions
- Number of Related Support Cases