Guest

Preview Tool

Cisco Bug: CSCvs75723 - CCP 5.1 Add-On dashboard fails to run - has Never worked in v3 clusters

Last Modified

Feb 28, 2020

Products (1)

  • Cisco Container Platform

Known Affected Releases

4.2.0 4.2.1 5.0.0 5.1 5.1.0

Description (partial)

Symptom:
ccpadmin@ccp-foxconncl02-0-master-0:~$ kubectl -n ccp get pod
NAME                                             READY   STATUS             RESTARTS   AGE
ccp-harbor-operator-64cd49fdbc-fxk27             1/1     Running            1          13d
ccp-helm-operator-96c65df4c-xqtsz                1/1     Running            1          13d
ccp-vip-manager-ccp-foxconncl02-0-master-0       1/1     Running            2          13d
cert-manager-799695b5bc-hqcv6                    1/1     Running            3          13d
kubernetes-dashboard-67d74b6485-xdm7j            0/1     CrashLoopBackOff   6          8m11s
nginx-ingress-controller-55zqk                   1/1     Running            2          13d
nginx-ingress-controller-6vkxs                   1/1     Running            2          13d
nginx-ingress-default-backend-74cf5545c6-dw5bh   1/1     Running            2          13d
ccpadmin@ccp-foxconncl02-0-master-0:~$

kubectl -n ccp describe pod/kubernetes-dashboard-67d74b6485-xdm7j
...
Events:
  Type     Reason     Age                 From                                  Message
  ----     ------     ----                ----                                  -------
  Normal   Scheduled  109s                default-scheduler                     Successfully assigned ccp/kubernetes-dashboard-67d74b6485-xdm7j to ccp-foxconncl02-1-node-gr-0
  Normal   Pulled     53s (x4 over 113s)  kubelet, ccp-foxconncl02-1-node-gr-0  Container image "registry.ci.ciscolabs.com/cpsg_ccp-charts/k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3" already present on machine
  Normal   Created    53s (x4 over 113s)  kubelet, ccp-foxconncl02-1-node-gr-0  Created container kubernetes-dashboard
  Normal   Started    53s (x4 over 112s)  kubelet, ccp-foxconncl02-1-node-gr-0  Started container kubernetes-dashboard
  Warning  BackOff    18s (x9 over 102s)  kubelet, ccp-foxconncl02-1-node-gr-0  Back-off restarting failed container
ccpadmin@ccp-foxconncl02-0-master-0:~$ kubectl -n ccp logs kubernetes-dashboard-67d74b6485-xdm7j
2020/01/22 01:13:50 Starting overwatch
2020/01/22 01:13:50 Using in-cluster config to connect to apiserver
2020/01/22 01:13:50 Using service account token for csrf signing
2020/01/22 01:13:50 No request provided. Skipping authorization
2020/01/22 01:13:50 Successful initial request to the apiserver, version: v1.14.8
2020/01/22 01:13:50 Generating JWE encryption key
2020/01/22 01:13:50 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2020/01/22 01:13:50 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2020/01/22 01:13:50 Synchronizer kubernetes-dashboard-key-holder-kube-system exited with error: unexpected object: &Secret{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Data:map[string][]byte{},Type:,StringData:map[string]string{},}
2020/01/22 01:13:52 Storing encryption key in a secret
panic: secrets is forbidden: User "system:serviceaccount:ccp:kubernetes-dashboard" cannot create resource "secrets" in API group "" in the namespace "kube-system"

goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.(*rsaKeyHolder).init(0xc4204d6fa0)
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:131 +0x2d3
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.NewRSAKeyHolder(0x1a7ee00, 0xc420208540, 0xc420208540, 0x127b962)
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:170 +0x83
main.initAuthManager(0x1a7e300, 0xc420593320, 0xc42058dc68, 0x1)
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:183 +0x12f
main.main()
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:101 +0x28c
ccpadmin@ccp-foxconncl02-0-master-0:~$

Conditions:
Fresh install of CCP 5.1, install Tenant cluster
Install Add-Ons > Dashboard
Bug details contain sensitive information and therefore require a Cisco.com account to be viewed.

Bug Details Include

  • Full Description (including symptoms, conditions and workarounds)
  • Status
  • Severity
  • Known Fixed Releases
  • Related Community Discussions
  • Number of Related Support Cases
Bug information is viewable for customers and partners who have a service contract. Registered users can view up to 200 bugs per month without a service contract.