Operational Defect Database
BugZero updated this defect 44 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 89324
vSphere with Tanzu Certificate Guide
Last update date:
4/5/2024
Affected products:
vSphere
Affected releases:
7.0
Fixed releases:
No fixed releases provided.
Description:
Purpose
This article provides guidance on vSphere with Tanzu Supervisor and Guest Cluster certificate management. --------------------------------------------------------------------- Certificates for vSphere with Tanzu are utilized in 4 places depending on where the authentication is being performed, these are: vCenter Server, Supervisor Cluster Nodes, Guest Cluster Nodes, ESXi hosts (used to support vSphere POD functionality) There are 2 trust domains in vSphere with Tanzu: VC trust domain: The default signer for certificates in this trust domain is the VMware Certificate Authority (VMCA) built into vCenter ServerK8s trust domain: The default signer for certificates in this trust domain is the Kubernetes CA. --------------------------------------------------------------------- WCP requires certificates for the following operations: Server certificate for wcpsvc (running on vCenter Server) to talk to the Supervisor Cluster api-servers over the management networkServer certificate for the authproxy server endpoint hosted on Supervisor Cluster VMServer certificate for the schedext endpoint hosted on Supervisor Cluster VMClient certificate for the WCP Solution user used by authproxy to query WCP vAPI Regular k8s operations: Client certificates for kubelet to authenticate to the API serverKubelet server certificates for the API server to talk to the kubeletsServer certificate for the API server endpoint (TLS over the workload network)Client certificates for administrators of the cluster to authenticate to the API serverClient certificates for the API server to talk to the kubeletsClient certificate for the API server to talk to etcdClient certificate/kubeconfig for the controller manager to talk to the API serverClient certificate/kubeconfig for the scheduler to talk to the API server.Client and server certificates for the front-proxy --------------------------------------------------------------------- The following command can be used from SSH to Supervisor or Guest Clusters to identify certificate expiry timeframe: # find / -type f \( -name "*.cert" -o -name "*.crt" \) -print 2>/dev/null | egrep -v 'ca.crt$|ca-bundle.crt$|kubelet\/pods|var\/lib\/containerd|run\/containerd' | xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
Impact / Risks
If certificates expire on the Supervisor or Guest Clusters, access and management of the clusters will fail.VMware engineering is aware of this and is working to build a script that will allow replacement of Supervisor Cluster certificates manually.NOTE: If your certificates expire, please notify VMware support for assistance with manual replacement.
Resolution
Delineation of WCP certificates, usage and locations On supervisor control plane nodes: WCP specific certificates Cert PathSigned ByUsed forCert Lifetime /etc/vmware/wcp/tls/vip.crtVMCA/CustomTLS certificate served by the nginx proxy running in front of each CP VM on the workload network1 Year / Custom/etc/vmware/wcp/tls/mgmt.crtK8s CATLS certificate served by the nginx proxy running in front of each CP VM on the management network1 Year /etc/vmware/wcp/tls/ncp/lb-default.cert VMCA/CustomCertificate applied to Service IP's built on the Ingress network in NSX-T 1 Year / Custom /etc/vmware/wcp/tls/wcpusr.certVMCAClient certificate for VC solution user for WCP2 Years*/etc/vmware/wcp/tls/schedext.certself-signedTLS certificate served by schedext2 Years /etc/vmware/wcp/tls/authproxy.crtK8s CATLS certificate served by authproxy2 Years/etc/vmware/wcp/tls/docker-reg.crtK8s CATLS certificate served by the internal docker registry2 Years/etc/vmware/wcp/tls/wcpagent.certVMCAWas: TLS certificate for docker registry and authproxy. No long in use after 7.0 U1 Kubernetes internal certificates Cert PathSigned ByUsed forCert Lifetime/var/lib/kubelet/pki/kubelet.crt K8s CACurrently not used. Kubelet serves "content" to metrics servers 1 Year/etc/kubernetes/pki/scheduler.crt K8s CAUsed to authenticate with the scheduler pod 1 Year/etc/kubernetes/pki/apiserver.crt K8s CAUsed to authenticate with K8s API server 1 Year /etc/kubernetes/pki/apiserver-etcd-client.crt K8s CAUsed by API server to authenticate with ETCD 1 Year /etc/kubernetes/pki/apiserver-kubelet-client.crt K8s CAUsed by API server to authenticate with kubelet 1 Year /etc/kubernetes/pki/front-proxy-client.crt K8s CA 1 Year /etc/kubernetes/pki/etcd/server.crt K8s CACert used for ETCD Server authentication 1 Year /etc/kubernetes/pki/etcd/peer.crt K8s CACert used for ETCD Peer server authentication 1 Year /etc/kubernetes/pki/etcd/healthcheck-client.crt K8s CA 1 Year /etc/kubernetes/pki/bootstrapper.crt K8s CAUsed for initial cluster bootstrap and customization n/a/etc/kubernetes/pki/front-proxy-ca.crt K8s CAK8s Front Proxy certificate authority 10 Year/etc/kubernetes/pki/etcd/ca.crt K8s CAK8s ETCD certificate authority 10 Year/etc/kubernetes/pki/ca.crt K8s CA K8s Cluster certificate authority 10 Year ---------------------------------------------------------------------------------- On ESXi Hosts (Only when WCP is deployed with NSX-T): Cert PathCert Lifetime/etc/vmware/spherelet/spherelet.crt1 Year/etc/vmware/spherelet/client.crt1 Year ---------------------------------------------------------------------------------- TKGS Guest Cluster Certificates: TKGS Guest Cluster Control Plane VMs Cert PathCert Lifetime/var/lib/kubelet/pki/kubelet.crt1 Year/etc/kubernetes/pki/apiserver.crt1 Year/etc/kubernetes/pki/apiserver-etcd-client.crt1 Year/etc/kubernetes/pki/etcd/server.crt1 Year/etc/kubernetes/pki/etcd/peer.crt1 Year/etc/kubernetes/pki/etcd/healthcheck-client.crt1 Year/etc/kubernetes/pki/front-proxy-client.crt1 Year/etc/ssl/certs/extensions-tls.crt10 Year TKGS Guest Cluster Worker Node VMs Cert PathExpiration Date/var/lib/kubelet/pki/kubelet.crt1 Year/etc/ssl/certs/extensions-tls.crt10 Year
Workaround
You can use the following kb to rotate the Certificates for Supervisor Control Plane VM's and ESXi hosts here: https://kb.vmware.com/s/article/90627 TKGS Guest Cluster Certificates can be rotated by upgrading the cluster. If they have expired then you can follow this kb to rotate them https://kb.vmware.com/s/article/95425
