x509: certificate has expired or is not yet valid

当对一正常运行中的k8s集群某个node kubelet重启后，发现kubelet启动失败，报错为

E1224 13:55:55.339147   46610 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://192.168.177.224:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dtes
t-node1&limit=500&resourceVersion=0: x509: certificate has expired or is not yet valid

原因

k8s证书已过期，可通过下述命令查看证书详细信息

[root@test-node2 k8s]# openssl x509 -in ca.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=192.168.177.224@1544090516
        Validity
            Not Before: Dec  6 10:01:56 2018 GMT
            Not After : Dec  6 10:01:56 2019 GMT
        Subject: CN=192.168.177.224@1544090516
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:d3:8e:76:00:74:70:97:10:87:f2:64:b8:8d:46:
                    15:37:e7:0d:9f:64:2c:85:87:40:53:fd:c7:c6:62:
                    03:55:c4:2c:06:df:ca:34:94:a0:ae:d2:5a:0a:7e:
                    6b:6e:fc:a3:92:2c:dd:72:41:60:60:ec:3f:9c:04:
                    ef:26:da:b3:af:68:5d:58:60:7a:60:5a:6d:6b:22:
                    ed:89:4b:af:dd:5e:06:60:6b:93:1a:66:50:b1:26:
                    20:83:46:e0:ff:0b:aa:b9:76:ff:b2:4e:6a:a9:ee:
                    05:e9:d2:82:05:ba:11:9d:6e:f9:93:ae:9a:ef:8a:
                    0e:ae:30:5a:5a:b5:b7:d7:20:33:1c:85:a9:47:02:
                    e4:1e:0e:54:ea:4c:ec:ba:34:1c:75:cf:71:29:dc:
                    b4:43:9d:27:f7:f4:68:21:cb:89:c4:aa:1d:33:28:
                    f2:a9:82:52:36:09:de:8f:75:1e:73:97:76:8c:25:
                    82:90:6a:e7:78:b8:19:32:9a:99:65:4e:4a:e9:11:
                    cd:58:a3:dc:4f:9d:8f:63:63:00:24:06:fd:ce:07:
                    c9:3d:8c:84:55:7b:31:49:81:a0:ca:3e:b2:06:e3:
                    bd:07:4a:f2:b7:c3:4c:d2:92:45:1a:9d:56:38:7e:
                    ab:15:31:16:85:fb:d1:41:3e:89:31:45:cc:d1:80:
                    81:3f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Alternative Name:
                DNS:kubernetes.default.svc, DNS:kubernetes.default, DNS:kubernetes, DNS:localhost, IP Address:192.168.177.224, IP Address:169.169.0.1
    Signature Algorithm: sha256WithRSAEncryption
         99:cd:d4:8a:54:d4:63:69:af:20:7e:1b:dc:03:a9:eb:14:99:
         5a:50:15:6d:e7:6d:42:60:53:7d:41:87:29:a0:7a:34:61:21:
         60:e7:18:f3:25:83:75:f7:53:ba:d9:03:05:f5:1c:93:ec:53:
         76:6d:7c:50:fa:1a:ce:2a:31:f5:26:1d:03:2c:63:2d:9d:e9:
         ab:1d:d1:61:8f:e0:46:f6:5c:cc:8d:93:70:dd:24:ee:fa:90:
         e9:29:cb:88:61:e5:99:0d:87:0b:b5:55:91:cc:6c:aa:d8:e3:
         1b:f2:d3:4e:9a:59:fe:ce:7e:a2:75:e4:73:b8:1f:e5:63:ae:
         55:25:37:82:15:b3:5f:e7:14:f7:37:a4:ed:ca:a8:f7:0c:b7:
         dc:9c:de:a0:6c:00:fa:00:b7:fe:94:14:9d:d8:91:7b:d4:1b:
         50:89:2a:ab:92:9b:1b:3d:b0:cb:83:55:b1:47:ff:ce:5e:a8:
         ef:73:8d:0b:ff:1d:ff:4c:8d:fc:e1:e8:30:27:89:d4:39:78:
         2d:a8:c7:06:68:e0:57:bb:67:3c:c0:6b:55:02:3b:75:c8:2c:
         37:ff:93:08:d1:a5:7b:f1:93:ec:7a:cd:c4:1f:39:cc:4b:65:
         b4:02:3a:ae:ad:06:a3:68:5c:d1:c0:77:89:a3:c1:0b:00:6a:
         bb:ba:99:cb

解决方案，升级证书

  • 删除服务端证书
rm /var/run/kubernetes/apiserver.crt
rm /var/run/kubernetes/apiserver.key
  • 重启apiserver
systemctl restart kube-apiserver

重启后，发现重新生成了服务端证书，并且证书有效

  • 查看账号对应的证书
kubectl get secret kube-node-token-wz588 -o json
cat ca.source |base64 -d > ca.crt
  • 替换kube-config中的crt

token不变

启动node节点的 kubelet 报以下错误

[invalid bearer token, [invalid bearer token, crypto/rsa: verification error]]

这个问题是在master节点生成了不同的 server.key 引起，由于 server.key 需要在 apiserver 启动时指定，在生成 service account 时会采用公钥认证。
在pod创建后，默认会在目录/var/run/secrets/kubernetes.io/serviceaccount/生成token及 ca.crt 等

解决方案，然后替换 token及crt文件

删除sa 重新创建