kubectl get cs查看组件状态kube-scheduler和kube-controller-manager显示unhealthy

kubectl get cs查看组件状态kube-scheduler和kube-controller-manager显示unhealthy

通过kubeadm安装的k8s集群获取kube-scheduler和kube-controller-manager组件状态异常

原因1:这两个pod的非安全端口没有开启，健康检查时报错，但是由于本身服务是正常的，只是健康检查的端口没启，所以不影响正常使用。

$ kubectl get csNAME STATUS MESSAGE ERRORcontroller-manager Unhealthy Get dial tcp 127.0.0.1:10252: connect: connection refusedscheduler Unhealthy Get dial tcp 127.0.0.1:10251: connect: connection refusedetcd-0 Healthy {"health":"true"}

kubernetes版本：v1.18.6  镜像信息

$ sudo docker imagesREPOSITORY TAG IMAGE ID CREATED SIZEnginx latest 8cf1bfb43ff5 4 hours ago 132MBk8s.gcr.io/kube-proxy v1.18.6 c3d62d6fe412 6 days ago 117MBk8s.gcr.io/kube-apiserver v1.18.6 56acd67ea15a 6 days ago 173MBk8s.gcr.io/kube-controller-manager v1.18.6 ffce5e64d915 6 days ago 162MBk8s.gcr.io/kube-scheduler v1.18.6 0e0972b2b5d1 6 days ago 95.3MBquay.io/coreos/flannel v0.12.0-amd64 4e9f801d2217 4 months ago 52.8MBk8s.gcr.io/pause 3.2 80d28bedfe5d 5 months ago 683kBk8s.gcr.io/coredns 1.6.7 67da37a9a360 5 months ago 43.8MBk8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 9 months ago 288MB

k8s组件pod状态

$ kubectl get pods -ANAMESPACE NAME READY STATUS RESTARTS AGEkube-system coredns-66bff467f8-22hgp 1/1 Running 0 45mkube-system coredns-66bff467f8-ck6qq 1/1 Running 0 45mkube-system etcd-node-14 1/1 Running 0 46mkube-system kube-apiserver-node-14 1/1 Running 0 46mkube-system kube-controller-manager-node-14 1/1 Running 0 17mkube-system kube-flannel-ds-amd64-lm7lt 1/1 Running 0 44mkube-system kube-proxy-5hghv 1/1 Running 0 45mkube-system kube-scheduler-node-14 1/1 Running 0 17m

排查思路：

1、先查看本地的端口，可以确认没有启动10251、10252端口

2、确认kube-scheduler和kube-controller-manager组件配置是否禁用了非安全端口

配置文件路径：/etc/kubernetes/manifests/scheduler.conf 、/etc/kubernetes/manifests/controller-manager.conf

如controller-manager组件的配置如下：可以去掉--port=0这个设置，然后重启sudo systemctl restart kubelet

重启服务之后确认组件状态，显示就正常了

$ kubectl get csNAME STATUS MESSAGE ERRORscheduler Healthy okcontroller-manager Healthy oketcd-0 Healthy {"health":"true"}

原因2: kubelet配置文件里没有指定静态pod的路径

解决办法：

（5.1）查找kubelet的配置文件位置 systemctl cat kubelet

[root@cka-node01 ~]# systemctl cat kubelet # /usr/lib/systemd/system/kubelet.service[Unit]Description=kubelet: The Kubernetes Node AgentDocumentation=[Service]ExecStart=/usr/bin/kubeletRestart=alwaysStartLimitInterval=0RestartSec=10 [Install]WantedBy=multi-user.target # /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 这是kubelet的配置文件 # Note: This dropin only works with kubeadm and kubelet v1.11+[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamicallyEnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.EnvironmentFile=-/etc/sysconfig/kubeletExecStart=ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

（5.2）vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf 打开配置文件，添加以下内容

# Note: This dropin only works with kubeadm and kubelet v1.11+[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests" 要添加这个配置Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamicallyEnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.EnvironmentFile=-/etc/sysconfig/kubeletExecStart=ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

（5.3）重新加载daemon, 重启kubelet [root@cka-node01 ~]# systemctl daemon-reload [root@cka-node01 ~]# systemctl restart kubelet

添加前 在node上 ps -ef |grep kubelet 查看加载的配置文件

[root@cka-node01 ~]# ps -ef |grep kubeletroot 689 1 2 Aug16 ? 00:51:35 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry--hangzhou.aliyuncs.com/google_containers/pause:3.2root 126462 126161 0 11:57 pts/0 00:00:00 grep --color=auto kubelet

添加后

[root@cka-node01 ~]# ps -ef |grep kubeletroot 4457 1 6 12:17 ? 00:00:02 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry--hangzhou.aliyuncs.com/google_containers/pause:3.2root 5089 126161 0 12:18 pts/0 00:00:00 grep --color=auto kubelet

复制

查找kubectl的帮助，grep manifest 可以看到

[root@cka-node01 ~]# kubelet --help |grep manimanifest can be provided to the Kubelet.(underspec'd currently) to submit a new manifest. --cgroup-driver string Driver that the kubelet uses to manipulate cgroups on the host. Possible values: 'cgroupfs', 'systemd' (default "cgroupfs") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See for more information.) --manifest-url string URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See for more information.) --manifest-url-header --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' Comma-separated list of HTTP headers to use when accessing the url provided to --manifest-url. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example: --manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful' (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See for more information.) --pod-manifest-path string Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See for more information.)

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。