k8s-kubeadm集群部署 一、环境准备：(1)虚拟机操作系统环境准备 参考链接: https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ ​ (2)关闭swap分区 1)临时关闭 swapoff -a && sysctl -w vm.swappiness=0 2)基于配置文件关闭 sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab ​ ​ (3)确保各个节点MAC地址或product_uuid唯一 ifconfig eth0 | grep ether | awk '{print $2}' cat /sys/class/dmi/id/product_uuid ​ 温馨提示: 一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。 Kubernetes使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装失败。 ​ ​ (4)检查网络节点是否互通 简而言之，就是检查你的k8s集群各节点是否互通，可以使用ping命令来测试。 ​ ​ (5)允许iptable检查桥接流量 cat <<EOF | tee /etc/modules-load.d/k8s.conf br_netfilter EOF ​ cat <<EOF | tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF sysctl --system ​ ​ (6)检查端口是否被占用 参考链接: https://kubernetes.io/zh-cn/docs/reference/networking/ports-and-protocols/ ​ ​ (7)检查docker的环境 参考链接: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.15.md#unchanged 1)配置docker源 curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo curl -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo yum list docker-ce --showduplicates ​ 2)安装指定的docker版本 yum -y install docker-ce-18.09.9 docker-ce-cli-18.09.9 yum -y install bash-completion source /usr/share/bash-completion/bash_completion ​ 3)打包软件包分发到其他节点部署docker（此步骤可跳过） mkdir docker-rpm-18-09 && find /var/cache/yum -name "*.rpm" | xargs mv -t docker-rpm-18-09/ ​ 4)配置docker优化 mkdir -pv /etc/docker && cat <<EOF | sudo tee /etc/docker/daemon.json { "registry-mirrors": [ "https://docker.1ms.run", "https://docker.xuanyuan.me" ], "data-root": "/data/docker" } EOF 5)配置docker开机自启动 systemctl daemon-reload systemctl enable --now docker systemctl status docker ​ ​ (8)禁用防火墙 systemctl disable --now firewalld ​ ​ (9)禁用selinux sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config grep ^SELINUX= /etc/selinux/config ​ ​ (10)配置host解析 cat >> /etc/hosts <<'EOF' 10.0.0.121 master02 10.0.0.122 node01 10.0.0.123 node02 EOF cat /etc/hosts二、软件包说明：你需要在每台机器上安装以下的软件包： kubeadm： 用来初始化集群的指令。 kubelet： 在集群中的每个节点上用来启动Pod和容器等。 kubectl： 用来与集群通信的命令行工具。 ​ kubeadm不能帮你安装或者管理kubelet或kubectl，所以你需要确保它们与通过kubeadm安装的控制平面(master)的版本相匹配。 如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。 ​ 然而，控制平面与kubelet间的相差一个次要版本不一致是支持的，但kubelet的版本不可以超过"API SERVER"的版本。 例如，1.7.0版本的kubelet可以完全兼容1.8.0版本的"API SERVER"，反之则不可以。三、所有节点安装kubeadm，kubelet，kubectl(1)配置软件源 cat > /etc/yum.repos.d/kubernetes.repo <<EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=0 repo_gpgcheck=0 EOF ​ ​ (2)查看kubeadm的版本（将来你要安装的K8S时请所有组件版本均保持一致!） yum -y list kubeadm --showduplicates | sort -r ​ ​ (3)安装kubeadm，kubelet，kubectl软件包 yum -y install kubeadm-1.15.12-0 kubelet-1.15.12-0 kubectl-1.15.12-0 ​ ​ (4)启动kubelet服务(若服务启动失败时正常现象，其会自动重启，因为缺失配置文件，初始化集群后恢复！此步骤可跳过！) systemctl enable --now kubelet systemctl status kubelet ​ ​ (5)温馨提示:（可以将k8s软件打包到其他节点安装哟，前提是得开启rpm包缓存。） mkdir k8s-rpm && find /var/cache/yum -name "*.rpm" | xargs mv -t k8s-rpm 参考链接: https://kubernetes.io/zh/docs/tasks/tools/install-kubectl-linux/四、初始化master节点(1)使用kubeadm初始化master节点 kubeadm init --kubernetes-version=v1.15.12 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.254.0.0/16 --service-dns-domain=example.com ​ ​ 相关参数说明: --kubernetes-version: 指定K8S master组件的版本号。 --image-repository: 指定下载k8s master组件的镜像仓库地址。 --pod-network-cidr: 指定Pod的网段地址。 --service-cidr: 指定SVC的网段 ​ --service-dns-domain: 指定service的域名。若不指定，默认为"cluster.local"。 ​ 使用kubeadm初始化集群时，可能会出现如下的输出信息: [init] 使用初始化的K8S版本。 [preflight] 主要是做安装K8S集群的前置工作，比如下载镜像，这个时间取决于你的网速。 ​ [certs] 生成证书文件，默认存储在"/etc/kubernetes/pki"目录哟。 ​ [kubeconfig] 生成K8S集群的默认配置文件，默认存储在"/etc/kubernetes"目录哟。 ​ [kubelet-start] 启动kubelet， 环境变量默认写入:"/var/lib/kubelet/kubeadm-flags.env" 配置文件默认写入:"/var/lib/kubelet/config.yaml" ​ [control-plane] 使用静态的目录，默认的资源清单存放在:"/etc/kubernetes/manifests"。 此过程会创建静态Pod，包括"kube-apiserver"，"kube-controller-manager"和"kube-scheduler" ​ [etcd] 创建etcd的静态Pod，默认的资源清单存放在:""/etc/kubernetes/manifests" [wait-control-plane] 等待kubelet从资源清单目录"/etc/kubernetes/manifests"启动静态Pod。 ​ [apiclient] 等待所有的master组件正常运行。 [upload-config] 创建名为"kubeadm-config"的ConfigMap在"kube-system"名称空间中。 [kubelet] 创建名为"kubelet-config-1.22"的ConfigMap在"kube-system"名称空间中，其中包含集群中kubelet的配置 ​ [upload-certs] 跳过此节点，详情请参考”--upload-certs" [mark-control-plane] 标记控制面板，包括打标签和污点，目的是为了标记master节点。 [bootstrap-token] 创建token口令，例如:"kbkgsa.fc97518diw8bdqid"。 如下图所示，这个口令将来在加入集群节点时很有用，而且对于RBAC控制也很有用处哟。 ​ [kubelet-finalize] 更新kubelet的证书文件信息 ​ [addons] 添加附加组件，例如:"CoreDNS"和"kube-proxy” (2)拷贝授权文件，用于管理K8S集群 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ​ ​ (3)查看集群节点 kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} 五、配置所有worker节点加入k8s集群(1)worker节点加入集群(如上图所示，下面的token及hash值需要根据您的集群环境而发生改变哟~) kubeadm join 10.0.0.121:6443 --token u6aq9n.4niu8w0byrppiafj \ --discovery-token-ca-cert-hash sha256:470779a9ab746a0cd2facf24e37956af9f844593734d55238c6ec0ad994cbd2c (2)查看集群现有的worker节点 kubectl get no NAME STATUS ROLES AGE VERSION master02 NotReady master 53m v1.15.12 node01 NotReady <none> 57s v1.15.12 node02 NotReady <none> 53s v1.15.12六、初始化网络组件cat > kube-flannel.yml << EOF --- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default spec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN', 'NET_RAW'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: rule: RunAsAny --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: flannel rules: - apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged'] - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flannel data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-amd64 namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/arch operator: In values: - amd64 hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.11.0-amd64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.11.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-arm64 namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/arch operator: In values: - arm64 hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.11.0-arm64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.11.0-arm64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-arm namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/arch operator: In values: - arm hostNetwork: true priorityClassName: system-node-critical tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.11.0-arm command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.11.0-arm command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN", "NET_RAW"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg EOF 在 master 节点执行以下命令： kubectl apply -f kube-flannel.yml 检查 Flannel Pod 状态 kubectl get pods -n kube-system | grep flannel kube-flannel-ds-amd64-2kfbn 1/1 Running 0 3m3s kube-flannel-ds-amd64-cwtpx 1/1 Running 0 3m3s kube-flannel-ds-amd64-fvdlk 1/1 Running 0 3m3s 检查节点状态： kubectl get nodes NAME STATUS ROLES AGE VERSION master02 Ready master 83m v1.15.12 node01 Ready <none> 31m v1.15.12 node02 Ready <none> 31m v1.15.12七、验证网络联通性# 创建测试 Pod kubectl run test-pod --image=busybox:1.28 --command -- sleep 3600 # 查看 Pod IP kubectl get pods -o wide # 进入 Pod 测试 ping 其他节点/Pod kubectl exec -it test-pod -- ping -c 3 10.244.1.2 # 替换为实际 Pod IP八、常见问题解决问题 1：Flannel Pod 启动失败，日志提示 "dial tcp 10.96.0.1:443: connect: no route to host" 原因：master 节点的 apiserver 端口未放行，或 kube-proxy 未正常运行。 # 检查 kube-proxy Pod 状态 kubectl get pods -n kube-system | grep kube-proxy # 重启 kube-proxy（若异常） kubectl rollout restart daemonset kube-proxy -n kube-system # 放行 master 节点 6443 端口 iptables -A INPUT -p tcp --dport 6443 -j ACCEPT 问题 2：Flannel Pod 卡在 Init 状态 原因：缺少 kube-flannel 目录权限 # 在所有节点创建目录并授权 mkdir -p /run/flannel chmod 777 /run/flannel kubectl get pods -n kube-system -w | grep flannel kubectl describe -n kube-system pods kube-flannel-ds-amd64-bvcf5 "network: failed to find plugin "flannel" in path [/opt/cni/bin]]..."这样的报错。 问题2：你安装的flannel 插件/opt/cni/bin 没有flannel程序 # 创建目录（如果不存在） sudo mkdir -p /opt/cni/bin #你有这个目录但是没有flannel这个执行程序 # 下载 CNI 插件（推荐使用 v0.8.6 版本） wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz # 解压到指定目录 sudo tar -xzvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/ # 验证 flannel 插件是否已存在 ls -la /opt/cni/bin/ | grep flannel # 设置执行权限 chmod +x /opt/cni/bin/flannel # 重启节点上的容器运行时 # 重启 containerd（根据你的容器运行时选择） sudo systemctl restart containerd # 重启 kubelet sudo systemctl restart kubelet