一、环境准备：
(1)虚拟机操作系统环境准备
参考链接:
https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
(2)关闭swap分区
1)临时关闭
swapoff -a && sysctl -w vm.swappiness=0
2)基于配置文件关闭
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
(3)确保各个节点MAC地址或product_uuid唯一
ifconfig eth0 | grep ether | awk '{print $2}'
cat /sys/class/dmi/id/product_uuid
温馨提示:
一般来讲，硬件设备会拥有唯一的地址，但是有些虚拟机的地址可能会重复。
Kubernetes使用这些值来唯一确定集群中的节点。 如果这些值在每个节点上不唯一，可能会导致安装失败。
(4)检查网络节点是否互通
简而言之，就是检查你的k8s集群各节点是否互通，可以使用ping命令来测试。
(5)允许iptable检查桥接流量
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
(6)检查端口是否被占用
参考链接: https://kubernetes.io/zh-cn/docs/reference/networking/ports-and-protocols/
(7)检查docker的环境
参考链接:
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.15.md#unchanged
1)配置docker源
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo
yum list docker-ce --showduplicates
2)安装指定的docker版本
yum -y install docker-ce-18.09.9 docker-ce-cli-18.09.9
yum -y install bash-completion
source /usr/share/bash-completion/bash_completion
3)打包软件包分发到其他节点部署docker（此步骤可跳过）
mkdir docker-rpm-18-09 && find /var/cache/yum -name "*.rpm" | xargs mv -t docker-rpm-18-09/
4)配置docker优化
mkdir -pv /etc/docker && cat <<EOF | sudo tee /etc/docker/daemon.json
{
"registry-mirrors": [
"https://docker.1ms.run",
"https://docker.xuanyuan.me"
],
"data-root": "/data/docker"
}
EOF
5)配置docker开机自启动
systemctl daemon-reload
systemctl enable --now docker
systemctl status docker
(8)禁用防火墙
systemctl disable --now firewalld
(9)禁用selinux
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config
grep ^SELINUX= /etc/selinux/config
(10)配置host解析
cat >> /etc/hosts <<'EOF'
10.0.0.121 master02
10.0.0.122 node01
10.0.0.123 node02
EOF
cat /etc/hosts
二、软件包说明：
你需要在每台机器上安装以下的软件包：
kubeadm：
用来初始化集群的指令。
kubelet：
在集群中的每个节点上用来启动Pod和容器等。
kubectl：
用来与集群通信的命令行工具。
kubeadm不能帮你安装或者管理kubelet或kubectl，所以你需要确保它们与通过kubeadm安装的控制平面(master)的版本相匹配。 如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。
然而，控制平面与kubelet间的相差一个次要版本不一致是支持的，但kubelet的版本不可以超过"API SERVER"的版本。 例如，1.7.0版本的kubelet可以完全兼容1.8.0版本的"API SERVER"，反之则不可以。
三、所有节点安装kubeadm，kubelet，kubectl
(1)配置软件源
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
(2)查看kubeadm的版本（将来你要安装的K8S时请所有组件版本均保持一致!）
yum -y list kubeadm --showduplicates | sort -r
(3)安装kubeadm，kubelet，kubectl软件包
yum -y install kubeadm-1.15.12-0 kubelet-1.15.12-0 kubectl-1.15.12-0
(4)启动kubelet服务(若服务启动失败时正常现象，其会自动重启，因为缺失配置文件，初始化集群后恢复！此步骤可跳过！)
systemctl enable --now kubelet
systemctl status kubelet
(5)温馨提示:（可以将k8s软件打包到其他节点安装哟，前提是得开启rpm包缓存。）
mkdir k8s-rpm && find /var/cache/yum -name "*.rpm" | xargs mv -t k8s-rpm
参考链接:
https://kubernetes.io/zh/docs/tasks/tools/install-kubectl-linux/
四、初始化master节点
(1)使用kubeadm初始化master节点
kubeadm init --kubernetes-version=v1.15.12 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.254.0.0/16 --service-dns-domain=example.com
相关参数说明:
--kubernetes-version:
指定K8S master组件的版本号。
--image-repository:
指定下载k8s master组件的镜像仓库地址。
--pod-network-cidr:
指定Pod的网段地址。
--service-cidr:
指定SVC的网段
--service-dns-domain:
指定service的域名。若不指定，默认为"cluster.local"。
使用kubeadm初始化集群时，可能会出现如下的输出信息:
[init]
使用初始化的K8S版本。
[preflight]
主要是做安装K8S集群的前置工作，比如下载镜像，这个时间取决于你的网速。
[certs]
生成证书文件，默认存储在"/etc/kubernetes/pki"目录哟。
[kubeconfig]
生成K8S集群的默认配置文件，默认存储在"/etc/kubernetes"目录哟。
[kubelet-start]
启动kubelet，
环境变量默认写入:"/var/lib/kubelet/kubeadm-flags.env"
配置文件默认写入:"/var/lib/kubelet/config.yaml"
[control-plane]
使用静态的目录，默认的资源清单存放在:"/etc/kubernetes/manifests"。
此过程会创建静态Pod，包括"kube-apiserver"，"kube-controller-manager"和"kube-scheduler"
[etcd]
创建etcd的静态Pod，默认的资源清单存放在:""/etc/kubernetes/manifests"
[wait-control-plane]
等待kubelet从资源清单目录"/etc/kubernetes/manifests"启动静态Pod。
[apiclient]
等待所有的master组件正常运行。
[upload-config]
创建名为"kubeadm-config"的ConfigMap在"kube-system"名称空间中。
[kubelet]
创建名为"kubelet-config-1.22"的ConfigMap在"kube-system"名称空间中，其中包含集群中kubelet的配置
[upload-certs]
跳过此节点，详情请参考”--upload-certs"
[mark-control-plane]
标记控制面板，包括打标签和污点，目的是为了标记master节点。
[bootstrap-token]
创建token口令，例如:"kbkgsa.fc97518diw8bdqid"。
如下图所示，这个口令将来在加入集群节点时很有用，而且对于RBAC控制也很有用处哟。
[kubelet-finalize]
更新kubelet的证书文件信息
[addons]
添加附加组件，例如:"CoreDNS"和"kube-proxy”
(2)拷贝授权文件，用于管理K8S集群
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
(3)查看集群节点
kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"}
五、配置所有worker节点加入k8s集群
(1)worker节点加入集群(如上图所示，下面的token及hash值需要根据您的集群环境而发生改变哟~)
kubeadm join 10.0.0.121:6443 --token u6aq9n.4niu8w0byrppiafj \
--discovery-token-ca-cert-hash sha256:470779a9ab746a0cd2facf24e37956af9f844593734d55238c6ec0ad994cbd2c
(2)查看集群现有的worker节点
kubectl get no
NAME STATUS ROLES AGE VERSION
master02 NotReady master 53m v1.15.12
node01 NotReady <none> 57s v1.15.12
node02 NotReady <none> 53s v1.15.12
六、初始化网络组件
cat > kube-flannel.yml << EOF
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
rule: RunAsAny
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- arm64
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-arm64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-arm64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/arch
operator: In
values:
- arm
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-arm
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-arm
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
EOF
在 master 节点执行以下命令：
kubectl apply -f kube-flannel.yml
检查 Flannel Pod 状态
kubectl get pods -n kube-system | grep flannel
kube-flannel-ds-amd64-2kfbn 1/1 Running 0 3m3s
kube-flannel-ds-amd64-cwtpx 1/1 Running 0 3m3s
kube-flannel-ds-amd64-fvdlk 1/1 Running 0 3m3s
检查节点状态：
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master02 Ready master 83m v1.15.12
node01 Ready <none> 31m v1.15.12
node02 Ready <none> 31m v1.15.12
七、验证网络联通性
# 创建测试 Pod
kubectl run test-pod --image=busybox:1.28 --command -- sleep 3600
# 查看 Pod IP
kubectl get pods -o wide
# 进入 Pod 测试 ping 其他节点/Pod
kubectl exec -it test-pod -- ping -c 3 10.244.1.2 # 替换为实际 Pod IP
八、常见问题解决
问题 1：Flannel Pod 启动失败，日志提示 "dial tcp 10.96.0.1:443: connect: no route to host"
原因：master 节点的 apiserver 端口未放行，或 kube-proxy 未正常运行。
# 检查 kube-proxy Pod 状态
kubectl get pods -n kube-system | grep kube-proxy
# 重启 kube-proxy（若异常）
kubectl rollout restart daemonset kube-proxy -n kube-system
# 放行 master 节点 6443 端口
iptables -A INPUT -p tcp --dport 6443 -j ACCEPT
问题 2：Flannel Pod 卡在 Init 状态
原因：缺少 kube-flannel 目录权限
# 在所有节点创建目录并授权
mkdir -p /run/flannel
chmod 777 /run/flannel
kubectl get pods -n kube-system -w | grep flannel
kubectl describe -n kube-system pods kube-flannel-ds-amd64-bvcf5
"network: failed to find plugin "flannel" in path [/opt/cni/bin]]..."这样的报错。
问题2：你安装的flannel 插件/opt/cni/bin 没有flannel程序
# 创建目录（如果不存在）
sudo mkdir -p /opt/cni/bin #你有这个目录但是没有flannel这个执行程序
# 下载 CNI 插件（推荐使用 v0.8.6 版本）
wget https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
# 解压到指定目录
sudo tar -xzvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin/
# 验证 flannel 插件是否已存在
ls -la /opt/cni/bin/ | grep flannel
# 设置执行权限
chmod +x /opt/cni/bin/flannel
# 重启节点上的容器运行时
# 重启 containerd（根据你的容器运行时选择）
sudo systemctl restart containerd
# 重启 kubelet
sudo systemctl restart kubelet
评论 (0)