K8s环境搭建

CentOS7下安装K8S 1.23.17 版本。需要提前安装 docker

一、环境准备

主机名 IP 用途 k8s版本 docker版本
harbor-server 192.168.10.250 www、镜像仓库
k8s231.bravexist.cn 192.168.10.231 controller-plane (Master) v1.23.17 20.10.24
k8s232.bravexist.cn 192.168.10.232 Worker Node v1.23.17 20.10.24
k8s233.bravexist.cn 192.168.10.233 Worker Node v1.23.17 20.10.24

k8s231、k8s232、k8s233都需要操作

  1. 关闭swap分区

临时关闭

1
swapoff -a && sysctl -w vm.swappiness=0

基于配置文件的关闭

1
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
  1. 确保各个节点MAC地址或product_uuid唯一
1
ip address show ens33 |grep ether |awk '{print $2}'
1
cat /sys/class/dmi/id/product_uuid
  1. 检查网络节点是否互通
1
ping 192.168.10.231 -c4
  1. 允许iptable检查桥接流量
1
2
3
4
5
6
7
8
9
cat <<EOF | tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
  1. 检查端口是否被占用

新机器没有服务。

  1. 检查docker的环境
  • docker
  • docker compose（不需要）

20.10.24版本没有自带compose

  1. docker配置的优化
1
2
3
4
5
6
mkdir -pv /etc/docker && cat <<EOF | tee /etc/docker/daemon.json
{
  "registry-mirrors": ["https://hub.bravexist.cn"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
  1. 将harbor服务器的客户端证书拷贝到k8s集群
1
mkdir -pv /etc/docker/certs.d/harbor.bravexist.cn

注意 ：接下来是在harbor机器上操作

1
2
3
scp /opt/softwares/harbor/certs/client/* 192.168.10.231:/etc/docker/certs.d/harbor.bravexist.cn
scp /opt/softwares/harbor/certs/client/* 192.168.10.232:/etc/docker/certs.d/harbor.bravexist.cn
scp /opt/softwares/harbor/certs/client/* 192.168.10.233:/etc/docker/certs.d/harbor.bravexist.cn
  1. 开机自启
1
2
3
systemctl daemon-reload
systemctl enable --now docker
systemctl status docker
  1. 关闭防火墙
1
systemctl disable --now firewalld
  1. selinux关闭
1
2
sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config 
grep ^SELINUX= /etc/selinux/config
  1. 配置host解析
1
2
3
4
5
6
7
cat >> /etc/hosts <<'EOF'
192.168.10.231        k8s231.bravexist.cn
192.168.10.232        k8s232.bravexist.cn
192.168.10.233        k8s233.bravexist.cn
192.168.10.250        harbor.bravexist.cn
EOF
cat /etc/hosts
  1. 验证是否能够登录harbor仓库
1
docker login -u admin -p 'Harbor12345' harbor.bravexist.cn

二、安装

所有节点安装kubeadm，kubelet，kubectl

  1. 设置仓库
1
2
3
4
5
6
7
8
cat  > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
  1. 查看kubeadm的版本
1
yum -y list kubeadm --showduplicates
  1. 安装kubeadm，kubelet，kubectl软件包
1
2
KUBE_VERSION=1.23.17-0
yum -y install kubeadm-${KUBE_VERSION} kubelet-${KUBE_VERSION} kubectl-${KUBE_VERSION}
  1. 验证启动,会启动失败
1
kubelet --version
1
2
systemctl enable --now kubelet
systemctl status kubelet
  1. 打包安装包（用于在离线环境）
1
2
3
cd ~
mkdir k8s-rpm-1_23_17-0
find /var/cache/yum -name "*.rpm" |xargs mv -t k8s-rpm-1_23_17-0/
1
tar -zcvf k8s-rpm-1_23_17-0.tar.gz k8s-rpm-1_23_17-0

三、初始化

先导入镜像

master节点执行

1
2
3
4
5
kubeadm init --kubernetes-version=v1.23.17 \
  --image-repository registry.aliyuncs.com/google_containers \
  --pod-network-cidr=10.100.0.0/16 \
  --service-cidr=10.200.0.0/16 \
  --service-dns-domain=bravexist.cn
  • --image-repository 使用阿里云镜像拉取镜像
1
2
3
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
1
2
kubectl get componentstatuses
kubectl get cs
1
kubectl get nodes

worker节点执行（需对应上述的初始化后的提示）

1
2
kubeadm join 192.168.10.231:6443 --token eqlig8.i19f9ug40bv3mpmt \
        --discovery-token-ca-cert-hash sha256:7a1e05d52def714e2e96d58f91507dd21e2b3c33111745af6f769eb157d3d30b

意外没跑起来的话，需要清理环境，重新启动或加入

1
2
3
rm -rf /var/lib/kubelet/*
rm -rf /etc/kubernetes
rm -rf /var/lib/etcd

导出controller-plane镜像

1
mkdir controller-control
1
2
3
docker images --format '{{.Repository}} {{.Tag}}' | \
awk '{f=$1"_"$2".tar";gsub("/","_",f);print $1":"$2, f}' | \
xargs -n2 sh -c 'docker save "$0" -o "$1"'

快速导入镜像的命令

1
for i in *.tar;do docker load -i $i ;done

添加kubectl的自动补全功能

1
echo "source <(kubectl completion bash)" >> ~/.bashrc && source ~/.bashrc

四、安装网络插件

安装网络插件，修改网段为pod的网段 10.100.0.0/16

1
# kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

or

1
2
# wget https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
wget https://g.bravexist.cn/https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
1
vim kube-flannel.yml
1
kubectl apply -f kube-flannel.yml

验证网络插件安装成功

1
kubectl get pods -A -o wide| grep kube-flannel

五、验证pod 间互通

  1. 创建资源的配置文件
1
vim test-network-linux-ds.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: linux-ds
spec:
  selector:
    matchLabels:
      school: tyust
      class: bitdata
  template:
    metadata:
      labels:
        school: tyust
        class: bitdata
    spec:
      containers:
      - image: alpine:3.23.2
        stdin: true
        name: mylinux

创建资源

1
kubectl apply -f test-network-linux-ds.yaml

查看资源

1
kubectl get pods -o wide
1
2
3
4
5
[root@k8s231.bravexist.cn ~]# kubectl get pods -o wide
NAME             READY   STATUS    RESTARTS   AGE   IP           NODE                  NOMINATED NODE   READINESS GATES
linux-ds-68ck2   1/1     Running   0          4s    10.100.2.4   k8s233.bravexist.cn   <none>           <none>
linux-ds-c5plq   1/1     Running   0          4s    10.100.1.2   k8s232.bravexist.cn   <none>           <none>
[root@k8s231.bravexist.cn ~]#

测试跨节点Pod之间的通信

kubectl exec  linux-ds-c5plq -- ping -c 3  10.100.2.4

删除ds资源

1
kubectl delete -f  test-network-linux-ds.yaml