现在位置: 首页 / 云容器 / containerd / 正文

kubeadm安装基于containerd的1.24.4版本kubernetes

  • 单手暴扣
  • 云容器
  • 2022-08-29
  • 82热度
  • 0评论

基础环境介绍

三台机器，1master+2node，均为 2C4G，Ubuntu 22.04.1 LTS
master01 192.168.88.131
node01 192.168.88.132
node02 192.168.88.133

基础环境配置

转发 IPv4 并让 iptables 看到桥接流量

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 设置所需的 sysctl 参数，参数在重新启动后保持不变
# 为了让 Linux 节点的 iptables 能够正确查看桥接流量
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 应用 sysctl 参数而不重新启动
sudo sysctl --system

Linux参数设置

禁用swap：

sed -ri 's/.*swap.*/#&/' /etc/fstab
swapoff -a

关闭 selinux

sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0

配置Kubernetes源

安装使用k8s仓库所需包

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

添加 Kubernetes apt 仓库

这里使用阿里源，谷歌的墙原因无法访问

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg
echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

如果是 CentOS 的话，配置如下：
yum -y install conntrack-tools ca-certificates curl

[root@hxyw-app-09 ~]# yum -y install conntrack-tools ca-certificates curl
[root@hxyw-app-09 ~]# cat /etc/yum.repos.d/k8s.repo
[aliyun-k8s]
name=aliyun k8s - $basearch
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
#gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

[root@hxyw-app-09 ~]# yum clean all
[root@hxyw-app-09 ~]# yum makecache
[root@hxyw-app-09 ~]# yum search kubeadm --showduplicates
[root@hxyw-app-09 ~]# yum install kubeadm-1.24.4-0.x86_64 kubelet-1.24.4-0.x86_64 kubectl-1.24.4-0.x86_64
[root@hxyw-app-09 ~]# yum install yum-versionlock
[root@hxyw-app-09 ~]# yum versionlock add kubectl kubeadm kubelet

安装 kubeadm、kubelet、kubectl

sudo apt-get update

# 可以查看版本
sudo apt-cache madison kubeadm | head
# 安装指定版本
sudo apt-get install -y kubelet=1.24.4-00 kubeadm=1.24.4-00 kubectl=1.24.4-00
# 保持版本不被更新
sudo apt-mark hold kubelet kubeadm kubectl

containerd 安装

安装

参考地址：https://github.com/containerd/containerd/blob/main/docs/getting-started.md

有三种方式：

  1. 通过官方二进制包安装
    第一步：安装 containerd
    下载地址：https://github.com/containerd/containerd/releases
    二进制包是通过基于 glibc 的 Linux 发行版进行动态构建的，例如 Ubuntu 和 Rocky Linux，二进制包可能不能在基于 musl 构建的发行版上运行，如 Alpine Linux，这种发行版的用户可能必须从源代码或第三方包中安装容器。
    由于Kubernetes CRI特性已经包含在 containerd 的 tar.gz 安装包中，所以不需要额外下载 cri-containerd 包来使用 CRI。
    cri-containerd 包已经被弃用，无法在老的 Linux 发行版中工作，也将会在 containerd 2.0 版本中移除。
    第二步：安装 runc
    下载地址：https://github.com/opencontainers/runc/releases
    二进制文件是静态构建的,可以在任何Linux发行版上工作。
    第三步：安装 CNI 插件
    下载地址：https://github.com/containernetworking/plugins/releases
    二进制文件是静态构建的,可以在任何Linux发行版上工作。
  2. 通过 apt-get 或者 dnf 安装
    CentOS
    Ubuntu

    • 安装docker仓库源
      • apt-get update
      • apt-get install ca-certificates curl gnupg lsb-release
      • sudo mkdir -p /etc/apt/keyrings
      • curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
      • echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    • 安装 containerd
      • apt install containerd.io（安装时会将依赖的 runc 一起安装了，通过 dpkg -L containerd.io 可以查看安装了哪些）
      • dpkg -L containerd.io 
        root@master01:~# dpkg -L containerd.io
/.
/etc
/etc/containerd
/etc/containerd/config.toml(配置文件)
/lib
/lib/systemd
/lib/systemd/system
/lib/systemd/system/containerd.service
/usr
/usr/bin
/usr/bin/containerd
/usr/bin/containerd-shim
/usr/bin/containerd-shim-runc-v1
/usr/bin/containerd-shim-runc-v2
/usr/bin/ctr
/usr/bin/runc
/usr/share
/usr/share/doc
/usr/share/doc/containerd.io
/usr/share/doc/containerd.io/changelog.Debian.gz
/usr/share/doc/containerd.io/copyright
/usr/share/man
/usr/share/man/man5
/usr/share/man/man5/containerd-config.toml.5.gz
/usr/share/man/man8
/usr/share/man/man8/containerd-config.8.gz
/usr/share/man/man8/containerd.8.gz
/usr/share/man/man8/ctr.8.gz
  3. 通过源码编译安装
    参考：https://github.com/containerd/containerd/blob/main/BUILDING.md

我们这边使用第 2 种，containerd.io的 deb 或者 rpm 格式的包是由 docker 发布的，不是 containerd 项目发布的，可以参考 docker 文档如何设置 apt-get 或者 dnf 来安装 containerd.io 包。containerd.io 包含了 runC 但是不包含 CNI 插件。

客户端

containerd 有几种客户端可与其交互

名称 社区 API 目标 网站
ctr containerd 原生 仅用作调试 (无, 参考 ctr --help 来学习使用)
nerdctl containerd (non-core) 原生 通用 https://github.com/containerd/nerdctl
crictl Kubernetes SIG-node CRI 仅用作调试 https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md

虽然 ctr 工具与 containerd 捆绑在一起,但应该注意到 ctr 工具仅用于调试容器。nerdctl 工具提供稳定和人性化的用户体验。

containerd 配置

重写配置文件中的仓库地址

  1. 重写默认配置文件

默认的配置文件 /etc/containerd/config.toml 中的内容很少，需要将 containerd 的所有默认配置输出来重写到默认的配置文件中：

mkdir -p /etc/containerd
containerd config default | tee /etc/containerd/config.toml
  1. 修改默认配置中的参数
vim /etc/containerd/config.toml

#修改以下的key为下面新值:

sandbox_image = "k8s.gcr.io/pause:3.6"
SystemdCgroup = false
修改为：
sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"
SystemdCgroup = true
  1. 重启containerd服务
root@master01:~# systemctl restart containerd.service 
root@master01:~# ctr images list

> ctr 是 containerd 自带的用于调试的客户端
  1. 修改crictl获得containerd的sock信息

crictl是对ctr命令的封装，通过 critools 包进行安装的，默认连接的是 dockershim.sock

root@master01:~# cat /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false

root@master01:~# systemctl restart containerd.service 
root@master01:~# systemctl enable containerd.service
root@master01:~# systemctl enable kubelet.service

通过kubeadm初始化K8S集群

初始化单节点 master

可以查看安装k8s需要拉取的镜像

root@master01:~# kubeadm config images list
I0831 21:57:23.561808    8802 version.go:255] remote version is much newer: v1.25.0; falling back to: stable-1.24
k8s.gcr.io/kube-apiserver:v1.24.4
k8s.gcr.io/kube-controller-manager:v1.24.4
k8s.gcr.io/kube-scheduler:v1.24.4
k8s.gcr.io/kube-proxy:v1.24.4
k8s.gcr.io/pause:3.7
k8s.gcr.io/etcd:3.5.3-0
k8s.gcr.io/coredns/coredns:v1.8.6

执行初始化

root@master01:~# kubeadm init --kubernetes-version=1.24.4 --apiserver-advertise-address=192.168.88.131 --apiserver-bind-port=6443 --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap --cri-socket=unix:///run/containerd/containerd.sock

[init] Using Kubernetes version: v1.24.4
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: missing optional cgroups: blkio
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master01] and IPs [10.96.0.1 192.168.88.131]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master01] and IPs [192.168.88.131 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master01] and IPs [192.168.88.131 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.511211 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 64ahht.0btvrnojx6ygjvvo
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.88.131:6443 --token 64ahht.0btvrnojx6ygjvvo \
    --discovery-token-ca-cert-hash sha256:2b082d313b7173a1f559f51aca9a438c7ef8757df6fded490f72e43155840047

初始化完成后，就可以查看拉取了哪些镜像

root@master01:~# crictl images ls
IMAGE                                                                         TAG                 IMAGE ID            SIZE
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   v1.8.6              a4ca41631cc7a       13.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.5.3-0             aebe758cef4cd       102MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.24.4             6cab9d1bed1be       33.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.24.4             1f99cb6da9a82       31MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.24.4             7a53d1e08ef58       39.5MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.24.4             03fa22539fc1c       15.5MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.6                 6270bb605e12e       302kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.7                 221177c6082a8       311kB

工作节点加入集群

# 此命令为在 master 初始化集群时自动生成并在控制台输出的信息中
root@node01:~# kubeadm join 192.168.88.131:6443 --token 64ahht.0btvrnojx6ygjvvo \
    --discovery-token-ca-cert-hash sha256:2b082d313b7173a1f559f51aca9a438c7ef8757df6fded490f72e43155840047
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: missing optional cgroups: blkio
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

网络插件安装

选择一：安装 Flannel 插件

https://kubernetes.io/docs/concepts/cluster-administration/addons/

这里选择使用 flannel
github地址：https://github.com/flannel-io/flannel#deploying-flannel-manually

yml文件下载到master：https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel.yml

执行：kubectl apply -f kube-flannel.yml

root@master01:~# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

如果要查看pod的运行状态的话，执行kubectl命令注意要指定命名空间，aliyun的yml文件命名空间都是 kube-system，但是 我们下载的yml文件的命名空间是 kube-flannel

安装完成后，kube-system空间下的 core-dns 为running状态，工作节点状态也会变成 Ready

方拾二：安装 Calico 插件

https://projectcalico.docs.tigera.io/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico

通过本地部署的资源清单方式进行安装
file

有三种类型安装：

  • 通过k8sapi，小于50个节点的推荐安装
  • 通过k8sapi，大于50个节点的推荐安装
  • 使用 etcd 数据存储进行安装

使用第一种类型：

  1. 获取资源清单文件
curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -O
  1. 修改 calico.yaml 中的 CALICO_IPV4POOL_CIDR 配置与初始化集群时的配置一致：
- name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
# Cluster type to identify the deployment type
- name: CLUSTER_TYPE
  value: "k8s,bgp"
# 以下配置为新增
- name: IP_AUTODETECTION_METHOD
  value: "interface=eth0"
  1. 应用资源清单
kubectl create -f calico.yaml

metric-server安装

参考文档及下载

# 下载 yaml 文件
root@master01:~# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml

# 修改镜像，因是从k8s镜像仓去下载，国内无法访问，所以调整镜像，从duckerhub的 bitnami中拉取，镜像tag不带v，需要注意修改。
root@master01:~# vim high-availability.yaml
    - 将： image: k8s.gcr.io/metrics-server/metrics-server:v0.6.1
      改为：image: bitnami/metrics-server:0.6.1
    - 在 containers.args 下新增参数：- --kubelet-insecure-tls

# 执行安装
root@master01:~# kubectl apply -f high-availability.yaml

# 验证
root@master01:~# kubectl top pod -l name=cpu-reload --sort-by=cpu -A