Freedom_ATX's Blog
0%

Kubernetes组件etcd相关知识点总结

发表于 更新于 分类于 ，

Kubernetes组件etcd相关知识点总结

本文总结补充自马哥教育相关视频

简介

etcd是CoreOS团队于2013年6月发起的开源项目，其的目标是构建一个高可用的分布式键值（key-value）数据库。Etcd内部采用raft协议作为一致性算法等基于Go语言实现。

官方网站：https://etcd.io/

github 地址：https://github.com/etcd-io/etcd

官方硬件推荐：https://etcd.io/docs/v3.4/op-guide/hardware/

etcd具有下面这些属性:

  • 完全复制:集群中的每个节点都可以使用完整的存档
  • 高可用性:Etcd可用于避免硬件的单点故障或网络问题
  • 一致性:每次读取都会返回跨多主机的最新写入
  • 简单:包括一个定义良好、面向用户的API(gRPC)
  • 安全:实现了带有可选的客户端证书身份验证的自动化TLS
  • 快速:每秒10000次写入的基准速度
  • 可靠:使用Raft算法实现了存储的合理分布Etcd的工作原理

配置文件

etcd可以直接将启动参数写在service文件中

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/  # 数据保存目录
ExecStart=/opt/kube/bin/etcd \  # 二进制文件路径
  --name=etcd-192.168.174.134 \
  --cert-file=/etc/kubernetes/ssl/etcd.pem \
  --key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --peer-cert-file=/etc/kubernetes/ssl/etcd.pem \
  --peer-key-file=/etc/kubernetes/ssl/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem \
  --initial-advertise-peer-urls=https://192.168.174.134:2380 \  # 通告自己的集群端口
  --listen-peer-urls=https://192.168.174.134:2380 \  # 集群之间通讯端口
  --listen-client-urls=https://192.168.174.134:2379,http://127.0.0.1:2379 \  # 客户端访问地址
  --advertise-client-urls=https://192.168.174.134:2379 \  # 通告自己的客户端端口
  --initial-cluster-token=etcd-cluster-0 \  # 创建集群使用的token，一个集群内的节点保持一致
  --initial-cluster=etcd-192.168.174.134=https://192.168.174.134:2380,etcd-192.168.174.128=https://192.168.174.128:2380,etcd-192.168.174.135=https://192.168.174.135:2380 \  # 集群所有节点信息 
  --initial-cluster-state=new \  # 新建集群的时候的值为new, 如果是已经存在的集群为existing
  --data-dir=/var/lib/etcd \  # 数据目录路径
  --snapshot-count=50000 \
  --auto-compaction-retention=1 \
  --max-request-bytes=10485760 \
  --auto-compaction-mode=periodic \
  --quota-backend-bytes=8589934592
Restart=always
RestartSec=15
LimitNOFILE=65536
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

或者可以不将启动参数写在service文件中，使用config文件方式定义，例如：

创建etcd配置文件

vim /etc/etcd/etcd.conf

# 节点名称
ETCD_NAME="etcd0"
# 指定数据文件存放位置
ETCD_DATA_DIR="/var/lib/etcd/"

创建systemd配置文件

vim /etc/systemd/system/etcd.service

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
 
[Service]
User=root
Type=notify
WorkingDirectory=/var/lib/etcd/
## 根据实际情况修改EnvironmentFile和ExecStart这两个参数值
## 1.EnvironmentFile即配置文件的位置，注意“-”不能少
EnvironmentFile=-/etc/etcd/etcd.conf
## 2.ExecStart即etcd启动程序位置
ExecStart=/usr/local/bin/etcd
Restart=on-failure
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target

etcd 的基本使用

etcd有多个不同的API访问版本，v1版本已经废弃，etcd v2 和 v3本质上是共享同一套raft 协议代码的两个独立应用，接口不一样，存储不一样，数据互相隔离，也就是说如果从etcd v2 升级到 v3 原来v2 的数据还是只能用v2 的接口访问，v3 的接口创建的数据也只能通过v3 的接口访问

增加和修改，如果存在则替换

查询
etcdctl put <键名> <键值> 
删除
etcdctl get <键名> 
查看集群状态
etcdctl del <键名>
watch命令
etcdctl endpoint status --write-out=table # watch是监听键或前缀发生改变的事件流。

对某个key监听操作，当/key1发生改变时，会返回最新值

etcdctl watch /key1 

# 在node1中监听/data
[root@node1 ~]# etcdctl watch /data
# 在node2中给/data节点赋值
[root@node2 ~]# etcdctl put /data "hello etcd"
OK
此时node1监听到节点对应值
[root@node1 ~]# etcdctl watch /data
PUT
/data
hello etcd

监听key前缀

etcdctl watch /key --prefix

监听到改变后执行相关操作

etcdctl watch /key1 -- etcdctl member list

etcd 集群成员的心跳信息

root@node1:~# export NODE_IPS="192.168.174.134 192.168.174.128 192.168.174.135"

root@node2:~# for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/local/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem endpoint health; done
https://192.168.174.134:2379 is healthy: successfully committed proposal: took = 14.611795ms
https://192.168.174.128:2379 is healthy: successfully committed proposal: took = 11.889947ms
https://192.168.174.135:2379 is healthy: successfully committed proposal: took = 12.589934ms

etcd 集群的成员信息

------------------------------+------------+
[root@node1 ~]# ETCDCTL_API=3 etcdctl --write-out=table member list --endpoints=https://192.168.174.134:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem
+------------------+---------+----------------------+------------------------------+------------------------------+------------+
|        ID        | STATUS  |         NAME         |          PEER ADDRS          |         CLIENT ADDRS         | IS LEARNER |
+------------------+---------+----------------------+------------------------------+------------------------------+------------+
| 65f8d952bfce7d85 | started | etcd-192.168.174.128 | https://192.168.174.128:2380 | https://192.168.174.128:2379 |      false |
| 9d16670b8c95b723 | started | etcd-192.168.174.134 | https://192.168.174.134:2380 | https://192.168.174.134:2379 |      false |
| b36760bf3ef3fc98 | started | etcd-192.168.174.135 | https://192.168.174.135:2380 | https://192.168.174.135:2379 |      false |
+------------------+---------+----------------------+------------------------------+------------------------------+------------+

显示etcd群集的详细信息

[root@node1 ~]# export NODE_IPS="192.168.174.128 192.168.174.134 192.168.174.135"
[root@node1 ~]# for ip in ${NODE_IPS}; do ETCDCTL_API=3 etcdctl --write-out=table endpoint status --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/etcd.pem --key=/etc/kubernetes/ssl/etcd-key.pem; done
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.174.128:2379 | 65f8d952bfce7d85 |  3.4.13 |   20 kB |     false |      false |       768 |         32 |                 32 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.174.134:2379 | 9d16670b8c95b723 |  3.4.13 |   25 kB |     false |      false |       768 |         32 |                 32 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|           ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.174.135:2379 | b36760bf3ef3fc98 |  3.4.13 |   20 kB |      true |      false |       768 |         32 |                 32 |        |
+------------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

查看所有的key

[root@node1 ~]# ETCDCTL_API=3 etcdctl get / --prefix --keys-only  #以路径的⽅式所有key信息
/data
/data/user
...

etcd 中查询k8s相关信息

查看所有的key

root@etcd01:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only  #以路径的⽅式所有key信息
......
/registry/services/endpoints/kubernetes-dashboard/kubernetes-dashboard

/registry/services/specs/default/kubernetes

/registry/services/specs/kube-system/kube-dns

/registry/services/specs/kubernetes-dashboard/dashboard-metrics-scraper

/registry/services/specs/kubernetes-dashboard/kubernetes-dashboard

查看kubernetes中所有pod的信息

root@etcd01:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep pods
/registry/pods/kube-system/calico-kube-controllers-647f956d86-srt9s
/registry/pods/kube-system/calico-node-7f2kc
/registry/pods/kube-system/calico-node-ccv26
/registry/pods/kube-system/calico-node-kw499
/registry/pods/kube-system/calico-node-r4kvx
/registry/pods/kube-system/calico-node-rqm8f
/registry/pods/kube-system/calico-node-vjm2k
/registry/pods/kube-system/coredns-55d54f7cfb-74vh8
/registry/pods/kubernetes-dashboard/dashboard-metrics-scraper-856586f554-595fb
/registry/pods/kubernetes-dashboard/kubernetes-dashboard-79b875f7f8-5qzn4
#在kubernetes中查看pod信息
root@k8s-master01:~# kubectl get pods -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
kube-system            calico-kube-controllers-647f956d86-srt9s     1/1     Running   0          42h
kube-system            calico-node-7f2kc                            1/1     Running   2          11d
kube-system            calico-node-ccv26                            1/1     Running   2          11d
kube-system            calico-node-kw499                            0/1     Running   1          6d
kube-system            calico-node-r4kvx                            1/1     Running   5          11d
kube-system            calico-node-rqm8f                            1/1     Running   2          11d
kube-system            calico-node-vjm2k                            1/1     Running   2          11d
kube-system            coredns-55d54f7cfb-74vh8                     1/1     Running   0          42h
kubernetes-dashboard   dashboard-metrics-scraper-856586f554-595fb   1/1     Running   0          42h
kubernetes-dashboard   kubernetes-dashboard-79b875f7f8-5qzn4        1/1     Running   1          42h

查看kubernetes中所有namespace的信息

root@etcd01:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep namespaces
/registry/namespaces/default
/registry/namespaces/kube-node-lease
/registry/namespaces/kube-public
/registry/namespaces/kube-system
/registry/namespaces/kubernetes-dashboard

#在kubernetes中查看namespaces信息
root@k8s-master01:~# kubectl get namespaces 
NAME                   STATUS   AGE
default                Active   11d
kube-node-lease        Active   11d
kube-public            Active   11d
kube-system            Active   11d
kubernetes-dashboard   Active   10d

查看kubernetes中所有deployments的信息

root@etcd01:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep deployments
/registry/deployments/kube-system/calico-kube-controllers
/registry/deployments/kube-system/coredns
/registry/deployments/kubernetes-dashboard/dashboard-metrics-scraper
/registry/deployments/kubernetes-dashboard/kubernetes-dashboard

#在kubernetes中查看deployments
root@k8s-master01:~# kubectl get deployments -A
NAMESPACE              NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
kube-system            calico-kube-controllers     1/1     1            1           11d
kube-system            coredns                     1/1     1            1           11d
kubernetes-dashboard   dashboard-metrics-scraper   1/1     1            1           10d
kubernetes-dashboard   kubernetes-dashboard        1/1     1            1           10d

查看calico网络组件信息

root@etcd01:~# ETCDCTL_API=3 etcdctl get / --prefix --keys-only | grep calico
/calico/ipam/v2/assignment/ipv4/block/172.20.122.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.135.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.32.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.58.192-26
/calico/ipam/v2/assignment/ipv4/block/172.20.85.192-26
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master01
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master02
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node01
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node02
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node03
/calico/ipam/v2/handle/k8s-pod-network.44820babdec34cd55e26d1f73bd1d62dbffc12cb48453380b9d37b8e27cacfbc
/calico/ipam/v2/handle/k8s-pod-network.5bf9b2f255631210515d4b6722cc317f0f81fdacf60f47f3746fbcde8a239c0d
/calico/ipam/v2/handle/k8s-pod-network.b33e81d8d1d9de9cd404d9de66c1615c4bcfcbe9d93fc92972f0fd79f6f0d983
/calico/ipam/v2/host/k8s-master01/ipv4/block/172.20.32.128-26
/calico/ipam/v2/host/k8s-master02/ipv4/block/172.20.122.128-26
/calico/ipam/v2/host/k8s-node01/ipv4/block/172.20.85.192-26
/calico/ipam/v2/host/k8s-node02/ipv4/block/172.20.58.192-26
/calico/ipam/v2/host/k8s-node03/ipv4/block/172.20.135.128-26
......

查看指定的key

#查看namespaces中default的key
root@etcd01:~#  ETCDCTL_API=3 etcdctl get /registry/namespaces/default
/registry/namespaces/default
k8s

v1    Namespace 

default"*$014daf97-460a-4bf1-8c45-7cd238da53532´Z&
ubernetes.io/metadata.namedefaultz{
kube-apiserverUpdatev´FieldsV1:I
G{"f:metadata":{"f:labels":{".":{},"f:kubernetes.io/metadata.name":{}}}} 


kubernetes 
Active"

#查看calico的key
root@etcd01:~# ETCDCTL_API=3 etcdctl get /calico/ipam/v2/assignment/ipv4/block/172.20.122.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.122.128-26
{"cidr":"172.20.122.128/26","affinity":"host:k8s-master02","allocations":[0,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null],"unallocated":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],"attributes":[{"handle_id":"ipip-tunnel-addr-k8s-master02","secondary":{"node":"k8s-master02","type":"ipipTunnelAddress"}}],"deleted":false}

查看所有calico的数据

root@etcd01:~# ETCDCTL_API=3 etcdctl get --keys-only --prefix /calico
/calico/ipam/v2/assignment/ipv4/block/172.20.122.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.135.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.32.128-26
/calico/ipam/v2/assignment/ipv4/block/172.20.58.192-26
/calico/ipam/v2/assignment/ipv4/block/172.20.85.192-26
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master01
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-master02
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node01
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node02
/calico/ipam/v2/handle/ipip-tunnel-addr-k8s-node03
......

etcd备份与恢复

etcd数据备份流程

ETCD 不同的版本的 etcdctl 命令不一样，但大致差不多，本文备份使用 napshot save , 每次备份一个节点就行。

命令备份（k8s-master1 机器上备份）：

$ ETCDCTL_API=3 etcdctl --cacert=/opt/kubernetes/ssl/ca.pem --cert=/opt/kubernetes/ssl/server.pem --key=/opt/kubernetes/ssl/server-key.pem --endpoints=https://192.168.1.36:2379 snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db

备份脚本（k8s-master1 机器上备份）：

#!/usr/bin/env bash

date;

CACERT="/opt/kubernetes/ssl/ca.pem"
CERT="/opt/kubernetes/ssl/server.pem"
EKY="/opt/kubernetes/ssl/server-key.pem"
ENDPOINTS="192.168.1.36:2379"

ETCDCTL_API=3 etcdctl \
--cacert="${CACERT}" --cert="${CERT}" --key="${EKY}" \
--endpoints=${ENDPOINTS} \
snapshot save /data/etcd_backup_dir/etcd-snapshot-`date +%Y%m%d`.db

# 备份保留30天
find /data/etcd_backup_dir/ -name *.db -mtime +30 -exec rm -f {} \;

etcd 数据恢复流程

当etcd集群宕机数量超过集群总节点数一半以上的时候（如总数为3台，宕机2台），就会导致集群宕机，后期需要重新恢复数据，则数据恢复流程如下：

  1. 恢复服务器系统
  2. 重新部署etcd集群
  3. 停止kube-apiserver/controler-manager/scheduler/kubelet/kube-proxy
  4. 停止ETCD集群
  5. 各ETCD节点恢复同一备份数据
  6. 启动各节点并验证ETCD集群
  7. 启动kube-apiserver/controler-manager/scheduler/kubelet/kube-proxy
  8. 验证k8s master 状态及pod 数据

准备工作

  • 停止所有 Master 上 kube-apiserver 服务
$ systemctl stop kube-apiserver  

# 确认 kube-apiserver 服务是否停止 
$ ps -ef | grep kube-apiserver
  • 停止集群中所有 ETCD 服务
$ systemctl stop etcd
  • 移除所有 ETCD 存储目录下数据
$ mv /var/lib/etcd/default.etcd /var/lib/etcd/default.etcd.bak
  • 拷贝 ETCD 备份快照
# 从 k8s-master1 机器上拷贝备份 
$ scp /data/etcd_backup_dir/etcd-snapshot-20191222.db root@k8s-master2:/data/etcd_backup_dir/ 
$ scp /data/etcd_backup_dir/etcd-snapshot-20191222.db root@k8s-master3:/data/etcd_backup_dir/

恢复备份

# k8s-master1 机器上操作
$ ETCDCTL_API=3 etcdctl snapshot restore /data/etcd_backup_dir/etcd-snapshot-20191222.db \
  --name etcd-0 \
  --initial-cluster "etcd-0=https://192.168.1.36:2380,etcd-1=https://192.168.1.37:2380,etcd-2=https://192.168.1.38:2380" \
  --initial-cluster-token etcd-cluster \
  --initial-advertise-peer-urls https://192.168.1.36:2380 \
  --data-dir=/var/lib/etcd/default.etcd
  
# k8s-master2 机器上操作
$ ETCDCTL_API=3 etcdctl snapshot restore /data/etcd_backup_dir/etcd-snapshot-20191222.db \
  --name etcd-1 \
  --initial-cluster "etcd-0=https://192.168.1.36:2380,etcd-1=https://192.168.1.37:2380,etcd-2=https://192.168.1.38:2380"  \
  --initial-cluster-token etcd-cluster \
  --initial-advertise-peer-urls https://192.168.1.37:2380 \
  --data-dir=/var/lib/etcd/default.etcd
  
# k8s-master3 机器上操作
$ ETCDCTL_API=3 etcdctl snapshot restore /data/etcd_backup_dir/etcd-snapshot-20191222.db \
  --name etcd-2 \
  --initial-cluster "etcd-0=https://192.168.1.36:2380,etcd-1=https://192.168.1.37:2380,etcd-2=https://192.168.1.38:2380"  \
  --initial-cluster-token etcd-cluster \
  --initial-advertise-peer-urls https://192.168.1.38:2380 \
  --data-dir=/var/lib/etcd/default.etcd

上面三台 ETCD 都恢复完成后，依次登陆三台机器启动 ETCD

$ systemctl start etcd

三台 ETCD 启动完成，检查 ETCD 集群状态

$ ETCDCTL_API=3 etcdctl --cacert=/opt/kubernetes/ssl/ca.pem --cert=/opt/kubernetes/ssl/server.pem --key=/opt/kubernetes/ssl/server-key.pem --endpoints=https://192.168.1.36:2379,https://192.168.1.37:2379,https://192.168.1.38:2379 endpoint health

三台 ETCD 全部健康，分别到每台 Master 启动 kube-apiserver

$ systemctl start kube-apiserver

检查 Kubernetes 集群是否恢复正常

$ kubectl get cs

kubeasz备份恢复

./ezctl backup k8s-01
kubectl delete pod net-test1 -n default  # 删除pod模拟数据丢失
./ezctl restore k8s-01  # 尝试从备份中恢复pod

Velero 结合mino 实现k8s 业务数据备份与恢复

Velero 是一个云原生的灾难恢复和迁移工具，它本身也是开源的,采用Go语言编写，可以安全的备份、恢复和迁移 Kubernetes集群资源数据。

Velero 是西班牙语意思是帆船，非常符合Kubernetes社区的命名风格，Velero的开发公司Heptio，已被VMware收购。

Velero 支持标准的K8S集群，既可以是私有云平台也可以是公有云，除了灾备之外它还能做资源移转，支持把容器应用 从一个集群迁移到另一个集群。

image-20220512142839572

minio 安装

# 拉取镜像
[root@manager ~]# docker pull minio/minio:latest
latest: Pulling from minio/minio
d46336f50433: Pull complete
be961ec68663: Pull complete
44173c602141: Pull complete
a9809a6a679b: Pull complete
df29d4a76971: Pull complete
2b5a8853d302: Pull complete
84f01ee8dfc1: Pull complete
Digest: sha256:d786220feef7d8fe0239d41b5d74501dc824f6e7dd0e5a05749c502fff225bf3
Status: Downloaded newer image for minio/minio:latest
docker.io/minio/minio:latest

# 创建数据目录
[root@manager ~]# mkdir -pv /data/minio
mkdir: 已创建目录 "/data"
mkdir: 已创建目录 "/data/minio"

# 创建minio容器,如果不指定,则默认用户名与密码为 minioadmin/minioadmin,可以通过环境变量自定义,如下:
[root@etcd-1 ~]# docker run --name minio -p 9000:9000 -p 9999:9999 \
> -d --restart=always \
> -e "MINIO_ROOT_USER=admin" \
> -e "MINIO_ROOT_PASSWORD=ddrbdgzy" \
> -v /data/minio/data:/data \
> minio/minio:latest server /data \
> --console-address '0.0.0.0:9999'

浏览器访问9999端口：

image-20220512164607646

使用创建容器时指定的用户名密码登录，并创建一个名为velerodata的bucket：

image-20220512185752491

velero 安装

配置velero认证环境：
# 工作目录：
[root@etcd-1 /]#:~# mkdir  /data/velero -p

# 认证文件：
[root@etcd-1 velero]#:/data/velero# vim velero-auth.txt 
[default]
aws_access_key_id = admin
aws_secret_access_key = ddrbdgzy

# 准备user-csr文件：
root@k8s-master1:/data/velero# vim awsuser-csr.json
{
  "CN": "awsuser",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

# 准备证书签发环境：
[root@etcd-1 velero]:/data/velero# apt install golang-cfssl
https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64 
https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64 
https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64

# 执行证书签发
[root@etcd-1 velero]# ./cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=./ca-key.pem -config=./ca-config.json -profile=kubernetes ./awsuser-csr.json | ./cfssljson -bare awsuser

签发完成后在当前目录有如下文件：

image-20220512195444916

#分发证书到api-server证书路径：
[root@master ~]# ll /etc/kubernetes/ssl/
总用量 32
-rw-r--r--. 1 root root 1679 4月  26 20:18 aggregator-proxy-key.pem
-rw-r--r--. 1 root root 1383 4月  26 20:18 aggregator-proxy.pem
-rw-r--r--. 1 root root 1679 4月  26 20:18 ca-key.pem
-rw-r--r--. 1 root root 1350 4月  26 20:18 ca.pem
-rw-r--r--. 1 root root 1675 4月  26 20:18 kubelet-key.pem
-rw-r--r--. 1 root root 1452 4月  26 20:18 kubelet.pem
-rw-r--r--. 1 root root 1675 4月  26 20:18 kubernetes-key.pem
-rw-r--r--. 1 root root 1610 4月  26 20:18 kubernetes.pem

#生成集群认证config文件：
# export KUBE_APISERVER="https://192.168.68.152:6443"
# kubectl config set-cluster kubernetes \
--certificate-authority=/etc/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=./awsuser.kubeconfig

# 查看生成的 awsuser.kubeconfig
[root@master ~]# cat awsuser.kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR1RENDQXFDZ0F3SUJBZ0lVVGlOV1o4REJEZGFkRTNkQkRGNVAwdVlJMVJvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qSXdOREkyTVRFeE9UQXdXaGdQTWpFeU1qQTBNREl4TVRFNU1EQmFNR0V4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEREQUtCZ05WQkFvVApBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQW5ueGY1Ly9yemg4V2FmeEI5NVljQ3VBaHY3VTIKSUl2cVJOT2g4UmlGTVFHcDVzdHFOaXREcEZ0Rk5BbzhsanJjNjBIQWttdXpwSkRJOXVxdG1vU1NXOFB2Si9VeQo4dzJBbDhxa0c3alhncjF3V2MrUU1qTzhRV1BiRXAzM3djUGE4ckVpZ1hOTzRDckxUM0NKQ2xydk1KbWNFWDkwCjhTUjlKb01hbGtnaEFnNkdsTlVFMTZHZi84dkwzQmdjOWI0ZzV0VWswNWRTV0R4SG5aenRGVHgrZTlxSThYemsKVXUzenF3Wll1MHdwQlpPb3F3bzBqQnQrdnhhbWxOTGFJdG5TaytPcFVXRk8xVGpmS016K3lmeFc2TUg0UThhVgphb2x0N3ZsTkhuZlRUakw3R3VHaCthbmZ6N2hkRUxFbkNJblJkMGY1ellVV1FBWnRhSlRnV0R4QTFRSURBUUFCCm8yWXdaREFPQmdOVkhROEJBZjhFQkFNQ0FRWXdFZ1lEVlIwVEFRSC9CQWd3QmdFQi93SUJBakFkQmdOVkhRNEUKRmdRVWFxVFhmL3VMK3VvcHBBYWFsaTIyY3UvdTlBNHdId1lEVlIwakJCZ3dGb0FVYXFUWGYvdUwrdW9wcEFhYQpsaTIyY3UvdTlBNHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBRGpldVBLTmFESEhKd3VaWDNiMmNocDliWW5KCkZlbXp4K3JPRzVpckpoeWt6cHprMTZPelJIZlJjMGd4SFdjOC9jZFBBd1hQK2dFdWkzZ1lOdGpSS3lTV2RjNmcKUVJ6bGZlWkovWUpjdFFjWG9QTU1YNlYxS2dWS3IwMlh0QjJNR1FQd0VHR2RrQUp3ekh0RmRaRnRmQmQ3dEQ4NQo3SUNrZ20xM0d0STVmLzBLRXh2VEJKdlhCUWthcjRITEtYTndRc3piUW5yeE14ajdISlo5ZFlidnZDZjgvUkpPCkFvdXBYVFppcURNaXJENnJUVzNUWTdEb05iY2FtcVpKZk5tc2NGdWZJcVBkZGw1L2lHS3VDckJEZXBmMjRVU3QKM1I4TzJSNjFZbHowYkk3TXVhWUk3aHFtWmg5QmFQUFU2T3kxZ2dlMjdpczRaT2h1anBoVkZ6MW9nVk09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://192.168.68.152:6443
  name: kubernetes
# 后续信息缺失，需要继续生成加入用户的认证信息
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null

# 设置客户端证书认证
[root@master ~]# kubectl config set-credentials awsuser --client-certificate=/etc/kubernetes/ssl/awsuser.pem --client-key=/etc/kubernetes/ssl/awsuser-key.pem --embed-certs=true --kubeconfig=./awsuser.kubeconfig
User "awsuser" set.

再次查看

image-20220512202610837

可以看到用户公钥私钥已经加入

#设置上下文参数:
# kubectl config set-context kubernetes \
--cluster=kubernetes \
--user=awsuser \
--namespace=velero-system \
--kubeconfig=./awsuser.kubeconfig

#设置默认上下文:
#kubectl config use-context kubernetes --kubeconfig=awsuser.kubeconfig

可以看到上下文信息已经写入

image-20220512203235386

# 执行velero server端安装：
velero --kubeconfig  ./awsuser.kubeconfig \
	install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.3.1 \
    --bucket velerodata  \
    --secret-file ./velero-auth.txt \
    --use-volume-snapshots=false \
	--namespace velero-system \
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.68.148:9000/

可以看到velero已经启动

image-20220512203837164

image-20220512203908533

此时velero安装完成

使用velero + minio 备份

#验证安装：
root@k8s-master1:/data/velero# kubectl  describe pod velero-6755cb8697-phfsr -n velero-system  

velero backup create magedu-n66-20220417  --include-namespaces myserver


DATE=`date +%Y%m%d%H%M%S`
velero backup create myserver-ns-backup-${DATE} \
--include-namespaces myserver \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

root@k8s-master1:/data/velero# kubectl  get pod -n velero-system 
NAME                  READY   STATUS    RESTARTS   AGE
velero-6755cb8697-phfsr   1/1     Running     0          5m45s

velero backup create myserver-ns-backup-2022041621 \
--include-namespaces myserver \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system


velero restore create --from-backup default-backup-20220416205931 --wait --kubeconfig=./awsuser.kubeconfig --namespace velero-system

velero restore create --from-backup myserver-ns-backup-20220417155913 --wait \
--kubeconfig=./awsuser.kubeconfig \
--namespace velero-system

image-20220512210852408

欢迎关注我的其它发布渠道