深度解析移动云容器服务KCS创建流程
myluzh 发布于 阅读：43
0x00 前言
作者对移动云订购容器服务CKS后，云主机是怎么自动构建一个k8s集群表示疑惑，所以花了点时间研究一下，
0x01 用户订购
用户在移动云控制台订购K8s集群 选择: 3Master + 2Worker + 规格/网络等配置。
0x02 EKI编排服务解析订单，生成安装配置
当你在云平台上点击“创建一个 K8s 集群”或“添加一个 Master 节点”时，底层其实是通过 OpenStack 生成了一台虚拟机。为了让这台纯净的虚拟机在开机后自动变成一个配置完毕的 K8s 节点，云平台使用了 ConfigDrive 机制和 cloud-init 工具。
生成ConfigDrive配置（挂载到每个虚机的/dev/sr0）
/mnt/cdrom 其实是一块虚拟光驱。云平台把节点需要的初始化数据打包成一个 ISO 文件挂载给虚拟机。虚拟机内部的 cloud-init 服务在操作系统启动时，会自动读取这里面的文件并执行自动化装机。
[root@kcs-k8s-test-m-tp88s /]# df -h | grep sr0
/dev/sr0 666K 666K 0 100% /mnt/cdrom
# ConfigDrive 完整结构
[root@kcs-k8s-test-m-tp88s /mnt/cdrom]# cd /mnt/cdrom && tree
/mnt/cdrom/
├── openstack/
│ ├── 2012-08-10/ # 旧版本格式（兼容）
│ ├── 2013-04-04/
│ ├── 2013-10-17/
│ ├── 2015-10-15/
│ ├── 2016-06-30/
│ ├── 2016-10-06/
│ ├── 2017-02-22/ # ← 实际使用的版本
│ │ ├── meta_data.json # 虚机元数据
│ │ ├── network_data.json # 网络配置
│ │ ├── user_data # 核心安装脚本
│ │ ├── vendor_data.json
│ │ └── vendor_data2.json
│ └── latest/ # 符号链接到最新版本
└── (其他版本目录)meta_data.json (元数据)：
定义了这台虚拟机的身份。包含它的 hostname (kcs-k8s-test-m-tp88s)、uuid 以及用于生成系统随机数的 random_seed。
[root@kcs-k8s-test-m-tp88s /mnt/cdrom/openstack/latest]# cat meta_data.json
{
"random_seed": "ekNPQTUy4qs7343bn34S/VPCaMG028SAT6fqdO+jwgHDTbySM/DVGgwyjqGzeNTR8J/1VMtVoqhlijwXzNDMEwjLV2S6F5xK0WAUkrWOWkfceSLR9OdpMX7iFcoFkYNA1CLsDMjPoCTggF3WJFcpd5uzgEqKiSJ5s6W1uhrZuDnFwW/VFuogtPdjDRlY4/Aeri/yIHxlJUbbW5ckHuYgnezIE1jexEDZyc7vzvKr8os6Ah1Fn+74WLpxnbKjI6BVDfIFU+qQhrIRzx5cZ7BVKL8DUbWK1M4n89nFb542Yr1cQLpcK8n4XNvkgegbjRSqLP08zyw5lCYJnMv5NYHVm6+52lx0bZgLF1l71FbBzVDbAd+zB8uFPd/zUKlyTx+tsriAFyPsa9vVNzJhvrbCXm/d6UaW0OwoRMg2BLnivphK2wKVGltWNvfYAyWdeT7NobgJGGQW+uK+sbyZ6aC8IVZAYjCzffYINGXqVolRfmLm8HwhxGoH603K98+I6nQjRvhQs9i8qut0gVgu+4hHeX90f61NDLqmJ35i3NULl3pObaC7s0BbkT52N7WxtBZVqnsv3Et7pMHEoie+ZXFvyQRYCGusv6TIuyZBuWOTkZdYYi1tMlAodaEwJAA3uKYEfGqtnjyDcCzVNSYruYUiPU5GXn7hBFsYQ9G6aHjh1/c=",
"uuid": "78b66a14-fb70-4fe3-ab2e-74c3b57f5c03",
"hostname": "kcs-k8s-test-m-tp88s",
"launch_index": 0,
"devices": [],
"name": "kcs-k8s-test-m-tp88s"
}network_data.json (网络数据)
定义了底层网络拓扑。cloud-init 会读取里面的 IPv4 (192.168.11.139)、IPv6、网关、DNS (211.136.17.107) 和 OVS 网卡 MAC 地址，自动写成操作系统的网卡配置文件并拉起网络，确保这台机器能通网。
[root@kcs-k8s-test-m-tp88s /mnt/cdrom/openstack/latest]# cat network_data.json
{
"services": [{
"type": "dns",
"address": "211.136.17.107"
}, {
"type": "dns",
"address": "211.136.20.203"
}],
"networks": [{
"network_id": "213e8096-d2ba-4dd6-b1d7-82e549507be9",
"type": "ipv4",
"services": [{
"type": "dns",
"address": "211.136.17.107"
}, {
"type": "dns",
"address": "211.136.20.203"
}],
"netmask": "255.255.255.0",
"link": "tap7399fc12-09",
"routes": [{
"netmask": "0.0.0.0",
"network": "0.0.0.0",
"gateway": "192.168.11.1"
}],
"ip_address": "192.168.11.139",
"id": "network0"
}, {
"network_id": "213e8096-d2ba-4dd6-b1d7-82e549507be9",
"type": "ipv6_dhcpv6-stateful",
"services": [],
"netmask": "ffff:ffff:ffff:ffff::",
"link": "tap7399fc12-09",
"routes": [{
"netmask": "::",
"network": "::",
"gateway": "2409:8c2f:3800:5795::1"
}],
"ip_address": "2409:8c2f:3800:5795::35a",
"id": "network1"
}],
"links": [{
"ethernet_mac_address": "fa:16:3e:ce:43:86",
"mtu": 1600,
"type": "ovs",
"id": "tap7399fc12-09",
"vif_id": "7399fc12-09ec-49ec-84c1-053df9d05833"
}]
}user_data（核心安装脚本）
这个文件是标准的 cloud-config 格式，分为三个主要阶段：
阶段 A：系统权限设置 (disable_root, ssh_pwauth)
开启了 Root 用户的密码登录权限，方便后续的自动化脚本或运维人员介入。
阶段 B：文件注入 (write_files)
云平台管控端会预先生成好集群所需的各种核心文件，经过 base64 编码后塞进这个列表里。cloud-init 会将它们解码并写入到这台机器的指定路径中。
1、Kubernetes PKI 证书体系 (/etc/kubernetes/pki/...)：
为了保证多个 Master 节点的高可用（HA），它们必须共享同一套 CA 证书和 ServiceAccount 密钥。云平台在这里直接将管控端生成好的 ca.crt/key、etcd/ca.crt/key、front-proxy-ca.crt/key 和 sa.pub/key 注入到了节点中，从而绕过了 kubeadm init 默认自己生成证书的步骤。
2、拉起脚本 (/usr/local/bin/deploycluster.sh)：
base64 解码粗略看了一下，主要功能是定义了一个下载函数，去内网文件服务器（FILE_SERVER）拉取名为 ecloud-k8s-script-v1.6.5.tar.gz 的核心部署工具包，并解压。
3、Kubeadm 配置文件 (/etc/kubeadm/kubeadm.cfg)：
这是 kubeadm 初始化/加入集群的配置文件。里面写满了这套商业化 K8s 发行版的深度定制参数，例如：
指定的镜像仓库：cis-hub-huadong-7.cmecloud.cn/ecloud
定制的 K8s 版本：v1.29.5-eki.4.1.0
网络和组件参数：开启了 IPVS 模式，挂载了特定的 Audit 审计日志路径，配置了 APIServer 的高可用 VIP 等。
阶段 C：最终执行 (runcmd)
这是机器启动时的最后一步，依次执行以下命令：
echo 'root:... | chpasswd -e：设置 Root 用户的密码。
deploycluster.sh --file-server 10.195.207.205:32092：执行刚才注入的下载脚本，去内网拉取并解压 Kubernetes 部署工具包。
kuberun.sh ...：这是终极安装命令。它接收了大量的参数（节点角色 deploy-masters、网络模式 calico、容器运行时 containerd、实例规格 c5.2xlarge.2 等）。这个脚本会调用系统里的 kubeadm，结合前面注入的证书和 kubeadm.cfg，正式将这台机器初始化为一个 K8s Master 节点。
[root@kcs-k8s-test-m-tp88s /mnt/cdrom/openstack/latest]# cat user_data
#cloud-config
disable_root: false
ssh_pwauth: True
write_files:
- path: /etc/kubernetes/pki/ca.crt
encoding: base64
owner: root:root
permissions: '0640'
content: |
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3RENDQWRTZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQ0FYRFRJMk1ETXlOVEF5TlRVek5Wb1lEekl4TWpZd016QXlNREkxTlRNMVdqQVZNUk13RVFZRApWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBCnljVXhSbk1DUVRGclFwRjBQejJaUUpqU1RlcFMrM2dKQVhPTExXajArRUNxTll3ZTJ1cWdtVjB1RFR5a3AvbFUKN2dWWnhNeTVUTTg2MDE0djFKRXd5dEVJeCtRTkp3NUV6Q1R4cFB1QVhETHVCNCtpUjd2RVEyalVvUEVrajE1RApvMFpPQkhNVlBNWmVtVmhyN0RhRUROS3ZoRTIrb2NFQlpsTEdsYmZ2L2NSdWtsbEUxcTdXcVBDbmtCNUt2V3RyCjBaZU5odDZRQUxZQ3BVODZCeHJwUGFTbUt1RzNuTFVSZzhnYXl2L2U4MUVDRnZOSVJpN2lieXFpRnRsU2tJUkMKckZBZjB2a093K3llZTdRRkg1NVZLQ29xdGs2NitTVkoyQ2QzV1pzZGJpVE5HbUdMYkdMWm4xSldscU10OVJ2RQptaXhCTHFJWXNTWkNTYjVLdHlKVUF3SURBUUFCbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQXFRd0VnWURWUjBUCkFRSC9CQWd3QmdFQi93SUJBREFkQmdOVkhRNEVGZ1FVN3MvejM5MkxBaHh1ZEJZVVYxeFNaL01oN2Fnd0RRWUoKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ2YvZ3BnRjAxTmJOL01tQTF6SkxIWXdzaHBKZlNLQ0hjK2RncEFsV3dLZApUbkpGaDRGZ2s1Y0FmbnhpQzRPSWxqUXBBVlFMWFl0MXYwalM1eEk2TmJxZlBEM0NNUUl0VWJ1UzcxdGVFdTZUCnlzdlFwYzRTSGt5Y3ROOXVuT2VjcSs2MVhkVzBnZmRDUzZ3a0dtai8rVkxkbzR1RmZiQUtsZUt5RE1SVVBTOWsKdHBNTHkxSDBVelptSDBWTC9GYnhpVkdZTUdlM1J2ZkYxYmVPY0ZvT3N2YllMVy8rZldMS2JVUEpDS2dtbGc5TAptb3ZFSlFjZ2RMOThzbmQvdU1EcWNia3RWaFM1VE5oNVNqL1V4ZVk4U1BsVGhSVm42ekM1dUUwdU9uSWVVZ2pmCkhvcEczZmw2L0VQWGxEbnQrbnBnd25hTTFxcHZjVG9KaDY5bnpTdm5yV2s9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
- path: /etc/kubernetes/pki/ca.key
encoding: base64
owner: root:root
permissions: '0600'
content: |
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBeWNVeFJuTUNRVEZyUXBGMFB6MlpRSmpTVGVwUyszZ0pBWE9MTFdqMCtFQ3FOWXdlCjJ1cWdtVjB1RFR5a3AvbFU3Z1ZaeE15NVRNODYwMTR2MUpFd3l0RUl4K1FOSnc1RXpDVHhwUHVBWERMdUI0K2kKUjd2RVEyalVvUEVrajE1RG8wWk9CSE1WUE1aZW1WaHI3RGFFRE5LdmhFMitvY0VCWmxMR2xiZnYvY1J1a2xsRQoxcTdXcVBDbmtCNUt2V3RyMFplTmh0NlFBTFlDcFU4NkJ4cnBQYVNtS3VHM25MVVJnOGdheXYvZTgxRUNGdk5JClJpN2lieXFpRnRsU2tJUkNyRkFmMHZrT3creWVlN1FGSDU1VktDb3F0azY2K1NWSjJDZDNXWnNkYmlUTkdtR0wKYkdMWm4xSldscU10OVJ2RW1peEJMcUlZc1NaQ1NiNUt0eUpVQXdJREFRQUJBb0lCQUJiM2ZiZU5yQ0l4N0Y5Mgo3QkJZeENMV0RRdUx2U2RjK2ZWeTFWTDFhL3VvRjFmTXNTUnRGRVp3djFHeXZKd0Jld1BlNkdGemhqSWRNbzU3CkNFbStGSERrUzF5K3lOVWx3Q1NyOHpUbGl5NWVLUThEbXFRU0FQSHdDRktnMnU4QTBBVW1vYkhLOXc0UEZ4Y3cKWStuTzAzTEc0VU5Ya0NLY1hycnF6WCtlOWlVMTdLbXNMbUZSbkZWSXl5NndDV3l5VW5vSFRrSk0xL2ZMYThTaApwT1lhS3QwSmNTbWgzc1UxOEVwdmhmSThCSzl2ZWNEWUJOTmVsRWs0bjM0cDU4RWJaOGQvWFo0RWRnMXdpM2Z6CmhYaGxTK3d2eE93NUdDUWVlcU01Z1o3aGtuMzdHb1oxckZ4V3JWNmFNelYvMFNiRG9haitJWWZvbGZwZFAvZ2oKcmJQTHI2VUNnWUVBem1ENGNrVWZka3VTdWF0S0haRVN1M0JxU0NObkVobnRMcEU4YmU0bVplSEdxemlRcFZFZQpqbmJqWDNDRTVhR2RiekhnZDcwdTNZaVpJSWo1d0c2YkhhYzdDWWZOaS9oUGpickY0N1lGK1ZWOHJyZWxWb1YyClYrWUx5V2VDUnEyeUorWjFiOXFMeUpFaUdZbldQTlRrNXBTeGtFUUxmZnVrSDJ0RHFGcnJzVzhDZ1lFQStraVAKSHNmOHdjempNZnJMc2o4OE9ub1FjTVh2ckM5UlNjNmxyZURteklRVS9zVTJIOXA5R3hMazJEWUZCMW5JaVB1OQo5eVZQNWZueGluajNYSExaRHp0WUhhMEMxNzFHN3NCUUZmRWs4MktWN1hobXJjazFmbVNicEZMWGdPQ0dmSEwyCk5YU3hMMUJEZnpIMEkyaTQ3LzVjSXZmTDM4WWdqak1jaG1NTlZLMENnWUVBaThMRHZhN3Q5WkNNVnN5WExwcTIKVXRWNFJFNGxXTzdSM3IxZ2JSbmdTeEt4RmZjQ2pkSDNuWWNKeC9KTkxhMWJEcGg2YU54blJvTmhIOVZqUFZ3cQpFOVRTZUV2TmVVSzVyVU9WQy9hUzZSMXBpSEM1dVhROGhwNDEwVGtWMG9PQ3FONjdIUHFsdXpmK0hjbG9tbDJhCmZrU29Vd2lodDdtWWxlWndOUzBOZkdVQ2dZQi84R2xneGNBNTNSOWlaQjZPUG03dVFZbDM3R2FvOFFNdnBIZmkKMjIxL3JDRURYeEpjMUJaUnFhWGJ0RGw3MlhSK09abVE1YnpqQlpKb1E0L0c3VnB4dzljMlRFT0F2dHVzbmhnUwpMMVBCS21zVG1oRjYwcmtLcENrL3BhMU56dmhRVTMveU1YV0ZoeFVKeHlKU20yeTJHYU5GcUwvSjR3Q3ZVQWRMCjF3UndmUUtCZ1FDK1VKVnhGcGlHcWI5cG1oWkhDcFliamQ1SldVZnZMRDF2Mi85UVlqL01jVDFDMHlGaklLdloKQy93SG1GV05kcGgwRjd1NXAvZGFaMzRWT1lLcFZYOXZxMVJ0eXVsNUdmRFBlcFRHMHNtZnA0QkhYanZEems1dgpTT0pnUTUrTTYxMEs1SFUxV2JMM2pCQm5GREliTmdDcy94ZjZUVDdUTXVRYWZJN1gycFF1Tnc9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
- path: /etc/kubernetes/pki/etcd/ca.crt
encoding: base64
owner: root:root
permissions: '0640'
content: |
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3RENDQWRTZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQ0FYRFRJMk1ETXlOVEF5TlRVek9Wb1lEekl4TWpZd016QXlNREkxTlRNNVdqQVZNUk13RVFZRApWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBCnJRV3FIZUVWaFRHTzFYWDJrd0JFVzZUbFhScit3WmRwUG9ud1dKL0p2eHBjNzJQRXFSOEdCaDkvOUJWWHJ0aGYKQkxhMFRWWTBMelZjTzIzaFV2Z1FRcDNQbTlLcmhnZjluNGtmY3lsUzB1cWFTUG5aS2dvYzVFUUE3eUxUTVZ1cgpFK0FtazdsNFFQczk5NVdNeG96WVUrdkxYeVYxUEpJdW1kaWlmNENSOTNieWU3Q01uRUQ0djNwYldudlB1UU96CkZrV09qMjBFNXhVN3BrWEtiejd3Y1Z5TWc2cEduN1pVVVVoUXRHMCtBWWtrTW5NRFlmaU1JNUNhZWgxSVVoV0cKSXRsU005a1k1cTRaNnFpQ0lId0JNVGtqWWlya21XV2dWRTRob3hRMjV4b1NJNGl4eDE0RG1lY3U0a2lNbmd2QQp1aW1NR1dlaVRBd2JVRkRqNzNmNyt3SURBUUFCbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQXFRd0VnWURWUjBUCkFRSC9CQWd3QmdFQi93SUJBREFkQmdOVkhRNEVGZ1FVb1dZM2VjODNEaEpid2FQbDg0QXQybTVsWkVnd0RRWUoKS29aSWh2Y05BUUVMQlFBRGdnRUJBSmdBc3dxWkI1cTJHYllJamVTMkpTUDhCSHI3RE5NaGNMd0hyUjNQejhtVwpHUDZjNVpqNjBiekdKaytnNEVIaTJDWTJXaGpZai81VEVNUVFoWkRhamtCN1A4eVFzMndBb1cwek96TFE4c2dVCjIyQTlMRk9NaXVKZWZibDJGWHpKUXNqODRsUW5uTmVxcGhVaFlZeFJmT0NoSDFLUkNaRUZLWURNNDd5Y1pMQ3gKZEJXbFhOdTVSZkRFa2xlb3B1K3hEbCtGNzFiYkZVK0FCRTRvNm1XMnVSVk8rbkd1d0xVTDc0NFZyd21tZ1AwTQpRcHpXbjExODB0ZTZRMzNRMDdrN1ovKzlkZERTOWZrWm16anpZckh6NDB6dW9TbGRGVnE0TlBiWTk4c3dHWkdwCmR3UHFReXdscFRROGdhZ0haOVRmV2x3S3hGUGIrSWhFeEZlYXQySGZxZHc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
- path: /etc/kubernetes/pki/etcd/ca.key
encoding: base64
owner: root:root
permissions: '0600'
content: |
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBclFXcUhlRVZoVEdPMVhYMmt3QkVXNlRsWFJyK3daZHBQb253V0ovSnZ4cGM3MlBFCnFSOEdCaDkvOUJWWHJ0aGZCTGEwVFZZMEx6VmNPMjNoVXZnUVFwM1BtOUtyaGdmOW40a2ZjeWxTMHVxYVNQbloKS2dvYzVFUUE3eUxUTVZ1ckUrQW1rN2w0UVBzOTk1V014b3pZVSt2TFh5VjFQSkl1bWRpaWY0Q1I5M2J5ZTdDTQpuRUQ0djNwYldudlB1UU96RmtXT2oyMEU1eFU3cGtYS2J6N3djVnlNZzZwR243WlVVVWhRdEcwK0FZa2tNbk1ECllmaU1JNUNhZWgxSVVoV0dJdGxTTTlrWTVxNFo2cWlDSUh3Qk1Ua2pZaXJrbVdXZ1ZFNGhveFEyNXhvU0k0aXgKeDE0RG1lY3U0a2lNbmd2QXVpbU1HV2VpVEF3YlVGRGo3M2Y3K3dJREFRQUJBb0lCQUF4M3ZJYnpxYnZUMHVHUgo0d2M3dlRGSFpCbTk4TDZkZlFGN0toMFF3cFpwUFdvb3E4cXVDQjZYMVg0T3JhZFZReCtSVk5PLzB2blY1QVFLClNuTFNta1ZhbnRPeExoZjE2bkk5RE0yZEhERkRvNE4vc1lUa2ZxbDZOd0VFWnVpSEhRQk5KaXA5OG1yb1Q5SlAKN2ZsK3U3WHNaMWEvV2IvWUh0Q0tPa2RxeWRubGszWVpTVTY5ZkdDWnF0ZFVCQW9vMU9LSGwreWJycUFWdXphRgpMcW5QZlF2ZmZvSW9FZUxkcmdNZEhzZ2dyRHBqOTlRMVROa25qNlNKdDNhVzN2aHQzTWNkZUdjbjcxMDNwWjJjCnVmS2tYSDhLbi8waXNiS2FLVzQzYW9JeUNlQ1p6dHg4VmZYU3UrbHFNNkg4KzVpV2kva0NsQk1jMDdWOHd3bmQKMUpjT0pYMENnWUVBeFlYOGxkcHJXUDFKV2JLSFRIMUlqVjNsSzBsZmRsa0NuTEdiZ05hdEtXTXFRUjhPUHpoTwo3Z2NwZjFEcDMzTWNneWxzME9SQ1VWS01pRGdHV0IxdzlLQzc3dHA4WURzUk9XZUR4eXdpYjd2N0p0aUwzNFF4CmJqa2loZGsxb1hOQVhEWXgvbDIyZ1IrbnN6SFh3WkhQZkY5SzI1eG9Ra0hHbG81TTlJV2o5OVVDZ1lFQTREN0QKUVh0MHNWZzJGQUdEOUtlL3BPMmNTbTlvaUlYNThKdDRmS2t0RjhlUGZFZnR6TnkxS3hyZTFKTFc5NENlSEExVApxSUZ1RnRFTTlpSnA1cnJmVWJ1cnZTT0R4dzN5VVpxN2x1bTFUd2ZYdUdrUEd1bXVtTStWdGFySFBqa3Z5OEhhClM0cGIwbDkyZWFWbkk3TzdiK0doTDJoRW91T3ROOXNsN1NTUVhJOENnWUVBbEFqb0ZmTTlzdE1aanlVUzY4dVYKZllXYWhJZVlDUjJLcko4YnVVS3JRckowYjV2ejFJUEIrL2pZSy9nYlg0RnBKQS8rNHN1L3ZDME83K1IxTk1MVAo3ak1zeGtWdkk3d0JHN0d0L0s3aUhEV1pkREtsR2Q1OElXeW1xQVB6Z3MzYXRZRlVsSnZ0ZFBhaGU5Wm1La2U2ClppOFE3bWhaWnhiZTIrVklYWlp2SGdVQ2dZQUZLQXhQWGlwaHhaaUF2MFFzaFFyNEhPcWlINHUwei9mZVc0VGEKd1AwamRkaEwwRStjalZxeElnNExyMUM0SWtJQWZTSDJWdnVVRkx5S2tHSUZCemtKWlJwZTRBa3dzNVpsMy92KwpUV040N01JK0lGUlRseG9IczRaS3hpR013YjNpbnBPSmR5WURZV1NWQ1lPa280WmszVGhhb2JncVVyZng5OTBZClplWFg2d0tCZ0RCWi8vOWFmSlZLUmJ1Zms0RzJKUlhpRDkyYzFVR25xUU1uT0NQOERjcURDVEVBTmtRS21EK1cKMG5ZZTRZSmtxODlWemc2OWM5WVFVTVVaZ2ZrdFFIdWg0MW5ZQy93TlA4NDhla3FCV05jbDJqWVNaUHp5NUxDYQpraEFXdzBiOEJpdDIrQU14clFTMVl1YTZ2NUZHTDJ1K21NYitHcHFKSFdueXRwNVRaZ0hRCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
- path: /etc/kubernetes/pki/front-proxy-ca.crt
encoding: base64
owner: root:root
permissions: '0640'
content: |
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM3RENDQWRTZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQ0FYRFRJMk1ETXlOVEF5TlRVek9Wb1lEekl4TWpZd016QXlNREkxTlRNNVdqQVZNUk13RVFZRApWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBCm9tVEZyQ2JQeFNwRktEWHVJUDU0OU9yUUhMUUlIMXFkM1lGWnVDTFg1NE9ESG1kNVdaUGtJTFVmOEF2RkZFQksKbGpPOEV6Y0xlR0w1OWI3bm9FLzdFeHNzdnovZGsxNVVJRWxya2sxOHpZNEpDQUNEeWZCbGZrT0xENzlqa1NBSQorbzYyS2E0S2J0bVJjZFJMSkhSOVc2TGxqMFBkKzhwb3ZuN0VuT3RqSmxRdG1oZ1BxV29YalgxTXF2MkFKTUVrCk9KbDBiTWRxeExaU2RiNVpHOEpvV0lUb1ZzRHFyckgyc1FSdmFDUGVDeWpCbjNwTW5nVmdiODMvNDU0Vk9URjIKS1BLeFlZQmcyWGptUVZXbGh4clNTMXA4UTI3bFpvS0hmT2k3dmJJY2piamFqS2NieDJxQWVrdW1BUWFmcTV0Rwp2V3JuQ3pnelRBNTE1cHErOTRMWFNRSURBUUFCbzBVd1F6QU9CZ05WSFE4QkFmOEVCQU1DQXFRd0VnWURWUjBUCkFRSC9CQWd3QmdFQi93SUJBREFkQmdOVkhRNEVGZ1FVUGZUUzROTDczSWpVWTZUOTdNUkdtNEpDVGVJd0RRWUoKS29aSWh2Y05BUUVMQlFBRGdnRUJBRDFYcFFaNU9QSXRjZEFLTmM1eVFPV1U2emxsK29ja25XREhPQ1FVZUNNQgpjb05YazNZV0QvOUpsb2JuVEFGSUpGQWJaL3BjbnlRZVl0WEJ3b0c4bExhMnNITEE1emxaYWkwdy9sc2w4VDk4ClpjT3ZPN0VobXFBS3NRRm1UTXdUQTlHMG1PeU9jOHI5enVSYUdkdnE5UWgxRWVYMkhIVVJxa3FZY0IvZXFnNlYKRjdJZDE0aDVqMEExRThvY3ZuMTBndEcwOE9YOFpGNGJxMDJwdXJ2dGVnNjB0OThnV2k3aWJLbkFHVWdoNGx5VApZdmRyYTQrK1ZsYUJyWnJocVFERXB0NGdieUQwUWlTc2dmUENPT1UxVi9hOUZDb3dpbzJIMUx5dStNMEtxSEMrCnExb1UyYTRLb0hBVmJ6NERaSzJieXRNN09BVTN3RUM2ZGtaUkNrVDBGRWs9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
- path: /etc/kubernetes/pki/front-proxy-ca.key
encoding: base64
owner: root:root
permissions: '0600'
content: |
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb2dJQkFBS0NBUUVBb21URnJDYlB4U3BGS0RYdUlQNTQ5T3JRSExRSUgxcWQzWUZadUNMWDU0T0RIbWQ1CldaUGtJTFVmOEF2RkZFQktsak84RXpjTGVHTDU5Yjdub0UvN0V4c3N2ei9kazE1VUlFbHJrazE4elk0SkNBQ0QKeWZCbGZrT0xENzlqa1NBSStvNjJLYTRLYnRtUmNkUkxKSFI5VzZMbGowUGQrOHBvdm43RW5PdGpKbFF0bWhnUApxV29YalgxTXF2MkFKTUVrT0psMGJNZHF4TFpTZGI1Wkc4Sm9XSVRvVnNEcXJySDJzUVJ2YUNQZUN5akJuM3BNCm5nVmdiODMvNDU0Vk9URjJLUEt4WVlCZzJYam1RVldsaHhyU1MxcDhRMjdsWm9LSGZPaTd2YkljamJqYWpLY2IKeDJxQWVrdW1BUWFmcTV0R3ZXcm5Demd6VEE1MTVwcSs5NExYU1FJREFRQUJBb0lCQUFueG03MFAzMWNXWUllMgp6YThOaGdDUlJFOE5veFd3YWN3L2VHdnpEajlwNlNSNmQ0N0pwSVZ3TWRWMEV3eExaNFhOQXk0MkI2akdmc0hTCmY4SnNRMWFIS25WSGh0ellRTlI5U20zNStyTm5pQndLVkFlUWhkWjJjbFJ6aHJoRE91bUV4WmpGeEhQSE5NWEEKbHgxVFdMMjh1c3ZML3hMRThTY0JBaThOcHRPWTZQMkkvaTFLekNIaWs2ZHdXVGZDZy9BeUZGenlLT0Q3UmtVVApyZ3E2Qmx3UlJEL0pPc1RIMFJET0RId3RaTGhUbEFRVUppOW5HenE3S25EMFgrc3dBaUdHbSt0a05Ra1QxQ2FLCkswTHdGNFBRNTFXRGRLZTJDM2hYZFEzK3BzVTZWT0VLYUN1NWJIbEY3UC9UWTdiczhENkNkdG1CSHh6Y2w0THgKNS90Rm9ORUNnWUVBekNLS0JmbjBYT3ZNOTRoelp1KzBWWUhWdTNDRDE4RWpQbXdpWkN3L05VZXFXbkRqZTFjagplaGhKMWt6RUN5Y0g0cWlpaHZMT3pGVnR6Y2lwUTRGOERDcXpFbURWMkYwM0cvcm9sWXNpbGpEUHhTUEI2YmFWClU0eVVmbEVHQWlreENHRjNoTUxZVHpJUkhhdlV6b2tFNmZBdzFMeGh6T2oxU3ZGRm1jWTV0STBDZ1lFQXk2ZEYKMThnRmplN2xnRkpSNjBVRDVWQ2hueVhTek0wVjl2MU02UHI5clFnUy9aaDBOSmFjTmtIbVYyTFVjWDloZUpsUAo4Tkk4VitTakdtNDNycGlFeUtDZUVkUlNlTmFJUkRrakF3Y01DNkNMSnYxT0RJU1JFeC9zUytCcE14YVJ4QXY1CkdQU0ZhOXozdVdlN3hCbnZCZ1BhR3JzSUhtNFc2VWdMaUlHZ0pLMENnWUFZTXV2N3cyTEZkU3FLR1lIY3JRUEsKc3laOEh0MXlRVElGWDFwQVY4SnlkWGxyV1VDT1NZa3FHeUQ5cDRJQjlIR0ozQVhRUzQ1YVNMSklsOFlBKzZPUgo2YW5xdnRINjRTbjhSaVUyUFJVdmlyL0dsZk9SMmhRZm9HV21COExYbEx4OFN0bVpRbVBVRjVKUjJ5SFNEZ29vCkZWSWtsZVJlSHl1YzQ3Y2xnSXNzclFLQmdDWWM2dlJFS2MzelBKNDBTY0ozQ3hDYWMzVGVWa0lmeTVHS3ZCOEsKQWZtay9qRFpuRDNQUmZMZGlHY29Sc3ZxNCtuMi96LzVpSE9HaFlQSHhzSDFKenlJMnF4SmlSbTJSSkJJQlNabQo1amt5MVhmNWhlYlAxSHE0eWJjMWkxcVZTYmhmNlVGaldhampGTFZ0RlhYUXlLdmVncTNuL00vOUdHcVdJaHBzCjcvU05Bb0dBSmhjMVRoZDVYYVphakJjdDY4ZzhEU2dsQTJ0ZTluRkZCUU5rdnZZZU9yd3R2emNlK01Yb3dJclMKNUt1RmloRk44VXRWRjI4NEFnaU9LTzJNcG56NDMxU3hTZ3ZvRU5ENjRONWFUZksrVEE1c1paMDZsNUJQNHg3ZApzZjczdk1BaTdKMnk4R2Vyc0hMaFZQQXF6cDRmdVljYkJneUtnMk1lL3hieWhDMjlyOHc9Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
- path: /etc/kubernetes/pki/sa.pub
encoding: base64
owner: root:root
permissions: '0640'
content: |
LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUExWUxyRWxlb04vSzdITnlwSnlkTgpBQW5lbTdkQk5veFl5bFdoUEh2S3R6UWlSZzVYYVcycXlOUDVqSDVTZG1hVlBxbEJ1NnN4RFV1MDBPZ0dWWnRKCm9UVStPZ09jNlN2Sm8yakFQd2J6QnJHUGUxMDFSRE5MNFB6dHpUOXVaZ1h3ZmdrMytZZlpqc0V2YjZsRTZISlUKSzBVUG9FVjBNZForeXpRQmsvUDFkbDluZ04vZWpFMkxaMWVsRXFLaEszeW9VY2p0bEQ2ZVQya0FQV1pOWFU3Wgo4ZFZFVlF2dG81SE1CUUdmNzZldWtuRXZERk4zV3UxdjMvdVdKSUhzSHRhbXk2UXFOM2RwVkF5T2RVWHgwM1lQCmVSSGFuVzRLUUVIOTFubG9vOUFpY1BKYUlvNFpZeE5wUEhzZlhDbXVYRHhJcjBwU29OMlB6NE83WGhRbmgzb2wKMndJREFRQUIKLS0tLS1FTkQgUFVCTElDIEtFWS0tLS0tCg==
- path: /etc/kubernetes/pki/sa.key
encoding: base64
owner: root:root
permissions: '0600'
content: |
LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBMVlMckVsZW9OL0s3SE55cEp5ZE5BQW5lbTdkQk5veFl5bFdoUEh2S3R6UWlSZzVYCmFXMnF5TlA1akg1U2RtYVZQcWxCdTZzeERVdTAwT2dHVlp0Sm9UVStPZ09jNlN2Sm8yakFQd2J6QnJHUGUxMDEKUkROTDRQenR6VDl1WmdYd2ZnazMrWWZaanNFdmI2bEU2SEpVSzBVUG9FVjBNZForeXpRQmsvUDFkbDluZ04vZQpqRTJMWjFlbEVxS2hLM3lvVWNqdGxENmVUMmtBUFdaTlhVN1o4ZFZFVlF2dG81SE1CUUdmNzZldWtuRXZERk4zCld1MXYzL3VXSklIc0h0YW15NlFxTjNkcFZBeU9kVVh4MDNZUGVSSGFuVzRLUUVIOTFubG9vOUFpY1BKYUlvNFoKWXhOcFBIc2ZYQ211WER4SXIwcFNvTjJQejRPN1hoUW5oM29sMndJREFRQUJBb0lCQUJpaWFjL3NlRGE0VlZsdgpwajZqdkxFZjhtVENBSTYwSTd4NG84bFFPU1BwS25rdHgyMGRINkxiUGtRMUFQdXpPMDRIQmxRS1hQYjlRS2dICjFVOUVRdnNNSXhsYmVGdTQxeU40L3hGbWtseTMyT2V4YWVkc0NibTBSUlcwMTE2REdldlkwWElEZUJrTjloU3EKa1k1R1BxcmRaWCttODlDYVFIZmVrTDRLM0V2allPNFBrS2tYS293TVlkcUR2bVgwcjd2bW9UYU43aUFXaHllNApwbE1FY0IwVVpSUlRiN0NFNVJJL1pLd3FjZVV4S2E1UEU2YTV4MTRoQ0RmV1pEYnFKT3ZLUmpmLzhmU2NhY2tjCjFrTjRpMGpTeTYwUFRiWi9WNXBWWXZuY2JBVHlBamM4ZkFZMUF4bGZNUmQ2TW14a0o3Y1FCNlJpWStFUVNGcU4KVWNEdVVyRUNnWUVBMW9UNDVNOThWOU45cU1JVk9aT2NzMlpuUjBOM29pbXVjRDVTeVI2MXdtWitEbDVQbDFnVQpPSWJ5YVBPL1NCNFBvc1U2cmN0QVdtZXdDWjVmNm9IL0RhL3lsdHlzandiM1YrdVUrbXB1bnZHa2pOS0lxZXgrCnVxS0REUkxkNG1vZTNrVUtGcUhWVnc4VldqUkhjK1JxYWJnZXZ1K0lnQWFZZks3S0dwUHd1RkVDZ1lFQS9zd00KSCt2cWxxN1I2aEhtS1E0VVJqaVR2RDl3Tk9mbkxSUVhIdVJMd3RZeUE5RG13ZjNPUktlUlY0akhDSWpDbmRtTQpSVUtCSG5xSWNZL3lnVVd2ZDdxSnZucEJkNzdHV1lLTEoxK1cvditVQkpXZjdkRWxRU1lVOCtpVjAvc0xHblJBCllBbXM4Q2dCTDZCVkkxZHpla1Z2dEdJZ0wrR3lJaEM0VysyZ1hHc0NnWUFNOXlKMzZkWnhGSFkrMGVRb2c3UnYKMzF1VW9nNUQvZEx1TThZYkk4RUdpOTFJandpdWRBTmMyME1oZHNIejRPVS9DRDZnckcwcVNhUUpJTXBaU1J3YQpQcTBoMHhxVzFtdnlvMmx3clNnY2NTeHAybnVxRVlJalU1a3FIQjdQQld6eU1DZ0k4Q1VOeXZxV1poeC9jNm0rCjFBTC90VWlCdkdSUS9OdDRPY0xOMFFLQmdHWGFmNFpMTW0ybDJMZnZDOGlobmkwcjlMS3QwVmIwMVE3S0Z5djgKS3VUcDV2aHJpN05FbUM0TnBpWU53VEtDS1BvY3V0djg1OHlkUXVuU2x5aGlDUENkbXU2UHhKZnZwUzZtNXFXSQpxcjJvd1N6TCt6Q0FDSnB3ZExQRDZCRGpLOThaVlpxT2c1bEZCS1JiUFcxeFNmSTR5NXlhRlMvTzB2eVhIbnR4CkZFZWRBb0dCQUpNSDFuN29LVWZzNmIwc0ZDbjNZZlU1bHJzZmNzWDdqaXF5RHRlYUpsaXR5d2R4UWFPVVduNUQKTUVzMzhJTXhpSUI5ZTBFMG1IazRuT3ZCSThmd0txTzlmNGJPdmRxSEZjbXVzdVVoaUUwWDJrSVhZUzdycTRNSgppb24wNkplZnh5L25aakZ5Ly8rKzhuOXRzeUNQR2Y2RGN5MVM2d0N5NWdQbDg4d1c5TmJJCi0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
- path: /usr/local/bin/deploycluster.sh
encoding: base64
owner: root:root
permissions: '0777'
content: |
CnNldCAtZSAteAoKZXhlYyAxPi92YXIvbG9nL2NsdXN0ZXJfZGVwbG95LmxvZyAyPiYxCgpwdWJsaWNfY29tbW9uX2xvZygpewogICAgICAgIGVjaG8gJChkYXRlICsiWyVZJW0lZCAlSDolTTolU106ICIpICQxID4+IC92YXIvbG9nL2NsdXN0ZXJfZGVwbG95LmxvZwp9CgpCT09UUEtHPS91c3IvbG9jYWwvYmluCmNkICRCT09UUEtHCgpldGgwaXBmbGFnPTAKd2hpbGUgWyAkZXRoMGlwZmxhZyA9PSAwIF07IGRvCiAgICAgICAgZXRoMGlwPSQoaWZjb25maWcgZXRoMCB8IGF3ayAnL2luZXQvIHtwcmludCAkMn0nIHwgY3V0IC1mMiAtZCAiOiIgfGF3ayAnTlI9PTEge3ByaW50ICQxfScpCiAgICAgICAgaWYgWyAhIC16ICRldGgwaXAgXTsgdGhlbgogICAgICAgICAgICAgICAgZXRoMGlwZmxhZz0xCiAgICAgICAgICAgICAgICBpZmNvbmZpZyBldGgwIG10dSAxNTAwIHVwCiAgICAgICAgZmkKZG9uZQoKCmZ1bmN0aW9uIGZpbGVzZXJ2ZXJfY3VybCgpIHsKICB0YXJnZXQ9JDEKICB0b25nPTEKICB0aW1lb3V0PTUKICB3aGlsZSBbICR0b25nID09IDEgXTsgZG8KICAgIHJldF9jb2RlPSQoY3VybCAtSSAtZyAtcyAtLWNvbm5lY3QtdGltZW91dCAkdGltZW91dCAkdGFyZ2V0IC13ICV7aHR0cF9jb2RlfSB8IHRhaWwgLW4xKQoKICAgIGlmIFsgIngkcmV0X2NvZGUiID0gIngyMDAiIF07IHRoZW4KICAgICAgdG9uZz0wCiAgICAgIGVjaG8gImN1cmwgJHRhcmdldCBpcyBvayIKICAgIGVsc2UKICAgICAgZWNobyAid2lsbCBjdXJsICR0YXJnZXQgYWZ0ZXIgNXMgIgogICAgICBzbGVlcCA1CiAgICBmaQogIGRvbmUKfQoKbW9kcHJvYmUgaXBfdnNfcnIKbW9kcHJvYmUgaXBfdnNfd3JyCm1vZHByb2JlIGlwX3ZzX3NoCnNldCArZSAreAptb2Rwcm9iZSBuZl9jb25udHJhY2tfaXB2NAptb2Rwcm9iZSBuZl9jb25udHJhY2sKc2V0IC1lIC14CgoKCnByZXBhcmVfaW5zdGFsbF9wa2coKXsKICAgICAgICBQS0dfVFlQRT0kMQogICAgICAgIFBLR19WRVJTSU9OPSQyCglta2RpciAtcCBwYWNrLwogICAgICAgIHB1YmxpY19jb21tb25fbG9nICJEb3dubG9hZCBwa2cgZnJvbSAke0ZJTEVfU0VSVkVSfSIKICAgICAgICBpZiBbICEgLWYgcGFjay8ke1BLR19UWVBFfS0ke1BLR19WRVJTSU9OfS50YXIuZ3ogXTsgdGhlbgogICAgICAgICAgICAgICAgZm9yICgoaW50ZWdlciA9IDA7IGludGVnZXIgPCAzOyBpbnRlZ2VyKyspKTsgZG8KICAgICAgICAgICAgICAgICAgICAgICAgaWYgWyAteiAkRklMRV9TRVJWRVIgXTsgdGhlbgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHB1YmxpY19jb21tb25fbG9nICJQS0dfRklMRV9TRVJWRVIgaXMgbm90IGNvbmZpZy4iCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgZXhpdCAxCiAgICAgICAgICAgICAgICAgICAgICAgIGZpCiAgICAgICAgICAgICAgICAgICAgICAgIGN1cmwgLWcgLS1yZXRyeSA1IC0tcmV0cnktZGVsYXkgMiAkRklMRV9TRVJWRVIvc2hhcmUvJHtQS0dfVFlQRX0tJHtQS0dfVkVSU0lPTn0udGFyLmd6IFwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA+IHBhY2svJHtQS0dfVFlQRX0tJHtQS0dfVkVSU0lPTn0udGFyLmd6IHx8IChwdWJsaWNfY29tbW9uX2xvZyAiZG93bmxvYWQgZmFpbGVkIHdpdGggNCByZXRyeSxleGl0IDEiICYmIGV4aXQpCiAgICAgICAgICAgICAgICAgICAgICAgIGlmICEgdGFyIC14dmYgcGFjay8ke1BLR19UWVBFfS0ke1BLR19WRVJTSU9OfS50YXIuZ3o7IHRoZW4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBpZiBbICIkaW50ZWdlciIgPT0gIjIiIF07IHRoZW4KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHB1YmxpY19jb21tb25fbG9nICJGYWlsZWQgdG8gZG93bmxvYWQgcGtnIGZpbGUuIiAmJiBleGl0IDEKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBmaQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHB1YmxpY19jb21tb25fbG9nICJVbnRhciAke1BLR19WRVJTSU9OfS50YXIuZ3ogZmFpbGVkISwgcmV0cnkiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcm0gLXJmICR7UEtHX1RZUEV9LSR7UEtHX1ZFUlNJT059LnRhci5negogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNsZWVwIDUKICAgICAgICAgICAgICAgICAgICAgICAgZWxzZQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVybiAwCiAgICAgICAgICAgICAgICAgICAgICAgIGZpCiAgICAgICAgICAgICAgICBkb25lCiAgICAgICAgZmkKICAgICAgICBwdWJsaWNfY29tbW9uX2xvZyAiRmluaXNoZWQgZG93bmxvYWQgdGhlIHBrZzogJFBLR19UWVBFIgp9CgpwdWJsaWNfY29tbW9uX3BhcnNlX2FyZ3MoKSB7CiAgICAgICAgd2hpbGUKICAgICAgICAgICAgICAgIFtbICQjIC1ndCAwIF1dCiAgICAgICAgZG8KICAgICAgICAgICAgICAgIGtleT0iJDEiCgogICAgICAgICAgICAgICAgY2FzZSAka2V5IGluCiAgICAgICAgICAgICAgICAtLWZpbGUtc2VydmVyKQogICAgICAgICAgICAgICAgICAgICAgICBleHBvcnQgRklMRV9TRVJWRVI9JDIKICAgICAgICAgICAgICAgICAgICAgICAgc2hpZnQKICAgICAgICAgICAgICAgICAgICAgICAgOzsKICAgICAgICAgICAgICAgICopCiAgICAgICAgICAgICAgICAgICAgICAgICMgdW5rbm93biBvcHRpb24KICAgICAgICAgICAgICAgICAgICAgICAgcHVibGljX2NvbW1vbl9sb2cgInVua25vdyBvcHRpb24gWyRrZXldIgogICAgICAgICAgICAgICAgICAgICAgICA7OwogICAgICAgICAgICAgICAgZXNhYwogICAgICAgICAgICAgICAgc2hpZnQKICAgICAgICBkb25lCn0KCm1haW4oKXsKCXB1YmxpY19jb21tb25fbG9nICJTdGFydCBpbnN0YWxsIGs4cyBzY3JpcHQuLi4iCglwdWJsaWNfY29tbW9uX3BhcnNlX2FyZ3MgIiRAIgoJZmlsZXNlcnZlcl9jdXJsICRGSUxFX1NFUlZFUgoJcHJlcGFyZV9pbnN0YWxsX3BrZyAiZWNsb3VkLWs4cy1zY3JpcHQiICJ2MS42LjUiCglwdWJsaWNfY29tbW9uX2xvZyAiU2NyaXB0IGhhcyBiZWVuIGRvd25sb2FkZWQuIgp9CgpoZWxwPSIKVXNhZ2U6CiIkMCIKLS1maWxlLXNlcnZlciBodHRwOi8vMTAuMTQyLjExMy44OAoiCm1haW4gIiRAIgo=
- path: /etc/kubeadm/kubeadm.cfg
encoding: base64
owner: root:root
permissions: '0640'
content: |
LS0tCmFwaVNlcnZlcjoKICBjZXJ0U0FOczoKICAtIGFwaXNlcnZlci5jbHVzdGVyLmxvY2FsCiAgLSAxMjcuMC4wLjEKICAtIDo6MQogIC0gMTkyLjAuMC4xCiAgLSBrOHMtdGVzdC5pdGhvLmNuCiAgZXh0cmFBcmdzOgogICAgYXVkaXQtbG9nLWZvcm1hdDoganNvbgogICAgYXVkaXQtbG9nLW1heGFnZTogIjciCiAgICBhdWRpdC1sb2ctbWF4YmFja3VwOiAiMTAiCiAgICBhdWRpdC1sb2ctbWF4c2l6ZTogIjEwMCIKICAgIGF1ZGl0LWxvZy1wYXRoOiAvdmFyL2xvZy9rdWJlcm5ldGVzL2F1ZGl0LmxvZwogICAgYXVkaXQtcG9saWN5LWZpbGU6IC9ldGMva3ViZXJuZXRlcy9hdWRpdC1wb2xpY3kueW1sCiAgICBkZWZhdWx0LW5vdC1yZWFkeS10b2xlcmF0aW9uLXNlY29uZHM6ICIyNDAiCiAgICBkZWZhdWx0LXVucmVhY2hhYmxlLXRvbGVyYXRpb24tc2Vjb25kczogIjI0MCIKICAgIGRpc2FibGUtYWRtaXNzaW9uLXBsdWdpbnM6IExpY2Vuc2UKICAgIGVuYWJsZS1hZG1pc3Npb24tcGx1Z2luczogTm9kZVJlc3RyaWN0aW9uLERlZmF1bHRUb2xlcmF0aW9uU2Vjb25kcyxQb2ROb2RlU2VsZWN0b3IKICAgIGV0Y2QtY29tcGFjdGlvbi1pbnRlcnZhbDogMTVtCiAgICBmZWF0dXJlLWdhdGVzOiBMb2FkQmFsYW5jZXJJUE1vZGU9dHJ1ZQogICAgZ29hd2F5LWNoYW5jZTogIjAuMDAxIgogICAgbWF4LW11dGF0aW5nLXJlcXVlc3RzLWluZmxpZ2h0OiAiMTAwMDAiCiAgICBtYXgtcmVxdWVzdHMtaW5mbGlnaHQ6ICIxMDAwMDAiCiAgICBwcm9maWxpbmc6ICJmYWxzZSIKICAgIHNlcnZpY2Utbm9kZS1wb3J0LXJhbmdlOiAzMDAwMC0zMjc2NwogICAgdGxzLWNpcGhlci1zdWl0ZXM6IFRMU19FQ0RIRV9FQ0RTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUsVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0LFRMU19FQ0RIRV9SU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNQogIGV4dHJhVm9sdW1lczoKICAtIGhvc3RQYXRoOiAvZXRjL2t1YmVybmV0ZXMvYXVkaXQtcG9saWN5LnltbAogICAgbW91bnRQYXRoOiAvZXRjL2t1YmVybmV0ZXMvYXVkaXQtcG9saWN5LnltbAogICAgbmFtZTogYXVkaXQKICAgIHBhdGhUeXBlOiBGaWxlT3JDcmVhdGUKICAgIHJlYWRPbmx5OiB0cnVlCiAgLSBob3N0UGF0aDogL3Zhci9sb2cva3ViZXJuZXRlcwogICAgbW91bnRQYXRoOiAvdmFyL2xvZy9rdWJlcm5ldGVzCiAgICBuYW1lOiBhdWRpdC1sb2cKICAgIHBhdGhUeXBlOiBEaXJlY3RvcnlPckNyZWF0ZQogIHRpbWVvdXRGb3JDb250cm9sUGxhbmU6IDEwbTBzCmFwaVZlcnNpb246IGt1YmVhZG0uazhzLmlvL3YxYmV0YTMKY2x1c3Rlck5hbWU6IGs4cy10ZXN0CmNvbnRyb2xQbGFuZUVuZHBvaW50OiBhcGlzZXJ2ZXIuY2x1c3Rlci5sb2NhbDo2NDQzCmNvbnRyb2xsZXJNYW5hZ2VyOgogIGV4dHJhQXJnczoKICAgIGJpbmQtYWRkcmVzczogMC4wLjAuMAogICAgY2x1c3Rlci1zaWduaW5nLWR1cmF0aW9uOiA4NzYwMDBoMG0wcwogICAgY29uY3VycmVudC1kZXBsb3ltZW50LXN5bmNzOiAiNTAiCiAgICBjb25jdXJyZW50LWVuZHBvaW50LXN5bmNzOiAiNTAiCiAgICBjb25jdXJyZW50LW5hbWVzcGFjZS1zeW5jczogIjEwMCIKICAgIGNvbmN1cnJlbnQtcmVwbGljYXNldC1zeW5jczogIjUwIgogICAgY29uY3VycmVudC1zZXJ2aWNlLXN5bmNzOiAiMTAwIgogICAgY29uY3VycmVudC1zdGF0ZWZ1bHNldC1zeW5jczogIjIwMCIKICAgIGt1YmUtYXBpLWJ1cnN0OiAiNTAwIgogICAga3ViZS1hcGktcXBzOiAiNTAwIgogICAgbm9kZS1tb25pdG9yLWdyYWNlLXBlcmlvZDogMjBzCiAgICBub2RlLW1vbml0b3ItcGVyaW9kOiAycwogICAgcHJvZmlsaW5nOiAiZmFsc2UiCiAgICB0bHMtY2lwaGVyLXN1aXRlczogVExTX0VDREhFX0VDRFNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2LFRMU19FQ0RIRV9FQ0RTQV9XSVRIX0FFU18yNTZfR0NNX1NIQTM4NCxUTFNfRUNESEVfRUNEU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNSxUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzEyOF9HQ01fU0hBMjU2LFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1CmRuczoge30KZXRjZDoKICBsb2NhbDoKICAgIGV4dHJhQXJnczoKICAgICAgYXV0by1jb21wYWN0aW9uLXJldGVudGlvbjogIjEiCiAgICAgIGNpcGhlci1zdWl0ZXM6IFRMU19FQ0RIRV9FQ0RTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUsVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0LFRMU19FQ0RIRV9SU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNQogICAgICBlbGVjdGlvbi10aW1lb3V0OiAiNTAwMCIKICAgICAgaGVhcnRiZWF0LWludGVydmFsOiAiNTAwIgogICAgICBsaXN0ZW4tbWV0cmljcy11cmxzOiBodHRwOi8vMC4wLjAuMDoyMzgxCiAgICAgIG1heC1yZXF1ZXN0LWJ5dGVzOiAiMTA0ODU3NjAiCiAgICAgIHF1b3RhLWJhY2tlbmQtYnl0ZXM6ICI4NTg5OTM0NTkyIgogICAgICBzbmFwc2hvdC1jb3VudDogIjEwMDAwMCIKaW1hZ2VSZXBvc2l0b3J5OiBjaXMtaHViLWh1YWRvbmctNy5jbWVjbG91ZC5jbi9lY2xvdWQKa2luZDogQ2x1c3RlckNvbmZpZ3VyYXRpb24Ka3ViZXJuZXRlc1ZlcnNpb246IHYxLjI5LjUtZWtpLjQuMS4wCm5ldHdvcmtpbmc6CiAgZG5zRG9tYWluOiBjbHVzdGVyLmxvY2FsCiAgcG9kU3VibmV0OiAxNzIuMjAuMC4wLzE2CiAgc2VydmljZVN1Ym5ldDogMTAuMjMzLjAuMC8xOApzY2hlZHVsZXI6CiAgZXh0cmFBcmdzOgogICAgYmluZC1hZGRyZXNzOiAwLjAuMC4wCiAgICBrdWJlLWFwaS1idXJzdDogIjUwMCIKICAgIGt1YmUtYXBpLXFwczogIjUwMCIKICAgIHByb2ZpbGluZzogImZhbHNlIgogICAgdGxzLWNpcGhlci1zdWl0ZXM6IFRMU19FQ0RIRV9FQ0RTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQsVExTX0VDREhFX0VDRFNBX1dJVEhfQ0hBQ0hBMjBfUE9MWTEzMDUsVExTX0VDREhFX1JTQV9XSVRIX0FFU18xMjhfR0NNX1NIQTI1NixUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0LFRMU19FQ0RIRV9SU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNQoKLS0tCmFwaVZlcnNpb246IGt1YmVhZG0uazhzLmlvL3YxYmV0YTMKY2VydGlmaWNhdGVLZXk6IDU2MTRlN2MxNTQ0OWFlNDI5MDQ0YzE1YTJlYTg4NmY2MGQzOWZkZGM1ZTEzYzg5OTIzODM3N2M0YjRkMmUyN2YKa2luZDogSW5pdENvbmZpZ3VyYXRpb24KbG9jYWxBUElFbmRwb2ludDoge30Kbm9kZVJlZ2lzdHJhdGlvbjoKICBjcmlTb2NrZXQ6IC9ydW4vY29udGFpbmVyZC9jb250YWluZXJkLnNvY2sKICBrdWJlbGV0RXh0cmFBcmdzOgogICAgY2xvdWQtcHJvdmlkZXI6IGV4dGVybmFsCiAgICBjb250YWluZXItcnVudGltZS1lbmRwb2ludDogdW5peDovLy9ydW4vY29udGFpbmVyZC9jb250YWluZXJkLnNvY2sKICAgIGhvc3RuYW1lLW92ZXJyaWRlOiBrY3MtazhzLXRlc3QtbS10cDg4cwogICAgbWF4LXBvZHM6ICIxMjgiCiAgICBub2RlLWlwOiAnOjonCiAgICBub2RlLWxhYmVsczogbWFjaGluZS5lY2xvdWQuY21zcy5jb20vbm9kZS10eXBlPWNlbnRlcixtYWNoaW5lLmVjbG91ZC5jbXNzLmNvbS9tYWNoaW5lLXJlZ2lvbj1OMDU3NC1aSi1OQlpEMDEsbm9kZS5rdWJlcm5ldGVzLmlvL2Nsb3VkPSxtYWNoaW5lLmVjbG91ZC5jbXNzLmNvbS9tYWNoaW5lLW5hbWU9azhzLXRlc3QtY29udHJvbHBsYW5lLWxtcGwyLG1hY2hpbmUuZWNsb3VkLmNtc3MuY29tL21hY2hpbmUtdHlwZT1WTSxtYWNoaW5lLmVjbG91ZC5jbXNzLmNvbS9zcGVjc25hbWU9YzUuMnhsYXJnZS4yCgotLS0KYXBpVmVyc2lvbjoga3ViZXByb3h5LmNvbmZpZy5rOHMuaW8vdjFhbHBoYTEKY2xpZW50Q29ubmVjdGlvbjoKICBidXJzdDogMTAwCiAgcXBzOiAxMDAKZmVhdHVyZUdhdGVzOgogIExvYWRCYWxhbmNlcklQTW9kZTogdHJ1ZQpob3N0bmFtZU92ZXJyaWRlOiBrY3MtazhzLXRlc3QtbS10cDg4cwppcHRhYmxlczoge30KaXB2czoKICBleGNsdWRlQ0lEUnM6CiAgLSAxOTIuMC4wLjEvMzIKa2luZDogS3ViZVByb3h5Q29uZmlndXJhdGlvbgptZXRyaWNzQmluZEFkZHJlc3M6IDAuMC4wLjA6MTAyNDkKbW9kZTogaXB2cwpwb3J0UmFuZ2U6ICIiCndpbmtlcm5lbDoge30KCi0tLQphcGlWZXJzaW9uOiBrdWJlbGV0LmNvbmZpZy5rOHMuaW8vdjFiZXRhMQphdXRoZW50aWNhdGlvbjoKICBhbm9ueW1vdXM6IHt9CiAgd2ViaG9vazoKICAgIGNhY2hlVFRMOiAwcwogIHg1MDk6IHt9CmF1dGhvcml6YXRpb246CiAgd2ViaG9vazoKICAgIGNhY2hlQXV0aG9yaXplZFRUTDogMHMKICAgIGNhY2hlVW5hdXRob3JpemVkVFRMOiAwcwpjZ3JvdXBEcml2ZXI6IGNncm91cGZzCmNvbnRhaW5lckxvZ01heEZpbGVzOiA1CmNvbnRhaW5lckxvZ01heFNpemU6IDEwTWkKY3B1TWFuYWdlclJlY29uY2lsZVBlcmlvZDogMHMKZXZpY3Rpb25IYXJkOgogIGltYWdlZnMuYXZhaWxhYmxlOiAxNSUKICBtZW1vcnkuYXZhaWxhYmxlOiAxMDBNaQogIG5vZGVmcy5hdmFpbGFibGU6IDEwJQogIG5vZGVmcy5pbm9kZXNGcmVlOiA1JQpldmljdGlvblByZXNzdXJlVHJhbnNpdGlvblBlcmlvZDogNW0wcwpmaWxlQ2hlY2tGcmVxdWVuY3k6IDBzCmh0dHBDaGVja0ZyZXF1ZW5jeTogMHMKaW1hZ2VHQ0hpZ2hUaHJlc2hvbGRQZXJjZW50OiA4NQppbWFnZUdDTG93VGhyZXNob2xkUGVyY2VudDogODAKaW1hZ2VNaW5pbXVtR0NBZ2U6IDBzCmtpbmQ6IEt1YmVsZXRDb25maWd1cmF0aW9uCmt1YmVBUElCdXJzdDogMTAwCmt1YmVBUElRUFM6IDEwMAprdWJlUmVzZXJ2ZWQ6CiAgY3B1OiAxODBtCiAgbWVtb3J5OiAyLjYwRwpub2RlU3RhdHVzUmVwb3J0RnJlcXVlbmN5OiAwcwpub2RlU3RhdHVzVXBkYXRlRnJlcXVlbmN5OiAwcwpyb3RhdGVDZXJ0aWZpY2F0ZXM6IHRydWUKcnVudGltZVJlcXVlc3RUaW1lb3V0OiAwcwpzZXJpYWxpemVJbWFnZVB1bGxzOiBmYWxzZQpzdHJlYW1pbmdDb25uZWN0aW9uSWRsZVRpbWVvdXQ6IDBzCnN5bmNGcmVxdWVuY3k6IDBzCnRsc0NpcGhlclN1aXRlczoKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9BRVNfMjU2X0dDTV9TSEEzODQKLSBUTFNfRUNESEVfRUNEU0FfV0lUSF9DSEFDSEEyMF9QT0xZMTMwNQotIFRMU19FQ0RIRV9SU0FfV0lUSF9BRVNfMTI4X0dDTV9TSEEyNTYKLSBUTFNfRUNESEVfUlNBX1dJVEhfQUVTXzI1Nl9HQ01fU0hBMzg0Ci0gVExTX0VDREhFX1JTQV9XSVRIX0NIQUNIQTIwX1BPTFkxMzA1CnZvbHVtZVN0YXRzQWdnUGVyaW9kOiAwcwo=
runcmd:
- echo 'root:$6$76/O5ToF$GkXliQEhjTWy4sO0sD7RMPKQzGGf8/Sjoycw9BbO/aGKQwLJjXU59WmeLxZ8qTj7./D09DeVhcWY82DBDZKlw1' | chpasswd -e
- 'deploycluster.sh --file-server 10.195.207.205:32092'
- 'kuberun.sh --m1Host 127.0.0.1 --kubeletCgroupDriver cgroupfs --npuNode false --dualStack false --role deploy-masters --starwayIpStack NULL --user NULL --imageServerPort 443 --fileServer 10.195.207.205:32092 --apiServerVIP 192.0.0.1 --gpuSchedule NULL --podCIDR 172.20.0.0/16 --mtuValue 1500 --spec c5.2xlarge.2 --serviceNodePortRange 30000-32767 --v4Enable true --ipv6SingleStack false --nodeRuntime containerd --v6Enable true --dockerIOAccPort 7999 --v6PrefixLen 64 --calicoDualStack false --imageServerIP 10.195.207.201 --cisNameServer NULL --timeServer 114.118.7.163,10.215.242.54,10.215.242.55 --kubeVersion v1.29.5-eki.4.1.0 --clusterNetworkMode calico --clusterRuntime containerd --imageRepo cis-hub-huadong-7.cmecloud.cn --dualStackIpv4First false'
0x03 手动把节点加入到集群
订购一台新的云主机
为了测试，手动订购了一台ecs
初始化分区
# 把sdb也格式化成xfs
root@k8s-test-gpu:~# sudo mkfs.xfs /dev/sdb -f
meta-data=/dev/sdb isize=512 agcount=4, agsize=6553600 blks
realtime =none extsz=4096 blocks=0, rtextents=0
Discarding blocks...Done.
# 查看UUID
root@k8s-test-gpu:~# lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
└─sda1 ext4 1.0 9cb9e9a5-e5a1-46e3-9ead-2330073ed95a 77.5G 17% /
sdb xfs f4a0c79c-f46e-4912-9bf9-e19e90f3d418
sr0 iso9660 Joliet Extension config-2 2026-03-26-14-28-31-00
# 创建挂在文件夹
root@k8s-test-gpu:~# mkdir -p /var/lib/paascontainer
# 设置自动挂载
root@k8s-test-gpu:~# echo "UUID=f4a0c79c-f46e-4912-9bf9-e19e90f3d418 /var/lib/paascontainer/ xfs defaults 0 0" >> /etc/fstab
# 查看自动挂载
root@k8s-test-gpu:~# cat /etc/fstab
/dev/disk/by-uuid/9cb9e9a5-e5a1-46e3-9ead-2330073ed95a / ext4 defaults 0 1
/swap.img none swap sw 0 0
UUID=f4a0c79c-f46e-4912-9bf9-e19e90f3d418 /var/lib/paascontainer/ xfs defaults 0 0
# 尝试挂载
root@k8s-test-gpu:~# mount -a
# 查看挂载
root@k8s-test-gpu:~# df -h | grep sd
/dev/sda1 99G 19G 76G 20% /
/dev/sdb 100G 2.0G 98G 2% /var/lib/paascontainer创建跟KCS一样的目录结构
mkdir -p /var/lib/paascontainer/{kubelet,containerd,docker,etcd} && \
ln -s /var/lib/paascontainer/containerd /var/lib/containerd && \
ln -s /var/lib/paascontainer/docker /var/lib/docker && \
echo "目录结构配置完成！"配置sysctl
加载内核模块
modprobe br_netfilter
modprobe overlay开启ipv4转发等参数
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF
sysctl --system安装contaierd
由于集群目前使用的containerd版本是1.6.28，为了兼容性，我这边选择使用使用二进制包安装。
VERSION="1.6.28"
wget https://github.com/containerd/containerd/releases/download/v${VERSION}/containerd-${VERSION}-linux-amd64.tar.gz前提条件检查
你的Worker节点需要满足：
1、网络互通 - 能访问Master节点的6443端口
2、基础环境 - 已安装containerd、kubelet、kubeadm
2、配置正确 - kubelet版本与集群兼容
第一步：获取集群加入信息
# 在任意Master节点上执行
# 登录Master节点
ssh root@36.134.185.103
# 1. 生成bootstrap token
kubeadm token create
# 输出示例: abcdef.0123456789abcdef
# 2. 获取CA证书哈希
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt -noout | \
openssl rsa -pubin -outform der | \
sha256sum | awk '{print $1}'
# 输出示例: 8a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0t1u2v3w4x5y6z7a8b9c0d1e2f
# 3. 获取API Server地址
cat /etc/kubernetes/kubelet.conf | grep server:
# 输出: server: https://apiserver.cluster.local:6443
# 或者使用控制平面端点
kubectl get cm -n kube-system kubeadm-config \
-o jsonpath='{.data.ClusterConfiguration}' | \
grep controlPlaneEndpoint
# 输出: controlPlaneEndpoint: apiserver.cluster.local:6443第二步：在你的Worker节点上执行
# 配置containerd使用移动云镜像仓库
cat > /etc/containerd/certs.d/cis-hub-huadong-7.cmecloud.cn/hosts.toml <<EOF
server = "https://cis-hub-huadong-7.cmecloud.cn"
[host."https://cis-hub-huadong-7.cmecloud.cn"]
capabilities = ["pull", "resolve"]
EOF
systemctl enable containerd --now
# 3. 安装kubelet/kubeadm/kubectl
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
EOF
# 注意：这里需要安装与集群兼容的版本
# 移动云EKI使用v1.29.5-eki.4.1.0，但你可以用标准的v1.29.x
yum install -y kubelet-1.29.* kubeadm-1.29.* kubectl-1.29.* --disableexcludes=kubernetes
systemctl enable kubelet --now
---第三步：创建kubeadm配置（可选但推荐）
# 在Worker节点上创建join配置
cat > /tmp/kubeadm-join.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta3
kind: JoinConfiguration
discovery:
bootstrapToken:
token: abcdef.0123456789abcdef # 替换为第一步获取的token
apiServerEndpoint: apiserver.cluster.local:6443 # API Server地址
caCertHashes:
- sha256:8a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0t1u2v3w4x5y6z7a8b9c0d1e2f # 替换为第一步获取的哈希
nodeRegistration:
name: your-worker-node # 可选：指定节点名称
criSocket: /run/containerd/containerd.sock
kubeletExtraArgs:
cloud-provider: external
container-runtime-endpoint: unix:///run/containerd/containerd.sock
EOF第四步：执行join命令
# 方法1: 使用配置文件
kubeadm join --config=/tmp/kubeadm-join.yaml
# 方法2: 使用命令行参数
kubeadm join apiserver.cluster.local:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:8a2b3c4d5e6f7g8h9i0j1k2l3m4n5o6p7q8r9s0t1u2v3w4x5y6z7a8b9c0d1e2f \
--node-name your-worker-node第五步：验证节点加入成功
# 在Master节点上验证
kubectl get nodes -o wide
# 应该能看到你的新节点：
# NAME STATUS ROLES AGE VERSION
# kcs-k8s-test-m-tp88s Ready master XXh v1.29.5-eki.4.1.0
# kcs-k8s-test-m-jj48s Ready master XXh v1.29.5-eki.4.1.0
# kcs-k8s-test-m-6thhn Ready master XXh v1.29.5-eki.4.1.0
# kcs-k8s-test-s-psfjf Ready <none> XXh v1.29.5-eki.4.1.0
# kcs-k8s-test-s-72lh6 Ready <none> XXh v1.29.5-eki.4.1.0
# your-worker-node Ready <none> XXs v1.29.x.x ← 你的节点
可能遇到的问题和解决方案
问题1: API Server地址无法解析
# 错误: couldn't find host apiserver.cluster.local
# 解决方案1: 使用Master节点IP
# 获取任意Master的IP (192.168.11.139/38/246)
kubeadm join 192.168.11.139:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:8a2b3c...
# 解决方案2: 配置hosts或DNS
echo "192.168.11.139 apiserver.cluster.local" >> /etc/hosts
问题2: 镜像拉取失败
# 错误: Failed to pull image
# 解决方案: 配置镜像仓库认证
mkdir -p /etc/containerd/certs.d/cis-hub-huadong-7.cmecloud.cn/
# 从Master节点复制镜像配置
scp root@36.134.185.103:/etc/containerd/certs.d/* \
/etc/containerd/certs.d/
# 或者手动配置
cat > /etc/containerd/certs.d/cis-hub-huadong-7.cmecloud.cn/hosts.toml <<EOF
server = "https://cis-hub-huadong-7.cmecloud.cn"
[host."https://cis-hub-huadong-7.cmecloud.cn"]
capabilities = ["pull", "resolve"]
EOF
问题3: kubelet版本不兼容
# 错误: Version skew
# 解决方案: 安装兼容版本
# 移动云EKI使用v1.29.5-eki.4.1.0
# 你的Worker应该使用v1.29.x版本
# 查看Master版本
kubectl version --short
# 安装匹配版本
yum install -y kubelet-1.29.* kubeadm-1.29.*
问题4: 云控制器标签问题
# 你的节点可能缺少移动云的标签
# 这不影响基本功能，但可能影响云平台集成
# 可选：手动添加标签
kubectl label node your-worker-node \
machine.ecloud.cmss.com/node-type=worker \
machine.ecloud.cmss.com/machine-type=VM快速参考命令清单
在Master节点执行：
# 1. 生成token
kubeadm token create
# 2. 获取CA哈希
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt -noout | \
openssl rsa -pubin -outform der | sha256sum | awk '{print $1}'
# 3. 获取API Server地址
kubectl get endpoints -n default kubernetes
# 4. 验证节点
kubectl get nodes -o wide
在Worker节点执行：
# 1. 准备环境
modprobe br_netfilter overlay
# 2. 安装containerd
yum install -y containerd
systemctl enable containerd --now
# 3. 安装k8s组件
yum install -y kubelet-1.29.* kubeadm-1.29.* kubectl-1.29.*
systemctl enable kubelet --now
# 4. 加入集群
kubeadm join apiserver.cluster.local:6443 \
--token <TOKEN> \
--discovery-token-ca-cert-hash sha256:<HASH>
# 5. 验证
kubectl get pods -A
---
💡 进阶技巧
1. 生成永久token（24小时有效期）
kubeadm token create --print-join-command --ttl=24h
2. 查看所有可用token
kubeadm token list
3. 删除节点（如果需要重新加入）
# 在Master上
kubectl delete node your-worker-node
# 在Worker上
kubeadm reset
rm -rf /etc/kubernetes/
4. 自动配置脚本（一键加入）
# 在Master上生成join脚本
cat > /tmp/join-worker.sh <<'SCRIPT'
#!/bin/bash
TOKEN=$(kubeadm token create)
HASH=$(openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt -noout | \
openssl rsa -pubin -outform der | sha256sum | awk '{print $1}')
API_SERVER=$(kubectl get endpoints -n default kubernetes -o jsonpath='{.subsets[0].addresses[0].ip}'):6443
cat <<EOF
# 在Worker节点执行以下命令：
kubeadm join $API_SERVER \
--token $TOKEN \
--discovery-token-ca-cert-hash sha256:$HASH
EOF
SCRIPT
chmod +x /tmp/join-worker.sh
/tmp/join-worker.sh
总结
将自定义Worker节点加入移动云EKI集群的关键点：
- 网络互通 - Worker能访问Master的6443端口
- 版本兼容 - kubelet版本与集群匹配（v1.29.x）
- 正确凭证 - token和CA哈希必须正确
- 镜像仓库 - 配置移动云镜像仓库认证
-
云控制器 - 可选配置云平台标签
成功加入后，你的节点和移动云的节点在功能上基本一致，除了可能缺少一些云平台特定的标签和监控集成。
docker k8s openstack ecloud kcs