K8S部署定制RabbitMQ镜像 & 多节点RabbitMQ集群

0x01 定制RabbitMQ镜像

本文章基于DockerHub镜像rabbitmq:3.12.8-management，由于官方镜像没有集成有些插件，需要在镜像中安装插件并重新打包。

rabbitmq_delayed_message_exchange延迟队列插件的安装，rabbitmq 插件下载（官网）：https://www.rabbitmq.com/community-plugins.html【根据rabbitmq版本选择对应的插件，web管理页面首页可以查看rabbitmq版本】，手动安装插件教程参考：https://blog.csdn.net/user2025/article/details/105620908

下载完成rabbitmq_delayed_message_exchange-3.12.0.ez后，放到跟dockerfile同一目录，然后使用docker build进行打包。

DockerFile如下：

FROM rabbitmq:3.12.8-management  
ADD ./rabbitmq_delayed_message_exchange-3.12.0.ez /plugins/rabbitmq_delayed_message_exchange-3.12.0.ez  
RUN rabbitmq-plugins enable rabbitmq_delayed_message_exchange

构建新的镜像：

docker build -t rabbitmq_xfsh:3.13.0-rc.2-management .

0x02 在K8S集群中安装单节点RabbitMQ

rabbitmq.yaml文件如下：

---
kind: Secret
apiVersion: v1
metadata:
  name: rabbitmq-secret
  namespace: mqtest
data:
  pass: UEA4OHcwcmQ=
  user: YWRtaW4=
type: Opaque
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: mqtest
  labels:
    app: rabbitmq
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rabbitmq
    spec:
      volumes:
        - name: host-time
          hostPath:
            path: /etc/localtime
            type: ''
      containers:
        - name: rabbitmq
          image: 'harbor.itho.cn:5443/mytest/rabbitmq_xfsh:3.13.0-rc.2-managemen'
          ports:
            - name: tcp-5672
              containerPort: 5672
              protocol: TCP
            - name: http-15672
              containerPort: 15672
              protocol: TCP
          env:
            - name: RABBITMQ_DEFAULT_USER
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: user
            - name: RABBITMQ_DEFAULT_PASS
              valueFrom:
                secretKeyRef:
                  name: rabbitmq-secret
                  key: pass
          resources:
            limits:
              cpu: '2'
              memory: 4000Mi
            requests:
              cpu: 100m
              memory: 500Mi
          volumeMounts:
            - name: host-time
              readOnly: true
              mountPath: /etc/localtime
  serviceName: rabbitmq-headless
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-headless
  namespace: mqtest
  labels:
    app: rabbitmq
spec:
  ports:
    - name: tcp-rabbitmq-5672
      protocol: TCP
      port: 5672
      targetPort: 5672
  selector:
    app: rabbitmq
  clusterIP: None
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-external
  namespace: mqtest
  labels:
    app: rabbitmq-external
spec:
  ports:
    - name: http-rabbitmq-external
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app: rabbitmq
  type: NodePort

kubectl apply -f rabbit.yaml

15672为http管理端口，5672是通讯端口，web端登录的账号密码写在yaml的Secret中。

服务名称的格式通常是..svc.cluster.local
pod内部容器可以直接通过 rabbitmq-headless.mqtest.svc.cluster.local
访问可以做持久化，把/var/lib/rabbitmq挂载出来即可。

0x03 在K8S集群中安装多节点RabbitMQ（mq集群）

官方提供了快速部署的例子

# 通过官网获取 RabbitMQ Cluster Operator 部署资源配置清单cluster-operator.yml
wget https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

# RabbitMQ Cluster 部署资源清单 rabbitmq-cluster.yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  namespace: mqtest
  name: rabbitmq-cluster
  labels:
    app: rabbitmq-cluster
spec:
  replicas: 3
  image: harbor.itho.cn:5443/mytest/rabbitmq_xfsh:3.13.0-rc.2-managemen
  resources:
    limits:
      cpu: 2
      memory: 4Gi
    requests:
      cpu: 100m
      memory: 500Mi
  rabbitmq:
    additionalConfig: |
      default_user=admin
      default_pass=P@88w0rd
    # 启用其他插件需要按照下面的写，使用rabbitmqOperator部署mq集群，在镜像里面通过命令启用是无效的。官方手册https://www.rabbitmq.com/kubernetes/operator/using-operator.html#additional-plugins
    additionalPlugins:
      - rabbitmq_delayed_message_exchange 

# 管理页面的外部访问服务 rabbitmq-cluster-external.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-cluster-external
  namespace: mqtest
  labels:
    app: rabbitmq-cluster-external
spec:
  ports:
    - name: management
      protocol: TCP
      port: 15672
      targetPort: 15672
      nodePort: 31672
  selector:
    app.kubernetes.io/name: rabbitmq-cluster
  type: NodePort

# 三个yaml文件一把梭
kubectl apply -f cluster-operator.yml
kubectl apply -f rabbitmq-cluster.yaml
kubectl apply -f rabbitmq-cluster-external.yaml

如果后续需要修改mq集群的数量 需要通过编辑rabbitmq资源来实现

# 查看资源名称
root@iZbp12bkuvg20e1j3y9gtxZ:~/rabbitmq-cluster# kubectl get RabbitmqCluster -n base-ops
NAME               ALLREPLICASREADY   RECONCILESUCCESS   AGE
rabbitmq-cluster   False              False              52m
# 编辑资源
root@iZbp12bkuvg20e1j3y9gtxZ:~/rabbitmq-cluster# kubectl edit  RabbitmqCluster rabbitmq-cluster -n base-ops

0x03 开启rabbitmq集群镜像队列功能

镜像队列配置只能mq集群部署完后，随便进入一个rabbitmq容器运行下面命令来实现
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'

rabbitmq@rabbitmq-cluster-server-0:/$ rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all"}'
Setting policy "ha-all" for pattern "^" to "{"ha-mode":"all"}" with priority "0" for vhost "/" ...

0x04 参考文章

消息队列RabbitMQ及其插件的安装
rabbitmq官方手册-Using the RabbitMQ Cluster Kubernetes Operator
RabbitMQ集群环境搭建-镜像模式
rabbitmq生产环境配置详细