MinIO多节点集群部署与负载均衡配置

0x01 MinIO单节点Docker部署

1、docker minio 单机挂载单盘

run起来后直接访问http9001就是面板

docker run -it -d --name minio \
--restart=always \
-p 9000:9000 \
-p 9001:9001 \
-v /data/minio/data:/data \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=admin123" \
minio/minio server /data --console-address ":9001"

2、docker minio 单机挂载多盘

4块盘分别为 data1、data2、data3、data4。

docker run -it -d --name minio \
--restart=always \
-p 9000:9000 \
-p 9001:9001 \
-v /data/minio/data1:/data1 \
-v /data/minio/data2:/data2 \
-v /data/minio/data3:/data3 \
-v /data/minio/data4:/data4 \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=admin123" \
minio/minio server /data{1...4} --console-address ":9001"

0x02 MinIO集群二进制部署

1、准备两台机子，每个节点都挂载了2块盘（/data1 /data2）

# 两个节点均使用lsblk查看挂载，并确保在/etc/fstab配置了自动挂载
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1 223.4M  0 rom  
vda    253:0    0    20G  0 disk 
└─vda1 253:1    0    20G  0 part /
vdb    253:16   0   100G  0 disk /data1
vdc    253:32   0   100G  0 disk /data2

2、每个节点都二进制安装minio，确保版本一样

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/
#开防火墙
firewall-cmd --permanent --zone=public --add-port=9000/tcp
firewall-cmd --permanent --zone=public --add-port=9001/tcp
firewall-cmd --reload

3、创建systemd服务文件

vi /usr/lib/systemd/system/minio.service

[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio

[Service]
WorkingDirectory=/usr/local

User=minio-user
Group=minio-user
ProtectProc=invisible

EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \\"${MINIO_VOLUMES}\\" ]; then echo \\"Variable MINIO_VOLUMES not set in /etc/default/minio\\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES

# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify

# Let systemd restart this service always
Restart=always

# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536

# Specifies the maximum number of threads this process can create
TasksMax=infinity

# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no

[Install]
WantedBy=multi-user.target

# Built for ${project.name}-${project.version} (${project.name})

4、设置权限

默认情况下，minio.service文件以minio-user用户和组身份运行。

groupadd -r minio-user
useradd -M -r -g minio-user minio-user
chown minio-user:minio-user /data1 /data2

5、写入启动配置文件

vi /etc/default/minio

MINIO_VOLUMES="http://10.206.10.1{1...2}:9000/data{1...2}"
MINIO_OPTS="--console-address :9001"
MINIO_ROOT_USER=admin
MINIO_ROOT_PASSWORD=admin123
#MINIO_SERVER_URL="https://minio.example.net:9000"

6、启动

systemctl daemon-reload
systemctl start minio
systemctl enable minio

0x03 为MinIO服务器配置NGINX负载均衡

nginx配置参考如下：

upstream minio_s3 {
   least_conn;
   server 10.206.10.11:9000;
   server 10.206.10.12:9000;
}

upstream minio_console {
   least_conn;
   server 10.206.10.11:9001;
   server 10.206.10.12:9001;
}

server {
   listen       80;
   listen  [::]:80;
   server_name  minio.itho.cn;

   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   client_max_body_size 0;
   # Disable buffering
   proxy_buffering off;
   proxy_request_buffering off;

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      proxy_pass https://minio_s3; # This uses the upstream directive definition to load balance
   }

   location /minio/ui/ {
      rewrite ^/minio/ui/(.*) /$1 break;
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-NginX-Proxy true;

      # This is necessary to pass the correct IP to be hashed
      real_ip_header X-Real-IP;

      proxy_connect_timeout 300;

      # To support websockets in MinIO versions released after January 2023
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      # Some environments may encounter CORS errors (Kubernetes + Nginx Ingress)
      # Uncomment the following line to set the Origin request to an empty string
      # proxy_set_header Origin \'\';

      chunked_transfer_encoding off;

      proxy_pass https://minio_console; # This uses the upstream directive definition to load balance
   }
}

注意：S3 API签名计算算法不支持托管MinIO Server API（如example.net/s3/的代理方案。
您还必须为MinIO部署设置以下环境变量，默认路径为/etc/default/minio：
将MINIO_SERVER_URL设置为MinIO服务器的代理主机FQDN（https://minio.example.net）
将MINIO_BROWSER_REDIRECT_URL设置为MinIO控制台的代理主机FQDN（https://example.net/minio/ui）

0x04 参考文档

Deploy MinIO: Multi-Node Multi-Drive
Configure NGINX Proxy for MinIO Server