[转帖]MinIO系列7 - Minio性能压测

minio,系列,性能 · 浏览次数 : 0

小编点评

**内容概述** 生成内容时需要考虑以下因素： * 小对象和大对象的主要性能指标需求。 * 性能指标需求的平衡。 *磁盘性能。 * CPU/内存资源的配备。 **内容分析** 1. 小对象和大对象的性能指标需求。 2. 性能指标需求的平衡。 3.磁盘性能。 4. CPU/内存资源的配备。 **内容设计** *定义MinIO集群的性能指标需求。 *平衡的考虑磁盘性能，以及CPU/内存资源的配备。 **内容结论** *生成内容时需要带简单的排版。

正文

https://www.zhihu.com/people/keen-wang

前言

声明：此文为本人历史笔记的整理，文章实际撰写时间为2021年2月份，所以本中所使用的相关组件版本较老。此文是通过压力测试以理解MinIO在当前硬件条件下能提供的性能，以及相关的资源消耗情况。转载请注明出处，谢谢!

前置条件：已有Kubernetes集群，并安装好了MinIO集群。

相关组件版本：

组件名	版本	说明
Kubernetes	1.19	运行环境
minio	RELEASE.2020-06-14T18-32-17Z
iozone	3-490	测试磁盘的吞吐量
s3-benchmark		测试MinIO的上传下载性能

Part 1, 测试 8Nodes/32Disks

1 集群介绍

8个节点，每节点4*1TiB的磁盘。磁盘底层为VMware的全SSD vSAN分布式存储。
minio集群的节点参数设置如下

# Number of drives attached to a node
drivesPerNode: 4
# Number of MinIO containers running
replicas: 8
# Number of expanded MinIO clusters
zones: 1

pod的资源没做限制，内存request为4G
Storage class为默认

mc admin config get  minio-test storage_class
storage_class standard= rrs=EC:2 dma=write

2 MinIO磁盘性能评估

2.1 测试单盘性能

参照官方的性能测试报告，对于单盘的性能使用dd测试
在k8s的node节点上找到挂载到minio的磁盘，如下面命令所示，minio使用的磁盘为/dev/sd{c.d.e.f}

kubectl get po -n minio -o wide | grep $(hostname)
minio-4                   1/1     Running   0          31m    10.233.102.21   prod-ops-hy-k8s-27-1-31   <none>           <none>

kubectl exec -it minio-4 -n minio -- df -Th|grep export
/dev/sde             ext4         1006.9G    214.4M    955.5G   0% /export-2
/dev/sdd             ext4         1006.9G    214.4M    955.5G   0% /export-3
/dev/sdc             ext4         1006.9G      2.1G    953.6G   0% /export-0
/dev/sdf             ext4         1006.9G     79.3M    955.6G   0% /export-1

查看本地/dev/sdc的挂载路径

df -Th|grep sdc
/dev/sdc                        ext4     1007G  2.1G  954G   1% /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-b7021959-4a18-42e6-b4c6-10eab427e5cc/globalmount

进入上面查找到的路径，执行dd

## 以16M块连续写入2GB数据
# dd if=/dev/zero of=./test bs=16M count=128 oflag=direct
128+0 records in
128+0 records out
2147483648 bytes (2.1 GB) copied, 2.12772 s, 1.0 GB/s

## 以16M块连续读取2GB数据
#  dd of=/dev/null if=./test bs=16M count=128 iflag=direct
128+0 records in
128+0 records out
2147483648 bytes (2.1 GB) copied, 2.01436 s, 1.1 GB/s

上面结果为，单磁盘以16M块连续写吞吐量为1GB/s；以16M块连续读吞吐量为1.1GB/s。

2.2 测试JBOD性能

下载iozone，并安装

wget http://www.iozone.org/src/current/iozone-3-490.x86_64.rpm

rpm -ivh iozone-3-490.x86_64.rpm

ls -l /opt/iozone/bin/iozone
-rwxr-xr-x 1 root root 336464 Jul 14  2020 /opt/iozone/bin/iozone

在/mnt下面创建挂载目录，并将/dev/sd{c.d.e.f}重新挂载，以方便测试

mkdir /mnt/drive{1..4}
mount /dev/sdc /mnt/drive1
mount /dev/sdd /mnt/drive2
mount /dev/sde /mnt/drive3
mount /dev/sdf /mnt/drive4

df -Th | grep /mnt/drive
/dev/sdd                        ext4     1007G  215M  956G   1% /mnt/drive2
/dev/sdc                        ext4     1007G  2.1G  954G   1% /mnt/drive1
/dev/sde                        ext4     1007G  215M  956G   1% /mnt/drive3
/dev/sdf                        ext4     1007G   80M  956G   1% /mnt/drive4

按照官方性能测试报告的推荐，执行iozone，同样使用32 thread。由于不清楚官方文档中取的读写吞吐量值是Children还是Parent，当前测试出来的取Children的话，4个盘聚合的写吞吐量可达到1.2GB/s，读吞吐量为2.3GB/s。

/opt/iozone/bin/iozone -t 32 -I -r 32K -s 256M -F /mnt/drive{1..4}/tmp{1..8}
        Iozone: Performance Test of File I/O

...

        Children see throughput for 32 initial writers  = 1290740.23 kB/sec
        Parent sees throughput for 32 initial writers   = 1030806.54 kB/sec

...

        Children see throughput for 32 readers          = 2365000.80 kB/sec
        Parent sees throughput for 32 readers           = 2362278.36 kB/sec

4. 这个与dd测试的单盘1.1GB/s没太大的差距，由于上面iozone使用的是32K块，可能存在iops的性能达到了极限，而吞吐量上不去的情况。所以改成16M的块再测试一遍，结果如下所示，写吞吐量达到了2.76GB/s，读吞吐量达到了2.74GB/s。

/opt/iozone/bin/iozone -t 32 -I -r 16M -s 256M -F /mnt/drive{1..4}/tmp{1..8}
        Iozone: Performance Test of File I/O

...

        Children see throughput for 32 initial writers  = 2895796.42 kB/sec
        Parent sees throughput for 32 initial writers   = 2450370.20 kB/sec
        Min throughput per process                      =   79270.73 kB/sec
        Max throughput per process                      =  115852.30 kB/sec
        Avg throughput per process                      =   90493.64 kB/sec
        Min xfer                                        =  180224.00 kB

...

        Children see throughput for 32 readers          = 2874974.30 kB/sec
        Parent sees throughput for 32 readers           = 2793761.68 kB/sec
        Min throughput per process                      =   81235.27 kB/sec
        Max throughput per process                      =  100890.62 kB/sec
        Avg throughput per process                      =   89842.95 kB/sec
        Min xfer                                        =  212992.00 kB

3 MinIO吞吐量Benchmark

3.1 准备客户机

从github上克隆wasabi的s3-benchmark到ansible的主控主机上

git clone https://github.com/wasabi-tech/s3-benchmark.git
Cloning into 's3-benchmark'...
remote: Enumerating objects: 40, done.
remote: Total 40 (delta 0), reused 0 (delta 0), pack-reused 40
Unpacking objects: 100% (40/40), done.

这们使用一个k8s测试集群的8个主机节点来做MinIO benchmark的客户机，在ansibie host中这些主机归属于ops-hy-k8s组

at /etc/ansible/hosts
[ops-hy-k8s]
prod-ops-hy-k8s-27-1-11
prod-ops-hy-k8s-27-1-12
prod-ops-hy-k8s-27-1-13
prod-ops-hy-k8s-27-1-14
prod-ops-hy-k8s-27-1-15
prod-ops-hy-k8s-27-1-16
prod-ops-hy-k8s-27-1-17
prod-ops-hy-k8s-27-1-18

执行如下ansible命令将s3-benchmark复制到客户机

cd s3-benchmark
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark/s3-benchmark dest=/usr/local/bin/s3-benchmark mode=700"

4. 测试下是否可以批量执行

ansible ops-hy-k8s -m shell -a "/usr/local/bin/s3-benchmark -h"
prod-ops-hy-k8s-27-1-14 | FAILED | rc=2 >>
Wasabi benchmark program v2.0Usage of myflag:
  -a string
        Access key
  -b string
        Bucket for testing (default "wasabi-benchmark-bucket")
  -d int
        Duration of each test in seconds (default 60)
  -l int
        Number of times to repeat test (default 1)
  -r string
        Region for testing (default "us-east-1")
  -s string
        Secret key
  -t int
        Number of threads to run (default 1)
  -u string
        URL for host with method prefix (default "http://s3.wasabisys.com")
  -z string
        Size of objects in bytes with postfix K, M, and G (default "1M")non-zero return code

...

3.2 启动benchmark

先生成一个脚本，用来启动s3-benchmark，并复制到每个客户机。-z 2G意思是使用2G的对象进行上传下载测试，-t 32意思是客户端开启32进程

echo "/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 2G -t 32 -b s3bench-\$HOSTNAME -d 1" > /tmp/s3-benchmark.sh
ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh"

启动s3-benchmark，请注意这里一定要带"-f 8"，不然ansible只会同样启动5个客户机上的脚本

ansible ops-hy-k8s -f 8 -m shell -a "/root/s3-benchmark.sh"

到任意客户机上查看s3-benchmark是否在运行

ps -ef|grep s3
root      4843  4815  0 22:22 pts/4    00:00:00 /bin/sh -c /root/s3-benchmark.sh
root      4844  4843 85 22:22 pts/4    00:00:02 /usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 2G -t 32 -b s3bench-prod-ops-hy-k8s-27-1-11 -d 1

运行完成后返回结果，有两个客户机因为tcp连接重置而退出了

prod-ops-hy-k8s-27-1-13 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-13, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/08 23:12:14 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-13/Object-27: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-13/Object-27: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.13:40542->10.27.1.31:443: write: connection reset by peernon-zero return code

prod-ops-hy-k8s-27-1-17 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-17, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/08 23:12:17 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-17/Object-15: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-17/Object-15: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.17:35888->10.27.1.31:443: write: connection reset by peernon-zero return code

prod-ops-hy-k8s-27-1-18 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-18, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 231.4 secs, objects = 32, speed = 283.3MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 83.1 secs, objects = 32, speed = 789MB/sec, 0.4 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 159.8 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-12 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-12, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 366.9 secs, objects = 32, speed = 178.6MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 115.5 secs, objects = 32, speed = 567.4MB/sec, 0.3 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.2 secs, 25.9 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-11 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 429.8 secs, objects = 32, speed = 152.5MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 165.5 secs, objects = 32, speed = 396MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 137.0 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-16 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-16, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 450.1 secs, objects = 32, speed = 145.6MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 145.6 secs, objects = 32, speed = 450.1MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.2 secs, 160.2 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-14 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-14, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 442.7 secs, objects = 32, speed = 148MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 153.9 secs, objects = 32, speed = 425.7MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 110.2 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-15 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-15, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 441.6 secs, objects = 32, speed = 148.4MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 155.5 secs, objects = 32, speed = 421.6MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 116.1 deletes/sec. Slowdowns = 0

3.3. 结果分析

客户机吞吐量统计

客户机	PUT	GET
节点1	152.5MB/s	396MB/s
节点2	178.6MB/s	567.4MB/s
节点3			失败
节点4	148MB/s	425.7MB/s
节点5	148.4MB/s	421.6MB/s
节点6	145.6MB/s	450.1MB/s
节点7			失败
节点8	283.3MB/s	789MB/
汇总	1056.4MB/s	3049.8MB/s

CPU、内存消耗峰值和带宽峰值

MinIO节点	CPU使用峰值/核	内存使用峰值/GiB	带宽峰值/Rx	带宽峰值/Tx
minio-0	1.37	11.13	354MB/s	425MB/s
minio-1	1.31	11.26	375MB/s	439MB/s
minio-2	1.81	10.73	375MB/s	449MB/s
minio-3	1.12	10.48	362MB/s	405MB/s
minio-4	1.52	10.91	402MB/s	463MB/s
minio-5	1.16	10.99	343MB/s	393MB/s
minio-6	2.07	11.42	362MB/s	434MB/s
minio-7	1.57	10.94	340MB/s	423MB/s
平均	1.5	11	364MB/s	429MB/s

4 小文件benchmark

4.1 修改参数并启动benchmark

修改s3-benchmark的启动脚本为如下，-z 1M将对象大小修改为1M，-d 300持续测试300秒

/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 1M -t 32 -d 300 -b s3bench-$HOSTNAME

重新复制到各个客户机

ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh mode=700"

启动脚本

ansible ops-hy-k8s -f 8 -m shell -a "sh /root/s3-benchmark.sh"

4.2. 结果分析

客户机吞吐量统计

客户机	PUT吞吐量	PUT ops	GET吞吐量	GET ops
节点1	62.2MB/sec	62.2	316.7MB/s	316.7
节点2	62.1MB/sec	62.1	320.3MB/s	320.3
节点3	62.5MB/sec	62.5	317.3MB/s	317.3
节点4	62.4MB/sec	62.4	319.2MB/s	319.2
节点5	62MB/sec	62	319.3MB/s	319.3
节点6	62.5MB/sec	62.5	300.6MB/s	300.6
节点7	62.4MB/sec	62.4	318.5MB/s	318.5
节点8	62.6MB/sec	62.6	319.5MB/s	319.5
汇总	498.7MB/s	498.7	2531.4MB/s	2531.4

CPU、内存消耗峰值和带宽峰值

MinIO节点	CPU使用峰值/核	内存使用峰值/GiB	带宽峰值/Rx	带宽峰值/Tx
minio-0	8.05	22.92	202.7MB/s	274.9MB/s
minio-1	7.81	22.15	199.6MB/s	276.5MB/s
minio-2	8.92	22.22	212.6MB/s	292.4MB/s
minio-3	8.15	22.19	195MB/s	275.1MB/s
minio-4	8.5	22.50	202.9MB/s	280.8MB/s
minio-5	7.32	22.01	196.4MB/s	273.2MB/s
minio-6	9.36	22.03	204.2MB/s	278.4MB/s
minio-7	8.64	22.49	198MB/s	268.9MB/s
平均	8.34	22.31	201.4MB/s	277.5MB/s

5 测试结论

整个集群的大对象（2G）吞吐量，写入为1056.4MB/s，读取为3049.8MB/s。
小对象（1M）的写入吞吐量为498.7MB/s，写入ops为498.7；读取吞吐量为2531.4 MB/s，读取ops为2531.4。
对于单节点4个磁盘的JBOD吞吐量，读写能达到2.7GB/s，8个节点的集群理论IO吞吐量能达到21.6GB/s，节点间的网络吞吐量为25Gb/s的硬件环境。这个性能只能说差强人意，也可能是由于没有针对性优化。

Part 2，测试有资源限制的8Nodes/32Disks

1 集群介绍

在《Part1，测试8Nodes/32Disks》中我们可以看到并发上升后，minio的资源使用还是比较厉害。在小文件压测中，集群中节点的CPU平均使用8.34核，内存平均使用为22.31GiB。正式生产使用中，如果我们不需要到这么高的吞吐量，但是资源不做限制时，pod很容易就使用过多的资源。
因此，本次测试我们将资源限制为如下，以观察这些资源能支撑多大的并发量

resources:
  requests:
    cpu: 1
    memory: 2Gi
  limits:
    cpu: 2
    memory: 4Gi

其他保持一致，然后重新构建集群

2 MinIO磁盘性能评估

磁盘性能与《测试1-8Nodes-32Disks》中的保持一致

3 吞吐量Benchmark

3.1 准备客户机

请参照《Part1，测试8Nodes/32Disks》中的3.1

3.2 启动benchmark

运行完成后返回结果，有8个客户机中有7个为tcp连接重置而退出了，只有1个执行成功

prod-ops-hy-k8s-27-1-11 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/10 12:58:05 WARNING: createBucket s3bench-prod-ops-hy-k8s-27-1-11 error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
        status code: 409, request id: 16624A187E2ED504, host id:
2021/02/10 12:59:35 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-8: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-8: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.11:51916->10.27.1.31:443: write: connection reset by peernon-zero return code

...

prod-ops-hy-k8s-27-1-16 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-16, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 197.5 secs, objects = 32, speed = 331.9MB/sec, 0.2 operations/sec. Slowdowns = 0
Loop 1: GET time 70.4 secs, objects = 32, speed = 931.3MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.4 secs, 77.0 deletes/sec. Slowdowns = 02021/02/10 12:58:06 WARNING: createBucket s3bench-prod-ops-hy-k8s-27-1-16 error, ignoring BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
        status code: 409, request id: 16624A1896EE10BC, host id:

查看ingress nginx的日志

2021/02/10 04:59:18 [error] 46#46: *29201 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.17, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-17/Object-11 HTTP/1.1", upstream: "http://10.233.119.28:9000/s3bench-prod-ops-hy-k8s-27-1-17/Object-11", host: "minio-test.xxx.com"
2021/02/10 04:59:20 [error] 43#43: *29225 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.17, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-17/Object-23 HTTP/1.1", upstream: "http://10.233.107.26:9000/s3bench-prod-ops-hy-k8s-27-1-17/Object-23", host: "minio-test.xxx.com"
I0210 04:59:21.486939       7 main.go:187] "Received SIGTERM, shutting down"
I0210 04:59:21.486977       7 nginx.go:372] "Shutting down controller queues"
I0210 04:59:21.740124       7 nginx.go:380] "Stopping admission controller"
I0210 04:59:21.740262       7 nginx.go:388] "Stopping NGINX process"
E0210 04:59:21.740321       7 nginx.go:319] "Error listening for TLS connections" err="http: Server closed"
2021/02/10 04:59:21 [notice] 590#590: signal process started
2021/02/10 04:59:26 [error] 87#87: *29196 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.17, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-17/Object-18 HTTP/1.1", upstream: "http://10.233.112.30:9000/s3bench-prod-ops-hy-k8s-27-1-17/Object-18", host: "minio-test.xxx.com"
2021/02/10 04:59:28 [error] 130#130: *29130 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.27.1.14, server: minio-test.xxx.com, request: "PUT /s3bench-prod-ops-hy-k8s-27-1-14/Object-12 HTTP/1.1", upstream: "http://10.233.114.25:9000/s3bench-prod-ops-hy-k8s-27-1-14/Object-12", host: "minio-test.xxx.com"

原因是ingress nginx的proxy_connect_timeout默认是5s，5s连不上upstream host就会超时，通过kubectl edit ingress的方式修改名为minio的ingress，添加proxy-connect-timeout、proxy-read-timeout和proxy-send-timeout这3个annotation

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:

...

    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"

然后重启测试，仍有4个客户机失败，4个成功

prod-ops-hy-k8s-27-1-11 | FAILED | rc=1 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-11, region=us-east-1, duration=1, threads=32, loops=1, size=2G2021/02/10 13:17:36 FATAL: Error uploading object https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-20: Put https://minio-test.xxx.com/s3bench-prod-ops-hy-k8s-27-1-11/Object-20: net/http: HTTP/1.x transport connection broken: write tcp 10.27.1.11:36496->10.27.1.34:443: write: connection reset by peernon-zero return code

prod-ops-hy-k8s-27-1-17 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-17, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 333.7 secs, objects = 32, speed = 196.4MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 70.3 secs, objects = 32, speed = 932.3MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.6 secs, 51.0 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-13 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-13, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 342.6 secs, objects = 32, speed = 191.3MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 70.0 secs, objects = 32, speed = 936.7MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 1.0 secs, 33.4 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-18 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-18, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 375.0 secs, objects = 32, speed = 174.8MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 74.9 secs, objects = 32, speed = 875.1MB/sec, 0.4 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.3 secs, 94.5 deletes/sec. Slowdowns = 0

prod-ops-hy-k8s-27-1-14 | CHANGED | rc=0 >>
Wasabi benchmark program v2.0
Parameters: url=https://minio-test.xxx.com, bucket=s3bench-prod-ops-hy-k8s-27-1-14, region=us-east-1, duration=1, threads=32, loops=1, size=2G
Loop 1: PUT time 413.1 secs, objects = 32, speed = 158.7MB/sec, 0.1 operations/sec. Slowdowns = 0
Loop 1: GET time 69.1 secs, objects = 32, speed = 947.9MB/sec, 0.5 operations/sec. Slowdowns = 0
Loop 1: DELETE time 0.4 secs, 74.8 deletes/sec. Slowdowns = 0

修改ansible的host配置文件，增加一个主机组为s3-client，将客户机的数量减少至4个后所有都执行成功了

[s3-client]
prod-ops-hy-k8s-27-1-11
prod-ops-hy-k8s-27-1-12
prod-ops-hy-k8s-27-1-13
prod-ops-hy-k8s-27-1-14

3.3. 结果分析

客户机吞吐量统计

客户机	PUT	GET
节点1	254.5MB/sec	669.3MB/sec
节点2	443.7MB/sec	755.1MB/sec
节点3	238.7MB/sec	717.6MB/sec
节点4	231.9MB/sec	779.7MB/sec
汇总	1168.8MB/s	2921.7MB/s

CPU、内存消耗峰值和带宽峰值

MinIO节点	CPU使用峰值/核	内存使用峰值/GiB	带宽峰值/Rx	带宽峰值/Tx
minio-0	1.16	3.17	299MB/s	354MB/s
minio-1	1.65	3.25	353MB/s	414MB/s
minio-2	1.71	3.24	268MB/s	331MB/s
minio-3	1.57	3.35	314MB/s	352MB/s
minio-4	1.46	3.1	293MB/s	308MB/s
minio-5	1.34	3.07	208MB/s	337MB/s
minio-6	1.5	3.17	288MB/s	316MB/s
minio-7	1.81	3.11	327MB/s	392MB/s
平均	1.53	3.18	294MB/s	350MB/s

4 小文件benchmark

4.1 修改参数并启动benchmark

修改s3-benchmark的启动脚本为如下，-z 1M将对象大小修改为1M/100K，-d 300持续测试300秒

/usr/local/bin/s3-benchmark -a minio -s minio123 -u https://minio-test.xxx.com -z 1M -t 32 -d 300 -b s3bench-$HOSTNAME

重新复制到各个客户机

ansible ops-hy-k8s -m copy -a "src=/tmp/s3-benchmark.sh dest=/root/s3-benchmark.sh mode=700"

启动脚本

ansible ops-hy-k8s -f 8 -m shell -a "sh /root/s3-benchmark.sh"

4.2 结果分析

4.2.1 1M对象大小

客户机吞吐量统计

客户机	PUT吞吐量	PUT ops	GET吞吐量	GET ops
节点1	19.5MB/sec	19.5	50.8MB/sec	50.8
节点2	19.7MB/sec	19.7	50.2MB/sec	50.2
节点3	19.5MB/sec	19.5	50.5MB/sec	50.5
节点4	19.4MB/sec	19.4	50.4MB/sec	50.4
节点5	19.5MB/sec	19.5	50.6MB/sec	50.6
节点6	19.6MB/sec	19.6	50.2MB/sec	50.2
节点7	19.6MB/sec	19.6	51MB/sec	51
节点8	19.5MB/sec	19.5	50.5MB/sec	50.5
汇总	156.3MB/s	156.3	404.2MB/s	404.2

CPU、内存消耗峰值和带宽峰值

MinIO节点	CPU使用峰值/核	内存使用峰值/GiB	带宽峰值/Rx	带宽峰值/Tx
minio-0	1.81	2.82	93.5MB/s	102.4MB/s
minio-1	1.71	2.96	68.2MB/s	114.2MB/s
minio-2	2	3.1	109.3MB/s	177.4MB/s
minio-3	1.69	2.8	111.1MB/s	140.6MB/s
minio-4	1.91	2.97	69.7MB/s	112.6MB/s
minio-5	1.65	2.92	79.6MB/s	102MB/s
minio-6	1.88	2.9	79.5MB/s	102.7MB/s
minio-7	1.87	2.79	96MB/s	197.2MB/s
平均	1.82	2.91	87.6MB/s	135.2MB/s

4.2.2 100KB对象大小

客户机吞吐量统计

客户机	PUT吞吐量	PUT ops	GET吞吐量	GET ops
节点1	2.4MB/sec	24.3	5.3MB/sec	54.7
节点2	2.4MB/sec	24.6	5.4MB/sec	55.2
节点3	2.4MB/sec	24.3	5.4MB/sec	55.5
节点4	2.4MB/sec	24.6	5.4MB/sec	55
节点5	2.4MB/sec	24.6	5.3MB/sec	54.8
节点6	2.4MB/sec	24.4	5.4MB/sec	55.4
节点7	2.4MB/sec	24.6	5.3MB/sec	54.8
节点8	2.4MB/sec	24.6	5.4MB/sec	55.2
汇总	19.2MB/s	196	42.9MB/s	440.6

CPU、内存消耗峰值和带宽峰值

MinIO节点	CPU使用峰值/核	内存使用峰值/GiB	带宽峰值/Rx	带宽峰值/Tx
minio-0	1.78	3.29	12.83MB/s	17.53MB/s
minio-1	1.72	3.31	14.93MB/s	20.09MB/s
minio-2	2	3.5	19.93MB/s	25.49MB/s
minio-3	1.71	3.09	18.36MB/s	23.36MB/s
minio-4	1.89	3.42	17.03MB/s	15.99MB/s
minio-5	1.68	3.07	13.45MB/s	15.63MB/s
minio-6	1.89	3.29	14.13MB/s	16.37MB/s
minio-7	1.83	3.17	16.21MB/s	25.99MB/s
Avg	1.81	3.27	15.86MB/s	20.06MB/s

5 测试结论

大对象的吞吐量数量与无资源限制相差不多。与小对象相比，操作数少很多，会造成资源消耗相对少不少，特别是内存。
1M对象和100K对象的ops性能基本在一个量级，100K略高。CPU使用率基本相同，100K的内存使用率略高。可见在磁盘的IO性能未达瓶颈时，小对象的ops处决于CPU/内存资源，与对象大小相关性不大。
在设计MinIO生产集群时，应考虑如下几点：

先预先评估集群的存储对象大小，是大对象为主（>100M），还是小对象为主（KB ~ MB）。
当以大对象为主时，应重点关注磁盘的吞吐量；当以小对象为主时，应重点关注磁盘的IOPS和MinIO节点的内存。
然后定义MinIO集群的性能指标需求，再平衡的考虑磁盘性能，以及CPU/内存资源的配备。

[转帖]MinIO系列7 - Minio性能压测

小编点评

正文

前言

Part 1, 测试 8Nodes/32Disks

1 集群介绍

2 MinIO磁盘性能评估

2.1 测试单盘性能

2.2 测试JBOD性能

3 MinIO吞吐量Benchmark

3.1 准备客户机

3.2 启动benchmark

3.3. 结果分析

4 小文件benchmark

4.1 修改参数并启动benchmark

4.2. 结果分析

5 测试结论

Part 2，测试有资源限制的8Nodes/32Disks

1 集群介绍

2 MinIO磁盘性能评估

3 吞吐量Benchmark

3.1 准备客户机

3.2 启动benchmark

3.3. 结果分析

4 小文件benchmark

4.1 修改参数并启动benchmark

4.2 结果分析

4.2.1 1M对象大小

4.2.2 100KB对象大小

5 测试结论

与[转帖]MinIO系列7 - Minio性能压测相似的内容：

# 热门排行