小编点评

**问题描述** 使用 StackExchange.Redis 客户端SDK 连接 Azure Redis 服务，长期运行后发现每天偶发 Timeout Error，错误消息如下：  StackExchange.Redis.RedisTimeoutException: Timeout performing HGETALL (15000ms), next: HGETALL new_town, inst: 0, qu: 0, qs: 17, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: xxxxxxxx.redis.cache.chinacloudapi.cn:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxxxxxxxxxxx, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=17,Free=8174,Min=2,Max=8191), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)   **排查方向** 1. **WORKER 的 CPU 占用率过高** - WORKER: (Busy=17,Free=8174,Min=2,Max=8191) 表示 WORKER 的 CPU 占用率很高，可能导致时间限制。 2. **客户端 CPU 及网络带宽不均衡** - 客户端可能出现 CPU 紧张或网络带宽限制，导致请求被阻塞或超时。 3. **大键值优化** - 使用多个小值来存储数据，以减少对大键值的访问，以降低内存占用和时间成本。 4. **更新 Azure Redis 版本** - 升级至 Azure Redis 5 或更高版本，因为 5 及更高版本支持更大的内存限制和性能提升。 5. **调整内存限制** - 在配置文件中设置 `Maxmemory-reserved` 和 `Maxfragmentationmemory-reserved` 属性以指定最大内存限制。 6. **监控慢指令** - 使用 `slowlogs` 选项查看 Redis 中的慢指令，并优化性能。

正文

问题描述

使用StackExchange.Redis 作为Redis客户端SDK，连接Azure Redis服务，长期运行后发现，每天都偶发 Timeout Error。

错误消息如下：

 

• StackExchange.Redis.RedisTimeoutException: Timeout performing HGETALL (15000ms), next: HGETALL new_town, inst: 0, qu: 0, qs: 17, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: xxxxxxxx.redis.cache.chinacloudapi.cn:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxxxxxxxxxxx, IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=17,Free=8174,Min=2,Max=8191), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

 

•  ERROR log - Timeout performing HSET (15000ms), next: HGET token, inst: 1, qu: 0, qs: 35, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: xxxxxxxx.redis.cache.chinacloudapi.cn:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxxxxxxxxxxx, IOCP: (Busy=0,Free=1000,Min=50,Max=1000), WORKER: (Busy=29,Free=8162,Min=100,Max=8191), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

 

•  ERROR log - Timeout performing EXPIRE (15000ms), next: HGET token, inst: 0, qu: 0, qs: 35, aw: False, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: xxxxxxxx.redis.cache.chinacloudapi.cn:6380, mc: 1/1/0, mgr: 10 of 10 available, clientName: xxxxxxxxxxxx, IOCP: (Busy=0,Free=1000,Min=50,Max=1000), WORKER: (Busy=29,Free=8162,Min=100,Max=8191), v: 2.1.30.38891 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

 

排查方向

在第一个错误中，可以发现 WORKER 的Busy 数量 远大于 Min 数量 WORKER: (Busy=17,Free=8174,Min=2,Max=8191),，所以可以通过设置Worker/IOCP的线程数来解决这个问题。详细的说明见文档：https://docs.azure.cn/zh-cn/azure-cache-for-redis/cache-management-faq#recommendation

 

而之后继续出现Timeout 问题，这需要从如下几个方面优化：

1）查看慢指令(slowlogs) : 

there are slowlogs like HGET, HGETALL, HSCAN on this cache. Some commands are more expensive than others to execute, depending on their complexity. Because Redis is a single-threaded server side system, the time needed to run some more time expensive commands may cause some latency or timeouts on client side, as server can be busy dealing with these expensive commands.
Please refer Troubleshoot Azure Cache for Redis latency and timeouts | Microsoft Learn

 

2）查看客户端CPU及网络带宽

Check client host CPU or Network bandwidth. Please refer https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-troubleshoot-timeouts#high-cpu-on-client-hosts

 

3）查看大键值(bigkeys)

• Optimize your application for a large number of small values, rather than a few large values.
• The preferred solution is to break up your data into related smaller values.

Please refer: https://docs.azure.cn/zh-cn/azure-cache-for-redis/cache-troubleshoot-timeouts#large-key-value

 

4）升级Azure Redis到更高的定价层

5）Additional suggestion:
The memory reservations are not configured properly: Maxmemory-reserved and Maxfragmentationmemory-reserved have only set 50 MB each. Recommend to update the maxmemory-reserved and maxfragmentationmemory-reserved [atleast equal to 10% of the cache size].
For more details refer : Best practices for memory management - Azure Cache for Redis | Microsoft Learn