[转帖]OutOfMemory自动重启程序

outofmemory,自动,重启,程序 · 浏览次数 : 0

小编点评

## Alternative Solutions to Handle OutOfMemoryError **1. Implement a Graceful Shutdown Mechanism:** - Modify the `start.sh` script to exit gracefully upon encountering an OutOfMemoryError. - Use `System.exit(0)` or `System.exit(1)` to signal success or failure. - Append an `at` command to the end of the `start.sh` script with a `sleep` of 10 seconds to allow the process to exit gracefully. **2. Configure Memory Monitoring and Restart:** - Use the `-XX:+HeapDumpOnOutOfMemoryError` flag with the `at` command. - Specify the path to the heap dump file as `/data/log/appOutOfMemory.hprof`. - Restart the service immediately upon reaching the OutOfMemoryError. **3. Use a Monitoring and Restart Tool:** - Install tools like `atop` or `vmstat` to monitor memory usage and process creation/termination. - Configure the service to restart automatically when memory usage reaches a set threshold. **4. Implement a Dedicated Monitoring Service:** - Create a separate service that continuously monitors memory usage and service health. - Send alerts or notifications when memory limits are reached. - Restart the service automatically upon reaching the memory limit. **5. Increase Memory Limit:** - If possible, increase the allocated memory for the service to accommodate larger memory requirements. - Remember that this option can have performance implications.

正文

OutOfMemory以后程序已经假死，无法再提供服务，最好的做法是dump内存，发送警告，然后重启服务

我的方案：利用at命令延迟启动

但有一个问题，at最多支持分钟操作，也就是说要1分钟以后才能启动，我的业务允许接受1分钟的延迟，总比收到警告然后等到人工去启动要好一点。

 -XX:OnOutOfMemoryError="at -f /data/deploy/start.sh now +1 minutes;kill -9 %p;"

at命令的安装和启动

yum -y install at
systemctl restart atd

知识点
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/log/appOutOfMemory.hprof
OnOutOfMemoryError命令要等到dump文件appOutOfMemory.hprof生成完成以后才会执行
注意直接执行restart.sh是行不通的，因为新启动的进程直接保留了原进程的端口引用，并且这个端口引用无法被kill，Spring boot在初始化完成以后就会发现端口已经被占用了就会停掉，如果使用随机端口的话则可以忽略这个问题。
下面的方案行不通

 -XX:OnOutOfMemoryError="at -f /data/deploy/restart.sh"

改进思路

停止进程可以用自带的参数： -XX:+ExitOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError
思路一：可以用监控脚本定时每隔10秒监控进程是否存在，如果不存在则启动程序
思路二：写一个专门启动服务的程序，XX:OnOutOfMemoryError发一条请求任务到这个启动服务的程序
上面的改进思路是有追求的运维干的事情，我不是