Memory Monitor config

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory Monitor config

TheQL
Hello!

Of course the memory monitor is documented, but I am a little confused about our current alerts.

In the past I had the memory monitor configured to trigger the GC timer at 2500MB.
We started to hit that regularly and it was time to run SwiftMQ with more total RAM. So I am now running SwiftMQ with -Xmx4096M instead of -Xmx3072M. It seems to make sense to also set the memory monitor to +1GB, roughly 3500MB. (I am aware that this is +1000MB and not +1024MB)

Now we still reach this limit a lot, although nothing has changed. Sure, Java uses available heap, why shouldn't it? But before I increase the memory monitor setting even more, we do want to prevent any OOM errors or running out of heap space, how do I interpret these mail warnings?

Memory Monitor on <xxx>_HA: Memory Usage CRITICAL! Starting GC Timer
ROUTER MEMORY STATUS:

Max Memory: 3641 MB
Total Memory: 3501 MB
Free Memory: 1781 MB

and then later this:
Memory Monitor on <xxx>_HA: Memory Usage back to NORMAL! Stopping GC Timer
ROUTER MEMORY STATUS:

Max Memory: 3641 MB
Total Memory: 3447 MB
Free Memory: 1960 MB

And so on... This repeats itself and each status lasts for several minutes.

Free memory doesn't seem to be alarming in either mails, but total memory is close to max memory in both cases. How am I supposed to act on this and why am I still getting this although I had the total and max memory increased? Or do I need to set alarms closer to max memory to avoid these alerts? Does this actually help me at all to avoid running out of memory? https://www.swiftmq.com/products/router/swiftlets/sys_monitor/introduction/index.html is not very precise on this.

Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: Memory Monitor config

IIT Software
Administrator
Hi,

the monitor sits on used memory and the mails you‘ve gotten are ok. I know it’s annoying to get these mails in high frequency if the usage vibrates around the threshold.

Same would happen if the monitor would sit on free memory. GC sometimes waits until all memory is used and then frees them at once.
Reply | Threaded
Open this post in threaded view
|

Re: Memory Monitor config

IIT Software
Administrator
In reply to this post by TheQL
And the sense of these alert mails is that you take a look at it.
Reply | Threaded
Open this post in threaded view
|

Re: Memory Monitor config

TheQL
I get that. I just wanted to know if there are reasons why I shouldn't put the threshold close to or at the Max Memory value. The only reason I really care about this is to prevent OOM, but maybe that is entirely not what this feature is for ;)
Reply | Threaded
Open this post in threaded view
|

Re: Memory Monitor config

TheQL
Tonight we got this:

ROUTER MEMORY STATUS:

Max Memory: 3641 MB
Total Memory: 3641 MB
Free Memory: 1263 MB

Router still survived. So I changed my alert to 3640MB.

But one other thing. We had a split brain this night due to missed heart-beats. This occured at 3:24:15 AM.
At the same time we could see this in memory consumption of the JavaVM on the active node. It makes me wonder...

SwiftMQ-RAM.jpg