| Author |
Message
|
| Esa |
 Posted: Thu Sep 27, 2012 11:16 am Post subject: memory issue with a large number of nonpersistent msgs |
 |
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
While waiting to be given something useful to do I repeated my little test from last May:
| Quote: |
I just created a little test flow that looped with MQGet, reading a 500 kB message repeatedly (with MQGMO_BROWSE_FIRST) as a BLOB, parsing it as XMLNSC and putting that on another queue.
I let the flow loop 200 000 times, which took about half an hour - during which the memory allocated by the DataFlowEngine process remained very stable, varying between 91620 and 91628 kB.
Broker V8 on 64-bit Win 7. |
I monitored the processes and the memory consumption with task manager and resource monitor. Everything worked the same way. But then I added another flow, a simple flow that would just consume messages from the output queue of the test flow. I started the second flow after the first flow had put over 100 000 messages in the output queue. After a while the physical memory usage shown by the performance tab in Task Manager started to rise. However, no processes in the Processes tab showed any increase. The memory consumption continued to rise untill almost all of my 8 GB was in use. And it would probably have risen even more if I hadn't stopped flow 2. That stopped the rise, but the memory consumption stayed on the same high level.
In brief, no processes showed high memory consumption, but the overall memory consumption was at its limit and the system was quite slow to respond. But there was a lot of activity in the Disk tab of Resource Monitor, especially the System process was very busy writing and reading at a huge rate.
I should actually have posted this in a Websphere MQ forum, because this is not a Message Broker issue (the memory allocated by the execution group stayed at the same levell all the time). But I suspect that few other applications than Message Broker have high enough performance to be able to produce this effect.
I think the situation takes place when the in-memory buffer for non-persistent messages of the queue has flown over and the queue manager is writing the messages in the physical queue file. Now, if applications are both writing and reading non-persistent messages, the way the queue manager tries to synchronize the in-memory buffer and the queue file makes the file system cache of the operating system allocate more and more memory.
The memory consumption returned to the normal level after I had closed applications that had an open handle on the queue. I added a third flow in the execution group so that I could stop both of my test flows without stopping the EG. And I had rfhutil keeping the queue open as well, to make sure this has nothing to do with the EG and the broker.
I'm sure this is a known issue. Probably operating system specific, too. I tried to search MQSeries.net, but could not find anything. It's just a bit difficult fo find proper search keywords, I suppose. |
|
| Back to top |
|
 |
| Esa |
| Posted: Thu Sep 27, 2012 2:05 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Yes, it's WMQ 7.1 which is not supported by WMB 8.0 but I'm sure this could be achieved with MB 7.0 on WMQ 7as well... |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 27, 2012 5:10 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| Yes, it's WMQ 7.1 which is not supported by WMB 8.0 |
8.0.0.1 supports WMQ 7.1. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Sep 27, 2012 11:37 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
| Esa wrote: |
| Yes, it's WMQ 7.1 which is not supported by WMB 8.0 |
8.0.0.1 supports WMQ 7.1. |
Anyway, the point is that getting messages from a queue with a large (huge) number of nonpersistent messages seems to bring the system on its knees, at least if it's Win 7. A practical example of this could be an MCA starting to get messages from a transmission queue after a longer network breakdown.
I was asking if this is a known issue. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Sep 28, 2012 1:18 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Does the loop increase the WMB node stack, or is it using the propagate command from a compute node? |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Sep 28, 2012 2:27 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| zpat wrote: |
| Does the loop increase the WMB node stack, or is it using the propagate command from a compute node? |
The latter. But this is not a Message Broker issue but a WMQ or operating system issue. It does not matter how the messages are put on the queue. The problems start when you start reading from the queue. You need to read messages with great speed to be able to produce the effect, for example using RFHUtil to read messages and keeping ctrl-Q down does not seem to be fast enough.
I think you could simulate it by loading 100 000 nonpersistent messages of size 500 KB in a queue and then starting a message flow that would just read and throw away messages. The overall memory usage of the system would start rising. |
|
| Back to top |
|
|
| kimbert |
| Posted: Fri Sep 28, 2012 4:01 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I think you could simulate it by loading 100 000 nonpersistent messages of size 500 KB in a queue and then starting a message flow that would just read and throw away messages. The overall memory usage of the system would start rising. |
The proof would be complete if you replaced WMB with a simple C program that uses the MQ API to get the messages. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Sep 28, 2012 4:04 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Support pac MA01 perhaps. |
|
| Back to top |
|
|
| Esa |
| Posted: Sun Sep 30, 2012 12:05 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| kimbert wrote: |
| Quote: |
| I think you could simulate it by loading 100 000 nonpersistent messages of size 500 KB in a queue and then starting a message flow that would just read and throw away messages. The overall memory usage of the system would start rising. |
The proof would be complete if you replaced WMB with a simple C program that uses the MQ API to get the messages. |
I tested it with a simple C program called the MCA. I modified my flow so that it put messages on a remote queue. Then I stopped the sender channel and let the flow fill the transmission queue with 200 000 messages. After that I stopped the broker and started the sender channel. The memory consuption rose gradually until it was over 95% of the available memory. No individual process showed high memory usage. The disk queue length stayed over 15, with System writing 25 MB/sec (filesystem cache for the receiving QM writing nonpersistent messages on disk) and amqzlaa0 reading messages from the XMITQ at 25 MB/sec.
When the flow was putting messages, it was only the System that kept the disk busy, writing almost 50 MB/sec. But that did not make the memory consumption rise.
When I stopped the channel, the memory usage sank to normal level. But if I had had another program getting messages from the same queue, the memory consumption would have stayed on the high level, even if the program did not read messages but just kept the queue handle open. I could not test this with the XMITQ because the MCA opens the queue in exclusive mode. But my theory is that the memory consumption will not return to normal until there are no more open input handles on the queue. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Sep 30, 2012 5:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| I tested it with a simple C program called the MCA. |
Your definition of "simple" and mine varies quite sharply. Think of all the handshaking and other activities the MCA performs. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Sep 30, 2012 11:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So you've shown that the problem is not in any way related to Broker, but is in fact a consequence of how your queue manager is configured and how MQ handles deep queues.
It therefore seems to me that you should focus your energy on understanding those topics better and determining if there are performance tuning adjustments you can make to your queue manager to better handle deep queues, or whether you should just make a note in your book that "deep queues lead to problems" (as most of the rest of us have already done...  ) |
|
| Back to top |
|
|
| Esa |
| Posted: Sun Sep 30, 2012 11:41 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Thanks, mqjeff, for the confirmation.
I must admit that tuning MQ did not cross my mind. My log size is certainly too small, among other things.
The case is not a customer case. I cannot even think of any kind of application that would have such deep queues (with the exception of MQ itself after a long network failure). I guess I'm focusing my energy on these things just because I'm a bit bored... |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Oct 01, 2012 4:06 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Esa wrote: |
| I cannot even think of any kind of application that would have such deep queues (with the exception of MQ itself after a long network failure). |
Yet I'm sure you like all of us have some app queues with high Max Q Depth values because potentially the consuming app could die and stay dead for quite a while before the producing app's were stopped. Understanding how things behave in this scenario if a worthwhile endeavor.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 6:36 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
|
| SAFraser |
| Posted: Tue Oct 02, 2012 1:29 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
esa, I very much appreciate you sharing this research.
Sadly, we do have a business case. We have an application that puts 1.4M messages into a queue without a consumer. We've just had quite a battle with the developers, and have finally made some progress. Now we only let them put 400,000 messages at a time.
Ridiculous, I know. <sigh> |
|
| Back to top |
|
|
|
|
