| Author |
Message
|
| Jhap |
 Posted: Thu May 28, 2015 10:16 am Post subject: Inactive queue load / unload messages to disc ? |
 |
|
Novice
Joined: 18 Feb 2010
Posts: 18
|
Hello !
Just need to clarify next claim.
If MQ has example one million persistent messages stored in local queue without continuously reading those messages out. After few minutes MQ start to behaves like that, it loads / unloads those messages to disc and also writes note from this operation to log between every 10 000 messages ?
During this operation all other functions concerning this queue, like receiver channel go to wait status ? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu May 28, 2015 10:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If MQ has 1 million messages loaded to a queue, your program is completely broken and you will get horrible horrible performance.
Your app developers should also be fired.
Or at least taken out back and trouted for a while.
 |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 28, 2015 10:30 am Post subject: Re: Inactive queue load / unload messages to disc ? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Jhap wrote: |
| If MQ has example one million persistent messages stored in local queue without continuously reading those messages out. After few minutes MQ start to behaves like that, it loads / unloads those messages to disc and also writes note from this operation to log between every 10 000 messages ? |
I fully agree with my most worthy associate. Even if the software can handle it, that's a truly horrible design.
| Jhap wrote: |
| During this operation all other functions concerning this queue, like receiver channel go to wait status ? |
All other functions will go into a wait state; the log is not at a queue level but at a queue manager level. It also contains one million duplicate copies of these messages so they can be recovered if the queue file is corrupted (you say log so I say persistent) so the I/O will be through the roof. Imagine the start up time if corruption occurs and if it has to roll through the log to recover. Imagine if you just restart it and it verifies one million persistent messages in the queue against the log images.
Even if they're not persistent, woe betide you if you do a controlled stop and controlled start of the queue manager and thus make the queue manager do an integrity check of the queue file. That's going to take a while. Woe will slightly less betide you if it crashes and just decides to restart - reinitializing a queue file that used to hold that much will still delay the start up.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Jhap |
| Posted: Thu May 28, 2015 11:00 am Post subject: |
|
|
Novice
Joined: 18 Feb 2010
Posts: 18
|
Thanks !
Problem is not poor program, problem is that there is no program at all  Program is goin to start after few weeks.
Actually I just heard this and this is not my problem, I just want to know does MQ behaves like I wrote.
If messages are persistent, mq start after few minutes make copies ? Is this true ?
I thought that MQ makes copy immediately when persistent message arrives to queue and delete copy when message is put another queue and commit ?
What about if there is one million messages in queue and some program read messages slowly . MQ still makes copy of those messages which are still in queue ? |
|
| Back to top |
|
|
| Jhap |
| Posted: Thu May 28, 2015 11:15 am Post subject: |
|
|
Novice
Joined: 18 Feb 2010
Posts: 18
|
Now I understand that point that mq writes log after few minutes....
MQ start make copies immediately but first mark to log appears after few minutes when messages are started put to queue  |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 28, 2015 11:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Jhap wrote: |
| I just want to know does MQ behaves like I wrote. |
Not exactly how you wrote.
| Jhap wrote: |
| If messages are persistent, mq start after few minutes make copies ? Is this true ? |
False. It logs the message at the same time as it arrives. It also compares the log copy with the queue copy at start up and during recovery operations (e.g. media recovery)
| Jhap wrote: |
| I thought that MQ makes copy immediately when persistent message arrives to queue and delete copy when message is put another queue and commit ? |
Yes, but it removes it when the message is read off the queue and the read is committed. It doesn't have to be put anywhere.
| Jhap wrote: |
| What about if there is one million messages in queue and some program read messages slowly . MQ still makes copy of those messages which are still in queue ? |
MQ will make a copy of any new message that arrives, and retains a copy of any message not yet read off. If messages arrive while they're being read off, this causes a huge I/O penalty as the file pointer is moved to where the copy is being removed, then moved back to the end to add a record, then moved forward and so forth.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 28, 2015 11:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Jhap wrote: |
| MQ start make copies immediately but first mark to log appears after few minutes when messages are started put to queue |
The queue manager doesn't report it in "real time", even though that's how the copies are made.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Jhap |
| Posted: Thu May 28, 2015 11:28 am Post subject: |
|
|
Novice
Joined: 18 Feb 2010
Posts: 18
|
Thanks a lot for clarifying  |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu May 28, 2015 3:27 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The message you see about 10,000 messages read/written is MQ buffer management internals telling you that it doing work for you - offloading/reloading buffers to/from disk.
When a persistent message is MQPUT to a queue, it is first written to the log portion of the file-system, then made available to any consuming (MQGETting) application in the message buffer in virtual storage.
If there is no MQGETting application, when the message buffer fills, buffer management software moves the message to the queue-holding portion of the file-system. So, worst case, two writes of the same message - one to the log and one to the queue-holding file-system.
Millions of messages produced, but no consumer? When, if ever, will the consumer come to life?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Andyh |
| Posted: Fri May 29, 2015 12:17 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
The AMQERRnn.LOG messages about "messages loaded" are nothing to do with MQ buffer management.
When a queue is first referenced, a representation of that queue is loaded into memory. One part of this memory representation is a queue index, and in order to create the queue index the queue manager needs to look at the control data (effectively the MQMD) of each message. The "messages loaded" messages are indicating how many 'MQMD's' have been accessed during this process.
The queue is loaded at first reference, and if the queue then becomes unreferenced during some sustained period of other activity the queue will be chosen to be unloaded, where the queue index will be discarded, and where buffers etc that haven't yet been written to disk would be written to disk. The intent here is to release memory that's associated with a queue that's no longer being used.
If you have a very deep queue, with an unusual access pattern (such as write only!) then it might be an idea to write a program that simply opens the queue and waits for queue manager termination. The presence of this open handle will be sufficient to prevent the queue from being unloaded, and thus the need to reload the queue at the next reference.
Of course the well known mantra that "MQ is not a database" is worth repeating here. There are some occasions where a deep queue is within acceptable messaging design goals (for example when some downstream system is unavailable a deep transmission queue), but storage of large volumes of long lived data isn't the sweet spot for a traditional messaging engine. |
|
| Back to top |
|
|
| Andyh |
| Posted: Sun Mar 18, 2018 12:23 pm Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
I cannot repeat the "MQ is not a database" mantra too frequently, but there's a little change in MQ 9.0.5 that might benefit some users of deep queues.
MQ 9.0.5 includes a small change to try to reduce the frequency of queue unload/reload.
Prior to 9.0.5 the algorithm that decides when a queue is unloaded used the time and amount of activity since the queue was last referenced in order to decide when to unload the queue.
From 9.0.5 the current depth of the queue is also taken into account in an attempt to reduce the frequency of unload/reload for deep queues. |
|
| Back to top |
|
|
| tczielke |
| Posted: Mon Mar 19, 2018 8:26 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
My two cents, but the feature that Kafka has to act as a psuedo-database (e.g. store the messages on disk for a week and allow the application to replay the messages, if needed) is a handy feature that our Kafka developers leverage and IBM MQ does not offer.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| Andyh |
| Posted: Mon Mar 19, 2018 10:15 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Kafka is more of a message streaming service than the traditional once and once only messaging service that MQ provides.
There are clearly applications more suited to one or the other, and neither approach plays particularly well in the wrong scenario. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Mar 19, 2018 3:23 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Andyh wrote: |
The AMQERRnn.LOG messages about "messages loaded" are nothing to do with MQ buffer management.
When a queue is first referenced, a representation of that queue is loaded into memory. One part of this memory representation is a queue index, and in order to create the queue index the queue manager needs to look at the control data (effectively the MQMD) of each message. The "messages loaded" messages are indicating how many 'MQMD's' have been accessed during this process.
The queue is loaded at first reference, and if the queue then becomes unreferenced during some sustained period of other activity the queue will be chosen to be unloaded, where the queue index will be discarded, and where buffers etc that haven't yet been written to disk would be written to disk. The intent here is to release memory that's associated with a queue that's no longer being used.
If you have a very deep queue, with an unusual access pattern (such as write only!) then it might be an idea to write a program that simply opens the queue and waits for queue manager termination. The presence of this open handle will be sufficient to prevent the queue from being unloaded, and thus the need to reload the queue at the next reference.
Of course the well known mantra that "MQ is not a database" is worth repeating here. There are some occasions where a deep queue is within acceptable messaging design goals (for example when some downstream system is unavailable a deep transmission queue), but storage of large volumes of long lived data isn't the sweet spot for a traditional messaging engine. |
This is correct. If the queue depth is large (>100,000), the queue index reload can take a long time (many minutes) and can impact performance of the qmgr. eg. DISPLAY QLOCAL of the queue will hang until the reload is complete, which effectively hangs the MQ command processor for other users.
The solution is to not let queues grow to a large depth. The messages should be consumed continuously or at a regular short interval, so the that the depth does not grow to (say) over 1000.
_________________
Glenn |
|
| Back to top |
|
|
|
|
