| Author |
Message
|
| wmbwmq |
 Posted: Wed Apr 04, 2012 12:24 pm Post subject: message load/unload to memory (AMQ7234) |
 |
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
What is the reason behind MQ's load/unload messages from memory?
I was doing some disaster preparedness testing. I loaded one queue with 1 million messages. And after the qmgr restart i had to wait for 14 minutes before i could put/get on that queue (apparently MQ was loading messages to memory : AMQ7234). Anyway I would have just popped a chill pill and assumed this never happened. But apparently MQ does this even when a queue is left unused for some time (couple of minutes may be).
can someone tell me why this is required? The queue file already must have stored the messages in some easily MQ readable format(i assume). And any FIFO get/put should have been as easy as grabbing a beer from the cooler. Why to load the entire queue to memory? Someone pls enlighten me |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Apr 04, 2012 9:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Again you see why a messaging system is not a database and is in fact quite different from it.
The best queue depth is 0. Messages just pass through as in a pipe!
Now what leads you to have 1 Million + messages in a queue?
And by the way being able to grab a beer from a cooler does in no way reflect to the cooler's integrity..., but you'd want to ensure the message store's integrity before removing, or adding messages to it, wouldn't you?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Thu Apr 05, 2012 6:02 am Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Saper,
I understand MQ is not a database. But the whole point of being asynchronous is storing messages temporarily. But again, if I store messages for weeks then I am abusing MQ but what if you have a heavily used queue @ 2000msgs/sec. And if the app is down for just 1 hour, we are looking at 7.2 million messages. Yuck.
Coming back to MQ's loading of messages to memory. I am not an expert in MQ internals. But say I am using some java to handle a large file. I add/delete text to it (hell i would go with hashing here). So to insert a message at the end of the file or delete a message from the start of the file would be simple one i assume (may be OS is overworking handling that file but not java). I understand there are transaction logs etc when it comes to MQ. But still i am not clear on the requirement to load the whole queue to memory as long as I am not doing priority get.
Sorry I am just trying to understand the rational here. Can you pls elaborate. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Apr 05, 2012 6:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
MQ doesn't necessarily load the entire contents of the entire queue in memory.
But it can't simply assume that all applications reading from a given queue are going to be reading from it in a strictly FIFO manner and not using MsgID or CorrelID to retrieve individual messages, or using Browse to read through messages.
So it has to make sure it knows at least some information about all messages in all queues. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 05, 2012 6:18 am Post subject: Re: message load/unload to memory (AMQ7234) |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Queues exist in virtual storage (memory). Messages are hardened when written to disk storage, and are loaded from disk into virtual storage queues so they can be processed by your applications.
So, did it take 14 minutes for your application to process the very first message? Or did it take 14 minutes for your app to browse through to the last message in the queue? How do you know this?
Are you saying that you didn't even start your (or any) application? And that there was no response from the qmgr to any MQSC commands for 14 minutes?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Thu Apr 05, 2012 6:41 am Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Thanks Jeff.
Actually I am unable to see any real reason behind using a priority/msgid/correlid based get. Which just goes against asynchronous concept(may be except priority based get; but if we shouldnot accumulate messages on queues then why bother with priority?) and not to mention the severe performance hit . Besides I think any such requirement can be easily and efficiently handled using a database.
Still the MQ's default msgdlvsq is priority. this blows my mind. (nothing against IBM btw  , they also set default maxdepth as 5000. what are we using MQ for ? counting crows?
Bruce,
Yes, I just did a browse on the queue (amqsbcg) and it took 14 minutes. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 05, 2012 6:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| what if you have a heavily used queue @ 2000msgs/sec. And if the app is down for just 1 hour, we are looking at 7.2 million messages. Yuck. |
Off topic, but this is where you step back from pure WMQ. If you have the heavily used queue you describe there are a couple of options:
- You (and your stakeholders) accept that it's going to take a while for the queue manager to come back after an outage, but that's ok because you're running a 2000 mps application on a single instance of a queue so it can't be important if they're processed quickly after arrival
- You handle your 2000 mps application by splitting the load across a cluster so when one instance crashes you only get 60,000 odd messages stuck
- You have HA on the instance so it's not down for an hour.
You pay your money, you make your choice.
At startup the queue manager sets itself up for what it thinks it's doing; passing messages which are flowing through it to applications. Typically this means there are not that many messages sitting on queues waiting to go because they're flowing through. In exception cases like yours, where there's a huge backlog, that doesn't go so well.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 05, 2012 6:51 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| wmbwmq wrote: |
Bruce,
Yes, I just did a browse on the queue (amqsbcg) and it took 14 minutes. |
Do the same test with one message in a queue. Repeat with 1000 messages in a queue. Repeat with 10,000 messages in a queue. 100,000 messages in a queue.
I'd expect the lag (delay) to increase, wouldn't you? I suspect that much of the lag is disk I/O.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 05, 2012 7:12 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wmbwmq wrote: |
| Actually I am unable to see any real reason behind using a priority/msgid/correlid based get. Which just goes against asynchronous concept(may be except priority based get; but if we shouldnot accumulate messages on queues then why bother with priority?) and not to mention the severe performance hit . Besides I think any such requirement can be easily and efficiently handled using a database. |
If you see no reason for a get based on msgid or correlid, how do you handle async request / reply scenatios? Correlating through a database with all the I/O that involves?
If the reason you have messages of differing priorities is because a batch application is dropping batches of low-priority messages onto the same queue identical but more urgent real time messages from a user facing application arrive on, why would you bother to put the urgent messages through the queue and have identical processing picking the rest off a database?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| wmbwmq |
| Posted: Thu Apr 05, 2012 7:29 am Post subject: |
|
|
Acolyte
Joined: 18 Jul 2011
Posts: 66
|
Bruce,
Sorry I meant it took 14 minutes before amqsbcg started dumping any(first) message. All this time MQ was reloading the entire queue to memory. And as per the logs(AMQ7234), it was loading at the rate of 1.2k msgs/ sec. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 05, 2012 9:55 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm not surprised by this behavior.
It's no more surprising than the first call to a database causes the DBMS to read the database table from disk into virtual storage (memory).
A difference worth noting here is that there is an expectation that dbs will be accessed randomly (so the DBMS may read portions of the db randomly from disk), while queues are not accessed randomly.
I paraphrase my worthy colleagues: a queue is not a database.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
