| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| How does MQ queue manager select message by CorrelationID? |
« View previous topic :: View next topic » |
| Author |
Message
|
| claire |
 Posted: Tue Aug 30, 2005 6:26 am Post subject: How does MQ queue manager select message by CorrelationID? |
 |
|
Apprentice
Joined: 10 Apr 2002
Posts: 32
|
Hello,
For MQ5.2 and 5.3, if I use MA88 or JMS to select a message by specifying its correlation id, how does queue manager locate the message? Does it inspect the messages in the queue one by one? Or it uses "index" to find the message? Is there any performance report for this operation? Thanks a lot. |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Tue Aug 30, 2005 6:10 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If you use something other than JMS, and you do a get by Correl ID, the QM will start at the top of the queue and look at each message one by one till it finds a match, or it gets to the bottom of the queue (MQRC=2033). On z/OS QMs only, you can have the queue Indexed by CorrelID or MessageID, which significantly increases performance for very deep queues on selective MQGETs. The MQSC manual has some info, as do the z/OS MQ manuals.
If you use JMS, you have 2 ways to ask for the message with a specific Correl ID. If you prefix the wanted Correl ID with "ID:", then it acts as described above. If you do not prefix it, then what JMS is really asking for is a copy of the 1st message to be retrieved, it inspects it, accepts it or reqjects it, gets the next message, etc. In other words, if your message is #1000, JMS will issue 1000 MQGETs! But if you insert that ID: in front of the Correl ID value, it acts like a traditional GET and that 1000th message will be reterned by only one MQGET atthe QM (obviously much better performing!).
See here for more:
http://www.mqseries.net/phpBB2/viewtopic.php?t=21530&highlight=jms
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hopsala |
| Posted: Wed Aug 31, 2005 2:27 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| PeterPotkay wrote: |
| But if you insert that ID: in front of the Correl ID value, it acts like a traditional GET and that 1000th message will be reterned by only one MQGET atthe QM (obviously much better performing!). |
Sorry to interrupt, but I didn't quite get what you meant by inserting the ID: "in front"...?
More importantly, what you describe is very strange behavior indeed, and this is not what the post you referenced to says. MQ gets requests in JMS or normal API and translates it to some internal procedure which gets from queue (an internal MQGET, if you will), I don't see why should a C language MQGET with CorrlId get msgs one by one and compare, and a JMS MQGET (the equiv) should somehow zone in on that one message and pull it out, why shouldn't both do it?
Regardless, I don't think MQ maintains any meta-table concerning queues, only some linked-list (within the files); to my knowledge, there is no table in memory which contains the CorrlId and MsgId of every message. So it doesn't really matter how you work with MQ, internally there is no way it can find a correllation id without going through the entire queue.
The only possible difference between JMS and other API's is that JMS only receives the MQMD instead of both the MQMD and message data when searching for the right CorrlID (although I don't see why a C MQGET shouldn't do the same) - this can indeed speed up performance if these are large messages, saving a lot of memcpy declarations. But still, there would always be 1000 internal gets from queue if msg1000 has the CorrlID you're looking for.
(Btw, An interesting experiment would be: put 10 msgs, the 5th is 10KB while the others are 50KB, now MQGET with CorrlId of the 5th msgs and a buffer size of 10KB... This will tell us if MQ draws the entire msg or only mqmd, and whether it uses the user buffer or its own memory)
I don't think i'm missing something, but if I do please advise  |
|
| Back to top |
|
|
| markt |
| Posted: Wed Aug 31, 2005 3:55 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
>> I don't think MQ maintains any meta-table concerning queues, only some linked-list (within the files); to my knowledge, there is no table in memory which contains the CorrlId and MsgId of every message. So it doesn't really matter how you work with MQ, internally there is no way it can find a correllation id without going through the entire queue.
You are wrong. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Aug 31, 2005 5:46 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
afaik .... each queue has a Message Chain, which is an in-memory list. Each message on the queue is in the chain, and for each message there is a hash of the msgId and correlID, as well as pointers to the message in the queue file. There is also a table called the message details cache, which contains details for the 512 most recently used messages, which contains the complete correlID and msgID.
When you issue a selective get, the message chain is searched based on the hash. If a possible match is found, then the details cache is checked to see if its an exact match. If the message isn't in the details chace, then the message is read directly out of the queue (becuase the location of the bits of the message is in the chain and other in-memory locations) and the messageID or correlID is checked.
_________________
-wayne |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Aug 31, 2005 6:05 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
hopsala, check out that post again that I referanced, especially the last 2 or 3 posts that I did in that thread, where I finally figured out what was going on.
The Transaction Vision API Exit clearly showed that on selective MQGETs by Correl ID on queues with more than 1 message:
1. non JMS apps caused the QM to only record 1 MQGET
2. JMS apps that prefixed the wanted CorrelID value with ID: caused the QM to only record 1 MQGET
3. JMS apps that did not use ID: would issue an MQGET for each and every message until the match was found.
In #1 and 2, the slection is done by the QM, in #3 the selection is done by the JMS app. When considering MQClients, you can see how much extra traffic is generated by #3. Because #3 might be having the app using message selectors that are not present in the MQMD, I suspect that #3 actually causes the entire message to be sent back to the app for inspection, but I did not put a sniffer on to see how much data was actually flowing on the wire back to the client app on each of those hopeful MQGETs.
A couple of years ago I did test the CorrelID indexing on z/OS, and found it *very* worthwhile, especially when the queues got over 5K deep.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bower5932 |
| Posted: Wed Aug 31, 2005 7:17 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
|
| Back to top |
|
|
| JLRowe |
| Posted: Wed Aug 31, 2005 8:00 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
The new messaging engine in WAS v6 stores messages in a database, I would expect retrieval by messageId/correlId to be a lot quicker as they are fully indexed.
Not sure about JMS message properties though, these may require a table scan, even though a database should be a lot quicker at this than MQ. |
|
| Back to top |
|
|
| hopsala |
| Posted: Wed Aug 31, 2005 12:21 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
Well, apparently I was wrong after all; Thanks all for the input, I am eternally grateful. 
Funnily enough, this hashing table wschutz described adds another clause to the ever-growing list I have under the title "a good queue is an empty queue" - A phrase I utter in the ears of my customers and students wherever I can.
I guess that like any other hashing table, since space is limited (both on the hashing table and on the 512 cache), as more hashed msgs are added, the number of collisions increases, thus worsening performance. This, coupled with other reasons we already know about (IO) would explain the non-linear degradation in MQGET performance when the queue is full and someone is getting by CorrlID. In fact, as long as there are less than 512 msgs on queue, you can get by CorrlId with no worries at all - that's good to know!
I wonder, does anyone know which hashing algorithem MQ uses? (overflow, clustering, quadric etc, and which h(k) function) Also, is there a seperate hash table for CorrlID, MsgID and GroupID?
(Btw, is any of this documented at all?) |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Aug 31, 2005 12:32 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
No, it's not a hash table, the MsgID and CorrelID of the message are stored as hashed values in the message chain. (so hash collisions aren't important)
The only "public" info I know of on this is the "Distributed MQ Internals" presentation given at the MQ Tech conference.
_________________
-wayne |
|
| Back to top |
|
|
| markt |
| Posted: Wed Aug 31, 2005 12:57 pm Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
| And it's different on the V6 64-bit qmgrs where more memory is available for internal indexing, hashing, caching etc. |
|
| Back to top |
|
|
| hopsala |
| Posted: Wed Aug 31, 2005 1:00 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| wschutz wrote: |
| No, it's not a hash table, the MsgID and CorrelID of the message are stored as hashed values in the message chain. |
Ah, I see; so I was right after all - Every MQGET by CorrlID has O(n) complexity, and although indeed there is only one MQGET, it is after going through the entire list. True, the hashed values speed it up significantly (no need for binary compares, just some XOR probably) - but still, that's a lot of operations, added that if it's not in the 512 cache, a lot of IO...
Did I understand you correctly?
| wschutz wrote: |
| The only "public" info I know of on this is the "Distributed MQ Internals" presentation given at the MQ Tech conference. |
Well, how about some "private" knowledge? Don't hold out on us, knowledge to the people!  |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
