| Author |
Message
|
| atheek |
 Posted: Sun Jun 07, 2009 11:58 pm Post subject: MQ Selective Gets on client side?? |
 |
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
Hi All,
We have an a MQ JMS client application that consumes messages from a queue in a remote mq server using selective gets. The gets are selected on an application specific string hashed to the message id field by the sender application. The client application uses the same hashing algoithm to retrieve the message.
[Understand this is a misue of messageID field. This application was designed by a third party and handed over to us recenty]
At any time this queue has atleast 30,000 messages pending. The message Id was used as this is one of the few header fields on which MQ does indexing.
Still it is observed that the response time for get's are very high. The explanation given by one of the designers is that on a client application, message selection is done at the client side ie. all the messages from the queue (30,000 or more) are retrieved to the client side over the network and the selection is done there. This, as per him, also explains the high network traffic observed during the get operations.
Is this theory correct? Please share your experiences if you have come across this.
Thanks,
Atheek |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Jun 08, 2009 12:26 am Post subject: Re: MQ Selective Gets on client side?? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| atheek wrote: |
| At any time this queue has atleast 30,000 messages pending. The message Id was used as this is one of the few header fields on which MQ does indexing. |
Only if the queue is hosted on z/OS and configured to hold an index. Queues on distributed platforms are not indexed.
| atheek wrote: |
| Is this theory correct? |
Yes - if the queue manager was a database, you'd be doing a table scan. You'd see the same thing on a binding connection, but expressed as high I/O.
I've no idea what would happen if you had a z/OS indexed queue and used CAF to do a selective read.
You've already said using the message ID is bad design. So is using a queue as a database (with 30,000 rows). Accept it's not going to perform all that well..... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jun 08, 2009 12:44 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Migrate to WMQ v7 (both client and QM) - the JMS selection now happens on the server side.
Using WMQ V7 is likely to make a huge performance difference to this application. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 08, 2009 1:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| Using WMQ V7 is likely to make a huge performance difference to this application. |
Storing these 30,000 messages in a database likely to make an even bigger difference... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jun 08, 2009 1:13 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Yes, but one is a simple upgrade to WMQ and the other is a complete application re-write. |
|
| Back to top |
|
|
| atheek |
| Posted: Mon Jun 08, 2009 1:27 am Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
Thanks Vitor and Zpat.
Is this only for JMS clients or for others as well?
They are using MQ as the data store to save few dollars on the database licenses. 
They were early using a cutom RFH Header in the message as the selector. Now they changed it to message id field and claims there is an improvement in response times for gets as they are able to leverage the indexing feature of MQ on these fields.
As per my understanding, IBM recommends to use MessageID and CorrelationID headers for selective get's. So is there (??) any indexing on these fields internally on distributed platforms? The result from their testing suggest so.
Thanks,
Atheek |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 08, 2009 1:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| Yes, but one is a simple upgrade to WMQ and the other is a complete application re-write. |
But 2 points occur:
- The application (given it relies on user-defined msgId and scanning 30,000 messages) could probably use a complete re-write
- Depending on the politics, an application re-write might be easier than upgrading the infrastructure! Imagine (and we can only imagine) if the site is on v5.3. An application project turns up and tells site management "our application will perform really badly unless we leapfrog directly to WMQv7". That could be a lot of forms & regression testing of other apps.
AFAIK despite the move of the JMS client select, the underlying queues are still only indexed on z/OS - am I right in that?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jun 08, 2009 1:38 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes, but a JMS selector (browse) does not use the index even if present, only the match msgid MQGET would use it.
This JMS performance problem was well-known to IBM, hence their changes to V7.
JMS function is now native to WMQ instead of being an after-thought bolt-on. |
|
| Back to top |
|
|
| markt |
| Posted: Mon Jun 08, 2009 1:41 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
Wrong on many levels.
On distributed platforms, indexing/hashing happens automatically for msgid retrieval.
Selection by msgid (provided JMS uses the right 24-byte format) has always been done server-side. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Jun 08, 2009 1:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
|
| Back to top |
|
|
| atheek |
| Posted: Mon Jun 08, 2009 1:46 am Post subject: |
|
|
Partisan
Joined: 01 Jun 2006
Posts: 327
Location: Sydney
|
| zpat wrote: |
| This JMS performance problem was well-known to IBM, hence their changes to V7. |
Does this implies that the issue is only with JMS clients?
Is the assumption that selective gets based on Message ID performs better than custom rfh headers on distributed platforms true? Or is it tend to be same as in both cases complete queue scan is required?
Please clarify.
Thanks,
Atheek |
|
| Back to top |
|
|
| markt |
| Posted: Mon Jun 08, 2009 1:49 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
On z/OS, indexing has to be explicitly enabled - hence it's an external function of the product.
On distributed platforms, it's internal. How the qmgr manages retrieval is entirely up to it, and there are no configuration attributes. It just so happens that the qmgr builds various internal tables and indexes. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jun 08, 2009 1:54 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| markt wrote: |
| Selection by msgid (provided JMS uses the right 24-byte format) has always been done server-side. |
Pre V7 - MQ is JMS unaware on the queue manager.
If the JMS client issues a MQGET match msgid instead of the normal JMS selector browsing then great. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jun 08, 2009 1:01 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| atheek wrote: |
| zpat wrote: |
| This JMS performance problem was well-known to IBM, hence their changes to V7. |
Does this implies that the issue is only with JMS clients?
Is the assumption that selective gets based on Message ID performs better than custom rfh headers on distributed platforms true? Or is it tend to be same as in both cases complete queue scan is required?
Please clarify.
Thanks,
Atheek |
Well it used to be that the only way to use a programmed selector without programming it yourself was by using the message selector in JMS/ XMS.
This message selector however still had to access each message as the selection was done in the client. V7 changes that. At the same time using the provider form (= ID:hexvalue) of the message selector for MsgId, CorrelId, GroupId has always been pretty fast compared to any other form of selection, letting us believe that this selection could actually happen on the qmgr... However only IBM can tell you exactly what the difference was...
V7, the doc claims, does property and MQMD field selection on the qmgr. This lessens the burden on the network for the clients and leads to considerable speedup.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Jun 08, 2009 1:51 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
This message selector however still had to access each message as the selection was done in the client. V7 changes that. At the same time using the provider form (= ID:hexvalue) of the message selector for MsgId, CorrelId, GroupId has always been pretty fast compared to any other form of selection, letting us believe that this selection could actually happen on the qmgr... However only IBM can tell you exactly what the difference was...
|
Even at MQ 6.0. if you used (= ID:hexvalue) you would see massive performance gains for selective gets by Message ID or Correl ID on deep queues. Using ID= made the selective MQGET perform like a typical non JMS one, i.e. the QM did the selction for you and sent you back the one message that matched. If you didn't do this, JMS asked for a copy of the 1st message on the q, if it didn't match it would ask for a copy of the 2nd, if it didn't match it would ask for a copy of the 3rd, etc. You can see how this would start performing worse and worse as the q got deeper.
I guess MQ 7.0 removes the oppourtunity for the developer to code it "wrong", so that every selective get is done on the QM side, and only the copy of the message you want goes over the wire, versus a copy of every message up to the one that finally matches. And MQ 7.0 allows this to happen for selectors other than Msg ID, Correl ID and Group ID. In MQ 6.0 it was only possible to boost the selective MQ GET performance if you were using these native MQ "selectors".
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
