| Author |
Message
|
| steffi2 |
 Posted: Wed Sep 09, 2009 4:04 pm Post subject: JMS Selector performance in 6.x? |
 |
|
Apprentice
Joined: 04 Sep 2009
Posts: 31
|
So at work I have a choice of either consuming from multiple queues asychronously with mutiple sessions, listeners within a single process where each QueueSession is to a distinctly different queue.
or I can choose to consume from 1 queue and use a selector to define what's relevant to a given QueueSession's receiver.
I suspect the first approach will be faster in MQ 6.x since there are performance issues with selectors prior to 7.x
Is that the case?
I can also go with one async consumer per process and have each process defined with a different selector or queue name (multiple queues)
What are the performance issues with MQ < 7.x? |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Sep 09, 2009 7:25 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
With MQ V 6x the best performance with selectors is obtained using the native provider form (ID:hexbinary value) and restricting the selection to some fields like
msg id
correl id
group id
etc... Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| manicminer |
| Posted: Thu Sep 10, 2009 1:01 am Post subject: |
|
|
Disciple
Joined: 11 Jul 2007
Posts: 177
|
In MQ6 selection is performed client side, where as in MQ7 selection is performed in the queue manager.
(see http://www-01.ibm.com/cgi-bin/common/ssi/ssialias?infotype=an&subtype=ca&htmlfid=897/ENUS208-068&appname=usn&language=enus )
The only caveat for that is as fjb_saper suggested and you selectors use standard MQ fields such as msg id, correl id, etc.
If you select based on a JMS property on your message, the client has to download the message, inspect the properties for a match, then move on to the next message if the match isn't made. I'm sure you can see for yourself what scenarios will introduce performance issues given that behaviour.
JMS performance in general, not just with selectors is much better in V7 with massive listener improvements and other improvements with server communications. If you are going JMS I would strongly recommend looking at the V7 upgrade path. There should be no changes required in your application as it is still standard JMS, it will just work faster in V7 in most cases 
_________________
Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime. |
|
| Back to top |
|
|
| steffi2 |
| Posted: Thu Sep 10, 2009 5:00 am Post subject: |
|
|
Apprentice
Joined: 04 Sep 2009
Posts: 31
|
Are mulitlple listeners attached the same QueueSession independently concurrent and not invoked serially in V7?
Also, in MQ/JMS is there anyway you can acknowledge the message and only that message and not all prior unacknowledged messages?
When I want to dispatch my message to multiple worker threads I ideally don't want to acknowledge the message until I know the work is finished and right now there's a flaw with my use of CLIENT_ACKNOWLEDGE because I queue up the messages to be acknowledged at the end of onMessage. ie. when a worker thread is finished it puts that message in a "queue" LinkedBlockingQueue which is then picked up by the delivery thread and acknowledged but this isn't reliable because when you call acknowledge on a a message you're not only acknwoeldging that message but all prior unacknowledged messages and in my case some of those messages might not be finished yet.
In JMS I'm never allowed to acknowledge a message outside of the delivery thread and I am looking for a way I can be guaranteed that before I acknowledge a message I now that I've fully processed it because I need to be able to recover from a restart and I can live with processing a duplicate but I cannot live with a message that was preivously acknowledge but not fully processed.
The only way I seem to be able to do this reliably is to maintain the entire state of my message in a working table and only when I know I've completed the processing the message can I delete or mark that entry deleted. This adds considerably overhead both my delivery thread and my worker threads as the delivery thread has to capture the state of the message before acknowledgement and the worker thread has the mark the state deleted upon completion.
I do not want to use synchronous APIs where I can consumer and acknowledge in the same thread. |
|
| Back to top |
|
|
| steffi2 |
| Posted: Thu Sep 10, 2009 5:21 am Post subject: |
|
|
Apprentice
Joined: 04 Sep 2009
Posts: 31
|
How do you get ID:0 or ID:1 in the correlationId? A queue publisher is a piece of C++ code and right now we're getting the large ID:x........... when attempting to write just a 0 or 1 there.
Got the selector working...
Is there anyway in MQ's java API that I can acknowledge a message from a different thread and not be forced to acknowledge the message in the delivery thread only?
| fjb_saper wrote: |
With MQ V 6x the best performance with selectors is obtained using the native provider form (ID:hexbinary value) and restricting the selection to some fields like
msg id
correl id
group id
etc... Enjoy |
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 10, 2009 3:26 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| steffi2 wrote: |
Also, in MQ/JMS is there anyway you can acknowledge the message and only that message and not all prior unacknowledged messages?
When I want to dispatch my message to multiple worker threads I ideally don't want to acknowledge the message until I know the work is finished and right now there's a flaw with my use of CLIENT_ACKNOWLEDGE because I queue up the messages to be acknowledged at the end of onMessage. ie. when a worker thread is finished it puts that message in a "queue" LinkedBlockingQueue which is then picked up by the delivery thread and acknowledged but this isn't reliable because when you call acknowledge on a a message you're not only acknwoeldging that message but all prior unacknowledged messages and in my case some of those messages might not be finished yet.
In JMS I'm never allowed to acknowledge a message outside of the delivery thread and I am looking for a way I can be guaranteed that before I acknowledge a message I now that I've fully processed it because I need to be able to recover from a restart and I can live with processing a duplicate but I cannot live with a message that was preivously acknowledge but not fully processed.
The only way I seem to be able to do this reliably is to maintain the entire state of my message in a working table and only when I know I've completed the processing the message can I delete or mark that entry deleted. This adds considerably overhead both my delivery thread and my worker threads as the delivery thread has to capture the state of the message before acknowledgement and the worker thread has the mark the state deleted upon completion.
I do not want to use synchronous APIs where I can consumer and acknowledge in the same thread. |
You need to follow the J2EE pattern.
Each Thread reading from the queue has it's own connection and session.
If you need to scale, the new thread needs to create it's own connection and session. Thus each onMessage call is handled by a single session. So when you commit the session only that message is committed. The thread that reads the message handles the message.
In J2EE you'd typically set a requiresNew as transaction handling on the onMessage method. As stand alone JMS you'll have to fake it.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| steffi2 |
| Posted: Thu Sep 10, 2009 3:56 pm Post subject: |
|
|
Apprentice
Joined: 04 Sep 2009
Posts: 31
|
| Am I forced to handle the message in the Thread that received the message? I already have two async threads but each async thread submits the work to a pool of worker threads and that's where things become complicated because I don't necessarily know I've finished the work so committing the session is not possible from my task thread. But if complete the work in the thread that takes delivery I then have to use selectors which apparently come with a performance cost in MQ 6.x which is what we are using. I have to use selectors because I then need to have message listener per thread per session and that requires selectors to define the work for each consuming thread. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 10, 2009 4:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| steffi2 wrote: |
| Am I forced to handle the message in the Thread that received the message? I already have two async threads but each async thread submits the work to a pool of worker threads and that's where things become complicated because I don't necessarily know I've finished the work so committing the session is not possible from my task thread. But if complete the work in the thread that takes delivery I then have to use selectors which apparently come with a performance cost in MQ 6.x which is what we are using. I have to use selectors because I then need to have message listener per thread per session and that requires selectors to define the work for each consuming thread. |
No, you are going after the wrong pattern.
If the worker threads execute different things in parallel, they could all have a shared dummy object with keys and set their respective keys to complete when done.
So your main thread has to wait until all keys are done to commit the session. Note that the worker thread should also have the possibility to notify the main thread to set the rollback only flag on the session.
If the work is not parallel threaded but just handed off to a worker pool you went after the wrong pattern. You should retrieve the connection from a connection pool and execute the work on the thread that received the message.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
