| Author |
Message
|
| nickt |
 Posted: Sun Apr 27, 2008 8:03 pm Post subject: Using backout queues and message selectors |
 |
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
Hi All,
I've noticed some curious behaviour using MQ 6.0 as a JMS provider
with WebSphere 6.1.
We're attempting to implement a retry-queue for messages that get
rolled back on their original queue which is hooked up to a message
driven bean. The desired effect is that a message comes in on the
original queue, gets rolled back perhaps due to an outage or load-
related issues or something so it gets requeued to the backout-retry
queue (the backout threshold on the original queue is set to 1).
This happens fine, and the message ends up on the retry queue in the
event of a rollback on the message driven bean. I now have a timed
bean that attempts to read messages from the retry queue and retry
them at defined retry intervals so I use message selectors like so:
message selector 1: "JMSXDeliveryCount = 1 AND JMSTimestamp < [current
time - first retry interval]"
message selector 2: "JMSXDeliveryCount = 2 AND JMSTimestamp < [current
time - second retry interval]"
message selector 3: "JMSXDeliveryCount = 3 AND JMSTimestamp < [current
time - third retry interval]"
Which works fine. I have set the backout threshold on the retry queue
to 3 so that if the third retry attempt results in a rollback the
message should finally be placed onto an error queue, never to be
retried again.
The problem I've encountered is that when the final retry attempt
fails, the message isn't being placed onto the error queue. The
message is effectively invisible to any QueueReceiver that has a
message selector (even if the message selector is just "TRUE" which
should match all messages) but it doesn't get put onto the backout-
requeue-queue.
It's not until I subsequently attempt to read from the queue with a
QueueReceiver that has no message selector that all the messages with
backout threshold exceeded get put onto the error queue.
Has anyone encountered this before? is there a way to get the message
to get requeued without attempting a read from the queue with no
message selector (as this would result in a message actually being
read and breaking the logic?)
Any help from someone who's seen this problem before would be greatly
appreciated, I'm not certain as to whether it's a bug in the IBM
libraries or if that's how it's supposed to behave.
Regards,
Nick Tindall |
|
| Back to top |
|
 |
| mvic |
| Posted: Mon Apr 28, 2008 5:27 am Post subject: Re: Using backout queues and message selectors |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| The way you have described this makes it sound like the message selector is ceasing to work when the backoutcount reaches a certain number. If that's a correct statement then I think you have reason to call IBM support, and ask for this to be investigated. If you can go to them with an MQ JMS trace that captures the problem happening then this will make things much quicker. |
|
| Back to top |
|
|
| nickt |
| Posted: Tue Apr 29, 2008 10:08 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| Thanks mvic... I'll report it and post back here when I know the answer |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Apr 30, 2008 6:54 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
looks to me as running as expected.
Drop the selector entirely. JMS would handle this behind the scene.
The rule: bothresh = X
MDB retry > X.
If MDB retry <= X the mdb will stop work when encountering a poison message.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nickt |
| Posted: Thu May 01, 2008 6:26 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| Can't drop the message selector... it's kinda a fundamental part of the logic... read the original message. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 02, 2008 1:36 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| nickt wrote: |
| Can't drop the message selector... it's kinda a fundamental part of the logic... read the original message. |
What I'm trying to tell you is that this is the standard behavior for an MDB in JMS. You can set the retry interval and you can set the retry times...
Setting different retry interval only makes sense if you are single threaded...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nickt |
| Posted: Mon May 05, 2008 4:02 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| fjb_saper wrote: |
| nickt wrote: |
| Can't drop the message selector... it's kinda a fundamental part of the logic... read the original message. |
What I'm trying to tell you is that this is the standard behavior for an MDB in JMS. You can set the retry interval and you can set the retry times...
Setting different retry interval only makes sense if you are single threaded...
Enjoy |
Thanks for that, unfortunately this queue isn't message driven, and I'd like to be able to process the messages on it out of order of arrival and at intervals like:
1st retry: 1 minute
2nd retry: 1 hour
3rd retry: 2 hours
What you have described above would require the messages to be processed in order and at regular intervals.
I've submitted the problem to IBM and they are taking a look at it. Will post the results..
Cheers,
Nick |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 05, 2008 8:25 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Sorry but I have a large concept problem here. Help me clarify it:
What is so important that if the first retry fails you need to wait one hour, the second 2 hours?
I would basically work here with multiple queues.
Have an MDB driven input queue with mutliple instances. You may reprocess the message up to 3 times in a very short time. If the resource is not available put to backout queue.
Run a scheduled process against the backout queue. If the process failed put message to backout queue 2 (2 hours).
Run a scheduled process against backout queue 2....
The problem with your design is that it is not scalable. The humongous delay forces you out of the most common scaling models...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nickt |
| Posted: Tue May 06, 2008 5:26 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| fjb_saper wrote: |
Sorry but I have a large concept problem here. Help me clarify it:
What is so important that if the first retry fails you need to wait one hour, the second 2 hours?
I would basically work here with multiple queues.
Have an MDB driven input queue with mutliple instances. You may reprocess the message up to 3 times in a very short time. If the resource is not available put to backout queue.
Run a scheduled process against the backout queue. If the process failed put message to backout queue 2 (2 hours).
Run a scheduled process against backout queue 2....
The problem with your design is that it is not scalable. The humongous delay forces you out of the most common scaling models...
Enjoy |
Some things are important my friend. You don't know where I work, you don't know what I'm working on.
We are doing what you describe... there's an initial queue, that has a backout queue and backout threshold of 1, there's a scheduled process running on the backout queue which has a backout threshold of 3, the scheduled process pulls messages out of the backout queue that have been there for 1 minute (the first time) 1 hour (the second time) and 2 hours (the third time) then if the third retry fails, the message goes to another backout queue where it is just left there for possible investigation later.
My concept problem with what you're saying is: Having 3 backout queues for every MDB queue is somehow better?!?! Not to mention three separate scheduled tasks each reading from these queues? you're on crack...
What I want to do is totally achievable (and scalable) using message selectors. Don't worry about it hey... I'm talking to IBM now. |
|
| Back to top |
|
|
| nickt |
| Posted: Tue May 06, 2008 5:28 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| nickt wrote: |
| fjb_saper wrote: |
Sorry but I have a large concept problem here. Help me clarify it:
What is so important that if the first retry fails you need to wait one hour, the second 2 hours?
I would basically work here with multiple queues.
Have an MDB driven input queue with mutliple instances. You may reprocess the message up to 3 times in a very short time. If the resource is not available put to backout queue.
Run a scheduled process against the backout queue. If the process failed put message to backout queue 2 (2 hours).
Run a scheduled process against backout queue 2....
The problem with your design is that it is not scalable. The humongous delay forces you out of the most common scaling models...
Enjoy |
Some things are important my friend. You don't know where I work, you don't know what I'm working on.
We are doing what you describe... there's an initial queue, that has a backout queue and backout threshold of 1, there's a scheduled process running on the backout queue which has a backout threshold of 3, the scheduled process pulls messages out of the backout queue that have been there for 1 minute (the first time) 1 hour (the second time) and 2 hours (the third time) then if the third retry fails, the message goes to another backout queue where it is just left there for possible investigation later.
My concept problem with what you're saying is: Having 3 backout queues for every MDB queue is somehow better?!?! Not to mention three separate scheduled tasks each reading from these queues? you're on crack...
What I want to do is totally achievable (and scalable) using message selectors. Don't worry about it hey... I'm talking to IBM now. |
PS. We're reading the messages out of order using the message selectors, I don't know what humungous delays you're talking about. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 07, 2008 1:26 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nickt wrote: |
| Some things are important my friend. You don't know where I work, you don't know what I'm working on. |
No, but we do know how the product works best.
| nickt wrote: |
| What I want to do is totally achievable (and scalable) using message selectors. Don't worry about it hey... I'm talking to IBM now. |
I admire your confidence, and please post your solution for the benefit of future readers. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed May 07, 2008 10:55 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| nickt wrote: |
| We're reading the messages out of order using the message selectors, I don't know what humungous delays you're talking about. |
When using JMS selectors the perfomance goes down and the CPU cycles used goes up very fast the deeper the queue gets. Do test your design by putting a thousand message in the backout queue and watch what happens. If you select on just MsgId and/or CorrelID you can avoid this if coded correctly.
-Peter (crack free, just like F.J.)
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| nickt |
| Posted: Wed May 07, 2008 3:02 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| PeterPotkay wrote: |
| nickt wrote: |
| We're reading the messages out of order using the message selectors, I don't know what humungous delays you're talking about. |
When using JMS selectors the perfomance goes down and the CPU cycles used goes up very fast the deeper the queue gets. Do test your design by putting a thousand message in the backout queue and watch what happens. If you select on just MsgId and/or CorrelID you can avoid this if coded correctly.
-Peter (crack free, just like F.J.) |
Thanks Peter,
so message selectors aren't really scalable in MQ... unless you're reading with MsgId or CorrelID... Interesting... We haven't had the chance to do any performance testing as we're still waiting for IBM to get back to us about the bug.. I had thought of doing this another way; upon the first rollback creating an EJB timer that when it expires (after the retry interval), reads the message off the queue and attempts to re-process it (would be done with MsgId) but that still leaves the same problem that I asked about in the first place, ultimately a queue that is only ever read from using message selectors will never requeue messages that have passed the backout threshold. Additionally, it would require an EJB timer running for every message on every retry queue that still had a pending retry. Sounds OK(?) in theory, but if MQ can't do message selection in a timely fashion with as few as 1000 messages then who knows how WAS will handle having 1000 timers running.
In any case, thanks for the information, I'm sure it will be invaluable when performance testing.
PS. You say that MsgId and CorrelationId work OK in message selectors, is there any way to tell a queue to index other header fields?
Cheers,
Nick |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed May 07, 2008 6:03 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| nickt wrote: |
You say that MsgId and CorrelationId work OK in message selectors, is there any way to tell a queue to index other header fields?
|
Only on z/OS can you opt to tell the Queue Manager to index on Message ID, Correl ID, Group ID or Message Token (this last one is a special case for WLM only). On the distributed platforms this indexing happens automatically and without option. On distributed the indexing is only done for Msg ID and Correl ID. Not sure if distributed indexes by Group ID as well.
The 1000 number I used was just a # out of the air. Your milage will vary. In some case we see CPU spike and response times drop as soon as there are only several dozen messages for the selectors to deal with. Message size has an effect as bigger messages take longer to go back and forth over the wire for all those browses.
In MQ 7.0 you can use selctors for the new MQ Message Properties. While there is no indexing done for this the selection is done at the QM side so the performance will be better than JMS using its selectors but still very slow comapred to Msg or Correl ID selective gets.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| nickt |
| Posted: Wed May 07, 2008 7:55 pm Post subject: |
|
|
Novice
Joined: 27 Apr 2008
Posts: 11
|
| PeterPotkay wrote: |
| nickt wrote: |
You say that MsgId and CorrelationId work OK in message selectors, is there any way to tell a queue to index other header fields?
|
Only on z/OS can you opt to tell the Queue Manager to index on Message ID, Correl ID, Group ID or Message Token (this last one is a special case for WLM only). On the distributed platforms this indexing happens automatically and without option. On distributed the indexing is only done for Msg ID and Correl ID. Not sure if distributed indexes by Group ID as well.
The 1000 number I used was just a # out of the air. Your milage will vary. In some case we see CPU spike and response times drop as soon as there are only several dozen messages for the selectors to deal with. Message size has an effect as bigger messages take longer to go back and forth over the wire for all those browses.
In MQ 7.0 you can use selctors for the new MQ Message Properties. While there is no indexing done for this the selection is done at the QM side so the performance will be better than JMS using its selectors but still very slow comapred to Msg or Correl ID selective gets. |
Thanks again Peter,
Given your input I'm working now on implementing this in a way that plucks messages out by MsgId at exactly the time they are due to be processed using the EJB timer service to schedule the retries.
This isn't the forum to ask this in obviously, but does anyone know if there will be any problem with creating potentially thousands of timers in WAS? I've done a little looking around but can't seem to find anything that says it would be a problem.
In the event of an outage that prevented messages being processed immediately the retry queues could easily swell to thousands of messages (the application is quite busy) and this design would see one EJB timer created for every message on the retry queue. It sounds dubious to me, because I don't have a lot of experience using the EJB timer service and nobody else seems to use it in such a way. Perhaps some trial and error is in order.
Cheers,
Nick |
|
| Back to top |
|
|
|
|
