| Author |
Message
|
| androidgalaxyman |
 Posted: Wed Feb 05, 2014 12:18 am Post subject: Multiple MQ Consumers accessing same queue on IBM MQ Series |
 |
|
Novice
Joined: 02 Jan 2014
Posts: 13
|
Dear All
i am having the problem regarding of duplicating.Assume multiple consumers accessing same Websphere MQ Queue "testQ". There is a Problem i could see , one consumer take the message which is already been taken another consumer . The Consumer are running different machines, only they are accessing the same queue name. Please help me!
SERVER 1: [consumer 1]
2014-01-28 12:09:52,399 [MY_Thread-25] DEBUG XXX - Set Connection as Unused.
2014-01-28 12:09:52,399 [MY_Thread-25] DEBUG XXX - Waiting for Request...
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Ensure Listener being locked.
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Processing request...
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Parsing OFSML Request ...
2014-01-28 12:09:52,404 [MY_Thread-25] DEBUG XXX - Parsing Done. SEC.TRADE,UBSVX/I/PROCESS,/********,P014012801167801,
SERVER 2: [Consumer 2]
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Notifying caller.
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Notification done.
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Set Connection as Unused.
2014-01-28 12:09:52,402 [MY_Thread-25] DEBUG XXX - Waiting for Request...
2014-01-28 12:09:52,404 [MY_Thread-25] DEBUG XXX - Ensure Listener being locked.
2014-01-28 12:09:52,404 [MY_Thread-25] DEBUG XXX - Processing request...
2014-01-28 12:09:52,404 [MY_Thread-25] DEBUG XXX - Parsing OFSML Request ...
2014-01-28 12:09:52,406 [MY_Thread-25] DEBUG XXX - Parsing Done. SEC.TRADE,UBSVX/I/PROCESS,/********,P014012801167801, |
|
| Back to top |
|
 |
| exerk |
| Posted: Wed Feb 05, 2014 12:25 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Pub/Sub
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Feb 05, 2014 1:03 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
MQ will not allow more than one application to consume the same message. Once and once only delivery is a primary design feature.
Unless you choose to use browse mode - which is generally intended for ulitities and not for applications.
It is perfectly safe to allow (and it's very good for resilience and load balancing) multiple consumers to access the same queue at the same time.
Browsing is a different and more complex area and has options like lock message under cursor - but there is rarely a need to use browse.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 05, 2014 6:35 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I'll have to agree with Exerk 
You seem to have the same message (content) present on 2 instances of the destination (whether clustered or not).
I would suspect that you have a pub/sub model and both consumers are subscribed to the same feed.
If you wanted a "load/balanced" subscription, you would need to do it in a cluster and provide a cluster alias as the subscribing qmgr name...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Feb 05, 2014 6:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| It's possible that one consumer has attempted to process the message, failed, and rolled it back. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Feb 05, 2014 6:57 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
It's not clear if the problem is seeing duplicates but not wanting them, or wanting duplicates but not getting them. Bringing Pub/Sub and clustering into the conversation can either be right or wrong, but first we need clarification on what the actual problem androidgalaxyman has.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Feb 05, 2014 7:04 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Sorry I am getting confused here...
Are we talking about accessing the same queue on different servers i.e. a different instance of the same queue, or multiple consumers accessing the same instance of the queue on the same server (in which case I wouldn't know if it is a rollback scenario or multiple subscriber scenario)...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Feb 05, 2014 7:20 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Speculation on my part, but breaking it down:
i am having the problem regarding of duplicating. Assume multiple consumers accessing same Websphere MQ Queue "testQ". - implies multiple instances of an application.
There is a Problem i could see , one consumer take the message which is already been taken another consumer . - could be that the 'consuming' application does a browse of a message, otherwise there's no way (other than a roll-back) it can happen.
But, as Peter so succinctly points out, we need the OP to clarify.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Feb 05, 2014 7:29 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Not wanting to over-complicate my answer, I ignored syncpointing.
If you get a message under syncpoint (transactionally) and you don't commit it - when either the app or when MQ backs out the transaction - then it will re-appear on the queue.
In this case the same message can be consumed more than once - but is easily detected by the application inspecting the MQMD.back value. This is incremented for each backout of the message.
So if the MQMD.back > 0 then this message has been previously given to an application but has been backed out to the queue. The action your app takes in this situation is up to you. If it is safe to re-process the message then do so, otherwise you could move it to another queue for manual intervention.
Use of two-phase commit can ensure consistency between a message transaction and a database transaction for example, ensuring that both are committed or rolled back together.
All this sort of stuff is done much better in MQ compared to the situation with web-services or (heaven forbid) file record processing.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| androidgalaxyman |
| Posted: Wed Feb 05, 2014 7:15 pm Post subject: |
|
|
Novice
Joined: 02 Jan 2014
Posts: 13
|
| zpat wrote: |
MQ will not allow more than one application to consume the same message. Once and once only delivery is a primary design feature.
Unless you choose to use browse mode - which is generally intended for ulitities and not for applications.
It is perfectly safe to allow (and it's very good for resilience and load balancing) multiple consumers to access the same queue at the same time.
Browsing is a different and more complex area and has options like lock message under cursor - but there is rarely a need to use browse. |
.
Thanks Mr. Zpat, i agree with your view. But my traces you could see , it seems there may be some chance. i need to pull my hair!  |
|
| Back to top |
|
|
| androidgalaxyman |
| Posted: Wed Feb 05, 2014 7:19 pm Post subject: |
|
|
Novice
Joined: 02 Jan 2014
Posts: 13
|
| fjb_saper wrote: |
I'll have to agree with Exerk
You seem to have the same message (content) present on 2 instances of the destination (whether clustered or not).
I would suspect that you have a pub/sub model and both consumers are subscribed to the same feed.
If you wanted a "load/balanced" subscription, you would need to do it in a cluster and provide a cluster alias as the subscribing qmgr name...
Have fun |
Hi @fjb_saper, Assume my case would be my application running in two different machine . Its an Listener which listens the same queue in a single instance of MQ which is running in one of the two machines. This is no implementation on cluster concept. Please let me know, if you need anything.  |
|
| Back to top |
|
|
| androidgalaxyman |
| Posted: Wed Feb 05, 2014 7:23 pm Post subject: |
|
|
Novice
Joined: 02 Jan 2014
Posts: 13
|
| PeterPotkay wrote: |
| It's not clear if the problem is seeing duplicates but not wanting them, or wanting duplicates but not getting them. Bringing Pub/Sub and clustering into the conversation can either be right or wrong, but first we need clarification on what the actual problem androidgalaxyman has. |
Dear @Peter , the actual problem is a different instance of Listening client application which is on different machine while accessing the same queue (which is actually running inside any of the machine) , the message consumed by both application. The MQ allowed to consume that message two times.  |
|
| Back to top |
|
|
| androidgalaxyman |
| Posted: Wed Feb 05, 2014 7:24 pm Post subject: |
|
|
Novice
Joined: 02 Jan 2014
Posts: 13
|
| exerk wrote: |
Speculation on my part, but breaking it down:
i am having the problem regarding of duplicating. Assume multiple consumers accessing same Websphere MQ Queue "testQ". - implies multiple instances of an application.
There is a Problem i could see , one consumer take the message which is already been taken another consumer . - could be that the 'consuming' application does a browse of a message, otherwise there's no way (other than a roll-back) it can happen.
But, as Peter so succinctly points out, we need the OP to clarify. |
Hi @exerk , your correct.two different instances of application which listens on a same queue , results this behavior.  |
|
| Back to top |
|
|
| androidgalaxyman |
| Posted: Wed Feb 05, 2014 7:41 pm Post subject: |
|
|
Novice
Joined: 02 Jan 2014
Posts: 13
|
| zpat wrote: |
Not wanting to over-complicate my answer, I ignored syncpointing.
If you get a message under syncpoint (transactionally) and you don't commit it - when either the app or when MQ backs out the transaction - then it will re-appear on the queue.
In this case the same message can be consumed more than once - but is easily detected by the application inspecting the MQMD.back value. This is incremented for each backout of the message.
So if the MQMD.back > 0 then this message has been previously given to an application but has been backed out to the queue. The action your app takes in this situation is up to you. If it is safe to re-process the message then do so, otherwise you could move it to another queue for manual intervention.
Use of two-phase commit can ensure consistency between a message transaction and a database transaction for example, ensuring that both are committed or rolled back together.
All this sort of stuff is done much better in MQ compared to the situation with web-services or (heaven forbid) file record processing. |
HI @zpat, thanks for your input. my code like this
| Code: |
MQGetMessageOptions gmo = new MQGetMessageOptions();
gmo.options = MQC.MQGMO_SYNCPOINT | MQC.MQGMO_WAIT;
gmo.waitInterval = this._listener.getWaitInterval();
/*
* Let's twist !
*/
while (!this._bStopping) { // && !bReadyToLeave){
MQMessage retrievedMessage = new MQMessage();
retrievedMessage.characterSet = MQC.MQCCSI_Q_MGR;
/*
* try{ _mqm.begin(); }catch(Exception e){ e.printStackTrace(); }
*/
/*
* Get the Message from the queue
*/
boolean bDone = false;
while (!this._bStopping && !bDone) {
try {
retrievedMessage.clearMessage();
} catch (IOException e1) {
this._listener.error("Error while clearing message.", e1);
}
long l = System.currentTimeMillis();
try {
this._listener.debug(new StringBuilder(" Retrieving message ...").append(_listener.getName()).toString());
this._inputQueue.get(retrievedMessage, gmo);
this._listener.debug(new StringBuilder(" Retrieving message ...").append(_listener.getName()).toString());
bDone = true;
} catch (MQException mqe) {
/*
* This is due to the connection getting suddenly down !
* This mean we have to raise it higher !
*/
if ((System.currentTimeMillis() - l) < 50) {
this._listener
.error("Error while reading message. Reason : "
+ mqe.reasonCode);
closeAll();
/*
* Retry every minutes.
*/
Initialize(60000);
}
/* Normal Error when no message - timeout */
} |
Are u saying the when under SYNCPOINT and don't commit - when either the app or when MQ backs out the transaction - then it will re-appear on the queue. ? can you please enlight little much?  |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Feb 05, 2014 10:27 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| androidgalaxyman wrote: |
| ...Are u saying the when under SYNCPOINT and don't commit - when either the app or when MQ backs out the transaction - then it will re-appear on the queue. ? |
When you get a message under SYNCPOINT the message is not available for other consumers to GET and unless you COMMIT the GET, i.e. when you have finished processing the message content, the message will not be deleted from the queue. If your application backs out the message, either explicitly or implicitly (e.g. broken connection) only then will it be available to another consumer to GET.
The above assumes your application doesn't do anything silly such as process the message content, act on it, and commit the content but not the message, e.g. GET, PROCESS, UPDATE DB, ROLL-BACK MESSAGE, and nowhere in your code do I see a COMMIT, explicit or otherwise.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
