| Author |
Message
|
| rxm8778 |
 Posted: Wed Aug 20, 2008 6:56 am Post subject: How does MQ handle mgs with mutiple consumer applications? |
 |
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
I have been asked the following question today and I realized that I could not answer it with certainty. With that, I am hoping you all can help me clear this MQ scenario:
I have a queue from which 7 or more consumer applications (non MDB) are reading.
The assumption is that the design requires that each consumer application resides on a seperate workstation and that they can only read 1 message at the time; Once they pick up 1 message, they have to complete the rest of the transaction (which involves parsing the data, inserting it in a database and other tasks) before grabbing the next message.
To my knowledge there are two ways to process the message from this queue:
1) We can use triggering, where it is the responsibility of the queue manager to let the applications know that messages have arrived to the queue.
2) We can use get with wait option and have all applications indefinitely reading a queue waiting for the arrival of messages.
The problem I see with the first scenario is that the queue can be triggered to be read by only 1 application (there is no such thing as a trigger list)....So that would not work!
For the second scenario if all 7 applications issue a get with wait on that queue, what is the mechanism in MQ that prevents application 2 and application 7 from reading the exact same message at the exact same time?
I know MQ prevents the duplication of messages but when asked the above question by the business partner, I realized that I did not have a definite answer.
Can you all enlighten me?
Also if you think of a better option than triggering or get with wait to address this scenario, please share it with me
thanks.
Last edited by rxm8778 on Wed Aug 20, 2008 10:55 am; edited 1 time in total |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Wed Aug 20, 2008 8:33 am Post subject: Re: How does MQ handle mgs with mutiple consumer application |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| rxm8778 wrote: |
The problem I see with the first scenario is that I queue can be triggered to be read by only 1 application (there is no such thing as a trigger list)....So that would not work!
|
Each of the 7 workstations can have a trigger monitor watching the same init queue.
| rxm8778 wrote: |
For the second scenario if all 7 applications issue a get with wait on that queue, what is the mechanism in MQ that prevents application 2 and application 7 from reading the exact same message at the exact same time? |
Don't worry about. IBM made sure this can't happen.
Scenario B is the way to go.
In either scenario there is no guarantee that the work will be distributed evenly betweent the 7.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Wed Aug 20, 2008 11:58 am Post subject: Re: How does MQ handle mgs with mutiple consumer application |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
| PeterPotkay wrote: |
Don't worry about. IBM made sure this can't happen.
Scenario B is the way to go.
In either scenario there is no guarantee that the work will be distributed evenly betweent the 7. |
Point taken Peter, scenario B is what I pointed them to. And I will certainly re-iterate that they should not be bugged down with the details on how MQ prevents duplicate reads. They should just trust  that two applications won't be able to grab the exact same message at the exact same time. (not always easy  to convey this message to non-believers without somewhat of a statement such as "well MQ puts a lock on the message" or some high level explanation of the sort but I will try)
thanks again |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Aug 20, 2008 12:18 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If they don't believe you, tell them to prove it!
Don't forget 2 or more apps can browse (read) the same message. But for destructive reads, MQ insures only 1 app wins that race.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Wed Aug 20, 2008 12:27 pm Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
...I will probably use this approach more often with business partners: "if you can't prove me wrong you just have to blindly trust me" 
Peter I thought some more about your suggestion:
| PeterPotkay wrote: |
Each of the 7 workstations can have a trigger monitor watching the same init queue.
|
I wanted to ensure I understood your proposal because it does not seem feasible to me.
See the example below from the sys admin manual:
----------------
DEFINE QLOCAL (MOTOR.INSURANCE.QUEUE) + PROCESS (MOTOR.INSURANCE.QUOTE.PROCESS) + MAXMSGL (2000) + DEFPSIST (YES) + INITQ (MOTOR.INS.INIT.QUEUE) + TRIGGER + TRIGTYPE (DEPTH) + TRIGDPTH (100)+ TRIGMPRI (5)
DEFINE QLOCAL(MOTOR.INS.INIT.QUEUE) + GET (ENABLED) + NOSHARE + NOTRIGGER + MAXMSGL (2000) + MAXDEPTH (1000)
DEFINE PROCESS (MOTOR.INSURANCE.QUOTE.PROCESS) + DESCR (’Insurance request message processing’) + APPLTYPE (UNIX) + APPLICID (’/u/admin/test/IRMP01’) + USERDATA (’open, close, 235’)
runmqtmc -m QMgrName -q MOTOR.INS.INIT.QUEUE
---------------------
According to the example above from the system admin manual, even if I start 7 trigger monitors), all the trigger messages will have information to trigger just one of the 7 applications: (/u/admin/test/IRMP01 as defined above). So there is no way to trigger more than one application from 1 queue. Above, the queue: MOTOR.INSURANCE.QUEUE is only associated to one process: MOTOR.INSURANCE.QUOTE.PROCESS not seven of them.
Am I missing something here...?
How would starting 7 trigger monitors trigger 7 different application to read 1 and only 1 queue?
Thanks |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Aug 20, 2008 2:09 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
1000 triggered app queues can specify the same INITQ. The trigger message that the QM places in the INITQ contains details about which q the triggered app should open.
You can connect 1000 trigger monitors to the same INITQ. The QM will only create 1 trigger messages for a trigger event. It then becomes a race condition. One and only one of this trigger monitors will pick up that one trigger message. Which one is nothing you can rely on. When you have 2 or more apps (a trigger monitor is just an MQ app you know) pulling from a queue, there is no documentation or guarantees as to the distribution of who gets the next message.
One trigger monitor can start 1000 different apps for 1000 different queues, but the typical trigger monitor only starts apps on the server that the trigger monitor actually executes on. If you have 1000 different client servers that each need triggering, then each one needs its own local copy of a client trigger monitor.
If the clients are Windows and you use triggering, insist on using the MA7K Support Pack.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Aug 20, 2008 2:37 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| MQ puts a lock on the message |
It does.
WMQ conforms to the ACID-test rules. WMQ is XA-compliant.
Atomicity - Either all of the changes occur or none of them do.
Consistency - Transactions always leave data in a consistent state.
Isolation - A transaction should appear as though it is running
all by itself, and the effects of this transaction are invisible to others until the transaction is committed.
Durability - Once a transaction is committed, its effects are
guaranteed to persist, even in the event of subsequent system failures.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Aug 20, 2008 4:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PeterPotkay wrote: |
1000 triggered app queues can specify the same INITQ. The trigger message that the QM places in the INITQ contains details about which q the triggered app should open.
You can connect 1000 trigger monitors to the same INITQ. The QM will only create 1 trigger messages for a trigger event. It then becomes a race condition. One and only one of this trigger monitors will pick up that one trigger message. Which one is nothing you can rely on. When you have 2 or more apps (a trigger monitor is just an MQ app you know) pulling from a queue, there is no documentation or guarantees as to the distribution of who gets the next message.
One trigger monitor can start 1000 different apps for 1000 different queues, but the typical trigger monitor only starts apps on the server that the trigger monitor actually executes on. If you have 1000 different client servers that each need triggering, then each one needs its own local copy of a client trigger monitor.
If the clients are Windows and you use triggering, insist on using the MA7K Support Pack. |
Peter wouldn't in this case a setup where each of the clients has a MessageListener running on the queue (JMS/XMS) guarantee a better distribution of the load?
It is still first come, first served, but the accent is not put on the initq and the app might be more scalable by running multiple instances from each server...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Aug 20, 2008 5:41 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Yes, up above I said I would not go with triggering as my first choice.
Still, with multiple consumers, the distribution will be uneven, unless the messages arrive faster than all the consumers can keep up with. If queuing occurs, then you can be fairly certain the workload will be relatively even, assuming each message takes about the same time to process regardless of message or consumer instance.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| rxm8778 |
| Posted: Thu Aug 21, 2008 4:33 am Post subject: |
|
|
Apprentice
Joined: 15 Sep 2005
Posts: 37
|
| PeterPotkay wrote: |
| 1000 triggered app queues can specify the same INITQ. The trigger message that the QM places in the INITQ contains details about which q the triggered app should open. |
In my case the scenario was more like: can 1 triggered app queue (just one queue) trigger 7 seperate applications? I did not see how that was possible when the relationship between application queue and process definition was 1 to 1. That was the clarification I was seeking in my previous post (because for a moment there Peter I thought you were saying it was possible.
But I do understand that multiple application queues can use the same INIT queue. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 21, 2008 4:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| You can reexamine what the application that is being triggered does. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 21, 2008 4:42 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| can 1 triggered app queue (just one queue) trigger 7 seperate applications? |
Are you asking if one triggered application queue can cause the same application to be launched for each message?
Trigger type EVERY will accomplish this.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Thu Aug 21, 2008 4:56 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
| bruce2359 wrote: |
| Quote: |
| can 1 triggered app queue (just one queue) trigger 7 seperate applications? |
Are you asking if one triggered application queue can cause the same application to be launched for each message?
Trigger type EVERY will accomplish this. |
Trigger type Every will not be a good idea with respect to performance level.
think it will keep on spawning new process for each and every message.
guess if there are 1 Lakh messages.... 
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Aug 21, 2008 5:09 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Gaya3 wrote: |
| ...guess if there are 1 Lakh messages.... |
I had to Google this! 1 lakh = 100,000
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Aug 21, 2008 5:12 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| Trigger type Every will not be a good idea with respect to performance level. |
I disagree. Trigger type EVERY will (likely) require more hardware resources to meet the demand. But if the service level agreement calls for more concurrency, EVERY is the solution.
| Quote: |
| ...think it will keep on spawning new process for each and every message. |
Not really. If the consuming application is written to MQGET with WAIT, then the first instance of the application will likely have finished processing the first message when the n-th message arrives. So there will not be a one-to-one relationship between messages and instances of the consuming program. This can be calculated.
If the server is under provisioned (needs more cpus, real storage, disk, ...), buy more as a solution to the business requirement. Havent we learned (over and over) not to design our business systems applications around (existing) hardware?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
