| Author |
Message
|
| iw.kuchin |
 Posted: Thu Dec 13, 2012 4:06 am Post subject: Browsing queue and deleting messages in one connection |
 |
|
Newbie
Joined: 13 Dec 2012
Posts: 3
|
We are planning to integrate our application with existing WebSphere MQ architecture. The functionality should be rather simple: application has one incoming queue + multiple processor threads (workers). We want to take messages from queue, process them and after that remove messages from queue. Messages should be removed from queue only after all the processing is done to ensure reliability in case of the application/server crashes. So algorithm looks like this:
1. Take first "untaken message"
2. Mark is as "taken"
3. Send it to worker thread
4. After message is fully processed remove processed message
I believe I can peek message while not removing it with "browse" option. But is it possible to remove message in same connection? If so, how can I navigate in queue to message-to-remove in some way? Main problem is that access is not sequential: I can pass messages to workers FIFO way, but processing times could differ significantly, so the need may arise to remove 3rd message before 1st.
Example: 4 worker threads, messages in queue:
| Code: |
Queue: F - E - D - C - B - A (HEAD)
Worker 1: no work
Worker 2: no work
Worker 3: no work
Worker 4: no work
|
After peeking and passing messages to workers:
| Code: |
Queue: F - E - D (HEAD) - C - B - A
Worker 1: A
Worker 2: B
Worker 3: C
Worker 4: D
|
And after some work is done I have to remove "B" message for example. Desired state:
| Code: |
Queue: F - E - D (HEAD) - C - A
Worker 1: A
Worker 2: no work
Worker 3: C
Worker 4: D
|
And after that:
| Code: |
Queue: F - E (HEAD) - D - C - A
Worker 1: A
Worker 2: E
Worker 3: C
Worker 4: D
|
I think that language/API should be irrelevant for this problem, so I would be grateful for any example. Many thanks in advance. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Dec 13, 2012 4:17 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
This is why MQ allows you to create a transaction.
Each of your threads should read the message in a transaction.
It's called a 'syncpoint'.
As you say, language is irrelevant. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 13, 2012 4:43 am Post subject: Re: Browsing queue and deleting messages in one connection |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| iw.kuchin wrote: |
| I believe I can peek message while not removing it with "browse" option. But is it possible to remove message in same connection? |
It is possible. It's the wrong thing to do. Browse is a very expensive operation and no application should ever want or need to do it.
If you don't want a message removed from the queue until it's been successfully processed then destructively read it from the queue under syncpoint and commit the transaction (or roll it back) at the appropriate point.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Dec 13, 2012 5:01 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You can "lock" the message under the browse cursor.
However browsing is really only for use for tools or special trigger monitor programs.
Unless you have a truly advanced requirement for handling messages - just use standard syncpointed gets. |
|
| Back to top |
|
|
| iw.kuchin |
| Posted: Thu Dec 13, 2012 5:22 am Post subject: |
|
|
Newbie
Joined: 13 Dec 2012
Posts: 3
|
1. In the initial design it was supposed that threads know nothing about MQ. They just get their piece of work to process and return processing result to "main" thread which is the only piece which communicates with MQ. I would prefer to stick to this design, but if it is necessary I can create multiple additional "reader" threads each running separate transaction for every message.
2. Also, I've found in WebShpere MQ for .NET RedBook some SYNCPOINT examples and in all examples QueueManager object is used to commit/rollback transactions. Does this mean that I need to create separate QueueManager objects in different threads for this pattern to succeed? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Dec 13, 2012 5:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
each instance of a QueueManager object represents a logical connection to the queue manager.
Would you run multiple worker threads all processing information in the same database using a single connection to the database and using a single understanding of the state of the interaction with the database?
Or would you ensure that each thread processing information in the database had it's own connection and could manage the state of that interaction according to it's own needs?
Otherwise, the boundary between code that understands MQ and code that doesn't is much more under your control than you think it is. In all cases you are creating a programmatic interface between the two portions - an object layer, an API, a set of method calls, etc.
There's no firm or particular reason why this boundary has to match the boundary between threads. |
|
| Back to top |
|
|
| iw.kuchin |
| Posted: Thu Dec 13, 2012 10:04 pm Post subject: |
|
|
Newbie
Joined: 13 Dec 2012
Posts: 3
|
| Thanks again for your replies. I have to redesign application a bit but may be it's not that bad. Transactions allow to solve some other problems in design more elegantly. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Dec 14, 2012 12:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Ideally re-design it, otherwise add MQGMO_LOCK to your browse operation - remember to unlock it later though, if you don't remove it. |
|
| Back to top |
|
|
|
|
