| Author |
Message
|
| jamesyu |
 Posted: Wed Jun 10, 2009 10:15 am Post subject: Multi threads processing messages in a single message flow |
 |
|
Acolyte
Joined: 31 Jan 2007
Posts: 70
|
The source application sends a big number of messages (over 10,000) to my message flow each time. The size of each message is large.
The last message sent from the source is a special message to indicate that this is the last message from the sender.
When the message flow retrieves this special message at the end, it will touch a trigger file on the AIX server to indicate all messages are ready on the target queues for another job to read being triggered by a File Watcher.
All of this is not an issue when running this message flow without additional instances.
This special message processing has become an issue when deploying the message flow with additional instances, in the case that, the thread to process the special message is finished too earlier than other threads are still busy with those large messages (3 minutes to process each large message).
Any way to make sure this special message will be processed right after all threads are done with the messages? |
|
| Back to top |
|
 |
| WMBDEV1 |
| Posted: Thu Jun 11, 2009 12:26 am Post subject: Re: Multi threads processing messages in a single message fl |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| jamesyu wrote: |
Any way to make sure this special message will be processed right after all threads are done with the messages? |
I suspect not with this design when running multithreaded. It should be possible to achieve if you are prepared to change your design though. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 11, 2009 12:38 am Post subject: Re: Multi threads processing messages in a single message fl |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jamesyu wrote: |
| Any way to make sure this special message will be processed right after all threads are done with the messages? |
Nothing springs to mind; it's a classic message affinity problem.
How does the consuming job (the one triggered by File Watcher) process the 10,000 messages correctly? Clearly it needs all 10,000 on 1-n queues (you do say queues) before it can start but how does it know the set(s) are complete? Totals, flags, what? Could this mechanism be used in place of this message?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jun 11, 2009 2:14 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
When this final message is received by one of the flow instances, the flow could wait long enough for any other instance to complete (e.g. using the MQGET node with a WAIT parameter on a empty queue).
If 3 minutes is the max for any flow to complete, then waiting 5 mins on receipt of the final message would cover it.
If you can't be certain of this then you need a way to signal if the flows are busy or not. They could use some shared variables to indicate their status. These are visible to other flow instances. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Jun 11, 2009 6:33 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| zpats idea sounds doable but just be careful incase a batch message gets backed out cus your final thread will then block. Bad times if another batch comes through! |
|
| Back to top |
|
|
| jamesyu |
| Posted: Thu Jun 11, 2009 6:33 am Post subject: |
|
|
Acolyte
Joined: 31 Jan 2007
Posts: 70
|
Thank you all for your response.
Customer owns the design. Nothing I can do about it to change the design.
In Java world, it's easy to come up with some method like AllThreadsIdle() this kind of monitoring stuff... |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 11, 2009 6:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jamesyu wrote: |
| In Java world, it's easy to come up with some method like AllThreadsIdle() this kind of monitoring stuff... |
In WMB world, we trout the customer until message affinity is removed from the design. We lessen the impact by pointing out that our way, they don't have to wait 30,000 minutes (using your figures of 10,000 messages at 3 minutes a message) before they start processing the first message. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jamesyu |
| Posted: Thu Jun 11, 2009 6:45 am Post subject: Re: Multi threads processing messages in a single message fl |
|
|
Acolyte
Joined: 31 Jan 2007
Posts: 70
|
| Vitor wrote: |
| Totals, flags, what? Could this mechanism be used in place of this message? |
That takes me back to 25 years ago when coding with Basic. Unfortunately, I am going with some sort of flag now...
The current idea is that we can split all of the input messages into 2 MQ message groups:
Group 1: all of the real messages
Group 2: one single special message
If the group 2 finished, we can always check to see if the group 1 finished or not by looking at the last message indicator in MQMD on the target queue and make sure no gaps between SeqNumber. If group 1 completed on the target queue, we go ahead and create a trigger file. Otherwise, we wait for another 30 seconds to check the group 1 again. All can be done within message flows… |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 11, 2009 7:09 am Post subject: Re: Multi threads processing messages in a single message fl |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jamesyu wrote: |
| That takes me back to 25 years ago when coding with Basic. |
I'm an old fashioned kinda guy. 
| jamesyu wrote: |
| If the group 2 finished, we can always check to see if the group 1 finished or not by looking at the last message indicator in MQMD on the target queue and make sure no gaps between SeqNumber. If group 1 completed on the target queue, we go ahead and create a trigger file. Otherwise, we wait for another 30 seconds to check the group 1 again. All can be done within message flows… |
So what value does group 2 add if you're checking for last message indicator in group 1 to fire off the next step in the processing? Also you need to ensure that the input doesn't wait for the group to be complete before starting processing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jun 11, 2009 7:31 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Remember to include a wait/sleep delay when waiting for the "in use" flag to change - don't loop tightly around in ESQL otherwise you will burn CPU.
You can still use the MQGET WAIT to cause the flow to wait for a period of time before rechecking the flag.
However if a really long time has elapsed and it still claims to be in-use - something is wrong - maybe generate an Email from the flow to alert to this condition and terminate without triggering the batch file? |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jun 11, 2009 9:20 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
To flag the in-use, I would get each flow to start by incrementing a certain shared integer variable and decrement it immediately before terminating.
The final message logic can wait until the value drops back to 1 before processing. Don't forget to adjust the shared variable value inside an ATOMIC clause to ensure integrity. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 11, 2009 9:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| To flag the in-use, I would get each flow to start by incrementing a certain shared integer variable and decrement it immediately before terminating. |
This does imply that the number of messages is known ahead of time; I wonder if this is true given they send a special message indicate transmission is complete.
Of course, a tweak to the design so this special message is sent first with a to-be-transmitted total in it would fix this and allow the use of a shared variable as indicated above.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jun 11, 2009 9:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
No it doesn't.
It simply records the number of flows currently running at any given moment in time. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jun 11, 2009 10:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| It simply records the number of flows currently running at any given moment in time. |
Ah - so it does! Apologies.
So you'd need a wait period in case there's a lull in messages where the single thread is idling
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jun 11, 2009 11:27 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
No, the final message makes the flow instance that picks it up check for the other flow instances that may be running at the same time and wait until they have finished their current message.
Only one flow is needed. Any number of instances can be set. |
|
| Back to top |
|
|
|
|
