| Author |
Message
|
| prasannanb |
 Posted: Sun Nov 27, 2011 12:34 am Post subject: MQ batch process |
 |
|
Apprentice
Joined: 17 Apr 2009
Posts: 35
|
Hi,
I have a requirement where have to process all the messages in the queue every 30 min(batch process).
How can I achieve this in MB?
If i use,
Timeout Notification -> MQGet -> Compute
This will process only one message every 30 min.
What is the alternative approach?
Thanks |
|
| Back to top |
|
 |
| zpat |
| Posted: Sun Nov 27, 2011 3:17 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Use WMQ and WMB as intended, as an event driven stateless messaging system? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Nov 27, 2011 4:07 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
| Use WMQ and WMB as intended, as an event driven stateless messaging system? |
Get out of bed the wrong side this morning?  
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Nov 27, 2011 3:09 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There are several bad ways to solve this problem.
You can have the timeoutnotification flow do something to "enable" the queue for real processing from a real MQInput node.
You can write a loop over the MQGet node using additional compute or JavaCompute nodes.
You can use a collection that is only complete at 30 minute intervals.
The only good way to solve this problem is to avoid it by processing all of the messages as they arrive, rather than in a batch. |
|
| Back to top |
|
|
| ydsk |
| Posted: Sun Nov 27, 2011 9:24 pm Post subject: |
|
|
Chevalier
Joined: 23 May 2005
Posts: 410
|
If only IBM can provide a polling interval on the MQInput node , just like they did for the FileInput and other nodes  This would be an option for all batch processing scenarios.
Another way, similar to the timer notification method listed above is to use a cron tab entry in unix to drop a message on a queue every 30 minutes so it kicks off a msgflow.
Looping around the MQGet node would be an interesting option. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Nov 28, 2011 1:45 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Looping around the MQGet node does present a nice set of potential pitfalls. Things like Units of Work and transactionality springto mind. There are others.
Enjoy
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| vmcgloin |
| Posted: Mon Nov 28, 2011 2:37 am Post subject: |
|
|
Knight
Joined: 04 Apr 2002
Posts: 560
Location: Scotland
|
| Why? What is the requirement? What is the outcome of the processing? Why can't you process the messages as they arrive and have the output consumed by a batch job if that really is a business requirement? How many messages are in your half-hourly batch, and what other non-functional requirements do you have? |
|
| Back to top |
|
|
| prasannanb |
| Posted: Mon Nov 28, 2011 11:18 pm Post subject: |
|
|
Apprentice
Joined: 17 Apr 2009
Posts: 35
|
| vmcgloin wrote: |
| Why? What is the requirement? What is the outcome of the processing? Why can't you process the messages as they arrive and have the output consumed by a batch job if that really is a business requirement? How many messages are in your half-hourly batch, and what other non-functional requirements do you have? |
My requirement is to re-process all the failed messages which will be there in the queue. This should be retried every 30 min. This is the business requirement. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 29, 2011 5:32 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| prasannanb wrote: |
| vmcgloin wrote: |
| Why? What is the requirement? What is the outcome of the processing? Why can't you process the messages as they arrive and have the output consumed by a batch job if that really is a business requirement? How many messages are in your half-hourly batch, and what other non-functional requirements do you have? |
My requirement is to re-process all the failed messages which will be there in the queue. This should be retried every 30 min. This is the business requirement. |
Use a cron job to move all the messages from the failed queue back into the original queue every 30 minutes.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| McueMart |
| Posted: Tue Nov 29, 2011 5:53 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
I had a similar requirement whereby I needed to process messages after they had been on a queue for X amount of time (for retry purposes).
The approach I ended up was is:
Hopefully the flow is quite self explanatory, but in brief:
RetryFlowTick: Set to Controlled operation. Invoke it initially however you want!
ForEachMessageOnRetryQ: Basically a FOR loop with a PROPAGATE in it. The loop will loop a maximum number of times to prevent hitting MQRC_SYNCPOINT_LIMIT_REACHED errors if you are doing transactional stuff.
GetOffRetryQ: Takes 1 message off the 'retry queue'
SetDestinationQ: Sets up the Destination list to put the message back on the queue it needs to go on (For my case this was encoded in the RFH2 header).
StopFlow: This comes out of the 'No messages' terminal and sets an ENVIRONMENT variable which stops the FOR loop in the ForEachMessageOnRetryQ from continuing.
SetupReinvoke: Sets up the necessary message to reinvoke this flow in X seconds.
This might not do exactly what you want as it is, but hopefully it gives you some ideas! |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 29, 2011 5:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
The usual warnings apply about using the timer nodes in restart situations.
As well as the warnings about repeatedly using an MQGet node in a flow.
I'm also interested to know how the ESQL node knows how many times to loop. Is it actually finding the depth, or just looping <syncpoint limit> times and relying on the 2033 to trigger the StopFlow node? If so, what happens if there are more messages on the retry queue than will fit in a unit of work?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Nov 29, 2011 6:28 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
It is always a bad idea to use WMB as a system administration tool for controlling start and stop of processing. WMB is a high speed messaging engine. Use Tivoli Omegamon, cron, MQ triggers or BMC Patrol to start and stop things.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| McueMart |
| Posted: Tue Nov 29, 2011 7:02 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
From my testing, as long as you don't actually 'loop' on a MQGet node (which increases the call stack internally resulting in a stackoverflow ) there are no major issues.
| Quote: |
| I'm also interested to know how the ESQL node knows how many times to loop. Is it actually finding the depth, or just looping <syncpoint limit> times and relying on the 2033 to trigger the StopFlow node |
It will stop the loop either by hitting the 2033, or by hitting the <syncpoint limit> (iterate a variable each time through a loop). [/quote] |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Nov 29, 2011 7:37 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| lancelotlinc wrote: |
| It is always a bad idea to use WMB as a system administration tool for controlling start and stop of processing. WMB is a high speed messaging engine. Use Tivoli Omegamon, cron, MQ triggers or BMC Patrol to start and stop things. |
I recall from my days as a Message Broker course instructor that the timeout nodes were added so that you have all control in one application - Message Broker - instead of spreading it in external tools, like cron. That's what the course material said.
As an ex course instructor I naturally don't put much weight on what the course materials say, but in this case it is consistent with this quote from http://www.ibm.com/developerworks/websphere/library/techarticles/0603_schutz/0603_schutz.html:
| Quote: |
To make a flow run hourly on the hour, you can use a UNIX cron job, which can provide rich semantics to control job scheduling. The job would presumably use a utility such as amqsput to send a (dummy) message to the input queue of the message flow that you want to run.
While this approach works, it adds complexity to the system compared to using timer nodes. You must co-ordinate the creation of the cron job with the deployment of the message flows. If many flows have this requirement, maintenance of the crontab becomes burdensome. Another point of failure is introduced into the system, and you must have a mechanism to ensure that the amqsput utility actually worked and didn't generate a bad return code such as 2035. This approach makes problem determination more complex.
Using timeout nodes provides a flexible, self-contained mechanism to start a message flow. No external coordination with the broker is required beyond what is normally required to deploy a flow. You can use built-in, broker-native tracing mechanisms to detect the failure to set the timers. Although cron is usually available on UNIX, not all installations have it, and other platforms such as Windows® and z/OS® have different job scheduling mechanisms. Timeout nodes provide a consistent mechanism across all WebSphere Message Broker V6 platforms.
|
I agree that timeout nodes perhaps should not be used for restarting things, but why not for starting and stopping flows at specific times?
I must have missed some discussion. You are in fact saying that TimeoutControl node should not be used at all and TimeoutNotification should only be used for implementing delay in processing? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Nov 29, 2011 9:03 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Would rather use the other stated options for controlling jobs. WMB is a high speed messaging engine. I know I seem to be repeating myself. Why bog down your Supersonic Jet with strap on model T engine parts?
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
