| Author |
Message
|
| g20720 |
 Posted: Tue Apr 07, 2009 2:33 pm Post subject: Batch Processing pattern |
 |
|
Newbie
Joined: 27 Oct 2006
Posts: 6
|
Undoubtedly this is an old question but I have not been able to locate any info. I'm curious about any common patterns/implementations for handling 'batches" of messages.
Situation: I have a broker queue/flow that accepts messages from a (non-IBM) mainframe. Normally a trickle of messages come to broker during the course of a day. Occassionally, there are batch jobs that can produce a large message volume (200K+) to this same queue. We already have implemented as part of the message flow an update to a database table to record the status of the message being processed. We generate a daily report from that data that we email to a recipient list.
What we would like to do is produce some sort of additional report at the completion of the large batch jobs to verify that all intended messages were processed and (preferably) send out an alert if there is a discrepancy in counts or errors encountered. The primary issue I'm having is knowing when to start this batch report process. I can't send a message to a separate queue for immediate processing because the broker flow may not have processed all of the input messages yet. I'm not fond of the idea of sending an additional message (with a different message format) on the same input queue.
I'm willing to accept that my primary assumptions are flawed. I'm more interested in how others may have dealt with similar situations. |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Tue Apr 07, 2009 3:16 pm Post subject: Re: Batch Processing pattern |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| g20720 wrote: |
| What we would like to do is produce some sort of additional report at the completion of the large batch jobs... |
How do you know the batch job has completed?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Wed Apr 08, 2009 1:50 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
|
| Back to top |
|
|
| Hari |
| Posted: Wed Apr 08, 2009 12:01 pm Post subject: |
|
|
Centurion
Joined: 21 Nov 2002
Posts: 117
Location: USA
|
Few of the possible solutions:
If the Source application is ok, you can have them but a sort of Trailer message at the end of their batch messages...which will have the total count. And of course have a Header message to indicate start of batch.
Match the Trailer count with the database count to verify the receipt of all messages.
How about monitoring your error/exception queues for possible errors....and have some alerts set thru some monitoring tools?
But at the end of the day it all depends what you are willing to change, the source app sending the messages or your architecture or both. |
|
| Back to top |
|
|
| g20720 |
| Posted: Wed Apr 08, 2009 5:42 pm Post subject: |
|
|
Newbie
Joined: 27 Oct 2006
Posts: 6
|
That was helpful in generating some new thoughts. The number of caveats in using groups and the impact on the flows is a bit surprising but generally understandable once described. Will have to consider the consequences further. Thank you for the input. |
|
| Back to top |
|
|
| g20720 |
| Posted: Wed Apr 08, 2009 5:52 pm Post subject: |
|
|
Newbie
Joined: 27 Oct 2006
Posts: 6
|
| Hari wrote: |
Few of the possible solutions:
If the Source application is ok, you can have them but a sort of Trailer message at the end of their batch messages...which will have the total count. And of course have a Header message to indicate start of batch. |
Most of my thinking has been in this direction, mostly because I could not identify any other reasonable alternatives. My hang-up on this option is that I don't necessarily like the idea of having 2 or 3 different logical message types (formats) on the same input queue - one for header, one for trailer and one for "real" data. If I ignore my personal preference, any suggestions on how best to identify header and trailer messages since the flow will have to take different actions depending on the logical msg type? |
|
| Back to top |
|
|
| HOMETOWN47 |
| Posted: Thu Apr 09, 2009 12:23 am Post subject: Batch Patterns |
|
|
Apprentice
Joined: 25 Mar 2003
Posts: 34
|
Previous client had a form of batch processing where a record was written to DB table each time one of the expected batch messages was processed. The flow would then trigger another 'check' flow that would interrograte the DB to see if the complete batch had been processed.
There has to be some way of distinguishing when the start of a batch and end of a batch occurs which then could be used in the above pattern... |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Apr 09, 2009 2:09 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| any suggestions on how best to identify header and trailer messages since the flow will have to take different actions depending on the logical msg type? |
An RFH2 header is a good solution. The settings in the <mcd> folder override the message set/message type/message format settings on the MQInput node. |
|
| Back to top |
|
|
| MQEnthu |
| Posted: Thu Apr 09, 2009 2:22 am Post subject: |
|
|
Partisan
Joined: 06 Oct 2008
Posts: 329
Location: India
|
| g20720 wrote: |
| My hang-up on this option is that I don't necessarily like the idea of having 2 or 3 different logical message types (formats) on the same input queue - one for header, one for trailer and one for "real" data. |
A flag set in the RFH2Header to indicate the start and end of the batch messaage wil do ... in that case you will have only one logical message.
_________________
-----------------------------------------------
It is good to remember the past,
but don't let past capture your future |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Apr 09, 2009 3:39 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Have fun writing your RFH2 headers on the mainframe!
Last edited by WMBDEV1 on Thu Apr 09, 2009 3:46 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Apr 09, 2009 3:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Why doesn't the logical message include information about the batch membership?
Shouldn't that be part of the business data somewhere, anyway? If it's batched by time - daily receipts or etc. - you still should be including enough information in the message to be carrying that criteria through to broker. |
|
| Back to top |
|
|
| g20720 |
| Posted: Thu Apr 09, 2009 3:50 am Post subject: |
|
|
Newbie
Joined: 27 Oct 2006
Posts: 6
|
| WMBDEV1 wrote: |
| Have fun writing your RFH2 headers on the mainframe! |
Indeed! I have tried it in the past and while successful it was a bit of a challenge. |
|
| Back to top |
|
|
| g20720 |
| Posted: Thu Apr 09, 2009 4:05 am Post subject: |
|
|
Newbie
Joined: 27 Oct 2006
Posts: 6
|
| mqjeff wrote: |
| Why doesn't the logical message include information about the batch membership? |
At present, there is no message content that would uniquely identify the source of these messages with the exception of the MQMD PutApplName. Since I can control what goes into these messages I can make a requirement to add additional information. I just have not yet arrived at what that might be. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Apr 09, 2009 4:07 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| g20720 wrote: |
| I just have not yet arrived at what that might be. |
How about batch id, position in batch, total messages in batch? |
|
| Back to top |
|
|
|
|
