| Author |
Message
|
| PEPERO |
 Posted: Mon Mar 16, 2015 8:05 am Post subject: Extract multiple messages from MQGET node under a transactio |
 |
|
Disciple
Joined: 30 May 2011
Posts: 177
|
Hi all;
I've two local queues one holding a summary message about some detailed messages residing in the second one. I mean there could be a 1 to many relationship between the messages residing in these queues. The relationship is implemented by the CorrelId of the summary message and the same CorrelId of the detailed messages.
Now consider a transactional message flow in which i want to read the summary message by an MQINPUT node , extract the CorrelId of it , then read all the related detailed messages from the second queue with the same CorrelId and putting them in a third queue.
Since there could be some detailed messages with no summary message related in the first queue , these detailed messages would never be copied into the third one.
The problem is that when i make the flow transactional, the third queue is loaded when all the detailed messages with the same CorrelIds are read from the second queue and this can impact the performance when there is so many detailed messages for a summary message with the same CorrelId in the second queue.
At the other hand i cannot consider the flow as non-transactional. So what do you suggest? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Mar 16, 2015 8:16 am Post subject: Re: Extract multiple messages from MQGET node under a transa |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PEPERO wrote: |
| So what do you suggest? |
Why not read all the messages from both queues and get a Collector node to group them by correlation id?
You'll need some downstream processing to do whatever it is you do when you have detail records but no summary records. One option being dumping them back into the queue to wait for their summary record a bit longer, but that doesn't like so great an idea....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Mon Mar 16, 2015 8:17 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
how you landed up in this design of having 2 queues
1 holding the correlID and other one having the payload, whats the context of it.
who is loading the messages in to the first queue and who is loading the messages to the second queue
are there any request - response flow going on in parallel to this where it sends the correlId
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| PEPERO |
| Posted: Mon Mar 16, 2015 8:36 am Post subject: |
|
|
Disciple
Joined: 30 May 2011
Posts: 177
|
| These are originated from some files. Each file has an identity record and the body records. The records format can vary from file to file and it depends on the file type identifier located at the first record. each body record can generate a single transaction in the host later. At the first phase the files contents (body records) are validated by the message dictionary after detecting the type of the records. After validating each record it would be put in a detailed queue. After validating all of the body records , a summary message would be inserted in a summary queue. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Mar 17, 2015 1:19 am Post subject: Re: Extract multiple messages from MQGET node under a transa |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| PEPERO wrote: |
Now consider a transactional message flow in which i want to read the summary message by an MQINPUT node , extract the CorrelId of it , then read all the related detailed messages from the second queue with the same CorrelId and putting them in a third queue.
|
What do you do with the summary message? Just throw it away after copying all detail messages? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Mar 17, 2015 4:49 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Sounds very much like some kind of batch processing with a batch identifier and a batch summary to verify that you have the complete batch...
The real question here is not how to process this, it is really much more, how are you going to be able to reconcile this ...
The way you set up your automated reconciliation process will determine how you need to process your inputs 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PEPERO |
| Posted: Wed Mar 18, 2015 7:32 am Post subject: |
|
|
Disciple
Joined: 30 May 2011
Posts: 177
|
| Quote: |
Sounds very much like some kind of batch processing with a batch identifier and a batch summary to verify that you have the complete batch...
|
Yes. 
When body records would be hundreds of thousands in a file type, handling under a single unit of work would impact the performance. The question is :
Is there any more convenient way to extract the related records from the queue under a single transaction ?
Last edited by PEPERO on Wed Mar 18, 2015 7:46 am; edited 2 times in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 18, 2015 7:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PEPERO wrote: |
| Quote: |
Sounds very much like some kind of batch processing with a batch identifier and a batch summary to verify that you have the complete batch...
|
Yes. |
I still plump for a Collector node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ganesh |
| Posted: Wed Mar 18, 2015 8:00 am Post subject: |
|
|
Master
Joined: 18 Jul 2010
Posts: 294
|
| Quote: |
| Is there any more convenient way to extract the related records from the queue under a single transaction ? |
Collector node is a good option and another not so good option would be to loop over the queue. |
|
| Back to top |
|
|
| PEPERO |
| Posted: Wed Mar 18, 2015 8:12 am Post subject: |
|
|
Disciple
Joined: 30 May 2011
Posts: 177
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 18, 2015 8:31 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PEPERO wrote: |
When body records would be hundreds of thousands in a file type, handling under a single unit of work would impact the performance. The question is :
Is there any more convenient way to extract the related records from the queue under a single transaction ? |
Well with your numbers, let me remind you that the default max uncommitted messages in a transaction is 10,000 ... For your sizes I would want to verify log space and other such considerations before even attempting to do anything in a single transaction...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
