| Author |
Message
|
| Ashimashi |
 Posted: Tue Jun 12, 2012 7:28 am Post subject: BatchCreator using Shared ROW variable |
 |
|
Novice
Joined: 06 Oct 2011
Posts: 20
|
Hi,
I'm trying to create a batchcreator flow which merges several incoming MQ messages to one outgoing message. I'm using a shared row variable as a placeholder. My approach is as follows:
1. If the received file is the first one in the group (MsgSeqNumber = 1): Add the body (BLOB) in a Shared Row variable.
2. If the received file is between the first message and the last message in group: Concatenate the new BLOB with the previously added BLOB in the Shared Row variable and so on.
The problem that I'm facing is the memory allocation for this procedure. It seems like the memory is built up little by little for each batch process.
I've seen another post regarding shared row variable and possible memory leakage.. Do you think that is the case? Is there other ways of creating batches without holding the final output body in a shared row as I'm doing?
Please let me know if you'd like me to show the code to visualize the problem even further.
Thanks,
A |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Jun 12, 2012 7:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Exactly what reason do you have for NOT using the Collector node, given that it is designed for exactly this usecase? |
|
| Back to top |
|
|
| Ashimashi |
| Posted: Tue Jun 12, 2012 11:54 pm Post subject: |
|
|
Novice
Joined: 06 Oct 2011
Posts: 20
|
Thank you jeff for the quick reply. Indeed, that is a solution. I remember trying to mess around with it without succeeding. But I'll give it another shot and see how it goes. Thanks again.
//A |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jun 13, 2012 12:33 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Ashimashi wrote: |
The problem that I'm facing is the memory allocation for this procedure. It seems like the memory is built up little by little for each batch process.
I've seen another post regarding shared row variable and possible memory leakage.. Do you think that is the case? Is there other ways of creating batches without holding the final output body in a shared row as I'm doing?
|
Yes, look here
But even if you applied the fix so that SHARED ROW variables did not leak parsers, I think it is bad architecture to collect messages there. Using the collector node as mqjeff (always) suggests is safer and easier, too. It's always better if you can configure functionality instead of coding it...
If you get memory problems with collector node, too (because it will have all the collected messages in memory simultaneously), you can store the messages in a queue instead. With MQGet node you can process the messages one at the time, and there is a trick you can use to release the memory after each message. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 13, 2012 1:52 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| If you get memory problems with collector node, too (because it will have all the collected messages in memory simultaneously), you can store the messages in a queue instead. With MQGet node you can process the messages one at the time, and there is a trick you can use to release the memory after each message. |
The end result desired is to create a single message that contains all the input messages.
You can't do that without putting all of the messages in memory at once. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jun 13, 2012 3:31 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
The end result desired is to create a single message that contains all the input messages.
You can't do that without putting all of the messages in memory at once. |
Yes, I can 
Of course the output message will have to contain the contents of all the input messages simultaneously, but all the input messages do not have to be in memory at the same time. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 13, 2012 4:11 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| Of course the output message will have to contain the contents of all the input messages simultaneously, but all the input messages do not have to be in memory at the same time. |
I am curious as to the difference between "all of the input messages being in memory at the same time" and "the output message that contains all of the input messages being in memory".
Just because you have reused the Input tree, that does not mean you do not still have a copy of the input message in memory. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Wed Jun 13, 2012 4:33 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| mqjeff wrote: |
I am curious as to the difference between "all of the input messages being in memory at the same time" and "the output message that contains all of the input messages being in memory".
Just because you have reused the Input tree, that does not mean you do not still have a copy of the input message in memory. |
I guess what Esa ment is that you will have only the final concatenated message (all of input message) once. Until then you can either put them in MQ or DB and concatenate.
When you use a SHARED Variable or the collector node, the complete message (from the first till the last) is held in the memory all times. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jun 13, 2012 4:45 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kash3338 wrote: |
| When you use a SHARED Variable or the collector node, the complete message (from the first till the last) is held in the memory all times. |
Perhaps you could expand this with respect to the Collector node, and with reference to this?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 13, 2012 4:46 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| kash3338 wrote: |
| the complete message (from the first till the last) is held in the memory all times. |
| kash3338 wrote: |
| When you use a SHARED Variable |
Yes
| kash3338 wrote: |
| or the collector node, |
Not as far as I know. AFAIK, the collector node uses a queue under the covers and it only reads all of the messages when the collection is complete or expires.
It's funny to think that the lab might have actually considered this, isn't it?
Or, in other words
| Vitor wrote: |
| kash3338 wrote: |
| When you use a SHARED Variable or the collector node, the complete message (from the first till the last) is held in the memory all times. |
Perhaps you could expand this with respect to the Collector node, and with reference to this? |
Last edited by mqjeff on Wed Jun 13, 2012 4:47 am; edited 1 time in total |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jun 13, 2012 4:47 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
I am curious as to the difference between "all of the input messages being in memory at the same time" and "the output message that contains all of the input messages being in memory".
Just because you have reused the Input tree, that does not mean you do not still have a copy of the input message in memory. |
Yes, it was a bit misleading to talk about input messages when I was in fact referring to messages that are read with an MQGet node.You can loop with MQGet nodes so that only one of the messages is in memory at the time. There is a safe way to do it, although not very commonly known, it seems. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 13, 2012 4:48 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| mqjeff wrote: |
I am curious as to the difference between "all of the input messages being in memory at the same time" and "the output message that contains all of the input messages being in memory".
Just because you have reused the Input tree, that does not mean you do not still have a copy of the input message in memory. |
Yes, it was a bit misleading to talk about input messages when I was in fact referring to messages that are read with an MQGet node.You can loop with MQGet nodes so that only one of the messages is in memory at the time. There is a safe way to do it, although not very commonly known, it seems. |
Yes. The safe way to do it is to use the Collector node. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jun 13, 2012 4:49 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| There is a safe way to do it, although not very commonly known, it seems. |
Which is unnecessary when you have the kind of situation described by the OP when a Collector node is a better choice.
You're welcome to increase the level of awareness on this "safe" way by posting it here.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Esa |
| Posted: Wed Jun 13, 2012 5:02 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
Not as far as I know. AFAIK, the collector node uses a queue under the covers and it only reads all of the messages when the collection is complete or expires.
|
I know the collector node stores messages in queues. I was talking about the compound message collection that the Collector node propagates. The output nodes are not capable of serializing it, so you will have to transform it. So in the message collection all the input messages will be in memory at the same time.
| Vitor wrote: |
| You're welcome to increase the level of awareness on this "safe" way by posting it here. |
I may do it some day. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 13, 2012 5:18 am Post subject: Re: BatchCreator using Shared ROW variable |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| So in the message collection all the input messages will be in memory at the same time. |
No. That's not true.
They're not input messages at that point. It's a single message tree.
I repeat that it is impossible to collect N messages, regardless of source, into a single MQ output message without holding them all in memory. |
|
| Back to top |
|
|
|
|
