| Author |
Message
|
| asraro |
 Posted: Mon Jan 19, 2009 3:10 pm Post subject: File Input Node |
 |
|
Novice
Joined: 25 Aug 2008
Posts: 15
|
| My input file is around 2MB, its taking around 40 minutes to process. Is this normal and is there a way to speed up the process ????? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Jan 19, 2009 3:14 pm Post subject: Re: File Input Node |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| asraro wrote: |
| My input file is around 2MB, its taking around 40 minutes to process. |
On a single processor Windows machine or an 8-way Unix box? With 1Mb of RAM or 4Mb? Processed as a single file or a record at a time? Is the process a single insert into a database or a complex transformation with reference to a view built on a cartesian join?
Help us to help you.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jan 19, 2009 4:29 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| How do you know that it is taking 40 minutes to process? I mean, what are you measuring to say that it takes 40 minutes versus 10 seconds? |
|
| Back to top |
|
|
| asraro |
| Posted: Tue Jan 20, 2009 7:28 am Post subject: |
|
|
Novice
Joined: 25 Aug 2008
Posts: 15
|
| Broker reads it from a Input File Node. When i put the file in the location, and broker starts processing the file, i can see the CPU % going up in the AIX box. After it finishes the processing and spits out the output file after 30 - 40 min, i can see the CPU going down. It has 16 Gigs and 2 cpu. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 20, 2009 7:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| asraro wrote: |
| broker starts processing the file, |
Again I ask what the processing involves and how the file is handled.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 20, 2009 7:40 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So you are measuring the time between when the FI "picks up" the file, and the time when the message flow produces output.
This provides no evidence that it is the FileInput node that is causing the delay. It could be anything in your flow between the FI and the FO nodes. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 20, 2009 7:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| It could be anything in your flow between the FI and the FO nodes. |
Hence my question about the processing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 20, 2009 7:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| It could be anything in your flow between the FI and the FO nodes. |
Hence my question about the processing. |
Yes.
It was a good question.
I was explaining the results of the answer to my question about measurement. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 20, 2009 7:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| I was explaining the results of the answer to my question about measurement. |
And rightly so sir.
We need clarity. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| asraro |
| Posted: Wed Jan 21, 2009 6:49 am Post subject: |
|
|
Novice
Joined: 25 Aug 2008
Posts: 15
|
Flow looks like this:
FileInput Node ---> Compute Node ---> FileOutput Node
In the compute node, file is read and then transformed accordingly to the message set. Code snippet is below:
DECLARE reffile REFERENCE TO InputRoot.MRM.FILE[1];
DECLARE cnt INTEGER 1;
WHILE LASTMOVE(reffile) DO
SET OutputRoot.MRM.FILE[cnt].UPCNUMBER = reffile.UPCNUMBER;
.
.
.
.
.
SET cnt = cnt + 1;
MOVE reffile NEXTSIBLING REPEAT TYPE NAME;
END WHILE; |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jan 21, 2009 1:50 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| asraro wrote: |
Flow looks like this:
FileInput Node ---> Compute Node ---> FileOutput Node
In the compute node, file is read and then transformed accordingly to the message set. Code snippet is below:
DECLARE reffile REFERENCE TO InputRoot.MRM.FILE[1];
DECLARE cnt INTEGER 1;
WHILE LASTMOVE(reffile) DO
SET OutputRoot.MRM.FILE[cnt].UPCNUMBER = reffile.UPCNUMBER;
.
.
.
.
.
SET cnt = cnt + 1;
MOVE reffile NEXTSIBLING REPEAT TYPE NAME;
END WHILE; |
Very innefficient shouldn't it rather be:
| Code: |
DECLARE reffile REFERENCE TO InputRoot.MRM.FILE[1];
SET OutputRoot.MRM.FILE[1] VALUE = NULL;
DECLARE outref REFERENCE TO OutputRoot.MRM.FILE[1];
WHILE LASTMOVE(reffile) AND LASTMOVE(outref) DO
SET outref.UPCNUMBER = reffile.UPCNUMBER;
..
MOVE reffile NEXTSIBLING REPEAT TYPE NAME;
IF LASTMOVE(reffile) THEN
CREATE NEXT SIBLING OF outref AS outref NAME 'FILE';
END IF;
END WHILE;
|
This should provide for a lot less tree walking... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| asraro |
| Posted: Tue Feb 03, 2009 3:02 pm Post subject: |
|
|
Novice
Joined: 25 Aug 2008
Posts: 15
|
I have changed the code, but its still taking around 30 - 40 min to process with input file node. I have even tried wtih MQInput node, result is same.
DECLARE reffile REFERENCE TO InputRoot.MRM.FILE[1];
DECLARE cnt INTEGER 1;
DECLARE targetCursor REFERENCE TO OutputRoot.MRM;
WHILE LASTMOVE(refInvSum) DO
SET targetCursor.FILE[cnt].UPCNUMBER = reffile.UPCNUMBER;
SET targetCursor.FILE[cnt].WMITEMNUMBER = reffile.WMITEMNUMBER;
.
.
.
SET cnt = cnt + 1;
MOVE reffile NEXTSIBLING REPEAT TYPE NAME;
END WHILE; |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Feb 03, 2009 3:38 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Is the processing time proportional to the size of the file ( i.e. if you reduce the file to a 10th of its size, does the processing time drop to a 10th )? Do a few tests, and plot the graph.
Are you holding the entire InputRoot.MRM message tree in memory? If you have not taken steps to avoid that, then you are probably using several Gb of RAM on the message tree alone. This technique would seem like a good idea if you are not already doing something similar: http://www.ibm.com/developerworks/websphere/library/techarticles/0505_storey/0505_storey.html
If the input message consists only of a single repeating structure ( no headers or trailers ) then you can do the same trick more easily using the Parsed Record Sequence feature of the FileInput node. |
|
| Back to top |
|
|
| asraro |
| Posted: Wed Feb 04, 2009 3:03 pm Post subject: |
|
|
Novice
Joined: 25 Aug 2008
Posts: 15
|
The code i used earlier to create the output using Refernce for Output.MRM creats a 0 byte file ??????
Time of processing was still the same, no improvements. |
|
| Back to top |
|
|
| asraro |
| Posted: Wed Feb 04, 2009 3:34 pm Post subject: |
|
|
Novice
Joined: 25 Aug 2008
Posts: 15
|
Processing time varies with the size of the file. But i have another flow with the same type of nodes and it process 4-5MB file in 3-4 minutes, thats what i expect from this flow too. Flow looks like this:
In orginal flow, i used file input node with no Referce to output.mrm. Flow looks like this:
File INput Node ---> Compute Node ----> FileOut Node.
In Modified flow, i used the MQ input node and used reference to Output.MRM. Processing time was still the same and it created a 0 byte file. Without using Referecne to output.MRM, it creats the file with output but its slow.
MQ INput node ---> Compute Node ----> FileOut Node. |
|
| Back to top |
|
|
|
|
