| Author |
Message
|
| giorginus80 |
 Posted: Mon Mar 16, 2009 5:58 am Post subject: FileInputNode Size limit? |
 |
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
Hello,
I have a flow with a FileInputNode, the files taken are tar file (so the next node is a java node to untar it) the flow is working good till a file is 110mb, but with a file of 150mb it goes in the mqsibackout directory, why? Can this limit increased? |
|
| Back to top |
|
 |
| jharringa |
| Posted: Mon Mar 16, 2009 12:22 pm Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
If I understand correctly, this is a limitation on the FileInput node and I do not believe it can be changed. In the past, I have use the FileInput to read line by line for files that can get larger than 100MB. I don't think that you're going to be able to untar this in broker. You may be able to chunk it apart and have something else process the file by reading a Fixed Length. This doc has some good information relating to this problem.
http://publib.boulder.ibm.com/infocenter/wmbhelp/v6r1m0/index.jsp?topic=/com.ibm.etools.mft.doc/au55470_.htm |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 16, 2009 12:30 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| There are some comments from mqpaul on increasing the size limits of the FileInput node. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Tue Mar 17, 2009 2:49 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| mqjeff wrote: |
| There are some comments from mqpaul on increasing the size limits of the FileInput node. |
I can't find it. Anyways I can't split the file, because it's a tar file. The FileInputNode size can be increased or it's a WMB limit? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 17, 2009 3:00 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Hrm... I *thought* there were comments...
My memory *also* tells me there's a known APAR and an "MQSI_" environment variable to enable larger than 100 MB files, in some later FP of Broker 6.1. But I don't have the details... |
|
| Back to top |
|
|
| jharringa |
| Posted: Tue Mar 17, 2009 4:47 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2007
Posts: 70
|
| giorginus80 wrote: |
| mqjeff wrote: |
| There are some comments from mqpaul on increasing the size limits of the FileInput node. |
I can't find it. Anyways I can't split the file, because it's a tar file. The FileInputNode size can be increased or it's a WMB limit? |
How big can the files get? They're just bytes, right? You would just need to reassemble them in order. We had a similar requirement but opted to have a Java component take care of the operation for us ahead of the broker (and for other reasons as well). Whenever we received large files it seemed that the FileInput node would quickly spike the CPU for as long as it took to process the file. This is a concern as we're operating in a shared environment to conserve costs. |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Tue Mar 17, 2009 5:34 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| jharringa wrote: |
| giorginus80 wrote: |
| mqjeff wrote: |
| There are some comments from mqpaul on increasing the size limits of the FileInput node. |
I can't find it. Anyways I can't split the file, because it's a tar file. The FileInputNode size can be increased or it's a WMB limit? |
How big can the files get? They're just bytes, right? You would just need to reassemble them in order. We had a similar requirement but opted to have a Java component take care of the operation for us ahead of the broker (and for other reasons as well). Whenever we received large files it seemed that the FileInput node would quickly spike the CPU for as long as it took to process the file. This is a concern as we're operating in a shared environment to conserve costs. |
Some files are 200mb, assemble it with an aggregation flow isn't working, so I don't know how to do it |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Tue Mar 17, 2009 7:23 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Why is the message being backed out? Have you managed to run a trace to try and determine this? If you dont connect the Java node does it manage to read the file ok?
Is there a core dump / abend?
Untarring a 100MB in memory at once, assuming a compression ratio of about 10:1 will require about 1GB. Out of the box I think the broker is configured to allow a maximum process size of 1GB per execution group (but this can be increased to 2GB and 3GB on Solaris). How much memory does the process use when it manages to read the largest file?
If memory is the limiting factor you will probably get an abend and the DataFlow Process will die (and then be automatically restarted). You can try increasing the memory used by the broker to see if this helps.
Which platform are you using?
Last edited by WMBDEV1 on Tue Mar 17, 2009 7:28 am; edited 1 time in total |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Tue Mar 17, 2009 7:26 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| WMBDEV1 wrote: |
Why is the message being backed out? Have you managed to run a trace to try and determine this? If you dont connect the Java node does it manage to read the file ok?
Is there a core dump / abend?
Untarring a 100MB in memory at once, assuming a compression ratio of about 10:1 will require about 1GB. Out of the box I think the broker is configured to allow a maximum process size of 1GB per execution group (but this can be increased to 2GB and 3GB on Solaris). How much memory does the process use when it manages to read the 110MB file?
If memory is the limiting factor you will probably get an abend and the DataFlow Process will die (and then be automatically restarted). You can try increasing the memory used by the broker to see if this helps.
Which platform are you using? |
I'm using linux. Anyway the file goes in mqsibackout directory, it isn't read, and it doesn't go on the failure terminal. It doesn't go in a terminal...it goes directly in mqsibacout dir. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Tue Mar 17, 2009 8:13 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| I think IC58202 was raised to allow the File Nodes to work with message > 100MB but after a quick search I still cant see the MQSI parameter to increase said size (although I do remember reading about it a couple of weeks ago). Sorry I cant help more atm. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Tue Mar 17, 2009 8:19 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Found it..... for once the IBM search was better than google 
MQSI_FILENODES_MAXIMUM_RECORD_LENGTH from http://www-01.ibm.com/support/docview.wss?uid=swg1IC58202
Hope that helps (but i'd still keep an eye on the memory size when the data leaves the Java node unless its streaming it back to the file system again) |
|
| Back to top |
|
|
| giorginus80 |
| Posted: Tue Mar 17, 2009 8:21 am Post subject: |
|
|
Centurion
Joined: 08 Jul 2008
Posts: 124
Location: Rome, Italy
|
| WMBDEV1 wrote: |
Found it..... for once the IBM search was better than google
MQSI_FILENODES_MAXIMUM_RECORD_LENGTH from http://www-01.ibm.com/support/docview.wss?uid=swg1IC58202
Hope that helps (but i'd still keep and eye on the memory size when the file leaves the Java node (unless its streaming it to the file system again) |
Ok thanks I founded it too....but It's strange, because I have the fixpack 6.1.0.3 so the apar is installed....I don't know |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Tue Mar 17, 2009 8:24 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| Have you set the paramater to say 200MB (and exported it) then restarted the broker..... Just checking |
|
| Back to top |
|
|
|
|
