| Author |
Message
|
| cwazpitt3 |
 Posted: Mon Apr 23, 2012 9:33 am Post subject: Detecting files > 100MB |
 |
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
I know there have been many discussions on here about how to handle large files, but I am wondering more about how Broker actually handles files over the size limit when read whole file. I am not looking for a workaround, just maybe an explanation or tips on how to detect that large file before it is processed or even after it is processed. Here is my situation...
I currently have a flow in production where the file size will exceed the limit only a couple times a year (for historical files). When these files drop, they get picked up by the input node and moved to mqsitransitin folder, but that's it. When this happens, I have not seen ANY record of the file not being processed. Nothing in any logs, no exception, the file will just sit there. Is there anything I am missing here? Does MB propagate to the failure terminal when a file too large is hit? What's the point of picking it up and doing absolutely nothing with it? I'd like to handle this situation with an email or error log or something so that folks know that the file needs manually moved. Hopefully I am missing something here... |
|
| Back to top |
|
 |
| Esa |
| Posted: Mon Apr 23, 2012 10:59 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Broker and OS version? Amount of RAM?
Where did you get the idea of a size limit? How much is it?
The InfoCenter does not mention any size limit at least when it comes to FileInput node. There may be an environment-dependent practical limit, though. If the eg cannot allocate enough memory, for example. In that case I would expect an entry in the logs telling that the eg has been restarted.
If a file is too large to be processed, I suppose it is too large to be propagated to failure terminal, too. Will the large file block the flow?
What do user trace or service trace say? It the node is smart enough not to try to process a file too large, it will most likely boast it in the trace. |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Tue Apr 24, 2012 11:16 am Post subject: |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
@Esa
We are running MB 7.0.0.2, I am not sure what OS and how much RAM because its a corporate environment, but these things should not be important as the file size limit is set as an environment variable (see http://www-01.ibm.com/support/docview.wss?uid=swg1IC58202).
Your point about the failure terminal is valid. It probably completely stops propagation at the point that it realizes the file is too big. I just wish there was a way to detect when that happens so as to be able to react to it via a Java node or shell script that could push the file along. In my case, all I need to do is FTP the file outbound via a FileOutput node, which would be entirely possible if the FileInput node didn't just crash completely. I don't think it "blocks the flow" per-say, it just keeps that file in mqsitransitin and I think (not confirmed) that any other files dropped with that name will be ignored until the blocked file is handled.
Its kind of tough around here to do a user trace or service trace and given that it won't help me too much, I will let you know if/when I decide to try it. Thanks for the response. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Apr 24, 2012 1:08 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
If all you need to do is to ftp the file out with a FileOutput node, why don't you configure the FileInput node to process the file in Fixed size chunks and run the FileOutput node in append mode. You would also get remarkably better performance. To start the ftp transfer you will have to wire FileInput nodes 'End of data' terminal to FileOutput nodes 'Finish file'.
This is the way its usually done. No size limits! |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Fri Apr 27, 2012 11:08 am Post subject: |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| Esa wrote: |
If all you need to do is to ftp the file out with a FileOutput node, why don't you configure the FileInput node to process the file in Fixed size chunks and run the FileOutput node in append mode. You would also get remarkably better performance. To start the ftp transfer you will have to wire FileInput nodes 'End of data' terminal to FileOutput nodes 'Finish file'.
This is the way its usually done. No size limits! |
Good point. I have flows that utilize this by do it line by line (delimiter based). Didn't think to use the block length or fixed size. What is a reasonable size? If you make it higher, does it eat more memory? Since I am not doing anything with these blocks of data, I am tempted to set the size at like 1MB or 1024 (something like that). What do you think? |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Apr 30, 2012 2:15 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
I was interested in the link provided by you, it was helpful for me. Thanks. But I would like to know two things here,
1. The link says to set the Evironment Variable "MQSI_FILENODES_MAXIMUM_RECORD_LENGTH" to override the length. Where do I need to set this environement?
2. What is the initial size limit of this FIle Input Node? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Apr 30, 2012 4:22 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| kash3338 wrote: |
| 1. The link says to set the Evironment Variable "MQSI_FILENODES_MAXIMUM_RECORD_LENGTH" to override the length. Where do I need to set this environement? |
In the shell that executes mqsistart.
If you don't know what that means, tell your Broker administrator and let them do it.
| kash3338 wrote: |
| 2. What is the initial size limit of this FIle Input Node? |
If it's not in that technote, then open a sev4 pmr to ask for the technote to be clarified. |
|
| Back to top |
|
|
| cwazpitt3 |
| Posted: Mon Apr 30, 2012 4:44 am Post subject: |
|
|
Acolyte
Joined: 31 Aug 2011
Posts: 61
|
| kash3338 wrote: |
I was interested in the link provided by you, it was helpful for me. Thanks. But I would like to know two things here,
1. The link says to set the Evironment Variable "MQSI_FILENODES_MAXIMUM_RECORD_LENGTH" to override the length. Where do I need to set this environement?
2. What is the initial size limit of this FIle Input Node? |
1. I am not sure how to set the environment variable, I have never tried it out. Might want to search google, you will likely find a way.
2. I believe the initial setting is 100MB, at least that is what I have always seen in my flows and I believe it is documented also. |
|
| Back to top |
|
|
|
|
