| Author |
Message
|
| francoisvdm |
 Posted: Wed Jun 15, 2011 2:29 am Post subject: Interrupt File Input node |
 |
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
Is it possible to interrupt the File Input node to stop reading a file after it red x records from it? In other words, I'm only interested in the first 10 records, but I do not need the rest of the file.
Thank you
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Jun 15, 2011 2:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
technically possible? yes.
A good idea? no.
Possible without doing things outside of normal broker operations - that is, without manipulating the file while broker is still trying to read it - no.
You *can* do this, by having the code that processes the 10th record turn around and cause something to move or rename the file. But it's not a good idea.
You'll have to evaluate whether the circumstances of doing that are better or worse than reading the whole file and skipping parts of it. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Wed Jun 15, 2011 2:39 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
Thanks, I do agree that is not a good option, I was hoping for maybe changing an environment var or something to force an end of file situation....
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 2:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If there were such an environment variable, it would be undocumented, and thus not available without opening a PMR.
I do not know one way or another whether such a "stop processing this file" switch exists. I doubt that it does, but that should not be taken as proof of anything.
I would look at either building a message definition that allowed you to treat all but the first ten records as a single blob field, or at using fixed length records and setting up a very fast method to end the flow instance if it's not the first 10. Something maybe that checks if a counter is greater than 0 and otherwise returns false. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jun 15, 2011 3:31 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| Is it possible to interrupt the File Input node to stop reading a file after it red x records from it? |
I think there's a fairly simple solution. Use 'Parsed Record Sequence' and define a 'Record' as x occurrences of the smaller 'record'. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Wed Jun 15, 2011 3:42 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
OK, not sure how that will work.... my incoming file is a blob, I need to identify the file depending on the first 6 characters in the file. Then, depending on the identification, I might need to read the file again, this time breaking it up into one or more records.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 3:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| kimbert wrote: |
| Quote: |
| Is it possible to interrupt the File Input node to stop reading a file after it red x records from it? |
I think there's a fairly simple solution. Use 'Parsed Record Sequence' and define a 'Record' as x occurrences of the smaller 'record'. |
That would read the entire file in blocks of X records.
It wouldn't read the first X records, and then stop.
| francoisvdm wrote: |
| OK, not sure how that will work.... my incoming file is a blob, I need to identify the file depending on the first 6 characters in the file. Then, depending on the identification, I might need to read the file again, this time breaking it up into one or more records. |
I would build a TDS model, and use the first 6 characters as a Tag or a Message Identifier or Key to identify the message type.
Then I would use the rest of the model to determine if I was breaking up the message into further records or not.
But that assumes you expect to read the whole file in general, which isn't what you said.
Otherwise I would build a straight forward MRM model that had two fields - the first 6 bytes and then something that covered the rest of the file as a BLOB. Then you could make your own determinations on what to do with the BLOB. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Jun 15, 2011 3:54 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| I think it's time for a complete description of the requirements. Over to you, francoisvdm. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Wed Jun 15, 2011 4:02 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
A file type gets identified by the first 6 bytes in the file. Depending on that I want to consume the file in different patterns, sometime not at all, or sometimes as fixed length records or parsed as an XML file.
The files can be VERY big, maximum at the moment 1.4Gb. Sometimes these big files need no processing in the broker, but it can only be determined after looking at the first 6 bytes. Hence my requirement not to burden the broker with a lot of work if it can be avoided.
Maybe the File Read node can play a role here?
Thanks for the effort all.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 4:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So if the file is NOT supposed to be "processed" by broker, what is broker expected to do with it?
Leave it where it was?
Move it or rename it somewhere else? |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Wed Jun 15, 2011 4:12 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
If not... hand it over to MQ FTE
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jun 15, 2011 4:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| francoisvdm wrote: |
| If not... hand it over to MQ FTE |
So in either way, Broker is expected to load the entire contents of the file and send it through one or more nodes.
It's just that in some cases, broker does more than that.
Is that correct? |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Jun 16, 2011 1:43 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
So you have several different types of input file, but they all arrive in the same directory. The type is identified in the first 6 bytes.
Sounds as if you would be better off with a small routing flow that inspects the first 6 bytes and then routes the file to another flow. Then you could design one message flow per file format, which sounds a lot more maintainable than putting it all into one flow.
A simple CWF message set with two fields would do the inspect-and-route operation. First field is 6 bytes long, second has length set to 'End of bitstream'. On-demand parsing prevents the second ( huge) field from being read at all. |
|
| Back to top |
|
|
| francoisvdm |
| Posted: Thu Jun 16, 2011 10:09 am Post subject: |
|
|
Partisan
Joined: 09 Aug 2001
Posts: 332
|
Sorry, public holiday here today, hence my slow reply.... thanks for replies, will try suggestion in the next week.
_________________
If you do not know the answer or you get the urge to answer with "RTFM" or "Search better in this forum", please refrain from doing so, just move on to the next question. Much appreciated.
Francois van der Merwe |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jun 16, 2011 10:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Just to provide some clarification
| kimbert wrote: |
| A simple CWF message set with two fields would do the inspect-and-route operation. First field is 6 bytes long, second has length set to 'End of bitstream'. On-demand parsing prevents the second ( huge) field from being read at all. |
On-demand parsing will prevent Broker from loading the entire contents of the huge file into memory at once. However, the FileInput and FileOutput nodes will use streaming parsing to move the entire contents of the file from one location to another, or if you pass the data to an MQFTE node or etc, then the contents will be dealt with in a manner consistent with the needs of that output node.
So it's not that using this method will only process the first 6 bytes of the file. It's just that it will only load the first 6 bytes into active memory and otherwise handle the file data efficiently. |
|
| Back to top |
|
|
|
|
