| Author |
Message
|
| Esa |
 Posted: Mon Feb 14, 2011 4:34 am Post subject: detecting end of files with FileInput |
 |
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Hello,
Does anyone know a simple way to detect when the input directory of FileInput node has become empty? All the ways I can think of are more or less complex.
To my surprise I realized that FileRead node does not support FTP or SFTP. There must be good reason for that but I cannot figure out what it could be. Some technical restriction? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Feb 14, 2011 5:17 am Post subject: Re: detecting end of files with FileInput |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| To my surprise I realized that FileRead node does not support FTP or SFTP. There must be good reason for that but I cannot figure out what it could be. Some technical restriction? |
Given that File Transfer Protocol only allows the transfer of an entire file, I can believe that does a specific read against a remote file would pose some challenges.
Ask yourself this: How would you perform such an action with a traditional application?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 14, 2011 6:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I would expect that FileRead will be enhanced with FTP/SFTP support at a later release. Whether that's an FP or a full product release, I wouldn't be able to guess.
Regardless, the question is "How do I detect a lack of input to a FileInput node within a certain period of time".
And a TimeoutNotification that can be reset by a TimeoutControl when a file is received seems like a reasonable solution. |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Feb 14, 2011 7:05 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
| I would expect that FileRead will be enhanced with FTP/SFTP support at a later release. Whether that's an FP or a full product release, I wouldn't be able to guess.. |
Nice.
| mqjeff wrote: |
| Regardless, the question is "How do I detect a lack of input to a FileInput node within a certain period of time". |
Actually, my question was "How do I detect a lack of input to a FileInput node without delay"
| mqjeff wrote: |
| And a TimeoutNotification that can be reset by a TimeoutControl when a file is received seems like a reasonable solution. |
At the moment, yes. If you have several hundred input files, the successive timeoutControls should override the previous, I suppose?
A lot of traffic in SYSTEM.BROKER.TIMEOUT.QUEUE? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 14, 2011 7:17 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| mqjeff wrote: |
| I would expect that FileRead will be enhanced with FTP/SFTP support at a later release. Whether that's an FP or a full product release, I wouldn't be able to guess.. |
Nice. |
Bear in mind I have no say in this. It's merely a personal expectation. The dev team frequently disappoints me.
| Esa wrote: |
| mqjeff wrote: |
| Regardless, the question is "How do I detect a lack of input to a FileInput node within a certain period of time". |
Actually, my question was "How do I detect a lack of input to a FileInput node without delay" |
You didn't mention anything about how long the delay you were willing to live with.
| Esa wrote: |
| mqjeff wrote: |
| And a TimeoutNotification that can be reset by a TimeoutControl when a file is received seems like a reasonable solution. |
At the moment, yes. If you have several hundred input files, the successive timeoutControls should override the previous, I suppose?
A lot of traffic in SYSTEM.BROKER.TIMEOUT.QUEUE? |
You didn't mention anything about "several hundred input files".  You could probably do something with a SHARED variable to determine if you should reset the Timeout in your flow to limit the number of resets.
It sounds like you really want something odd like an "Empty Directory" terminal on the FileInput node, where anytime it does an internal directory scan, and detects no files that match, it can trigger an event in the flow.
You might also be able to accomplish this using monitoring or something. Or something odd like writing an empty file to the input directory as part of the final parts of the flow, and then check to see if the FileInput has read your empty file.... |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Feb 14, 2011 7:35 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
It sounds like you really want something odd like an "Empty Directory" terminal on the FileInput node, where anytime it does an internal directory scan, and detects no files that match, it can trigger an event in the flow.
|
Yes, why not? It would be easy to implement.
On the other hand, why don't we have a "no more messages" terminal on MQInput? I think it's because it does not fit so well with the idea of messaging ans asynchronous processing. And you can always implement that kind of bath processing with MQGet. Or with a Collector node if you have a limited number of small messages.
And you could implement it with File Read node if it supported FTP...
But FileInput node actually detects this situation and it would be the most performant solution if it had that "no more files" or "empty directory" terminal.
| mqjeff wrote: |
You might also be able to accomplish this using monitoring or something. |
Which would just move all the traffic from SYSTEM.BROKER.TIMEOUT.QUEUE to some other system queue. |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Feb 14, 2011 7:41 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| mqjeff wrote: |
It sounds like you really want something odd like an "Empty Directory" terminal on the FileInput node, where anytime it does an internal directory scan, and detects no files that match, it can trigger an event in the flow.
|
Of course not that, that would be firing every time the node polls. So not an "empty directory" terminal, but "no more messages". So the terminal would fire only once immediately after the node has processed all matching files. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 14, 2011 7:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Esa wrote: |
| On the other hand, why don't we have a "no more messages" terminal on MQInput? I think it's because it does not fit so well with the idea of messaging ans asynchronous processing. |
Well you do, because if you had a batch of messages that were in some way related to each other they'd be in the same WMQ group, and the MQinput node only passes them forward when the group is complete.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 14, 2011 7:46 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| mqjeff wrote: |
It sounds like you really want something odd like an "Empty Directory" terminal on the FileInput node, where anytime it does an internal directory scan, and detects no files that match, it can trigger an event in the flow.
|
Of course not that, that would be firing every time the node polls. So not an "empty directory" terminal, but "no more messages". So the terminal would fire only once immediately after the node has processed all matching files. |
Which could still fire very very very often if Broker processes files faster than they are produced, but they are produced at a rapid rate.
This is partly why I suggested a delay... Think of the difference between short retry and long retry on a channel... |
|
| Back to top |
|
|
| Esa |
| Posted: Mon Feb 14, 2011 7:59 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| Vitor wrote: |
| Esa wrote: |
| On the other hand, why don't we have a "no more messages" terminal on MQInput? I think it's because it does not fit so well with the idea of messaging ans asynchronous processing. |
Well you do, because if you had a batch of messages that were in some way related to each other they'd be in the same WMQ group, and the MQinput node only passes them forward when the group is complete. |
I agree.
| mqjeff wrote: |
Which could still fire very very very often if Broker processes files faster than they are produced, but they are produced at a rapid rate.
|
It depends. Think of a flow that is started only after all the files are ready.
| mqjeff wrote: |
Which could still fire very very very often if Broker processes files faster than they are produced, but they are produced at a rapid rate.
This is partly why I suggested a delay... Think of the difference between short retry and long retry on a channel... |
I guess that is the way I will have to do it. I recall there is a wishlist forum somehere, I should perhaps add a wish for "no more messages" terminal. Unless it's already there. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Feb 14, 2011 8:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Esa wrote: |
| mqjeff wrote: |
Which could still fire very very very often if Broker processes files faster than they are produced, but they are produced at a rapid rate.
|
It depends. Think of a flow that is started only after all the files are ready. |
If you are producing a set of files in one batch from somewhere, add an eye catcher file or a trailer record to the last file that says "the batch is done".
| Esa wrote: |
| I guess that is the way I will have to do it. I recall there is a wishlist forum somehere, I should perhaps add a wish for "no more messages" terminal. Unless it's already there. |
I don't believe it's an existing feature.
File a requirement. |
|
| Back to top |
|
|
|
|
