| Author |
Message
|
| Bichu |
 Posted: Mon Oct 17, 2016 4:04 am Post subject: maximum value of polling interval for FileInput node |
 |
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Guys,
Does anyone know what is the maximum value that we can give for the parameter Polling Interval for File Input node. I googled a lot and couldn't find any where it as mentioned.
Thanks in advance. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Oct 17, 2016 4:28 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Given that it's an integer, I would put it to you that the maximum value is the maximum value of an integer as defined in ESQL.
I further put it to you that the maximum is not explicitly mentioned in the FileInput documentation because no reasonable design would have a polling interval even slightly approaching that value.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Mon Oct 17, 2016 6:04 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks Vitor.
I need to fetch a file once in a week and hence I come out with the value of 604800. I am also thinking of a cron or control-m job to start/stop message flow once in a week. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Oct 17, 2016 6:10 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You don't need to fetch a file once a week.
How often is the EG going to be restarted during that week?
You need to fetch a file when it's ready.
You should use some other method than the FileInput node and a long running polling interval.
Perhaps you can do something with MQ to put a message on a queue when it's been a week. Then you can use the FileRead node.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 17, 2016 7:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| I need to fetch a file once in a week and hence I come out with the value of 604800. |
And if by some chance the broker or EG is restarted in that week, that will restart / reset the timer.
| Bichu wrote: |
| I am also thinking of a cron or control-m job to start/stop message flow once in a week. |
How long are you going to let the flow run? How do you know the file will be processed in that time frame? How do you know another file won't arrive in that frame.
I agree with my most worthy associate. You don't want to fetch a file once a week; you want to fetch the file when it's ready to be processed. I would put it to you that the better solution is to have the file pushed to IIB when it's ready to be processed by the upstream process.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Oct 17, 2016 7:21 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Yes. Make it event driven.
Whatever creates the file should initiate a file transfer into the broker.
Most scheduling packages include a file watcher agent if you want this to be done by your scheduler software.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Bichu |
| Posted: Mon Oct 17, 2016 7:34 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
I totally agree with you all...In that case, I think the cron should trigger a message only if a file exists in the particular directory.
| Quote: |
| I would put it to you that the better solution is to have the file pushed to IIB when it's ready to be processed by the upstream process. |
Also, May i know what exactly you meant by file pushed to IIB. I know using fileinput/read node, IIB can pull files, but how files can be pushed to IIB.
Sorry for my ignorance.
[/quote] |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Oct 17, 2016 7:39 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
The fileinput node can scan a local directory. When the file is pushed to the IIB host, it will appear in this directory (assuming it is sent there).
Local scans can be much more frequent without wasting resources, however the default scan rate is still too frequent for a weekly transfer.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 17, 2016 7:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| Also, May i know what exactly you meant by file pushed to IIB. I know using fileinput/read node, IIB can pull files, but how files can be pushed to IIB. |
Two possible use cases, both alluded to by my worthy associate and which I endorse :
- a scheduled task / file watcher notices the file has been placed in the remote directory and moves it to the IIB local file system, where the FileInput node processes it
- the process which produces the remote file, once it decides the file is complete, transfers the file to the IIB local file system.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Oct 17, 2016 8:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
In the interests of full disclosure, the latter method is the one I've implemented here. Trying to work out when a remote file is "complete" is a PITA; the people best positioned to know that are the people writing it. It's a trivial task for them, having finished with the file, to initiate the transfer.
This is especially useful for files which accumulate through the business day (or week). Whenever it polls, IIB would be able to see a file and could pull it. There's no way for IIB to know if all of the records for a business day (or week) are present.
We've also had numerous incidents where the file which is available "by 6pm Friday without fail, and must be processed by 7pm" hasn't arrived by 6:30pm due to "upstream holdups". In each case we were told we could pull the file with confidence at the scheduled time "because the file is never late". Each time I insisted they push rather than we pull, and each time I pointed out that if we'd listened to them the business process would not have the complete data.
On one notable occasion I was told that it wasn't a problem "because we've developed a manual reconciliation process to handle when the file's pulled too soon by the existing system". I pointed out that we'd been told it was never late, to the response "oh yes, it's a key SLA to have the file ready by 6pm Friday. We hit it almost 70% of the time"
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Oct 17, 2016 5:14 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Vitor wrote: |
- the process which produces the remote file, once it decides the file is complete, transfers the file to the IIB local file system. |
How does IIB know the file transfer to the local file system is complete?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Oct 17, 2016 11:08 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
IIB uses file locking.
However the standard FTP/SFTP does not, so yes the processing of partial files is a risk, one that can be reduced by increasing the scan interval.
The same risk is present if files are pulled from the source system.
In general - don't use files if you want transactional integrity.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Oct 18, 2016 4:38 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| PeterPotkay wrote: |
| Vitor wrote: |
- the process which produces the remote file, once it decides the file is complete, transfers the file to the IIB local file system. |
How does IIB know the file transfer to the local file system is complete? |
Transfer to a staging area on the target, and use a post-transfer step to move it to the directory IIB is watching.
Yes it's complex with a lot of moving parts. Cost of using files not MQ/web services/etc.
Sucks to be the business users.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Tue Oct 18, 2016 6:54 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
We like to have the sending side send the file with an initial name starting with a period. This hides the file from a normal scan on a *nix system. And then when their transfer is complete they rename the file to remove the period. The renaming operation (ie mv) takes milliseconds (as long as you don't rename the file to another disk.
We've never had a problem processing an incomplete file using this method. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Oct 18, 2016 7:52 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Rename after transfer is one way around this, and you can set the fileinput node to only scan for the renamed files.
Then you only have to worry about missed files, files processed out of sequence, codepage issues, files part processed when failures occur in the flow and all other wonderful issues that using a transactionally unsafe protocol entails.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
|
|
