| Author |
Message
|
| Bichu |
 Posted: Thu Sep 08, 2016 5:48 am Post subject: Start and stop message flows in definite intervals |
 |
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Hi Guys,
I need to monitor a remote directory for files via FTP protocol once in a day and hence I cant work with Scan delay property. I tried searching for starting and stopping a message flow at a particular time or for a particular period, but no luck. I am left with only the Timer node option. Do you guys have any other options.
I am working in IIB v9.0.0.1 and MQ v7.5. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Sep 08, 2016 6:04 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You don't need to look for files once a day. You need to look for files when they are ready to be processed.
Files are only ready to be processed when they are completely written.
The program sending the files needs to make sure that they only have the right names or are in the right directory only when they are ready to be processed.
If you check the directory once a day, then you can't guarantee that all of the files are ready to be processed.
If you really really really need to make sure that you only look for the file once a day, you can either write a script that will start and then stop the flow, and use an OS scheduler to run this script. Or you can use something other than a FileInput node to start the flow, and use the FileRead node.
But no matter what, this is a terrible design and open to a lot of problems all the time, and almost always the fault of the sender. So you will get paged in production for all of their failures. And will have to prove, every time, that it is their failure and not yours.
This is a terrible design.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 08, 2016 6:13 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Many Thanks Jeff.
I have told my clients to have an MQ client at their end so as to process files as messages. But they prefer FTP with security token option to get files since they are already having FTP client and dont want to install MQ client on their end. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 08, 2016 6:40 am Post subject: Re: Start and stop message flows in definite intervals |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| Do you guys have any other options. |
I echo my most worthy associate. Say (by some method) you start the flow at 3am to look for the file; how do you know the file has been fully written? If you employ some locking strategy to prevent IIB reading a partially completed file, how long does IIB wait - presumably it can't wait indefinitely and must at some point consider the file "not arrived today".
The same problem comes with stopping the flow. You don't want to stop it too soon (i.e. while it's still processing) but you must stop it before the next file arrives and from the requirement, I would not be surprised to discover that they start writing the next file quite soon, and just keep appending to it through the business day (see above re: reading incomplete files).
This is a can of worms.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 08, 2016 6:47 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks Vitor and Jeff.
But I wonder how IIB can work with File nodes then. Is MQ the only alternative for this requirement. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 08, 2016 6:54 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Explain to them the difficulties with file transfer.
If they explain that they've done this already with lots of companies/partners, then tell them you will do things this way, but any errors in the process are their responsibility first.
You can set the Polling Interval on the FileInput node to run once an hour or once every five hours, to reduce the number of times you check for files.
In theory, you could set it to poll for 24 hours in seconds... 24*60*60 or sommat.
But again. You're in for a lot of problems.
The FileInput node moves files it's processing to another directory. The sender of files needs to take steps to ensure that files are not able to be read before they're done being written. For example. set the FileInput node to read .txt files and have the sender write files to .tmp until it's ready and then rename it to .txt
But the worms, the can, and you.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 08, 2016 7:08 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks Jeff.
Adding to this, once I fetch the files, I need to SFTP the file to another remote directory. Will it be fine if I go with Record is Unmodified Data option in Records and Elements tab in File Output Node. Bcz my data size is around 40 MB. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 08, 2016 7:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| But I wonder how IIB can work with File nodes then. Is MQ the only alternative for this requirement. |
No. We use files extensively (with IIB and with other products) but we have a raft of stuff to mitigate these problems. Crucially, we don't allow IIB to poll for files. We mandate that when files are ready to be processed, they're pushed to IIB.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 08, 2016 7:16 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks Vitor.
Thats clear to me. But how can we still assure file is available even if you mandate the files to be pushed to IIB.
Lets assume that IIB polled the directory when a file was copying(pushed), wont the same issue occur. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 08, 2016 7:18 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Yes. If the sender of a file doesn't make sure that it's ready to be processed - finished writing - in *some* way, then *no* program that tries to read the file can do so without a large chance of error.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 08, 2016 7:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| Bcz my data size is around 40 MB. |
That's not a very big file.
If all you're using IIB for is to pull a file from one location and push it, unmodified, to another location then you're using a very expensive sledgehammer to crack a very small nut. There are easier and cheaper ways to move files around in a scheduled fashion.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Sep 08, 2016 7:29 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Bichu wrote: |
| Lets assume that IIB polled the directory when a file was copying(pushed), wont the same issue occur. |
No, because we have had to build a locking strategy to prevent that. The difference with your scenario is we have the flow running constantly so as soon as the lock is released by the pushing application we know it's complete. Likewise we know that any file that's been pushed is, by definition, ready to be processed. So we don't need to stop or start the flows, or worry about the state.
Before my most worthy associate comments:
- yes, we have no guarantee that the file, as pushed, is actually complete and has not been truncated in transit
- yes, we have no guarantee that the file really was complete in a business sense before being pushed
- yes, we've pushed the whole file problem upstream rather than solving it
I'm responsible for the IIB estate, not the upstream systems. Sympathetic as I am to their problems and issues, I'm fine and thanks for asking.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 08, 2016 7:51 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks Jeff & Vitor for all your explanaitons.
| Vitor wrote: |
The difference with your scenario is we have the flow running constantly so as soon as the lock is released by the pushing application we know it's complete. Likewise we know that any file that's been pushed is, by definition, ready to be processed. So we don't need to stop or start the flows, or worry about the state.
|
As jeff mentioned earlier and now you, If I specify the polling interval as 1 hour(or in hours), the problem of message flow state will be resolved. But will it create memory problems/leaks. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 08, 2016 7:54 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The fileInput node will basically sleep until the interval has expired. shouldn't cause any real issues.
But you really need to talk to the upstream provider about all of the issues we've discussed, and make sure that either they've handled it or accepted that any problems with files transferred are their problems and not yours.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 08, 2016 7:56 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
| Many Thanks Jeff and Vitor. I am pretty clear now. |
|
| Back to top |
|
|
|
|
