| Author |
Message
|
| paustin_ours |
 Posted: Fri May 20, 2016 4:43 am Post subject: timed file transfer |
 |
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
I am looking at fileinput node to do a sftpget from remote server. this works fine. What we want to now do is only get files are specific times of the day and send it to another application.
i know you guys hate using IIB for this and would rather want us to use MFT solutions but we dont have control over the remote site and cant have them install any agents.
i see fileread dont have a sftp so am not sure how i can make this work on specific times of the day. Please share your thoughts. thanks in advance. |
|
| Back to top |
|
 |
| mayheminMQ |
| Posted: Fri May 20, 2016 4:50 am Post subject: |
|
|
Voyager
Joined: 04 Sep 2012
Posts: 77
Location: UK beyond the meadows of RocknRoll
|
I have to ask the version here as IIB 9 and 10's FileRead does have sftp option.
If that is present, you can always use TimerControl and TimerNotification set of nodes to kickstart FileRead and go on from there.
_________________
A Colorblind man may appear disadvantaged but he always sees more than just colors... |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 20, 2016 4:57 am Post subject: Re: timed file transfer |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| paustin_ours wrote: |
| i know you guys hate using IIB for this |
Because it's a PITA to administer.
| paustin_ours wrote: |
| Please share your thoughts |
Tell the remote side to do an sftp put rather than you do an sftp get.
After that, we're in the world of hateful. It's easy enough (technologically) to set up a scheduled job that starts the flow with the FileInput node at the time it has to do a get, and another to stop it when processing is complete. You then simply have to face the laundry list of non-technical problems, including (but not limited to):
- how you handle the situation where the file isn't ready on the remote end at the time appointed
- how you handle the situation where the file is much larger than you expect so the "stop" schedule goes off before the flow has finished processing
- how you handle the situation where the remote side starts writing a new file before the flow has reached it's scheduled stop (because you increased the window to cater for the situation above)
- how you cater for the situation where the file transfer fails (because ftp sucks) and you need to retry outside the schedule
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 20, 2016 5:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mayheminMQ wrote: |
| If that is present, you can always use TimerControl and TimerNotification set of nodes to kickstart FileRead and go on from there. |
Accepting that the Timer nodes are timers not schedulers. If any broker components are restarted, the Timer will restart from the original value so (for instance) if you want the file read every hour on the hour (and set a time value of 60 minutes to achieve this) then any of the following scenarios will cause a problem:
- an mqsireload is issued not at the top of the hour (i.e. at 10:06)
- a new version of the flow is deployed not at the top of the hour
- the broker is restarted (for an mqsisetdbparms not related to this flow, or indeed OS maintenance not related to the broker) not at the top of the hour
The timer nodes are designed to allow a flow to wait for a period before retrying an action. They're not designed to synchronize with a wall clock, and don't work that well for that use case.
Note the OP said "specific times of the day" not "every 60 minutes". Hence my comment.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri May 20, 2016 6:02 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
- an mqsireload is issued not at the top of the hour (i.e. at 10:06)
- a new version of the flow is deployed not at the top of the hour
- the broker is restarted (for an mqsisetdbparms not related to this flow, or indeed OS maintenance not related to the broker) not at the top of the hour
|
Actually, you can get a repeat timer to fire at a designed time thus negating the need to to things at a set time such as an EG restart etc.
You work out how long it is until the first correct time to fire and then code that into the timer node code.
We do that for our general scheduler flow. This runs once a minute and sends out all sorts of timed messages to different flows according to a schedule (and queue info) that is held in a DB. Thankfully, we use a common 'Timer' message format.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 20, 2016 6:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
I'll stick to telling people to use cron or CA7.
That kind of coding is not going to be done correctly by every developer in every instance, certainly not in my environment, though I accept the design is sound.
I stand by my previous comments on running this kind of solution through IIB. Even if you use a FileRead node.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri May 20, 2016 11:52 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
I'll stick to telling people to use cron or CA7.
|
I would too but finding a decent 'cron' for windows is not easy. There are ones out there but so far I have not found one that meets our requirements.
So for all flow triggering we use this flow. Adding a new trigger is as simple as adding a row to a table.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
