| Author |
Message
|
| paustin_ours |
 Posted: Fri May 06, 2016 7:33 pm Post subject: file transfer |
 |
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
We are looking into a solution where files coming on one server needs to be transferred to another server. The plan is to use the broker to listen on in the input directory and once a file comes in, pick it up and do a sftp to the destination server. we are doing sftp within java compute node calling jsch jar.
Does this sound like a good use of the broker? |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri May 06, 2016 7:42 pm Post subject: Re: file transfer |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| paustin_ours wrote: |
We are looking into a solution where files coming on one server needs to be transferred to another server. The plan is to use the broker to listen on in the input directory and once a file comes in, pick it up and do a sftp to the destination server. we are doing sftp within java compute node calling jsch jar.
Does this sound like a good use of the broker? |
Not the right thing to do. If you are using broker (transforming the input in any way) you should not be doing an sftp from within a JCN but calling the File Output node, and letting it handle the sftp. This way you will be able to transfer huge files if needed. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Sun May 08, 2016 4:35 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| what about broker being a router of files? should broker be in the business of routing files? |
|
| Back to top |
|
|
| vennela |
| Posted: Mon May 09, 2016 12:44 am Post subject: |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| Yes, intelligent routing is one of the best usages of broker |
|
| Back to top |
|
|
| zpat |
| Posted: Mon May 09, 2016 1:17 am Post subject: Re: file transfer |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| paustin_ours wrote: |
| we are doing sftp within java compute node calling jsch jar. |
Why would you even consider calling a jar file? The whole point of the broker is to provide everything built-in (other than in truly exceptional cirumstances).
The file output node (and you should use a FTP configurable service to hold the details - this can use SFTP) does everything you need.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon May 09, 2016 6:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Are you doing anything with the files other than moving them?
Are you using any information in them to figure out where they go? Are you doing anything to change them at all?
Are the files very large?
If all the answers to these questions are No, except the last one which is Yes, then you should consider something other than Broker. Because you aren't doing any function in Broker that can't be handled better, faster, and easier in a real file transfer solution.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| sumit |
| Posted: Mon May 09, 2016 9:21 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
I have seen one project using IIB *only* to sftp files from one folder to another. The file size was large and they faced a lot of issues.
If the intention is only to ftp a file, IIB is not the tool.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Mon May 09, 2016 1:31 pm Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
we have a job that kicks of a trigger message and puts the message on a queue. This queue is read by a message flow. The message flow does a sftpget of files from a remote server and it all files are gotten it commits if not the flow sleeps and puts the message back to the input queue. then it tires ths sftget again until all files are gotten or a time of the say has reached.
we do sftp via jsch within the JCN. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue May 10, 2016 1:34 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
What is the requirements ?
JUST COPY the files from Source (A) to destination (B).
Any operating system has better tool to copy files rather then using IIB/Broker ...
Please elaborate on your requirements 
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Tue May 10, 2016 1:50 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
requirement is to get files from source server and send to destination server. Source side can make the file available anytime. we poll all day or a specific amount of time. we do this by doing a sftp ls via the broker JCN calling jsch. then when we see files we sftp to broker server, then sftp again to destination server.
that is the requirement. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue May 10, 2016 2:06 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| paustin_ours wrote: |
requirement is to get files from source server and send to destination server. Source side can make the file available anytime. we poll all day or a specific amount of time. we do this by doing a sftp ls via the broker JCN calling jsch. then when we see files we sftp to broker server, then sftp again to destination server.
that is the requirement. |
You are doing simple file copy operations. I think broker is not the right tool for this job 
_________________
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 10, 2016 5:34 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ruimadaleno wrote: |
| paustin_ours wrote: |
requirement is to get files from source server and send to destination server. Source side can make the file available anytime. we poll all day or a specific amount of time. we do this by doing a sftp ls via the broker JCN calling jsch. then when we see files we sftp to broker server, then sftp again to destination server.
that is the requirement. |
You are doing simple file copy operations. I think broker is not the right tool for this job |
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Tue May 10, 2016 6:58 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| we don't want to write one of sftp applications either. We want this solution to be reusable, configurable and the destination de-coupled as in can we looked up and then routed to. how could we achieve this is we have sftp applications/scripts running on the server. |
|
| Back to top |
|
|
| timber |
| Posted: Tue May 10, 2016 7:10 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
I don't think an SFTP tool + custom scripts is the correct answer either. But there are some great 'managed file transfer' tools out there, including one from IBM that works really nicely with MQ and IIB:
http://www-03.ibm.com/software/products/en/wmq-fte |
|
| Back to top |
|
|
| paustin_ours |
| Posted: Tue May 10, 2016 7:38 am Post subject: |
|
|
Yatiri
Joined: 19 May 2004
Posts: 667
Location: columbus,oh
|
| I have been pushing for that as well but wanted to know if there are other options. thanks for the input. |
|
| Back to top |
|
|
|
|
