| Author |
Message
|
| ldtluxury |
 Posted: Wed Jul 29, 2015 9:19 am Post subject: FileInput node problem |
 |
|
Apprentice
Joined: 22 Oct 2014
Posts: 29
|
Hello everyone.
Could you please assist with this problem:
In the flow I have FileInput node which reads files from SFTP server. But SFTP server has special feature: it deletes file when it was downloaded.
Here's the problem: FileInput node throws an exception whilst deleting downloaded file because there is no more file on SFTP server.
Is it possible to switch off deleting file in FileInput node or ignore this exception (to continue work of the flow)?
Thanks a lot. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jul 29, 2015 9:24 am Post subject: Re: FileInput node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ldtluxury wrote: |
| But SFTP server has special feature: it deletes file when it was downloaded. |
To be clear - you mean that the SFTP server which is the source for the file detects that the file has been downloaded and deletes it, independently of any activity on the broker side?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ldtluxury |
| Posted: Wed Jul 29, 2015 10:17 am Post subject: Re: FileInput node problem |
|
|
Apprentice
Joined: 22 Oct 2014
Posts: 29
|
| Vitor wrote: |
| ldtluxury wrote: |
| But SFTP server has special feature: it deletes file when it was downloaded. |
To be clear - you mean that the SFTP server which is the source for the file detects that the file has been downloaded and deletes it, independently of any activity on the broker side? |
Yes. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 29, 2015 10:27 am Post subject: Re: FileInput node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ldtluxury wrote: |
| Vitor wrote: |
| ldtluxury wrote: |
| But SFTP server has special feature: it deletes file when it was downloaded. |
To be clear - you mean that the SFTP server which is the source for the file detects that the file has been downloaded and deletes it, independently of any activity on the broker side? |
Yes. |
I think you're hosed. AFAIK that behavior is not configurable, and the error occurs before control is passed to the flow, meaning you can't trap it. It's worth testing that last and see if anything comes out of the Failure terminal of the FileInput node that enables you to continue your processing.
But IMHO your best bet is to tell the SFTP not to do that. It can't possibly be sure (given the nature of FTP) that the download was successful; even if it knows it's transmitted all of the file contents that's no guarantee they arrived.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ldtluxury |
| Posted: Wed Jul 29, 2015 10:37 am Post subject: Re: FileInput node problem |
|
|
Apprentice
Joined: 22 Oct 2014
Posts: 29
|
| Vitor wrote: |
| ldtluxury wrote: |
| Vitor wrote: |
| ldtluxury wrote: |
| But SFTP server has special feature: it deletes file when it was downloaded. |
To be clear - you mean that the SFTP server which is the source for the file detects that the file has been downloaded and deletes it, independently of any activity on the broker side? |
Yes. |
I think you're hosed. AFAIK that behavior is not configurable, and the error occurs before control is passed to the flow, meaning you can't trap it. It's worth testing that last and see if anything comes out of the Failure terminal of the FileInput node that enables you to continue your processing.
But IMHO your best bet is to tell the SFTP not to do that. It can't possibly be sure (given the nature of FTP) that the download was successful; even if it knows it's transmitted all of the file contents that's no guarantee they arrived. |
Yep, there is nothing in failure terminal of the InputFile node.
And there is no possibility to switch off deleting file on SFTP (it's special SFTP server which works as a gateway)
Also I tried to use FileRead node, but there I must to specify the file name, but I don't know the exact file name, and there are lots of files.
Could you advice something, please  |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 29, 2015 10:57 am Post subject: Re: FileInput node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ldtluxury wrote: |
| Could you advice something, please |
Don't pull the file; get them to push it and read it with a FileInput node from the local system when it arrives.
Obviously if the transfer fails or it's a large file and you start reading it before the file is complete, this will result in a problem. It's up to you to make sure that, politically, it's not your problem because you were forced into this position by their unreasonable behavior. 
Or you could find another job and let the next poor fool figure this out.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ldtluxury |
| Posted: Wed Jul 29, 2015 11:04 am Post subject: Re: FileInput node problem |
|
|
Apprentice
Joined: 22 Oct 2014
Posts: 29
|
| Vitor wrote: |
| Don't pull the file; get them to push. |
How to do that? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 29, 2015 11:30 am Post subject: Re: FileInput node problem |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| ldtluxury wrote: |
| Vitor wrote: |
| Don't pull the file; get them to push. |
How to do that? |
Convince them to do the FTP instead of you, and then use the FileInput node against a local directory. |
|
| Back to top |
|
|
| ldtluxury |
| Posted: Wed Jul 29, 2015 11:38 am Post subject: Re: FileInput node problem |
|
|
Apprentice
Joined: 22 Oct 2014
Posts: 29
|
| mqjeff wrote: |
| ldtluxury wrote: |
| Vitor wrote: |
| Don't pull the file; get them to push. |
How to do that? |
Convince them to do the FTP instead of you, and then use the FileInput node against a local directory. |
Well, there is another problem: usage of local directories is prohibited according to security policy.
So, the result is that it's not possible to solve that problem in Integration Bus? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 29, 2015 11:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The result is that the gateway SFTP server behaves badly.
You can open a PMR and ask IBM to provide a "take no action" option on the finish file.
But you should also be aware that even when using FTP to read files, IIB copies them locally before it hands them to the FileInput node.
You should also discuss the business process of this file - to see if it counts as business critical - and then determine if that means that you should be using a managed file transfer tool rather than an integration tool to move the file into your organization. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 29, 2015 11:58 am Post subject: Re: FileInput node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ldtluxury wrote: |
| Well, there is another problem: usage of local directories is prohibited according to security policy. |
Then you can't use the FileInput node; when it performs the SFTP it stores the file on the local file system while it deletes it from the remote, then stages it in another directory while it's processing it.
| ldtluxury wrote: |
| So, the result is that it's not possible to solve that problem in Integration Bus? |
It's not possible, using that security policy, to solve it at all with any technology. Any SFTP from that server will result in a file on the local file system in contravention of the security policy. FTP is not a streaming protocol and the file will always traverse the disc. The only way to ensure the file never hits the disc is to send it via HTTP or other streaming protocol, which has it's own problems for a file of any size.
You might want to ask other users of this server how they solve the problem, and what they do with the file when they pull it from this SFTP gateway while they're waiting for the transfer to finish. Indeed if the processing is record based, what do they do with the part of the file they haven't got to yet?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 29, 2015 12:12 pm Post subject: Re: FileInput node problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Vitor wrote: |
It's not possible, using that security policy, to solve it at all with any technology. Any SFTP from that server will result in a file on the local file system in contravention of the security policy. FTP is not a streaming protocol and the file will always traverse the disc. The only way to ensure the file never hits the disc is to send it via HTTP or other streaming protocol, which has it's own problems for a file of any size.
You might want to ask other users of this server how they solve the problem, and what they do with the file when they pull it from this SFTP gateway while they're waiting for the transfer to finish. Indeed if the processing is record based, what do they do with the part of the file they haven't got to yet? |
To be clear, I accept unreservedly that this isn't a policy you came up with and you're just trying to implement it.
But it sounds like the kind of sweeping policy management comes up with, and that everyone on the ground goes, "yeah, yeah, sure" and gets an exception from. So you may be trying to adhere to a policy you don't need to.
War story: back in the day, an edict came down:
"128 bit SSL is no longer sufficiently secure. All SSL must be 256 bits or longer, by the end of this month"
Very few applications had managed to move by the end of that month. By the end of the year, all the 3rd party applications that couldn't support 256 bit SSL were still on 128 bit. A year later, the same management team that wanted 256 bit SSL were signing off on exceptions because they wouldn't pay the upgrade fees so we could get the version that did support 256 bit.
In the same 12 months, we had edicts on use of indexes in databases, use of persistence in WMQ (!) and Java naming standards. All implemented with equal efficiency.
Processing files without ever storing them on the local file system? Sounds great in a management white board session I'd imagine.....
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
