| Author |
Message
|
| sranjoy |
 Posted: Thu Dec 15, 2016 2:49 am Post subject: file input node is picking up the file before the copy is co |
 |
|
Newbie
Joined: 15 Dec 2016
Posts: 3
|
Hi Team,
Environment : IIB9 broker on Windows
SFTP server is on unix.
We have requirement to process files via file input node.I am facing an issue when business is trying to copy a large file (50 MB) in input file path.File input node is picking up the file(zero byte file) before the copy is complete.Third party can change the file filename after copy is successfully complete or any other process .i have discussed with them .
Can we do any thing in IIB to resolve the issue ?please suggest |
|
| Back to top |
|
 |
| timber |
| Posted: Thu Dec 15, 2016 4:40 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Quote: |
| Third party can change the file filename after copy is successfully complete or any other process .i have discussed with them . |
That sounds like a viable solution to me.
btw, if you use the Search button you will find many other questions similar to yours. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Dec 15, 2016 5:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
|
| sranjoy |
| Posted: Thu Dec 15, 2016 5:23 am Post subject: |
|
|
Newbie
Joined: 15 Dec 2016
Posts: 3
|
| Quote: |
btw, if you use the Search button you will find many other questions similar to yours. |
I found some but of different scenerios.
I can only perform any action in IIB only. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 15, 2016 5:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sranjoy wrote: |
| I can only perform any action in IIB only. |
Ignore IIB for a moment. Imagine you had a Java (or programming language of your choice) application that needed to read a large file, but couldn't start processing until the file was complete even though the file did actually exist on the file system as soon as the copy started. How would you fix this problem in that application?
Do the same with IIB.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Thu Dec 15, 2016 6:40 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| sranjoy wrote: |
| Quote: |
btw, if you use the Search button you will find many other questions similar to yours. |
I found some but of different scenerios.
I can only perform any action in IIB only. |
So what you said earlier
| Quote: |
| Third party can change the file filename after copy is successfully complete |
is no longer true?
The above is a typical way this is handled in a *nix environment.
Most places I've been at use .<filename> for the file in progress and then when the SFTP PUT is done it renames the file to <filename> (ie removes the period as the first character).
Dot files are also hidden in *nix 'ls' output unless you add the '-a' option. |
|
| Back to top |
|
|
| sranjoy |
| Posted: Thu Dec 15, 2016 11:04 pm Post subject: |
|
|
Newbie
Joined: 15 Dec 2016
Posts: 3
|
| still waiting for any solution in IIB |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Dec 15, 2016 11:36 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| sranjoy wrote: |
| still waiting for any solution in IIB |
There is no solution in IIB that anyone here would recommend for production.
The solution is as has been suggested already and alluded to by yourself.
i.e.
Create the file with the wrong filename and when it is complete rename it so that the FileInput Node will pick it up.
This thread highlights a fundamental issue that you will encounter when using files.
Other message streams (eg MQ) do not make the end result available to a reader until it is complete and comitted to the queue.
I have banged my head against many brick walls in one part of the world where everyone insits on using FTP for everything including one site where there was already an MQ Channel between both ends. One PHB wanted us to implement a DR solution (where the DR Sites were 1500km apart) using files.
In the end, I gave up and retired.
To parody Douglas Adams,
Files are not the answer to anyhing including Life, the Universe and Everything. They are just a PITA even on a good day. Not even the Vogons would consider using files when planning their interstellar highway.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 16, 2016 5:57 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sranjoy wrote: |
| still waiting for any solution in IIB |
| Vitor wrote: |
Ignore IIB for a moment. Imagine you had a Java (or programming language of your choice) application that needed to read a large file, but couldn't start processing until the file was complete even though the file did actually exist on the file system as soon as the copy started. How would you fix this problem in that application?
Do the same with IIB.
|
How much more of a solution do you want?? 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| adubya |
| Posted: Fri Dec 16, 2016 6:28 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
Can you collect the file from the remote site rather than have them push it to you ? The FileInput node could do that.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 16, 2016 6:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| adubya wrote: |
| Can you collect the file from the remote site rather than have them push it to you ? The FileInput node could do that. |
And how do you know the file is complete when you start to pull it? How do you know the process that creates it isn't running late?
This doesn't fix it the problem, it just moves it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| adubya |
| Posted: Fri Dec 16, 2016 6:48 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
OP, which SFTP server are you using on Windows ?
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
|
|
