| Author |
Message
|
| gauravg |
 Posted: Tue Aug 09, 2016 11:49 pm Post subject: FileRead Node unable to read remote file after Brokerrestart |
 |
|
Novice
Joined: 12 Jan 2016
Posts: 14
|
Hi,
We have a simple message flow to read remote file from ftp server. We are using ftp configurable service in which servername, securityIdentity,protocol,scanDelay are specified.
Remote ServerDirectory and Directory are set using localenvironment before hitting fileread node. Protocol is SFTP
At first deployment, this flow works perfectly fine by fetching file from remote directory. However if Integration Node(broker) is restarted, then it is not able to fetch files from remote directory until there is next deployment.
Once redeployed, again it starts fetching files.
In trace, it shows no file match and flow controls passes from "no match" terminal of fileread node. There is no exception also anywhere.
Node is configured to read file as "Whole File". Since before node restarts, it is working fine, that means sftp connection is properly set and server is responding to ftp request by fileread node.
Please suggest what can be the root cause for this. where we need to dig into more detail.
thanks
gaurav |
|
| Back to top |
|
 |
| gauravg |
| Posted: Wed Aug 10, 2016 1:51 am Post subject: |
|
|
Novice
Joined: 12 Jan 2016
Posts: 14
|
Just to update, IIB version is 10.0.0.4
Thanks |
|
| Back to top |
|
|
| adubya |
| Posted: Wed Aug 10, 2016 3:34 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
Have you checked the broker error log at startup for any FTP related errors ?
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| gauravg |
| Posted: Wed Aug 10, 2016 4:13 am Post subject: |
|
|
Novice
Joined: 12 Jan 2016
Posts: 14
|
| There are no ftp errors at broker startup. |
|
| Back to top |
|
|
| gauravg |
| Posted: Thu Aug 11, 2016 7:42 am Post subject: |
|
|
Novice
Joined: 12 Jan 2016
Posts: 14
|
Can someone please suggest here where can be the problem.
Thanks |
|
| Back to top |
|
|
| adubya |
| Posted: Thu Aug 11, 2016 7:50 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
Sounds like PMR territory to me.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 11, 2016 7:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| gauravg wrote: |
| Can someone please suggest here where can be the problem. |
So if I'm reading your post correctly, the problem is not that the flow tries to read the remote file and hangs, it's that the flow opens the FileRead node, fails to detect the remote file and passes control to the "no match" terminal until the flow is redeployed, when it correctly reads the file the first time and then repeats the "no match" behavior.
I'm going to assume that your design is that you read "data.txt" from the remote location, process it and then at some later point you're trying to read the new/latest/next copy of "data.txt" from that location, which doesn't work. If so, the problem is the sftp server at the remote end. It's either believing that you've already got the file because the connection is still open, or because you still have a file lock on "data.txt", albeit the copy with the previous contents. The redeploy fixes it by dropping the connection / releasing the lock / cleaning up.
If the FileRead node is going to the "no match" terminal, it's because the remote sftp server is telling it there's no file. Welcome to the world of ftp, enjoy your stay.
I'll also refer to you to @mqjeff's many excellent expositions on why this kind of file based processing is problematic even when the protocol doesn't collapse under you.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 11, 2016 7:56 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| adubya wrote: |
| Sounds like PMR territory to me. |
Sounds more like a problem for whoever administers the remote FTP server to me. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 11, 2016 8:01 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Compare the contents of the directory before the first file read and after the first file read.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Aug 11, 2016 8:05 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
Welcome to the world of ftp, enjoy your stay.
|
Yet, in some parts of the world they insist on using FTP to move ALL data around. I could name (but won't) a place where they do this even though they have MQ at both ends AND channels between the servers.
This is real time data that needs to be delivered promptly and correctly. Deaf ears result when they get told that FTP is not the way to do this day in, day out.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 11, 2016 8:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| Yet, in some parts of the world they insist on using FTP to move ALL data around. |
And it's a pain for us all, including the OP.
Well not me, I got it banned here. Still working on eliminating the transfer of data in file form; as an interim measure I've made the problems elucidated by our most worthy associate the purview of a different team.
If I don't get paged for it, it's not a problem. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 11, 2016 9:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| I've made the problems elucidated by our most worthy associate the purview of a different team. |
I imagine it's interesting to see the faces of a development team when they hear the words "Halting Problem".
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 11, 2016 11:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
| I've made the problems elucidated by our most worthy associate the purview of a different team. |
I imagine it's interesting to see the faces of a development team when they hear the words "Halting Problem". |
It's fun. I sit the 2 teams in a room and watch through the window, often with popcorn.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Aug 11, 2016 11:15 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Vitor, you're one of a shrinking number of people who appreciate the dangers of using files and file transfer for important purposes.
The other one is a certain Mr Murphy.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Aug 11, 2016 11:31 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
Vitor, you're one of a shrinking number of people who appreciate the dangers of using files and file transfer for important purposes.
The other one is a certain Mr Murphy. |
that is probably because we started out in this industry when paper tape and/or punched cards were being used as a data transfer method.
Us  will soon be put out to grass and the file transfer fans will  themselves. good riddance I say
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
