ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportDoes FileInput Node support file locking with FTP servers ?

Post new topicReply to topic
Does FileInput Node support file locking with FTP servers ? View previous topic :: View next topic
Author Message
next
PostPosted: Thu Jan 25, 2018 9:12 am Post subject: Does FileInput Node support file locking with FTP servers ? Reply with quote

Acolyte

Joined: 02 May 2010
Posts: 73

Hi,

We have 2 brokers and we are planning to deploy flows to both the brokers which read files from a FTP server.

Would the FileInput nodes place locks so that both the flows are not picking the same files?

Thanks.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 25, 2018 10:06 am Post subject: Re: Does FileInput Node support file locking with FTP server Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

next wrote:
Would the FileInput nodes place locks so that both the flows are not picking the same files?


No. File locking is always handled by the OS; this is true for any file reading application, not just IIB. In your scenario, it's not the File Input node that would be doing any potential locking, as what's interacting with the remote files is the FTP software on the remote server.

I would strongly doubt that any FTP solution is bright enough to notice that 2 FTP get requests for the same file are coming from 2 sibling threads on (what it considers to be) a remote server which it has no knowledge of and hence to have any connection with each other.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 25, 2018 10:11 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

This (to clarify the position) is why a FileInput node reading a file moves the file to the mqsitransitin directory and renames it; it can't rely on the OS not giving the file to another instance of itself and duplicating the processing.

And that's on a local system where the OS (theoretically) can determine where the file open commands are coming from. Never mind FTP.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
next
PostPosted: Thu Jan 25, 2018 10:37 am Post subject: Reply with quote

Acolyte

Joined: 02 May 2010
Posts: 73

Thanks Vitor,

We thought of handling this with different scan delays for the flows but this is going to coincide at some point.

Is there any better way to handle this scenario ?
Back to top
View user's profile Send private message
next
PostPosted: Thu Jan 25, 2018 10:46 am Post subject: Reply with quote

Acolyte

Joined: 02 May 2010
Posts: 73

We have a NFS storage as well which holds the QM and Broker data. Would it be right to use the same server as the file landing area if a common path is mounted to the broker servers ?
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 25, 2018 11:08 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

next wrote:
Is there any better way to handle this scenario ?


Well not using files has to be the best solution. It's not 1980, we can do better these days.

Given that this solution is probably impractical, I'd suggest using a non-IIB single threaded process (such as a script) to pull the files onto the broker's local file system and let the FileInput node do it's thing as described above.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Jan 25, 2018 11:10 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

next wrote:
We have a NFS storage as well which holds the QM and Broker data. Would it be right to use the same server as the file landing area if a common path is mounted to the broker servers ?


How would you know that whatever's writing the file has finished? It's a danger with the FTP solution, but worse if the broker has direct access to the partially written file.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
zpat
PostPosted: Fri Jan 26, 2018 12:14 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5645
Location: UK

Unix being a rather basic OS, does not lock files unless the application specifically requests it - many applications do not, such as the SFTP service.

IIB File nodes do request file locking - but that only protects it against other well-behaved applications.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Jan 26, 2018 5:39 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24972
Location: Ohio, USA

zpat wrote:
Unix being a rather basic OS, does not lock files unless the application specifically requests it


And sometimes not even then - the classic gotcha is if the file is already opened by an application that didn't request a lock. Many versions / distros just shrug and give both applications a file handle without any message or logging.

The only good thing is watching developers who relied on this banging their heads against desks. I like to heighten the pleasure by talking about more modern and better methods of data transmission loudly, but that's just me.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportDoes FileInput Node support file locking with FTP servers ?
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.