| Author |
Message
|
| next |
 Posted: Thu Jan 25, 2018 9:12 am Post subject: Does FileInput Node support file locking with FTP servers ? |
 |
|
Voyager
Joined: 02 May 2010
Posts: 75
|
Hi,
We have 2 brokers and we are planning to deploy flows to both the brokers which read files from a FTP server.
Would the FileInput nodes place locks so that both the flows are not picking the same files?
Thanks. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jan 25, 2018 10:06 am Post subject: Re: Does FileInput Node support file locking with FTP server |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| next wrote: |
| Would the FileInput nodes place locks so that both the flows are not picking the same files? |
No. File locking is always handled by the OS; this is true for any file reading application, not just IIB. In your scenario, it's not the File Input node that would be doing any potential locking, as what's interacting with the remote files is the FTP software on the remote server.
I would strongly doubt that any FTP solution is bright enough to notice that 2 FTP get requests for the same file are coming from 2 sibling threads on (what it considers to be) a remote server which it has no knowledge of and hence to have any connection with each other.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 25, 2018 10:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
This (to clarify the position) is why a FileInput node reading a file moves the file to the mqsitransitin directory and renames it; it can't rely on the OS not giving the file to another instance of itself and duplicating the processing.
And that's on a local system where the OS (theoretically) can determine where the file open commands are coming from. Never mind FTP.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| next |
| Posted: Thu Jan 25, 2018 10:37 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
Thanks Vitor,
We thought of handling this with different scan delays for the flows but this is going to coincide at some point.
Is there any better way to handle this scenario ? |
|
| Back to top |
|
|
| next |
| Posted: Thu Jan 25, 2018 10:46 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
| We have a NFS storage as well which holds the QM and Broker data. Would it be right to use the same server as the file landing area if a common path is mounted to the broker servers ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 25, 2018 11:08 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| next wrote: |
| Is there any better way to handle this scenario ? |
Well not using files has to be the best solution. It's not 1980, we can do better these days.
Given that this solution is probably impractical, I'd suggest using a non-IIB single threaded process (such as a script) to pull the files onto the broker's local file system and let the FileInput node do it's thing as described above.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 25, 2018 11:10 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| next wrote: |
| We have a NFS storage as well which holds the QM and Broker data. Would it be right to use the same server as the file landing area if a common path is mounted to the broker servers ? |
How would you know that whatever's writing the file has finished? It's a danger with the FTP solution, but worse if the broker has direct access to the partially written file.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jan 26, 2018 12:14 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Unix being a rather basic OS, does not lock files unless the application specifically requests it - many applications do not, such as the SFTP service.
IIB File nodes do request file locking - but that only protects it against other well-behaved applications.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jan 26, 2018 5:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| Unix being a rather basic OS, does not lock files unless the application specifically requests it |
And sometimes not even then - the classic gotcha is if the file is already opened by an application that didn't request a lock. Many versions / distros just shrug and give both applications a file handle without any message or logging.
The only good thing is watching developers who relied on this banging their heads against desks. I like to heighten the pleasure by talking about more modern and better methods of data transmission loudly, but that's just me.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
