| Author |
Message
|
| akil |
 Posted: Sat Aug 29, 2015 7:33 am Post subject: IIB9: FileInput SFTP |
 |
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
Hi
We've a scenario where we have 2 brokers (HA), active-active , with the same set of message flows deployed.
One of the message flows uses FileInput, with SFTP.
This means that 2 message flows ( on 2 different brokers ) are waiting for files to come on the same remote location.
The question is , will that file be processed by only 1 broker? Or is it possible that both the brokers pick the file?
_________________
Regards |
|
| Back to top |
|
 |
| nelson |
| Posted: Sat Aug 29, 2015 4:41 pm Post subject: |
|
|
Partisan
Joined: 02 Oct 2012
Posts: 313
|
Hope this helps:
| Quote: |
| While a file is being processed, the file system is used to lock the file. As a result, other programs (including other execution groups) are prevented from reading, writing, or deleting the file while it is being processed by the file nodes. |
|
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sat Aug 29, 2015 10:04 pm Post subject: Re: IIB9: FileInput SFTP |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| akil wrote: |
Hi
The question is , will that file be processed by only 1 broker? Or is it possible that both the brokers pick the file? |
The uniqueness you need depends upon the underlying file system locking.
Some older filsystems can't give the required locking (eg NFS 3).
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| akil |
| Posted: Mon Aug 31, 2015 7:10 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
The files are picked up from a RHEL 6.5 box, that OS supports file locking.
In case of local files, we've tested this, and the files are picked up only by 1 EG a time, my question was specifically when the SFTP is configured.. Haven't been able to figure out how to do that test..
_________________
Regards |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 31, 2015 7:27 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Well, in test you need to configure an SFTP server.
And it's not the OS that is responsible for file locks. It's the File System - so NFS, SAMBA, GFCS, etc. I believe SFTP gives you some file locking as well, but I don't remember off the top of my head.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Aug 31, 2015 9:22 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| akil wrote: |
The files are picked up from a RHEL 6.5 box, that OS supports file locking.
|
Whilst the OS supports file locking NOT repeat NOT all valid filesystems available on that FS support the desired level of file locking.
Be careful about where the files reside. For example in a Linux multi-host environment a shared file directory needs NFS V4 in order to provide the right file locking.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| akil |
| Posted: Tue Sep 01, 2015 7:15 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
| smdavies99 wrote: |
| akil wrote: |
The files are picked up from a RHEL 6.5 box, that OS supports file locking.
|
Be careful about where the files reside. For example in a Linux multi-host environment a shared file directory needs NFS V4 in order to provide the right file locking. |
So in case there is no control on the FS, I would assume that 2 different EG's / brokers could pick up the same file , and that needs to be handled within the flow ( some status/flag in a database table to record that a file by that name has been processed ) ..
But in this case, we need a way to tell the FileInput node to stop processing/reading the file - I've not been able to find a way to do that ..
_________________
Regards |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Sep 01, 2015 7:17 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If the files are going to the same location, the movement of the file into the mqsiarchive (or transit, whichever it is for fileinput) should prevent two nodes from reading the same file.
If the files are going to separate locations on separate disks/etc... then again, the same file doesn't exist in the same place, so two nodes won't read them at the same time.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| akil |
| Posted: Tue Sep 01, 2015 11:53 pm Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
| mqjeff wrote: |
If the files are going to the same location, the movement of the file into the mqsiarchive (or transit, whichever it is for fileinput) should prevent two nodes from reading the same file.
|
Since this is 'SFTP', I suppose the 'movement' is a 'copy' followed by a 'delete', and that I suppose could result in 2 nodes getting the same file ?
_________________
Regards |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Sep 02, 2015 12:16 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
What does your testing show?
Can the file be read by both brokers?
What does putting 200,000 files through show?
Are there any duplicates?
This would prove the validity of your solution on way or the other.
If there are duplicated then you may have to raise a PMR to get to the bottom of it.
I am sure that there are people here who would like to know the answer.
Setting up some tests would save a lot of time.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 02, 2015 5:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
|
| Back to top |
|
|
| akil |
| Posted: Wed Sep 02, 2015 8:52 am Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
| mqjeff wrote: |
| http://www-01.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ac55400_.htm |
This is about local folders, I understand that..
I am not clear on the behaviour when SFTP comes into the picture. I would suspect it would also depend on the SFTP server ( numerous posts of StackOverflow talk about the OpenSSH SFTP server not supporting file locking ), not been able to arrive at documentation that talks about this..
I could have just changed this to HTTPInput --> FileRead ( in which case only one of the servers would receive the HTTP request, and initiate the transfer ) , but FileRead with SFTP isn't available till 9.0.0.3 .. 
_________________
Regards |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 02, 2015 8:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I believe that the mqsitransit directory is created on the remote server, not on the local server when using FTP/SFTP.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| akil |
| Posted: Wed Sep 02, 2015 8:49 pm Post subject: |
|
|
Partisan
Joined: 27 May 2014
Posts: 338
Location: Mumbai
|
| mqjeff wrote: |
| I believe that the mqsitransit directory is created on the remote server, not on the local server when using FTP/SFTP. |
In 9.0.0.2, the mqsitransit is only on the local (ESB) machine, and not on the remote folder..
_________________
Regards |
|
| Back to top |
|
|
| timber |
| Posted: Thu Sep 03, 2015 12:53 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
| Quote: |
I am sure that there are people here who would like to know the answer.
Setting up some tests would save a lot of time. |
I agree - please post again if and when you find out the answer. |
|
| Back to top |
|
|
|
|
