| Author |
Message
|
| santhoshramesh |
 Posted: Tue Sep 01, 2015 12:51 am Post subject: Lock in NFS version 4 folders due to message flows |
 |
|
Novice
Joined: 26 Feb 2014
Posts: 17
|
Hi,
I have two different messages where Flow1 picks messages from MQ and parse the message and create a comma separated file. The Flow2 picks the output of flow1 (i.e. csv file) and inserts the required data into DB. Flow1 stores output file in a folder which is present in NFS version 4 env. The expected behavior is that Flow1 and Flow2 should work parallely and indepentently. There is no sequencing to be followed.
But in the current environment, where flows are deployed is not behaving as expected. Its behaving as, whenever the Flow1 creates a file in the output folder, until the Flow2 picks that message from folder, Flow1 is not processing any files from MQ as it is waiting for the Flow2 to pick the first file. Because of this, the overall time taken is increasing. Then we moved the Flow1 output folder from NFS version 4 environment to the folder present in local server(not in NFS env), then the flows started behaving as expected (i.e. flow1 picks message from MQ and create a file and flow 2 parallelly picks the messages from the folder and inserts into DB).
I am not sure of the reason for the folder locking in NFS environment and what is the change that needs to be done in Message flow level as well as environment level?
Broker version: 8.0.0.1
NFS version: 4
MQ version: 7.5
OS: Linux |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Tue Sep 01, 2015 1:18 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Have you used 'lsof' to determine this?
This is obviously related to your other post about file locking with ODBC.INI
Did you create a PMR as was suggested?
I find it strange that no other Linux users have reported this problem here.
Are you really, really sure that the NFS mounting is done with all the required options?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| santhoshramesh |
| Posted: Tue Sep 01, 2015 2:01 am Post subject: |
|
|
Novice
Joined: 26 Feb 2014
Posts: 17
|
The other post was related to a similar issue, where the flow has MQ to DB. This is another interface which is having two flows 1) MQ to FTP and 2) FTP to DB. In the second interface we faced the issue while the flows are directly interacting with FTP in NFS environment.
I am not sure about the NFS mounting, because the environment is handled by different team and i will request them about to check it.
Haven't raised the PMR yet, will ask the corresponding team to raise it.
But have you faced any similar kind of issue where the flow behaves unexpectedly while interacting with NFS environment. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Sep 01, 2015 5:09 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If the FileOutput node is not ending the message flow when writing to an FTP server, it's much more likely that the FTP service is the issue than that the FileOutput node is the issue.
Did you see what happened if you added additional isntances to the MQ->FTP flow? Did that read more messages from MQ that couldn't complete until the second flow read them?
In general, using files to communicate between applications is problematic. It sounds like you are running into a case where the second application is reading the file before the first one has quite finished with it - and so when the second application (or the FTP server) locks the file, the first one is unable to perform the necessary closing and cleanup of the file.
The FileOutput node is written to prevent this as much as possible - by writing things to the mqsitransit directory. But when talking to FTP, it has to wait for the FTP server to acknowledge that the file has been transferred.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| santhoshramesh |
| Posted: Tue Sep 01, 2015 5:16 am Post subject: |
|
|
Novice
Joined: 26 Feb 2014
Posts: 17
|
| The file is actually written to mqsitransit and on the time of completion the file will be passed to output directory only then the flow2 picks the file.. The performance of the flows were decreased considerably when the file is in NFS folder. Then we moved the folder from NFS to local server and the process was taking less amount of time. Hence we identified interaction of Message flow with NFS environment is creating an issue. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Sep 01, 2015 5:18 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
NFS is a, you know, Network File System. So disk operations can be affected by network speed and issues and configuration and etc. etc. etc.
It can also be affected by other use of the NFS server by other systems, and it's own local disk speed and etc. etc. etc.
Switching to local disk is an excellent way to speed things up. But you have to revisit your capacity requirements to make sure the local disk can handle things.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
