| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Filepath for Trace nodes |
« View previous topic :: View next topic » |
| Author |
Message
|
| mqxplorer |
 Posted: Wed Dec 17, 2014 8:17 pm Post subject: Filepath for Trace nodes |
 |
|
Master
Joined: 22 Jun 2009
Posts: 206
|
Hi All,
We have installed and configured Multi Instance brokers and queue managers. We have two servers in QA and one active broker instance runs on first server and its passive instance runs on second server. On second server, there is an active instance broker and its passive instance runs on the first server. Under normal circumstances, we have two active brokers on two servers. Both brokers share the same NFS4 file system. In our case, the file path is /autofs/iibha/CS/. We were testing a flow for high availability and fail over scenarios. In order to test this, I wrote a sample flow with trace node in it that prints the broker name, EG name, process ID, and time stamp and the file path specified on the trace node is the shared file system path /autofs/iibha/CS/HAInfo.log. The same flow is deployed on both brokers. We see one issue with this.. when we send 100 messages using DP as the load balancer, load is balanced well between the two brokers, however, the entries in trace file HAInfo.log does not match the total no of requests sent by DP to two brokers running on two servers. If I understood correct, when the flow running on first broker is writing the trace information to the file specified on the trace node, the flow running on second broker cannot write to the same file (shared file system) as the file is locked by flow running on first broker. Is that correct? If this is correct, then how can we use the NFS shared file system for trace nodes, file nodes?
Please give me some clarity on this. I am doing continuous trial and error testing and thought of posting it here so that I can get some inputs to make my testing more efficient.
Thanks
mqxplorer |
|
| Back to top |
|
 |
| mqxplorer |
| Posted: Thu Dec 18, 2014 12:13 pm Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
Can anybody help me with this? Please point me to a correct information center where I read this info.
Thanks
mqxplorer |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Dec 18, 2014 12:27 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqxplorer wrote: |
| Can anybody help me with this? Please point me to a correct information center where I read this info. |
- We're not a help desk with an SLA. Don't ping threads because you don't get a response.
- You shouldn't be using Trace nodes like this to collect statistical information, and you shouldn't be trying to share them across multi-instance brokers the way you are. Trace nodes are for investigating problems and they work fine, even in multi-instance, for that.
- File nodes use the transit directories to create unique copies of the file that are used by singluar instances of a flow. Hence file locking is not an issue on NFS.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqxplorer |
| Posted: Thu Dec 18, 2014 8:29 pm Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
Thank you Vitor for your explanation. I love this forum. Whenever, I am confused about the concepts I read, I can find answers or the information hubs where I can get the info I am looking for.
Thanks
Mqxplorer |
|
| Back to top |
|
|
| mqxplorer |
| Posted: Sun Dec 28, 2014 4:09 pm Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
| Vitor wrote: |
- File nodes use the transit directories to create unique copies of the file that are used by singluar instances of a flow. Hence file locking is not an issue on NFS. |
Vitor,
I guess, your comment holds good only, if you select the option "Stage in mqsitransit directory and move to output directory on "Finish File".
what happens if I selct the other option: "Write directly to the output file (append if file exists)?
Doesn't this create a file locking issue on NFS?
Thanks
mqxplorer |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Dec 29, 2014 5:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
| mqxplorer wrote: |
| Vitor wrote: |
- File nodes use the transit directories to create unique copies of the file that are used by singluar instances of a flow. Hence file locking is not an issue on NFS. |
Vitor,
I guess, your comment holds good only, if you select the option "Stage in mqsitransit directory and move to output directory on "Finish File".
what happens if I selct the other option: "Write directly to the output file (append if file exists)?
Doesn't this create a file locking issue on NFS?
Thanks
mqxplorer |
It should not if you make the file name unique (like adding date and time etc...).
The point of the output is to be unique to the flow instance... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqxplorer |
| Posted: Thu Apr 30, 2015 5:11 pm Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
| fjb_saper wrote: |
| mqxplorer wrote: |
| Vitor wrote: |
- File nodes use the transit directories to create unique copies of the file that are used by singluar instances of a flow. Hence file locking is not an issue on NFS. |
Vitor,
I guess, your comment holds good only, if you select the option "Stage in mqsitransit directory and move to output directory on "Finish File".
what happens if I selct the other option: "Write directly to the output file (append if file exists)?
Doesn't this create a file locking issue on NFS?
Thanks
mqxplorer |
It should not if you make the file name unique (like adding date and time etc...).
The point of the output is to be unique to the flow instance... |
| Quote: |
Files that are created and written by a FileOutput node are put in the output directory when they are finished. While records are being added to a file, it is kept in the mqsitransit subdirectory.
Each record is written by a single message flow instance. All message flow instances that are configured to write records to a specific file can append records to that file. Because instances can run in any order, records that they write might be interleaved, which means that the sequence of records might be altered. If you require the sequence of records in the output file to be maintained, ensure that only one FileOutput node instance uses the file. To ensure that only one FileOutput node instance uses the file, configure the message flow that contains the node to use the additional instances pool with zero instances, and ensure that other message flows do not write to the same file.
While a file is being processed, the file system is used to lock the file. As a result, other programs (including other integration servers) are prevented from reading, writing, or deleting the file while it is being processed by the file nodes. This lock is retained for a short period after a FileOutput node writes to the file without finishing it, leaving it in the transit directory. If message flows that are in the same integration server use the same output file and run sufficiently quickly, the broker does not relinquish the lock before the file is finished. However, if the message flows have longer intervals between them, the broker relinquishes the lock and another process or integration server can acquire a lock on the file. To prevent this situation, ensure that output directories are not shared between integration servers. |
https://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ac55400_.htm
Everywhere the documentation talks about transit directories only... As far as I understood, the transit directories come into picture, if the second option "Stage in mqsitransit directory and move to output directory on "Finish File". is selected .. the above explanation holds good.
What about the situation when we select "Write directly to the output file (append if file exists)"?
Our requirement is to write records to the same file with pattern "XYZFile_Utilization_CCYYMMDD" for that whole day and we have multi instance brokers configured with NFS4 file system. We deployed the flow on both brokers (active / active) on two different servers and specified the file path as shared NFS file system. Let's say /aut/dir1/subdir and we have selected the option "Write directly to the output file (append if file exists)" ..... another thing is we have additional instances running on both brokers. The behaviour we observed is .. when we have both brokers processing the messages at the same time and write records to the same file located on the shared file system, we lose some records and there is no information about this in the broker logs. However, if only one broker with multiple instances of the flow are processing messages and write to the same file, we do not see any message/record loss while appending to the file on NFS shared file system. We are on IIB 9.0.02.
Are we doing something wrong? I have tried reading different forums, I could not really understand the internal behaviour of broker handling the files.
Please help me with this.
Thanks in advance.
Thanks
mqxplorer |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 01, 2015 4:26 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
| Quote: |
| To prevent this situation, ensure that output directories are not shared between integration servers. |
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqxplorer |
| Posted: Fri May 01, 2015 7:50 am Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
| fjb_saper wrote: |
| Quote: |
| To prevent this situation, ensure that output directories are not shared between integration servers. |
|
Thanks for the confirmation. This is what I explained to my team members, however, they wanted some proof from IBM and asked me to raise a PMR with IBM which I did and IBM confirmed the same... Sometimes, people believe it is right if they hear the word "IBM said" .... though IBM's documentation says the same thing ...
Thanks
mqxplorer |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
