| Author |
Message
|
| simon.starkie |
 Posted: Tue Oct 02, 2012 10:46 am Post subject: Unconnected Trace nodes ${ExceptionList} filling up /tm? |
 |
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
Problem:
I have a flow with 3 Trace Nodes.
None of these Trace Nodes are connected to the flow.
And yet two of them, the ones for ${ExceptionList}, are filling up /tmp.
Question:
Why are these two unconnected Trace Notes for ${ExceptionList} writing to the filesystem?
Trace Node properties:
m236259_Environment
File
/tmp/m236259_Environment.trc
${Environment}
m236259_ExceptionList
File
/tmp/m236259_ExceptionList1.trc
${ExceptionList}
m236259_ExceptionList2
/tmp/m236259_ExceptionList2.trc
${ExceptionList}
Evidence that /tmp was written to:
ktazd2251.crdc.kp.org:/home2/ibadm> ls -alt /tmp/m236259*
-rw-rw---- 1 qanwhb03 mqm 184498 Oct 2 11:20 /tmp/m236259_ExceptionList1.trc
-rw-rw---- 1 qanwhb03 mqm 184498 Oct 2 11:20 /tmp/m236259_ExceptionList2.trc
[img] [/img]
Last edited by simon.starkie on Tue Oct 02, 2012 10:53 am; edited 1 time in total |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Tue Oct 02, 2012 10:53 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Each trace node in your flow should point to a unique file, not the same file. You may have other flows writing that you are unaware to those files. Change the names of the Trace nodes redeploy and bounce the broker. Don't leave unconnected Trace nodes in flows. Connect them or remove them. Turn trace off on the EG if you don't want Trace output.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 10:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| Each trace node in your flow should point to a unique file, not the same file. You may have other flows writing that you are unaware to those files. |
You may also have not managed to deploy the flow with the disconnected nodes, or someone may have deployed over top of the disconnected flow with the new flow.
Change the file names for these nodes to a unique value. completely remove the flow from all EGs. make sure you successfully build the flow and add it to a new bar file (so you know you don't have an old copy in the bar file from a bad build). Redeploy it.
THEN see if the new file is written to. |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Tue Oct 02, 2012 10:58 am Post subject: |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
Thanks lancelotlinc.
Understood and I removed them already and confirmed they no longer log.
But why do unconnected ${ExceptionList} Trace nodes log to the filesystem?
While unconnected ${Environment} Trace nodes don't?
And why do unconnected Trace nodes log if they are not connected and not included in the flow?
I've uploaded a screen print to prove I'm not halucinating 
And I'm 100% sure that the above flow is what was deployed.
We have Starteam labels to prove that the above flow was deployed.
And I can recreate the problem by adding the unconnected Trace nodes back in.
Or resolve it again by removing the same.
Last edited by simon.starkie on Tue Oct 02, 2012 11:00 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 11:00 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| simon.starkie wrote: |
And why do unconnected Trace nodes log if they are not connected and not included in the flow?
I've uploaded a screen print to prove I'm not halucinating |
They shouldn't. The screen print, however, just shows the toolkit's idea of the flow, not what the runtime has currently deployed - which could be entirely different... |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Tue Oct 02, 2012 11:02 am Post subject: |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
But why do unconnected ${ExceptionList} Trace nodes log to the filesystem?
While unconnected ${Environment} Trace nodes don't? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 11:05 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| simon.starkie wrote: |
But why do unconnected ${ExceptionList} Trace nodes log to the filesystem?
While unconnected ${Environment} Trace nodes don't? |
They *shouldn't*.
I repeat.
You haven't *proven* that they *do*. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Oct 02, 2012 11:09 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
It's not enough to redeploy a flow. You must stop the EG then start the EG in order to reset Trace node file handles. You could have had those Trace nodes active then redeployed a flow with them disconnected, which case the runtime still has open file handles and still writes the files. You must bounce the broker in order to reset open file handles from Trace nodes.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Tue Oct 02, 2012 11:13 am Post subject: |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
| mqjeff wrote: |
| simon.starkie wrote: |
But why do unconnected ${ExceptionList} Trace nodes log to the filesystem?
While unconnected ${Environment} Trace nodes don't? |
They *shouldn't*.
I repeat.
You haven't *proven* that they *do*. |
Than I'm not sure what else I can do to prove it.
I already knew that removing the Trace nodes solves the problem.
It is what it is.
| lancelotlinc wrote: |
| It's not enough to redeploy a flow. You must stop the EG then start the EG in order to reset Trace node file handles. You could have had those Trace nodes active then redeployed a flow with them disconnected, which case the runtime still has open file handles and still writes the files. You must bounce the broker in order to reset open file handles from Trace nodes. |
That's an interesting idea.
But the flow was never deployed previously to QA or PROD environments.
And yet the problem occurred there also. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 11:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| It's not enough to redeploy a flow. You must stop the EG then start the EG in order to reset Trace node file handles. |
That does not match my understanding.
Simon - you could try deploying a brand new flow that uses unique file names for each trace node... i.e.
MQinput->MQoutput
Trace1 (environment)
Trace2(exceptionlist)
And see if either of them write anything... |
|
| Back to top |
|
|
| simon.starkie |
| Posted: Tue Oct 02, 2012 11:23 am Post subject: |
|
|
Disciple
Joined: 24 Mar 2002
Posts: 180
|
| mqjeff wrote: |
| lancelotlinc wrote: |
| It's not enough to redeploy a flow. You must stop the EG then start the EG in order to reset Trace node file handles. |
That does not match my understanding.
Simon - you could try deploying a brand new flow that uses unique file names for each trace node... i.e.
MQinput->MQoutput
Trace1 (environment)
Trace2(exceptionlist)
And see if either of them write anything... |
This was a brand new flow with unique file names in PROD.
I was astonished that /tmp eventually filled to 98% after running for about a month.
And that was because of these unconnected Trace nodes.
The fact that it happened in PROD has made alot of people madder than a wet hen  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 11:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So there had been no version of the flow previously deployed, that had had the trace nodes connected in it? Other than having an old copy deployed, I can't see how the trace nodes would write anything at all, much less different behavior.
I just ran a quick test on v8 and did not get anything from an unconnected trace node. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Oct 02, 2012 11:28 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mqjeff wrote: |
| lancelotlinc wrote: |
| It's not enough to redeploy a flow. You must stop the EG then start the EG in order to reset Trace node file handles. |
That does not match my understanding. |
| rekarm01 wrote: |
| Cogito-Ergo-Sum wrote: |
| What am I missing? |
Is somebody deleting the trace file after the first transaction?
For Unix file systems, once the execution group process opens the trace file for writing, deleting the file won't stop the execution group from writing to it. The OS detaches the file from the file system, (so that new processes can't open the trace file to read it), but the OS won't actually delete the file until the execution group closes it.
The net effect of redeploying a barfile is to close the trace file, and then re-open (re-create) it.
Other file systems may behave similarly. |
My experience with 7.0.0.3 is that an EG must be stopped to release file handles. Since I don't have access to the Trace node source code, I cannot verify if this in fact what is happening.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 02, 2012 11:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| At 7.0.0.3 you should be able to release the file handles just as easily by undeploying the flow that contains the trace node - without needing to restart the EG. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Oct 02, 2012 11:40 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| mqjeff wrote: |
| At 7.0.0.3 you should be able to release the file handles just as easily by undeploying the flow that contains the trace node - without needing to restart the EG. |
rekarm intimates in the above quote that the EG holds the handle. I agree with you, the design should be at the flow level, but my experience and rekarm's quote point to the EG level.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
