| Author |
Message
|
| Gralgrathor |
 Posted: Wed Mar 16, 2016 7:22 am Post subject: IIB9 / WMB7+ : turn off automatic restarting of EGs |
 |
|
Master
Joined: 23 Jul 2009
Posts: 297
|
Good morning!
Does anybody know how to prevent an EG from automatically bouncing back when it's crashed? (Ie. through proper configuration rather than killing processes left and right?)
I'm trying to collect a service trace, but due to the integration server popping back up after dying the trace is many times bigger than I can comfortably handle. I'd like the EG to remain dead after the first deployment attempt, so that I can collect data before starting it back up.
Thanks!
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Wed Mar 16, 2016 7:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If you're trying to do it during a deploy, you maybe can tell the applications not to start automatically.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Wed Mar 16, 2016 7:42 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
Hi Jeff, thanks for responding.
My problem is that the EG will, immediately after bouncing back, attempt the deployment for a second time, crash a second time, and come back up again. If you're trying to gather traces of the initial crash (which is what IBM's problem mgt team asked me to do), you need a positively huge buffer size. I got up to 3G worth of trace, and I'm hesitant to use more of the system's disk - the only space I have access to is on the same volume as the broker's logs.
Telling the flows not to start will probably give me a better look on the failure mode during the final shutdown though - I hadn't thought of that. Thanks!
Gr,
Gr.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 16, 2016 9:10 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
...
Yeah.
Follow up with your PMR, ask the L2 rep...
(And say Hi to them for me...  )
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Wed Mar 16, 2016 12:14 pm Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
I'll give it a go.
I made some progress. First, I split up the offending app into smaller components. Setting the BAR's to 'manual start' allowed me to deploy the apps in question (with a single crash, but one that the EG recovered from with the deployed app in place) - meaning it's not the deployment itself that's the main problem, but starting the apps - or at least, starting more than one of the apps in the same EG.
However, I'm no closer to producing a usable service trace: the first app starts without a hitch, the second start command is accepted, the EG goes full throttle, crashes, restarts and tries to process the same start command again, before I can disable the trace. So with any reasonable buffer size, I still only capture the tail end of the startup, not the crash itself.
Still, something new to tell the PMR team.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 16, 2016 12:53 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Is it always "the second" app?
Or just "the app that happens to come in second when I deploy, but would fail if I tried it first, or third" ?
If it's the first choice, then... deploy the first app. shut down the EG, clear the trace. Enable the trace again, start the EG. deploy. Kill things quick.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Wed Mar 16, 2016 1:06 pm Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| mqjeff wrote: |
| Is it always "the second" app? |
It's whatever app I start second. Problem with killing things is that they'll bounce back up and attempt to start the same application again - only this time with the 'first' app in stopped state, since it was deployed not to start when the EG comes up. So I still wouldn't have a trace of the failure mode.
Perhaps I could try a mqsireadlog right after the first shutdown, but I doubt it'll start processing before the shutdown itself has left the buffer again.
Perhaps I should have RFE'd disabling the autorestart of an EG a couple of years ago. Anybody know how to build a time machine?
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| stoney |
| Posted: Wed Mar 16, 2016 4:26 pm Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
I'm not sure that it's available on WMBv7, but certainly is on IIBv9:
| Code: |
| mqsichangetrace IB9NODE -e default -t -l debug -m temp |
Service trace will be disabled when the integration server restarts (or crashes in your case!). |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Thu Mar 17, 2016 1:26 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| stoney wrote: |
| Code: |
| mqsichangetrace IB9NODE -e default -t -l debug -m temp |
|
I did not know that. Tried it, and again got different results:
| Code: |
Mar 17 09:49:28 usa20155 user:info IIB[66453802]: IBM Integration Bus v9003 (BROKER.EXECUTIONGROUP) [Thread 11052] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'TheApplication' of type '.APPZIP'.
Mar 17 09:49:36 usa20155 user:err|error IIB[66453802]: IBM Integration Bus v9003 (BROKER.EXECUTIONGROUP) [Thread 11052] (Msg 1/1) BIP2112E: Message broker internal error: diagnostic information '12', 'There is not enough memory available now.'.
Mar 17 09:49:36 usa20155 user:warn|warning IIB[12255972]: IBM Integration Bus v9003 (BROKER) [Thread 15421] (Msg 1/1) BIP2060W: The broker has detected that the Execution Group EXECUTIONGROUP, process ID 66453802, has shutdown. |
I'm learning new things all the time. Unfortunately nothing gets me closer to resolving the original problem
Reducing the buffer size of the trace got me to the shutdown without any obvious complaints about memory. Let's hope the new traces can tell IBM's PMR team something useful: they don't tell me anything.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
| stoney |
| Posted: Thu Mar 17, 2016 2:37 am Post subject: |
|
|
Centurion
Joined: 03 Apr 2013
Posts: 140
|
It seems that you're running on AIX.
The '12' in that error message corresponds to ENOMEM, and 'There is not enough memory available now' is the message for ENOMEM.
You've probably already checked to see if the system is actually running out of memory, but have you checked the output of 'ulimit -a'?
In particular, look for the current limits for 'data seg size' and 'virtual memory'.
You must run the 'ulimit' command as the user running IIB. |
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Thu Mar 17, 2016 4:02 am Post subject: |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
ulimit -a gave me:
| Code: |
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) 2048
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited |
But this error only occurs when I choose a large buffer (-c #####) for the mqsichangetrace command with -m temp as additional parameter.
_________________
A measure of wheat for a penny, and three measures of barley for a penny; and see thou hurt not the oil and the wine. |
|
| Back to top |
|
|
|
|
