| Author |
Message
|
| Gideon |
 Posted: Thu Mar 29, 2012 10:48 am Post subject: Can no longer deploy to broker |
 |
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
I am using Broker v7.0.0.1 on Linux
I can no longer deploy to the broker. I simply get a message saying its trying to deploy my bar.
I have tried starting and stopping the broker
as well as disconnecting and reconnecting
How can I snap the borker and TK out of this issue ? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Mar 29, 2012 10:52 am Post subject: Re: Can no longer deploy to broker |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Gideon wrote: |
| How can I snap the borker and TK out of this issue ? |
Cancel the deployment it's doing (or it thinks it's doing)?
Starting and stopping the broker is unlikely to stop it trying to deploy whatever it's found on the queue to deploy.
_________________
Honesty is the best policy.
Insanity is the best defence.
Last edited by Vitor on Thu Mar 29, 2012 10:53 am; edited 1 time in total |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:52 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 29, 2012 10:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| What is the primary group Id of the service Id that started broker? |
| Gideon wrote: |
| I can no longer deploy to the broker |
From which we can infer that it used to work. We can stretch that assumption to a change to the service id's principle group to something other than mqbrkrs would have been noticed. Likewise an HA failover would have been noticed.
I accept the possibility of noticed but not mentioned.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Mar 29, 2012 10:59 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| Vitor wrote: |
| lancelotlinc wrote: |
| What is the primary group Id of the service Id that started broker? |
| Gideon wrote: |
| I can no longer deploy to the broker |
From which we can infer that it used to work. We can stretch that assumption to a change to the service id's principle group to something other than mqbrkrs would have been noticed. Likewise an HA failover would have been noticed.
I accept the possibility of noticed but not mentioned. |
The symptoms don't appear as long as the mqsi commands are not run after startup, which is one possible reason it could work for some time before the semaphore issue crops up.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 29, 2012 11:04 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| The symptoms don't appear as long as the mqsi commands are not run after startup, which is one possible reason it could work for some time before the semaphore issue crops up. |
Fair point if this is a production or other platform that typically runs unattended.
Less valid if this is a development machine where mqsideploy is run frequently (either from the command line or via the Toolkit).
As the OP mentions the Toolkit I lean towards a non-production environment. Especially given the "can no longer deploy to broker", implying that the OP has been deploying all day & it's suddenly quit. If a production deploy has failed I would have assumed "can't deploy to broker".
Of course, the OP will soon respond & end the speculation and assumption.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Mar 29, 2012 7:58 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Also check the system.broker queues to make sure there is no "poison" message on it. And I mean here a message not at all formatted for the broker... What does the system log have to say? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Gideon |
| Posted: Mon Apr 02, 2012 9:22 am Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
I moved to this configuration, but am still having problems
| Code: |
[root@xxx02 log]# su - brk
MQSI 7.0.0.1
/opt/ibm/mqsi/7.0
[brk@xxx02 ~]$ groups brk
brk : mqbrkrs mqm |
|
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Apr 02, 2012 10:09 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| mapa |
| Posted: Mon Apr 02, 2012 10:17 am Post subject: |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
A colleague of mine ran into trouble when patching from 7.0.0.1 to 7.0.0.3 on 32-bit CentOS 6.2 some weeks ago. Didn't get time to follow it up yet though. Problem with the EGs misbehaving so it wasn't possible to deploy if I remember correctly. Unfortunately it is on the to-do list to do a follow up.
What does the broker log say (the system user log).
There was a lot errors in it when I looked briefly on our CentOS broker.
Don't have the image available as of now. |
|
| Back to top |
|
|
| Gideon |
| Posted: Mon Apr 02, 2012 3:57 pm Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
lancelotlinc
That data is quite long, so I did 2 versions one witha grep for Dataflow, and the other mq
Would you like me to post anything else or the entire contents ?
| Code: |
[root@hostname02 xxx]# ps -elf | grep DataFlow
0 S brk 3545 3439 0 80 0 - 569659 futex_ 14:13 ? 00:00:03 DataFlowEngine AAA 1fad1f75-3601-0000-0080-876c2672843a e0
0 S brk 3582 3439 0 80 0 - 569617 futex_ 14:14 ? 00:00:03 DataFlowEngine AAA fce71f75-3601-0000-0080-876c2672843a e1
0 S brk 3619 3439 0 80 0 - 569666 futex_ 14:14 ? 00:00:03 DataFlowEngine AAA 0a192075-3601-0000-0080-876c2672843a e2
0 S brk 3655 3439 0 80 0 - 569633 futex_ 14:14 ? 00:00:03 DataFlowEngine AAA b8472075-3601-0000-0080-876c2672843a e3
0 S brk 3691 3439 0 80 0 - 569611 futex_ 14:14 ? 00:00:02 DataFlowEngine AAA 76712075-3601-0000-0080-876c2672843a e4
0 S brk 3728 3439 0 80 0 - 569681 futex_ 14:14 ? 00:00:03 DataFlowEngine AAA 469d2075-3601-0000-0080-876c2672843a e5
0 S brk 3764 3439 0 80 0 - 569683 futex_ 14:15 ? 00:00:03 DataFlowEngine AAA 8dca2075-3601-0000-0080-876c2672843a e6
0 S brk 3800 3439 0 80 0 - 569634 futex_ 14:15 ? 00:00:03 DataFlowEngine AAA 29f22075-3601-0000-0080-876c2672843a e7
0 S brk 3909 3439 0 80 0 - 569664 futex_ 14:15 ? 00:00:03 DataFlowEngine AAA 95712175-3601-0000-0080-876c2672843a d0
0 S brk 3945 3439 0 80 0 - 569649 futex_ 14:15 ? 00:00:03 DataFlowEngine AAA 28982175-3601-0000-0080-876c2672843a d1
0 S brk 3982 3439 0 80 0 - 569668 futex_ 14:16 ? 00:00:03 DataFlowEngine AAA 5acd2175-3601-0000-0080-876c2672843a d2
0 S brk 4018 3439 0 80 0 - 569665 futex_ 14:16 ? 00:00:03 DataFlowEngine AAA 26062275-3601-0000-0080-876c2672843a d3
0 S brk 4054 3439 0 80 0 - 569682 futex_ 14:16 ? 00:00:03 DataFlowEngine AAA 0f332275-3601-0000-0080-876c2672843a d4
0 S brk 4092 3439 0 80 0 - 569643 futex_ 14:16 ? 00:00:02 DataFlowEngine AAA 54592275-3601-0000-0080-876c2672843a d5 |
| Code: |
mqm/bin/runmqchi -m AAA -q SYSTEM.CHANNEL.INITQ -r
0 S mqm 2881 2872 0 80 0 - 53173 semtim 14:04 ? 00:00:00 /opt/mqm/bin/amqpcsea AAA
0 S mqm 2905 2846 0 80 0 - 93949 semtim 14:04 ? 00:00:00 amqfqpub -mAAA
0 S mqm 2909 2905 0 80 0 - 158873 semtim 14:04 ? 00:00:00 amqfcxba -m AAA
0 S mqm 2923 1 0 80 0 - 65997 inet_c 14:04 pts/0 00:00:00 runmqlsr -m AAA -t tcp -p 1414
0 S mqm 3531 2923 0 80 0 - 103312 poll_s 14:13 ? 00:00:00 /opt/mqm/bin/amqrmppa -m AAA
0 S mqm 3536 2820 0 80 0 - 112159 semtim 14:13 ? 00:00:00 amqzlaa0 -mAAA -fip1
0 S mqm 4509 2820 0 80 0 - 410896 semtim 14:19 ? 00:00:08 amqzlaa0 -mAAA -fip2
0 S mqm 4578 2820 0 80 0 - 211718 semtim 14:19 ? 00:00:01 amqzlaa0 -mAAA -fip6
0 S mqm 4723 2820 0 80 0 - 378875 semtim 14:20 ? 00:00:04 amqzlaa0 -mAAA -fip7
0 S mqm 8968 2820 0 80 0 - 114439 semtim 15:39 ? 00:00:00 amqzlaa0 -mAAA -fip8
4 S root 9061 2553 0 80 0 - 25826 pipe_w 15:54 pts/0 00:00:00 grep mq
[root@hostname02 xxx]# |
|
|
| Back to top |
|
|
| Gideon |
| Posted: Mon Apr 02, 2012 4:00 pm Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
mapa
The qmgr is AAA, as is the broker name
Java is crashing. I have only seen this happen in RHEL 5.x. I used the same install package on RHEL 5.6 with no problems.
| Code: |
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Heap dump using '/tmp/heapdump.20120402.153602.8792.0002.phd' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20120402.153602.8792.0003.txt' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Snap dump using '/var/mqsi/common/errors/Snap.20120402.153602.8792.0001.trc' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Heap dump using '/tmp/heapdump.20120402.153602.8792.0002.phd' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20120402.153602.8792.0003.txt' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Snap dump using '/var/mqsi/common/errors/Snap.20120402.153602.8792.0001.trc' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Heap dump using '/tmp/heapdump.20120402.153602.8792.0002.phd' in response to an event
Apr 2 15:36:02 hostname02 IBM Java[8792]: JVMDUMP032I JVM requested Java dump using '/tmp/javacore.20120402.153602.8792.0003.txt' in response to an event
Apr 2 15:36:03 hostname02 WebSphere Broker v7001[8792]: (AAA.e8)[1]BIP2116E: Message broker internal error: diagnostic information 'Fatal Error; exception thrown before initialisation completed', 'JVM Startup', '8792', '1', '6', '5'. : AAA.87192175-3601-0000-0080-876c2672843a: /build/S700_P/src/DataFlowEngine/ImbMain.cpp: 224: ImbMain::ProgressChecker::~ProgressChecker: :
Apr 2 15:36:03 hostname02 WebSphere Broker v7001[8792]: (AAA.e8)[1]BIP2203E: An execution group has encountered a problem whilst starting. : AAA.87192175-3601-0000-0080-876c2672843a: /build/S700_P/src/DataFlowEngine/ImbMain.cpp: 1135: ImbMain::start: :
Apr 2 15:36:03 hostname02 WebSphere Broker v7001[8792]: (AAA.e8)[1]BIP7064S: Failed to create Java VM -4. : AAA.87192175-3601-0000-0080-876c2672843a: /build/S700_P/src/JNI/ImbJVM.cpp: 1034: ImbJVM::createJVM: :
Apr 2 15:36:03 hostname02 WebSphere Broker v7001[8792]: (AAA.e8)[1]BIP2204I: Execution group using process '8792' thread '1' stopped. : AAA.87192175-3601-0000-0080-876c2672843a: /build/S700_P/src/DataFlowEngine/ImbMain.cpp: 1222: ImbMain::start: : |
|
|
| Back to top |
|
|
| Gideon |
| Posted: Mon Apr 02, 2012 4:02 pm Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
mapa
One of the heapdumps:
| Code: |
vi /tmp/heapdump.20120402.153602.8792.0002.phd
^@^Rportable heap dump^@^@^@^E^@^@^@^G^A^D^@^SJRE 6.0 Linux amd64^B^B^Fð^@^@^_é^@ÃÃL^@^@^@hKÃ^@^@^?¤^C^O¢Zsun/nio/ch/FileChannelImpl^@^@^@^Cÿÿÿÿÿÿ^Y¶^@^@^@^@^@^@^@6^@^@^@^@^@^@^@¬^F0B^@^@^@^XL^K^@^@^?¤^C^H/nio/ch/Util^@^@^@^Fÿÿÿÿÿÿ^Yt^@^@^@^@^@^@^@^\^@^@^@^@^@^@^@6^@^@^@^@^@^@^@Z^@^@^@^@^@^@^@bÿÿÿÿÿþU2^@^@^?¤^C^H 8^@^X[Ljava/lang/ThreadLocal;^@^@^@^Aÿÿÿÿÿÿ^YL^Fp^Gâ^@^@^@^X\^^@^@^?¤^C^H 8^@^Qsun/ni^@^@^@^Bÿÿÿÿÿÿ^Qj^@^@^@^@^@^@^@^^^F0*^@^@^@^X\<87>^@^@^?¤^C^H 8^@^[sun/nio/ch/NativeDispatcher^@^@^@^Q@^F0^N^@^@^@^X\<9e>^@^@^?¤^C^O<9f>^H^@^Ysun/nio/ch/FileDispatcher^@^@^@^Aÿÿÿÿÿÿ^Q2^F0^N^@^@^@(\µ^@^@8^@^Zsun/nio/ch/NativeThreadSet^@^@^@^Bÿÿÿÿÿÿ^Q$^@^@^@^@^@^@^@"^Fpõ^T^@^@^@@IÃ
attach/javaSE/IPC^@^@^@^Uÿÿÿÿÿÿ^\^P^@^@^@^@^@^@^@^N^@^@^@^@^@^@^@"^@^@^@^@^@^@^@L^@^@^@^@^@^@^@j^@^@^@^@<80>^@^@^@^@^@^@^@<98>^@^@^@^@^@^@^@¬^@^@^@^@^@^@^@Ã
^@^@L^@^@^@^@^@^@^A ^@^@^@^@^@^@^A6^@^@^@^@^@^@^AN^@^@^@^@^@^@^Af^@^@^@^@^@^@^A<88>ÿÿÿÿÿÿålÿÿÿÿÿÃp^AÃ^@^@^@^XK^Q^@^@^?¤^C^H 8^@+com/ibm/tools/attach/javaSE/FileLockManager^@^@^@^Aÿÿÿÿÿÿ^ZP^F0^N^@^@^^?¤^C^HȈ^@3com/ibm/tools/attach/javaSE/BlockingFileLockManager^@^@^@^Bÿÿÿÿÿÿ^ZB^@^@^@^@^@^@^@.^F0:^@^@^?¤^C^H 8^@Acom/ibm/tools/attach/javaSE/BlockingFileLockManager$LockingStatus^@^@^@^Aÿÿÿÿÿÿ^Z^H^F0^XK<95>^@^@^?¤^C^H 8^@^Rjava/io/DataOutput^@^@^@^Aÿÿÿÿÿÿ^Yú^F0^N^@^@^@0Kª^@^@^?¤^C^H 8^@^Xjava/iosFile^@^@^@^Aÿÿÿÿÿÿ^Yì^FpÃü^@^@^@@^]ò^@^@^?¤^C^H 8^@ java/util/Hashtable$HashIterator^@^@^@^AÿÿÃ@^@X^^¿Mð^F0^N^@^^ |
|
|
| Back to top |
|
|
| Gideon |
| Posted: Mon Apr 02, 2012 4:12 pm Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
Getting a FDC that says I do not have enough of a resource
Possibly Semaphore or futex, etc
If I know what it was, I could do a sysctl and fix the problem
Any idea what resource its looking for
| Code: |
| Operating System :- Linux 2.6.32-220.4.2.el6.x86_64 |
| PIDS :- 5724H7230 |
| LVLS :- 7.0.1.0 |
| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |
| Vendor :- IBM |
| Probe Id :- XC037008 |
| Application Name :- MQM |
| Component :- xcsExecProgram |
| SCCS Info :- lib/cs/unix/linux_2/amqxprmx.c, 1.144 |
| Line Number :- 1135 |
| Build Date :- Aug 13 2009 |
| CMVC level :- p000-L090813 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 495 (mqm) |
| Real UserID :- 500 (brk) |
| Program Name :- amqzxma0 |
| Addressing mode :- 64-bit |
| Process :- 2820 |
| Process(Thread) :- 2820 |
| Thread :- 1 |
| ThreadingModel :- PosixThreads |
| QueueManager :- AAA |
| UserApp :- FALSE
| ConnId(1) IPCC :- 2 |
| ConnId(2) QM :- 2 |
| ConnId(3) QM-P :- 2 |
| ConnId(4) App :- 2 |
| Last HQC :- 2.0.0-199264 |
| Last HSHMEMB :- 2.1.1-110400 |
| Major Errorcode :- xecP_E_PROC_LIMIT |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6026 |
| Probe Severity :- 2 |
| Probe Description :- AMQ6026: A resource shortage prevented the creation of |
| a WebSphere MQ process. |
| FDCSequenceNumber :- 0 |
|
|
| Back to top |
|
|
| Gideon |
| Posted: Mon Apr 02, 2012 4:20 pm Post subject: |
|
|
Chevalier
Joined: 18 Aug 2009
Posts: 403
|
After looking at the data I posted, I deleted all but 5 execution groups
Then I had no problems populating them
But if I create more there is a problem. Dont know what the magic line is yet
So my question is: what resource is the Execution group taking up that the OS is running out of. If I know that I can fix the problem |
|
| Back to top |
|
|
|
|
