| Author |
Message
|
| nareshm13 |
 Posted: Fri Dec 07, 2012 6:11 am Post subject: QMGR is getting Hang In production |
 |
|
Apprentice
Joined: 07 Dec 2012
Posts: 29
|
Hi All,
In our production environment one of our QMGR hangs on every 3 to 6 months.
Version: 7.0.1.7
And when ever this hangs happen we can't able to execute any runmqsc commands....after verifying we found the below FDC generated saying creating a lock to the QMGR...from this FDC we can identify that "Program Name :- bulkDeact " had created the lock to the qmgr "xecL_W_LONG_LOCK_WAIT" So every time when ever this issue comes we tell the application team to stop there application inorder to restart the QMGR.
We already raised the issue to the development team regarding the process which is creating the lock to the QMGR...but no luck.
Requesting everyone is there any permanent fix for this issue.
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Mon December 03 2012 22:37:22 AST |
| UTC Time :- 1354563442.616284 |
| UTC Time Offset :- 180 (AST) |
| Host Name :- bisha |
| Operating System :- SunOS 5.10 |
| PIDS :- 5724H7223 |
| LVLS :- 7.0.1.7 |
| Product Long Name :- WebSphere MQ for Solaris (SPARC platform) |
| Vendor :- IBM |
| Probe Id :- XC307070 |
| Application Name :- MQM |
| Component :- xlsRequestMutex |
| SCCS Info :- lib/cs/unix/solaris_2/amqxlfmx.c, 1.69.1.2 |
| Line Number :- 2363 |
| Build Date :- Nov 17 2011 |
| CMVC level :- p701-107-111117 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 1029 (meduser) |
| Real UserID :- 1029 (meduser) |
| ProjectID :- 3 |
| Program Name :- bulkDeact |
| Addressing mode :- 32-bit |
| Process :- 19618 |
| Thread :- 1 |
| QueueManager :- EESO821P |
| UserApp :- TRUE |
| ConnId(1) IPCC :- 1192102101 |
| Last HQC :- 1.0.0-501136 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- xecL_W_LONG_LOCK_WAIT |
| Minor Errorcode :- OK |
| Probe Type :- MSGAMQ6150 |
| Probe Severity :- 3 |
| Probe Description :- AMQ6150: WebSphere MQ semaphore is busy. |
| FDCSequenceNumber :- 0 |
| Comment1 :- OwningProcess(6581) Status(ACTIVE) |
--More--(9%) |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Fri Dec 07, 2012 6:41 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Does the application use fail if quiescing flag?
>> is there any permanent fix for this issue.
Yes, proper application design and peer code reviews.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| nareshm13 |
| Posted: Fri Dec 07, 2012 6:46 am Post subject: |
|
|
Apprentice
Joined: 07 Dec 2012
Posts: 29
|
Thanks..!!
Apart from application point of view is there any thing can be done from MQ level side...like any steps need to be done before restarting the QMGR..!! |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Dec 07, 2012 6:51 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| nareshm13 wrote: |
Thanks..!!
Apart from application point of view is there any thing can be done from MQ level side...like any steps need to be done before restarting the QMGR..!! |
Thats like asking: "My car ran out of gas, is there anything I can do to make the engine run? I know it needs gasoline, but I want to do something besides buy more gasoline."
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| nareshm13 |
| Posted: Fri Dec 07, 2012 6:54 am Post subject: |
|
|
Apprentice
Joined: 07 Dec 2012
Posts: 29
|
|
| Back to top |
|
|
| nareshm13 |
| Posted: Fri Dec 07, 2012 6:56 am Post subject: |
|
|
Apprentice
Joined: 07 Dec 2012
Posts: 29
|
| Because this time i removed some semaphores |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Dec 07, 2012 7:48 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Ending a QMGR involves internal termination of several processes, not the least of which is the critical process manager, the execution controller, and and the command server. When a queue manager does not end in a reasonable time, its because one of those three processes is not able to safely terminate, or processes they are monitoring are not ready to leave.
Do a ps -elf | grep <qmgr name> to see what other processes might be hanging around.
The MQ software is very well refined piece of code. While no code is perfect, IBM MQ product has been in existence since 1993. The MQ product was based off of concepts developed for the IBM System/360 in 1964 (BTAM/QTAM).
Your best bet is to include the fail if quiscing flag on your MQOPEN/MQGET calls.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| nareshm13 |
| Posted: Fri Dec 07, 2012 5:59 pm Post subject: |
|
|
Apprentice
Joined: 07 Dec 2012
Posts: 29
|
--> to include the fail if quiscing flag on your MQOPEN/MQGET calls.
This we need to put across many on many application which are running or connecting to the server. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Dec 07, 2012 6:51 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| nareshm13 wrote: |
--> to include the fail if quiscing flag on your MQOPEN/MQGET calls.
This we need to put across many on many application which are running or connecting to the server. |
This you should have been putting in all applications since day one.
You can either fix the code (and ensure future applications are coded correctly) or accept that you're going to keep trashing your queue manager.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Dec 10, 2012 6:14 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
I think you discovered the root cause of your hang problem. Of course, we can always turn the power off on the server. That will bring the QMGR down without hanging.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Dec 10, 2012 1:38 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
The OP says the QM hangs every 3 to 6 months. He did not say it only hangs when they try and stop it for whatever reason. Apparently the thing is working fine, then it hangs and dumps FDCs, not necessarily in that order.
If this is the case, the root cause of the hang is not the ommision of the fail if quiscing flag on the MQOPEN/MQGET calls.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Mon Dec 10, 2012 2:06 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
If OP only stops the QMGR every 3 to 6 months, then no- your point not. If the practice is to ungracefully terminate things, then the accumulation of hanging connections in the QMGR could be an issue, then yes- your point plus.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
|
|
