| Author |
Message
|
| jf647 |
 Posted: Tue Aug 27, 2002 5:20 pm Post subject: Queue Manager locks up unpredictably - MQ 5.0 AIX 4.3.3 |
 |
|
Newbie
Joined: 27 Aug 2002
Posts: 2
|
I'm running MQ 5.0 on AIX 4.3.3. Recently, the queue manager has begun to lock up, becoming completely unresponsive to requests. This situation continues until all application processes connected to MQ are killed, and the queue manager is brought down and restarted.
From the application's perspective, we see one of two situations:
- if the app was already connected when the queue manager went south, then MQGET calls return.
- if an application was not connected and attempts to connect, the MQCONN call blocks forever, and the process spins, eating up CPU time at at 1:1 ratio with real time.
From an admin standpoint, we sometimes see a single agent (amqzlaa0) process spinning. Sometimes we see all of the agents spinning, and sometimes we see no errors at all.
When this happens, there are generally a ton of FDC files created (no access to them from where I'm writing this - I can post a header if anyone thinks it will help), all of which indicate failed returns from semop or semctl.
We've been trying for a couple of weeks to get the problem to repeat predictably with no success. We are prepping for an upgrade to v5.2, but due to our environment the box exhibiting the most problems can't be upgraded until all of the production systems already have v5.2
A couple of thoughts that we haven't been able to determine a final answer on:
- is there a possibility that we are exceeded some OS-imposed limit on sems or shared memory segments?
- the Queue Manager has MAXHANDS defined as 256. Could this be causing issues?
Any help from people who've experienced symptoms like this would be greatly appreciated. These problems started happening sporadically at the start of our current development cycle (though there have not been many MQ-specific changes in this cycle), and we are very afraid that they will follow us into production if we can't find a specific cause and resolution.
Thanks.
_________________
--
j.
James FitzGibbon
Ajilon Consulting |
|
| Back to top |
|
 |
| oz1ccg |
| Posted: Tue Aug 27, 2002 11:29 pm Post subject: |
|
|
Yatiri
Joined: 10 Feb 2002
Posts: 628
Location: Denmark
|
Have you been stopping applications using kill -9, I saw a similar sympthom a year ago, that was because there have been many kill -9 of processes using MQ.
The reason was shared storage was not freed after the kill, and therefore caused the systems to go down, when we figured that tou no more problems.
just my $0.02 
PS: This was on HP-UX
_________________
Regards, Jørgen
Home of BlockIP2, the last free MQ Security exit ver. 3.00
Cert. on WMQ, WBIMB, SWIFT. |
|
| Back to top |
|
|
| jf647 |
| Posted: Wed Aug 28, 2002 2:36 am Post subject: |
|
|
Newbie
Joined: 27 Aug 2002
Posts: 2
|
Our standard "take the daemons down" practice is to send SIGTERM several times, and only resort to SIGKILL if they are wedged. So in answer to your question, not as a matter of practice but when it is required.
Are you saying that when a program connected to MQ comes down because of SIGKILL that the shared MQ memory segments are not freed? If so, is there any way to determine if this is the case (i.e. get a report of which SHM ids are considered to be 'in use' by MQ and compare that against the output of ipcs?
Thanks.
_________________
--
j.
James FitzGibbon
Ajilon Consulting |
|
| Back to top |
|
|
| mrlinux |
| Posted: Wed Aug 28, 2002 3:51 am Post subject: |
|
|
Grand Master
Joined: 14 Feb 2002
Posts: 1261
Location: Detroit,MI USA
|
Well on HPUX and Iam sure it close to AIX, this will remove all the shared
memory (PS If you have multiple queue managers on the box dont do this)
ipcs | grep mqm | while read type id
do
ipcrm -$type $id
done
_________________
Jeff
IBM Certified Developer MQSeries
IBM Certified Specialist MQSeries
IBM Certified Solutions Expert MQSeries |
|
| Back to top |
|
|
| sjensen |
| Posted: Tue May 24, 2005 1:36 am Post subject: |
|
|
Centurion
Joined: 18 Dec 2003
Posts: 134
Location: London
|
Hi,
Or use amqiclen
Cheers
Stefan |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 24, 2005 12:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
MQ 5.0 on AIX 4.3 ???
If you are contemplating an upgrade don't go to MQ 5.2 which is already out of support. Go to MQ 5.3 or even better 6.0 (should come out end of this month) and AIX 5.2 maint level 3 or AIX 5.3 !!
 |
|
| Back to top |
|
|
| Anirud |
| Posted: Tue May 24, 2005 1:09 pm Post subject: |
|
|
Master
Joined: 12 Feb 2004
Posts: 285
Location: Vermont
|
| fjb_saper wrote: |
MQ 5.0 on AIX 4.3 ???
If you are contemplating an upgrade don't go to MQ 5.2 which is already out of support. Go to MQ 5.3 or even better 6.0 (should come out end of this month) and AIX 5.2 maint level 3 or AIX 5.3 !!
|
Incase you didn't notice, the original post is dated 27th Aug., 2002. |
|
| Back to top |
|
|
| bower5932 |
| Posted: Tue May 24, 2005 1:10 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| Check the date out on this. This is from 2002 with a stray append from yesterday.... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 24, 2005 1:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Sorry I saw Stefan's posting and thought that amqiclen only came with 5.3.  |
|
| Back to top |
|
|
|
|
