| Author |
Message
|
| kwelch |
 Posted: Thu Aug 29, 2002 11:42 am Post subject: WMQI freezes and we can't do anything |
 |
|
Master
Joined: 16 May 2001
Posts: 255
|
Hi,
We have been experiencing some strange behavior with WMQI 2.1 at our company and aren't sure what to make of it.
There are two separate scenarios which happened but our SA's think they may be related.
Scenario 1 happened in production. One day our production HUB where all of our MQ transactions flow through and also where WMQI broker and configmgr live got the blue screen of death. This is a Windows 2000 box. The memory was completely maxed out. They discovered that the box had not been rebooted for awhile and rebooted it and it was ok. What they have noticed is that as the week progresses more and more memory gets used. I am wondering if there are memory issues with WMQI 2.1?
Scenario 2 happened in our Development Environment. We did not get the blue screen of death, but we were using about half the available memory of the box and things just come to a complete halt. We can't deploy, stop/start message flows. Everything comes back with a time out message. The only way we are able to resolve this issue is to bring the configmgr and broker down and sometimes this doesn't not help and the server has to be rebooted. I actually pcanywhered into the box during this and cpu was almost idle. It almost seems like WMQI is confused and doesn't know what to do so does nothing and times out. Could it be a database issue?
Has anyone else experienced this or had similar problems? I would be interested to hear what the resolution was if you have.
Thanks,
Karen |
|
| Back to top |
|
 |
| MQSIBM |
| Posted: Thu Aug 29, 2002 9:12 pm Post subject: |
|
|
Novice
Joined: 31 Oct 2001
Posts: 18
|
this may happen sometimes if you have a "forever while loop".
check all the while blocks. |
|
| Back to top |
|
|
| Tibor |
| Posted: Fri Aug 30, 2002 12:24 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Karen,
We ran a problem with MQSI using 'Logical Order' on MQInput node (default setting: checked  ). When the sender application and/or the broker confused the grouping info (MsgSeqNr or LastMsg) the message flow hung up. Moreover, I couldn't stop broker normally. |
|
| Back to top |
|
|
| kwelch |
| Posted: Fri Aug 30, 2002 4:59 am Post subject: |
|
|
Master
Joined: 16 May 2001
Posts: 255
|
Hi,
Thanks for your suggestions.
This is definitely not something looping. I have seen loops before and when an app is looping you can see it sucking up all the cpu on the box. In the Dev environment when this happens the cpu is about 98% idle. This seems to happen when we have 4 or 5 people doing some heavy duty developing and are issuing a lot of deploys or stopping/starting message flows, or turning trace on/off.
Karen |
|
| Back to top |
|
|
| Lisa |
| Posted: Fri Aug 30, 2002 7:33 am Post subject: WMQI Freeze |
|
|
Master
Joined: 07 Jun 2002
Posts: 287
Location: NJ
|
Hi Karen,
You are not the only one with this problem. Our broker is on the Unix server, and I've seen the same problem. We re-boot our server every other week in order to clean up the system.
If I'm performing a lot of deployments, I will re-boot the system sooner.
Hope this helps!
Lisa |
|
| Back to top |
|
|
| kwelch |
| Posted: Fri Aug 30, 2002 7:49 am Post subject: |
|
|
Master
Joined: 16 May 2001
Posts: 255
|
Hi Lisa,
Yes it helps! I know that we are not the only one experiencing it now! But what is the cause? Does anyone know? IBM is unaware of this problem. When we talked with them they said no one else had ever reported this problem except us. Rebooting may not always be an option for us. What if you have 5 or 6 developers working at once doing breakthrough testing and things are constantly being deployed? Our broker and config mgr live on our MQ Hub. We are planning to move them to their own server but right now they are together. We don't always have the luxury of being able to reboot whenever we want.
Karen |
|
| Back to top |
|
|
| Tibor |
| Posted: Fri Aug 30, 2002 8:09 am Post subject: Re: WMQI Freeze |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| Lisa wrote: |
| You are not the only one with this problem. Our broker is on the Unix server, and I've seen the same problem. We re-boot our server every other week in order to clean up the system. |
Lisa,
Did you re-boot only your ConfigMgr server (NT/2K), didnt't you? Just because I never heard about cleaning up on a Unix sever  ...
Tibor |
|
| Back to top |
|
|
| kwelch |
| Posted: Fri Aug 30, 2002 8:12 am Post subject: |
|
|
Master
Joined: 16 May 2001
Posts: 255
|
| Also, our server does get a scheduled reboot every week and we still experience these problems. |
|
| Back to top |
|
|
| Lisa |
| Posted: Fri Aug 30, 2002 12:48 pm Post subject: WMQI |
|
|
Master
Joined: 07 Jun 2002
Posts: 287
Location: NJ
|
Hi All,
We only re-boot the Unix server. |
|
| Back to top |
|
|
| Tibor |
| Posted: Sun Sep 01, 2002 11:20 pm Post subject: Re: WMQI |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Lisa,
| Lisa wrote: |
| We only re-boot the Unix server. |
It seems mysterious for me. When any problem was with the broker on a Unix server (AIX), I stopped all of broker components then made a cleaning for shared resources (memory, semaphores & queues).
When you ran your broker under user named mqsi:
| Code: |
$ ipcs -a | grep mqsi
s 35 0x41008802 --ra-ra---- mqsi mqbrkrs mqsi mqbrkrs 2 18:07:21 11:30:16
s 36 0xffffffff --ra-ra---- mqsi mqbrkrs mqsi mqbrkrs 2 11:30:41 11:30:17
|
then
| Code: |
$ ipcrm -s 35
$ ipcrm -s 36
|
Or automatically:
| Code: |
#!/bin/ksh
ipcs -a |
grep $LOGNAME |
while read line
do
ipc=`print $line | awk '{print $1}'`
id=`print $line | awk '{print $2}'`
print "ipcrm -$ipc $id"
ipcrm -$ipc $id
done
|
Warning: use this method if you have only one broker per user!
Tibor |
|
| Back to top |
|
|
| mapa |
| Posted: Mon Sep 02, 2002 12:40 am Post subject: CSD 03 released |
|
|
Master
Joined: 09 Aug 2001
Posts: 257
Location: Malmö, Sweden
|
Hi,
To me it sounds like there is a memory leak somewhere.
You could make snapshots of memory usage by different processes and try to narrow it down to that level at least.
CSD03 is released 30/8 and contains alot of fixes, at least one contains a fix for a memory leak, (only just read the details very hastily).
Note that if you are using SuportPacs in your flows, the leak might be in one of those...
Best regards Mapa |
|
| Back to top |
|
|
|
|
