MQSeries.net :: View topic

spfdgio · Posted: Tue Mar 21, 2006 5:36 am Post subject: Broker Hang on AIX v5.3 ML4

We migrated our WBIMB (message broker) v5 fixpack6 from AIX v5.2 ML5 to AIX v5.3 ML4 level with 64bit enabled. Immediately afterwards we started to encounter intermittent hangups of the broker. External symptoms of what we saw were:
1) Deployment of a bar file to an EG hangs up and the toolkit does not respond with a message saying you exceeded your defined timeout values. You can see messages in the SYSTEM.BROKER.* queues. In this case, a recycle of the broker is required to recover - as an mqsireload to the EG hangs also. All other msgflows of other EG's work ok though.
2) Deployment of a bar file to an EG works fine, however some or all msgflows of the EG fail to consume messages from their input queues. There are no messages seen in SYSTEM.BROKER.* queues during this hang. A recycle of the broker is required to clear the issue. During this hang, all other EGs are working fine.
3) An EG will just stop consuming messages and fail to respond to start or stop of msgflows from the toolkit. Sometimes an mqsireload to the EG worked, but most did not.
Also note that after the broker hangups, you must issue mqsistop without the -i option. If you issue mqsistop with -i, we consistently saw the broker restart with 2 copies of the hung msgflow. When that happens, you must kill -9 one of them after issuing mqsistop (no -i) and get a good stop - then the broker will restart ok.
-
This issue held us up from rolling out AIX v5.3 in our shop. The fix was open apar IY79612 (since Dec 05), and IBM created an ifix for us which seems to have corrected the issue. We spent WEEKS trying to gather dbx and kdb output of the hung processes to give to IBM for them to find this was the culpret.
-
I hope this entry helps those of you who may hit upon this issue to save some of your hair from being pulled out. We are now back on track....

Take care, Dan in Harrisburg, PA