| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| A strange MQ case for all your advice |
« View previous topic :: View next topic » |
| Author |
Message
|
| bay hoe san |
 Posted: Fri Apr 27, 2007 12:53 am Post subject: A strange MQ case for all your advice |
 |
|
Centurion
Joined: 27 Nov 2006
Posts: 117
|
Hello,
( 1) I have a strange case, it happen at least twice per year and till to date no solution and cannot simulate in development. Recently, I was being "arrowed" to take on WMQ admin role and the case is in my radar now, worse thing is I have to solve it.
( 2) The background:
At mainframe, qmgr called QM1, local queue = H.QLOCAL with queue depth=1000, trigger type=FIRST, trigger set=YES, trigger msg priority=0, trigger depth=1, process = to trigger a CICS txn called CIS1.
IBM AIX, qmgr called SM1, qremote = S.QREMOTE.
AIX sends msg to mainframe, txn CIS1 will be triggered. CIS1 program name is called CIS1. CIS1 will do a MQGET I can say that in 1 second there are at least 10 messages to mainframe.
( 3) The situation:
It occured 1 day morning, the queue depth hits 1000 and flow over to dead letter queue i.e. QM1.DEAD.LETTER.QUEUE, fortunately my operators saw the messages flooding the console and so alerted me.
H.QLOCAL still register 1000 while QM1.DEAD.LETTER.QUEUE keeps increasing. There is NO CICS txn CIS1.
I read the program coded by application team. Is just a very simple MQGET program with MQGMO-WAITINTERVAL=10000 (i.e. 10 secs).
( 4) My query:
What could be the possible cause? Program bug? CICS slow? MQ slow? Program technique not correct?
Any MQ parameter to set?
( 5) Pls advise. Thank you.
.Hoe San. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Apr 27, 2007 1:07 am Post subject: Re: A strange MQ case for all your advice |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bay hoe san wrote: |
It occured 1 day morning, the queue depth hits 1000 and flow over to dead letter queue i.e. QM1.DEAD.LETTER.QUEUE, fortunately my operators saw the messages flooding the console and so alerted me.
H.QLOCAL still register 1000 while QM1.DEAD.LETTER.QUEUE keeps increasing. There is NO CICS txn CIS1.
|
So something stopped the application from being triggered... 
Because it's a trigger type FIRST, the trigger will only go off when the queue depth goes from 0 -> 1. If the trigger fails at that point, or the application fails to start / process, no further trigger messages will be generated & you'll get the symptoms you're experiencing.
Investigate the trigger monitor process, CICS log & application log for the time in question.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bay hoe san |
| Posted: Fri Apr 27, 2007 1:39 am Post subject: |
|
|
Centurion
Joined: 27 Nov 2006
Posts: 117
|
Hello,
( 1) I have investigated the followings:
1. CICS log
- no abend, no sys dump, no sys slowness
2. application
- a point to add, in the start of the program CIS1, it will do an enqueue on XXX, if return code is enqueue busy it will do an application abend MQEN.
- On that day, there is ONLY 1 appl abend MQEN meaning to say the txn CIS1 is active and another CIS1 is being triggered.
3. in QM1 started task, first warning msg on dead letter queue starts at about 9:32 am while the last CICS txn CIS1 started at 8:57:30.5419 & completed at 8:57:34.0316 i.e. ave response time is abt 3.4897.
( 2) I need your advice on how to check the trigger monitor process?
( 3) We suspect the msg was corrupted and hence prevent the triggering. Can this scenario happen? We double confirm with appl, the msg format is the same, so chance of msg corruption is very slim.
( 4) What's else can I check and what could be the possible cause?
( 5) Pls advice. Appreciate greatly. Thank you.
.Hoe San. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Apr 27, 2007 1:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bay hoe san wrote: |
| - a point to add, in the start of the program CIS1, it will do an enqueue on XXX, if return code is enqueue busy it will do an application abend MQEN. |
Please see my comments above regarding when trigger messages are generated. If the application if on a FIRST trigger and fails, the message will roll back onto the queue. The next message will take the depth from 1->2 which will not generate a trigger message. Please see the Application Programming Guide for a detailed description of this and all other triggering behaviours, along with strategy for dealing with this.
| bay hoe san wrote: |
| ( 2) I need your advice on how to check the trigger monitor process? |
It's a CICS task CKTI
| bay hoe san wrote: |
| ( 3) We suspect the msg was corrupted and hence prevent the triggering. Can this scenario happen? |
Please see my comments above regarding application failure.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Apr 27, 2007 12:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And remember to set your trigint (qmgr property) to something meaningful! (default is 11 days...) 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bay hoe san |
| Posted: Mon Apr 30, 2007 1:55 am Post subject: |
|
|
Centurion
Joined: 27 Nov 2006
Posts: 117
|
Hello,
( 1) Thank you for all your advises and references.
( 2) Current qmgr trigger interval is 7200000 ms i.e. 2 hours.
( 3) I am pondering, for the scenario which I have described:
1. Will it be better if I will to set the trigger type=EVERY for the local queue H.QLOCAL? What's the advantage and dis-advantage? Will there be any performance issue or etc?
( 4) If dead letter queue of a qmgr is full, will the qmgr freeze? What will be the consequence?
( 5) I have read the appl pgm guide/ref, perhaps I have not really grasp the concepts. Can experts like u suggest how can I tell application team to code? Or is there some sample whereby application team can refer to?
( 6) Thanks and have a nice day ahead.
.Hoe San. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Apr 30, 2007 3:38 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bay hoe san wrote: |
1. Will it be better if I will to set the trigger type=EVERY for the local queue H.QLOCAL? What's the advantage and dis-advantage? Will there be any performance issue or etc? |
The big disadvantage is that you get 1 CICS transaction for each message. Unless they're throttled in CICS there's a risk this will flood your system and cause resource shortage issues.
| bay hoe san wrote: |
| If dead letter queue of a qmgr is full, will the qmgr freeze? What will be the consequence? |
IIRC the receiver MCA will shut down and the channel will stop. This will (obviously) cause messages to start backing up on the sending system xmitq.
| bay hoe san wrote: |
| Can experts like u suggest how can I tell application team to code? |
It's really just good programming practice. The triggered task should always read the queue until empty and have proper "poison" message handling. Most of the supplied samples will demonstrate this behaviour.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bay hoe san |
| Posted: Thu May 03, 2007 5:06 pm Post subject: |
|
|
Centurion
Joined: 27 Nov 2006
Posts: 117
|
Hello,
( 1) What is "IIRC" in 1 of the advice u have given?
( 2) Do u mean that once dead letter queue for that qmgr is full (as determined by the predetermined queue depth), messages will start to accumulate at the sender xmitq? So will the queue mgr at z/OS hang when it's dead letter queue is full?
( 3) Does "poison" message mean "corrupted" message?
( 4) Yes, that pgm in trouble has coded in such a way that it loop until no message in queue.
( 5) Thank you for your advice.
.Hoe San. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu May 03, 2007 5:38 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| bay hoe san wrote: |
| What is "IIRC" in 1 of the advice u have given? |
This is an acronym for "If I Recall Correctly". It means "This is what I remember, which is probably right, but might not be".
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 03, 2007 11:55 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bay hoe san wrote: |
| ( 1) What is "IIRC" in 1 of the advice u have given? |
See post from jefflowrey. Apologies for my use of unknown language.
| bay hoe san wrote: |
| ( 2) Do u mean that once dead letter queue for that qmgr is full (as determined by the predetermined queue depth), messages will start to accumulate at the sender xmitq? So will the queue mgr at z/OS hang when it's dead letter queue is full? |
If my recollection is correct and the MCA does stop, it means the queue manager will refuse new messages when the maxdepth+1 message arrives. It doesn't mean the queue manager has frozen as locally connected applications will still be able to read and write messages. Mostly read, as they should be clearing the queue off.
On a personal note, if the dead letter queue fills up then you have a serious problem with your MQ administration. 1 message on the dead letter queue is a problem to be investigated urgrently. A full DLQ is a crisis that could and should have been prevented.
| bay hoe san wrote: |
| ( 3) Does "poison" message mean "corrupted" message? |
No it means a message that can't be processed. The message could be perfectly valid, just that the business data within it causes the application to roll back.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
