| Author |
Message
|
| aravindtk |
 Posted: Sat Feb 17, 2024 10:20 am Post subject: MQ Triggering z/OS |
 |
|
Novice
Joined: 05 Jun 2023
Posts: 17
|
Hi All,
I was going through the many conditions of the Triggering event on the IBM site
https://www.ibm.com/docs/en/ibm-mq/9.0?topic=triggers-conditions-trigger-event
One of them (No.  is that the Trigger Monitor must be running and polling the INITQ.
We recently had a case when the INITQ actually became full as the Trigger Monitor wasn't running. The Trigger messages that came after actually hit the DLQ. This seems to contradict the condition No.8 above; as the trigger messages should have stopped as soon as the Trigger Monitor had stopped running (it had actually crashed).
The trigger type on the Local Queue was set to EVERY.
Is there any reason why this could have happened?
Thoughts, ideas ... please.
Thanks muchly |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Sat Feb 17, 2024 3:58 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Maybe the trigger monitor had the INITQ open for input (IPROCS > 0 ) but the trigger monitor was not healthy and thus not pulling messages from the INITQ.
Or, maybe the trigger monitor was pulling messages from the INITQ just not at a rate that could keep up with the incoming trigger messages.
Trigger Type "Every" is rarely a good idea, in my opinion.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Feb 17, 2024 8:01 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| PeterPotkay wrote: |
| Trigger Type "Every" is rarely a good idea, in my opinion. |
Where TRIGTYPE(EVERY) is a good ides: In well-provisioned servers, and in circumstances where multiple concurrent instances of the consuming app are required to keep up with inbound workload - to meet SLA's, TRIGTYPE(EVERY) is an effective configuration.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| zpat |
| Posted: Sun Feb 18, 2024 12:46 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Assuming you don't run out of resources - otherwise it's a way of having a very bad day when a burst of traffic arrives.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| hughson |
| Posted: Sun Feb 18, 2024 1:58 am Post subject: Re: MQ Triggering |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| aravindtk wrote: |
One of them (No. 8) is that the Trigger Monitor must be running and polling the INITQ.
We recently had a case when the INITQ actually became full as the Trigger Monitor wasn't running. The Trigger messages that came after actually hit the DLQ. This seems to contradict the condition No.8 above; as the trigger messages should have stopped as soon as the Trigger Monitor had stopped running (it had actually crashed). |
If the program crashed, it can take time for the queue manager to notice that it is not there any more and clean up, remove hConn and hObjs etc. In that time, with Trigger Every it might well be possible to accrue a large number of trigger messages.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Feb 18, 2024 5:39 am Post subject: Re: MQ Triggering |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| aravindtk wrote: |
| We recently had a case when the INITQ actually became full as the Trigger Monitor wasn't running. |
Runmqtrm writes errors to qmgr error logs. What did you discover when you researched your error logs?
| aravindtk wrote: |
| The Trigger messages that came after actually hit the DLQ. |
What (reason code) did you discover when you researched the dead-letter header (DHL)? What was maxdepth of the triggered (business) queue? The initq?
| aravindtk wrote: |
| This seems to contradict the condition No.8 above; as the trigger messages should have stopped as soon as the Trigger Monitor had stopped running (it had actually crashed). |
Why did the trigger monitor crash? Did someone cancel it?
| aravindtk wrote: |
| The trigger type on the Local Queue was set to EVERY. |
Did TRIGTYPE change recently? In poorly provisioned environments (insufficient RAM, insufficient processors) TRIGTYPE(EVERY) will stress your system.
| aravindtk wrote: |
| Is there any reason why this could have happened? |
Has runmqtrm been modified?
Did you restart runmqtrm? Was the restart successful?
| aravindtk wrote: |
Thoughts, ideas ... please.
Thanks muchly |
More research is needed.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Andyh |
| Posted: Sun Feb 18, 2024 6:12 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
The condition specified in the IBM docs is "A trigger monitor currently has the initiation queue open for removing messages (that is, the OpenInputCount local queue attribute is greater than zero)". The queue manager has no idea what programs are trigger monitors, and so all that's really tested is that the OpenInputCount of the initiation queue is non zero.
You should be able to use runmqsc to examine what handles are open on the initiation queue to confirm what programs are opening the initq and it would be sensible to check that there are no unexpected handles open against this queue. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Feb 18, 2024 8:03 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Just a word of caution.
If you don't have a way to throttle the resources, a trigger every can lead to a catastrophy. Imagine there was a problem and now you are getting thousands of messages in the application queue. With a trigger every, you are trying to run thousands of instances of the application. This might exhaust the resources to the point of freezing the application server.
Some mainframes do use trigger every, but always with a cap on resources, like running a max of x instances of the triggered application...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Feb 18, 2024 8:34 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| fjb_saper wrote: |
Just a word of caution.
If you don't have a way to throttle the resources, a trigger every can lead to a catastrophy. Imagine there was a problem and now you are getting thousands of messages in the application queue. With a trigger every, you are trying to run thousands of instances of the application. This might exhaust the resources to the point of freezing the application server.
Some mainframes do use trigger every, but always with a cap on resources, like running a max of x instances of the triggered application...
|
TRIGTYPE(EVERY) is a design choice.
IBM MQ includes runmqtrm source code. It can be modified to limit the number of consumer apps launched.
A poorly written consumer will be launched, open the queue, get a message from the triggered queue, process it, then end the app. The heavy lifting is the o/s fetching the app and passing control to it. A well-written app instance will process the first message, then try (wait) to get another message - and then repeat, repeat, repeat.
Statistically speaking,1,000 messages arriving per second will not result in 1,000 instances of the well-written consumer app. A subset of 10-20 consumers should be able to meet SLAs. If the server is poorly provisioned, buy more OR renegotiate SLAs.
The TRIGTYPE(EVERY) scare originated decades ago on single- or dual-processor systems, with limited RAM and I/O bandwidth, and with badly written apps.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Feb 18, 2024 3:53 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
If you are going to rely on the MQ Trigger Monitor for critical interfaces, you should have monitoring and alerting on the triggered queue, the INITQ, the Trigger Monitor process, and the logs that it creates.
Good monitoring means that you will less likely be in a situation of rising (or full) queues, and a backlog in timely processing of messages.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Feb 19, 2024 8:03 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| gbaddeley wrote: |
If you are going to rely on the MQ Trigger Monitor for critical interfaces, you should have monitoring and alerting on the triggered queue, the INITQ, the Trigger Monitor process, and the logs that it creates.
Good monitoring means that you will less likely be in a situation of rising (or full) queues, and a backlog in timely processing of messages. |
I agree wholeheartedly.
The same level of monitoring should be in place for all mission-critical resources, including message channels.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| aravindtk |
| Posted: Mon Feb 19, 2024 3:57 pm Post subject: MQ Triggering runmqtrm |
|
|
Novice
Joined: 05 Jun 2023
Posts: 17
|
Thanks all for your responses.
The trigger monitor in question here is actually a mainframe TM - CSQQTRMN - to trigger IMS transactions.
Apologies for any confusion.
I couldn't really figure out why it crashed though it appears related to a change performed to upgrade another product from what MF SYSPROGS told me- thought TM was brought back there was a delay. Its quite likely that in this time it could have gone in a limbo when it had the INITQ open ; but was not processing any of these Trigger messages that were piling up ! I cannot think of any other explanation.
Just another related question though- if the TM does not run and QMGR doesnt generate any trigger message; what happens when the TM resumes? Does the triggering resume from that point? i.e if the TRIGTYPE was FIRST on the QLOCAL ; and there are already say 100 messages on the QLOCAL; when the TM starts running this would mean that there wont be any trigger messages generated (as with FIRST this only happens when the QDEPTH goes from zero to 1; and we already are at a 100).
Is this correct? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Feb 19, 2024 4:45 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
No. There is a qmgr trigger message interval attribute value that addresses so-called missing trigger events while the qmgr runs. At qmgr restart, the qmgr will fire the trigger for queues where a trigger should have fired - trigtype(fiirst) and depth > 0 in your case.
What starts your trigger monitor?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| aravindtk |
| Posted: Tue Feb 20, 2024 1:07 am Post subject: MQ Triggering runmqtrm |
|
|
Novice
Joined: 05 Jun 2023
Posts: 17
|
| Its an STC like the MSTR and CHIN on the mainframe; and starts up during the IPL along with the other 2 which starts the MF qmgr. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Feb 20, 2024 4:15 am Post subject: Re: MQ Triggering runmqtrm |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| aravindtk wrote: |
| Its an STC like the MSTR and CHIN on the mainframe; and starts up during the IPL along with the other 2 which starts the MF qmgr. |
Please be precise. What "other 2"?
You mentioned in your OP that the trigger monitor crashed. How did the trigger monitor STC task end? What abend code?
Was a dump created? What did the dump tell you?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
