| Author |
Message
|
| rparti |
 Posted: Sun May 06, 2007 12:47 pm Post subject: Messages flow very slow. Lots of messages in init Q |
 |
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
Hi,
Was wondering if anyone dealt with a similar issue.
Our team noticed that messages were flowing through a triggered queue very slowly...We fired 2 messages and there was a delay of over ten minutes to pick them up.
On investigation we noticed that there were over 16000 messages in the initiation queue. The number was increasing between 2 and 10 for 10 min.
When we looked at the processes, we saw that there were new trigger monitor processes being fired during the latency.
After the 10 minutes the two messages flowed normally. Also messages fired later were flowing normally.
I am new to this team. I also noticed that there was one trigger monitor service associated with all the queue managers. This was through an MQ service that was running seperately. I have never noticed this before. In other environment I have always seen a single trigger monitor program associated with one queue manager.
Also we are running on a Windows environment.
My questions are:
1) What could cause so many messages in the initiation queue, which in my mind caused the delay?
2) How does one trigger monitor service for one queue manager compare with one trigger monitor service for all queue managers.
Thanks in advance
--Heath |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Sun May 06, 2007 1:13 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The trigger monitor application is just another MQ application. This (usually) means it's connected to only one queue on only one queue manager. The default trigger monitor (runmqtrm/runmqtmc) can't be pointed at more than one queue on more than one queue manager, certainly.
There is a Support Pack that provides a Windows "service" that acts as a client trigger monitor. It can talk to more than one queue or queue manager - but runs each trigger service as a separate thread and should not have problems processing each thread independently.
However, I believe all of these trigger monitors will wait until the triggered process has completed before trying to start another instance of the triggered program.
And MQ itself will only generate a new trigger message if the queue being triggered is not open for input already.
What all of this really means is - the problem is most likely in the application that was triggered, and not in the trigger monitor.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rparti |
| Posted: Sun May 06, 2007 1:59 pm Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
Thanks a lot for your response
| jefflowrey wrote: |
| There is a Support Pack that provides a Windows "service" that acts as a client trigger monitor. It can talk to more than one queue or queue manager - but runs each trigger service as a separate thread and should not have problems processing each thread independently. |
You are right about the trigger service being a Support pack that runs as a Windows service. As I understand then, there is no "functional" difference running the trigger monitor as the default service, or running it as a Windows support pack service.
| jefflowrey wrote: |
However, I believe all of these trigger monitors will wait until the triggered process has completed before trying to start another instance of the triggered program. |
During the latency mentioned, I certainly noticed that more "Windows Processes" were being added and removed. These processes were certainly the trigger monitor processes (which could be the trigger service threads) and the some of the applications linked with triggering.
| jefflowrey wrote: |
And MQ itself will only generate a new trigger message if the queue being triggered is not open for input already. |
When we fired a couple of messages to the trigger queue, they were sitting there waiting to be processed. Only when the initiation queue started dequeuing, were they processed
| jefflowrey wrote: |
What all of this really means is - the problem is most likely in the application that was triggered, and not in the trigger monitor. |
I think so too. However I am told by the team here that this issue occurred 2 weeks back also. It might have happened prior to that too. There would be some reason as to why this could happen? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sun May 06, 2007 5:06 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I'm sure there is some reason. But I don't reallly know.
As a guess, maybe, the triggered application got stuck. It didn't finish, so either the queue manager didn't fire a new trigger because the queue was still open, or the trigger monitor didn't process the initiation queue because the application hadn't finished.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rparti |
| Posted: Mon May 07, 2007 7:45 am Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
| Thanks a ton for the response |
|
| Back to top |
|
|
| jdye |
| Posted: Tue May 08, 2007 12:25 pm Post subject: |
|
|
Apprentice
Joined: 14 Jun 2002
Posts: 31
Location: Kansas City
|
I would think that if a large batch of messages were put in a queue that had 'trigger=every" the init queue would be flooded with messages. Maybe this is what is causing the init queue to accumulate so many messages.
Janet |
|
| Back to top |
|
|
| rparti |
| Posted: Tue May 08, 2007 2:37 pm Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue May 08, 2007 7:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Consider defining your process with "start program name" to put the process in the background and continue with trigger processing.
To debug run the trigger monitor in the foreground and keep your process definition as is. You should find that either the program defined in the process never starts (never opens the queue for input ... Open a PMR) or it hangs (fix your triggered app).
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rparti |
| Posted: Wed May 09, 2007 6:57 am Post subject: |
|
|
Voyager
Joined: 05 Oct 2005
Posts: 86
Location: U.S.A.
|
| fjb_saper wrote: |
Consider defining your process with "start program name" to put the process in the background and continue with trigger processing.
|
MQ is running in a windows server enviornment. Was looking at the system admin manual to see how to define a process with "start program name", but could not find anything. I guess I am not sure what you mean.
| fjb_saper wrote: |
To debug run the trigger monitor in the foreground and keep your process definition as is. You should find that either the program defined in the process never starts (never opens the queue for input ... Open a PMR) or it hangs (fix your triggered app).
|
Would appreciate if you could explain this further or point to a read. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed May 09, 2007 7:01 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
|
| Back to top |
|
|
| Vitor |
| Posted: Wed May 09, 2007 7:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| rparti wrote: |
| fjb_saper wrote: |
Consider defining your process with "start program name" to put the process in the background and continue with trigger processing.
|
MQ is running in a windows server enviornment. Was looking at the system admin manual to see how to define a process with "start program name", but could not find anything. I guess I am not sure what you mean.
| fjb_saper wrote: |
To debug run the trigger monitor in the foreground and keep your process definition as is. You should find that either the program defined in the process never starts (never opens the queue for input ... Open a PMR) or it hangs (fix your triggered app).
|
Would appreciate if you could explain this further or point to a read. |
I think fjb_saper means change the process definition from "program name" to "START program name". For further details, look up the START command in the Windows help.
Running the trigger monitor in foreground is starting a command window and running it from the prompt. This allows you to see the sysout and what the code is doing. You can then act as he suggests. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu May 10, 2007 11:47 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
rparti,
Although a bit far fetched, here is a scenario that could cause the symptoms you see.
Your process definition does not have START at the head of it, so the Trigger Monitor (TM) will wait for a triggered app to finish before going back to the Init queue for another message.
Message lands on the app queue. QM generates a trigger message. TM picks it up and starts your app. Your app opens the q, gets the messages, tries to process it, can't, and for whatever reason starts closing and opening the queue over and over. Each time it does, trigger conditions are met and the QM generates another trigger message. The INITQ piles up, because the TM is still waiting on the app to end. Eventually the app finds that it can process the message (maybe a DB became available), does so, and finally ends. The TM then starts going thru the INITQ and retriggering your app over and over until it finishes the INITQ. Your app starts and ends quickly each time, gracefully handling a 2033 each time it finds the now empty app queue.
Your process def should have something like this in the Application field:
START E:\Programs\MyApp.exe.
If your app is doing that crazy loop described up above, that also needs to be looked at.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
