| Author |
Message
|
| RaviKrG |
 Posted: Wed Jan 21, 2009 4:11 am Post subject: Message Flow not processing the messages - Discussion Level |
 |
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Hi, I would like to keep this on the discussion level only.
I have a flow where all was fine but one day there were 3 messages on the input queue and it was not being processed and Qpasa threw an alert for Message flow not processing the messages.This i faced on production
I checked the following :
1) Broker was running
2) Execution group was running
3) Message flow was running
4) MQ services were all fine
5) There was no message on to the Backout queue
6) Checked the broker logs : info, warning and error, found that there is no entry of any problem infact there was no logs in the broker log for the time stamp
7) Checked the MQ log and no problem
 Since broker was on solaris checked the process through 'prstat' and found all was fine.
Finally I did a 'mqsireload BROKERNAME -e EXECUTION GROUP NAME' but still there were three messages on the input queue and no effect.
Now I was still in search that why the messages were not being processed. ?
I think as for
1) If the message flow is hung then how can I know that MF is hung and for this issue what steps need to be taken.
2) If there was a poisson message then how can I say that the messages not being processed are poisson messages and how to deal with this.
Thanks for your help. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Jan 21, 2009 4:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Do you have a BOTHRESH on your input queue? If not set it to 3 and check that the messages are not going to the DLQ. It could be that the flow is not able to process the messages and backs them out every time.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Wed Jan 21, 2009 4:25 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Thanks fjb, at present this can not be done in production but sure i will try this on my normal pre env system.
also BOTHRESh is zero. and since the BO queue is defined why this should go to the DLQ.
Since the backout queue is defined why the message was not put to BO queue when it was not processed. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Wed Jan 21, 2009 4:28 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
" It could be that the flow is not able to process the messages and backs them out every time. "
This itself is my question ? WHY ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 21, 2009 4:34 am Post subject: Re: Message Flow not processing the messages - Discussion Le |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RaviKrG wrote: |
I checked the following :
1) Broker was running
2) Execution group was running
3) Message flow was running
|
How did you determine the above? Was the queue actually opened for input? What is the criteria for the QPasa alert?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Wed Jan 21, 2009 4:43 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Vitor :
How did you determine the above
--------------------------------
1) checked the bip service : ps -ef | grep bip
2) checked through mqsilist BN and also ps -ef
3) checked through mqsilist BN -e EG and also ps -ef
Was the queue actually opened for input?
--------------------------------------
Yes.
Also the rest of the messages before were processed and if this was the case why this did not happen in whole one year
What is the criteria for the QPasa alert?
------------------------------------
At present I cannot tell this I will have to check since we dont do much for Qpasa . |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 21, 2009 4:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
These 3 messages are apparently valid in all respects, and have been properly committed by the putting application? They're not just sitting on the queue making the depth go up, they're real messages?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Wed Jan 21, 2009 4:57 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Vitor, For your last update I will just say that when I browsed the queue, It had the real data and the data contained was also fine.
About the "committed by the putting application" I will not be able to tell as I did not check (say i didnot know how to do this) but I would also like to say that the message was there for about 6 hrs and thats a long time. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 21, 2009 5:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RaviKrG wrote: |
| I would also like to say that the message was there for about 6 hrs and thats a long time. |
Which supports my hypothisis. A putting application / thread puts the message to the queue, then abends for some unknown reason. The UOW is not committed, so the message is not committed and available for get, and is not rolled back, so appears on the queue.
(It's this behaviour that's one of the reasons reading messages by getting queue depth and using a for..next loop is frowned on; queue depth is not as absolute as you think! Better to read until the queue manager says the queue is empty.)
Because your position seems to be this: the components are running normally, other messages are being processed and yet these messages sit there, not being processed, not going to a backout queue, not being dead lettered. If the flow is running (and we seem to have established it is) then when the MQInput node is running it's not being returned these messages by the queue manager, and yet there's not an error message to be seen. So there's an "acceptable" reason for these messages not being returned, and the most likely reason here is that the queue manager believes they're not committed and not eligable for return.
I could of course be hopelessly wrong.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 21, 2009 5:13 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RaviKrG wrote: |
| About the "committed by the putting application" I will not be able to tell as I did not check (say i didnot know how to do this) |
You may find this valuable, both in this specific question, and with your more general problem:
http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/topic/com.ibm.mq.csqzax.doc/rtmqavbl.htm
This makes a point which I'd overlooked; that the putting application might not have abended, but simply still be running with the UOW open!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 21, 2009 5:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
There are easy ways to tell if a queue has uncommitted messages on it.
It's not clear from "the messages stay on the queue" whether this has been done or not.
But it is clear that you can't browse uncommitted messages.
It's entirely possible for a message flow, particularly if it only has a single instance, to not read messages off an input queue during the time it takes for it to process it's current message.
It seems more likely that either someone told the message flow to stop, and didn't tell RaviKrG that they had done so, or that there was some kind of long running loop inside the flow or a long running transaction against a database. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Wed Jan 21, 2009 7:16 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Thanks Vitor, I feel the link provided will surely give an idea if the messgaes are uncommited and so that goes for the application error which is putting the messages to the input queue.
But mqjeff post "But it is clear that you can't browse uncommitted messages" if this is the case that the uncommitted messages canot be browsed so in my case I browsed all the three messages and the messages looked to be fine. and then the problem lies some where else.
"some kind of long running loop inside the flow or a long running transaction against a database." this might be a possible cause.
"someone told the message flow to stop, and didn't tell RaviKrG that they had done so" - is this really good to see these that too on production and does not log any entry.
But again if the messages are committed then can these be a possible reason 1) MF is hung 2) message is poison message |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 21, 2009 7:37 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RaviKrG wrote: |
| But again if the messages are committed then can these be a possible reason 1) MF is hung 2) message is poison message |
1) The flow will appear to hang if there's a loop in the ESQL and the coder needs to be trouted or there's a database issue
2) If the message is poison, it should go to a backout queue (or at least have the backout count increasing)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Wed Jan 21, 2009 8:12 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Viotr,
1) Do you say that loop in ESQL is the only possibility for a flow hung ?
2) That should be enough for the poison as this should be recorded in broker logs. (Poison message update closed) |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 21, 2009 8:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| RaviKrG wrote: |
| 1) Do you say that loop in ESQL is the only possibility for a flow hung ? |
No, I said database issue as well. If there was a loop in the ESQL you'd find messages not being processed (if there was only a single thread in the execution group) but you'd also see the EG chewing CPU. Depending on the exact circumstances of course.
| RaviKrG wrote: |
| 2) That should be enough for the poison as this should be recorded in broker logs. |
I didn't think poison messages were logged, just shoved onto the backout queue. Hence my comment about backout count.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
