| Author |
Message
|
| RaviKrG |
 Posted: Sun Sep 21, 2008 3:04 am Post subject: Message Flow not processing the messages |
 |
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Well, I have a message flow which was processing the messages very with no problem, but I found that at one time there were 3 messages on the input queue and the flow was not processing the messages as the queue was pilling up
I checked up all the following things on the production as
* Message flow was running
* The execution to which the flow was deployed was also running
* memory usage of the execution group and also the message flow
* Checked up the broker (no error logs or info or warning logs)
* Broker was running
* Queue manager was running
* Checked the queue manager log , there was no error or any problem reported
Can somebody tell why was the messages not processed as I also tried to restart the flow as
mqsireload broker_name -e execution group
even then the flow didnot process the messages.
Thanks |
|
| Back to top |
|
 |
| bower5932 |
| Posted: Sun Sep 21, 2008 3:59 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| Try browsing the queue with amqsbcg to see if the messages are really there. Could they be uncommitted? Look at the backout count of the messages to see if it is increasing. Does the actual queue have an open input count? |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Sun Sep 21, 2008 4:06 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Thanks for your reply bower5932.
Yes we have browsed the messages and found that there were 3 messages on the input queue, even this has not been transfered to backout queue like in some other cases.
Also the actual queue has the open input count as one |
|
| Back to top |
|
|
| broker_new |
| Posted: Sun Sep 21, 2008 5:33 am Post subject: |
|
|
Yatiri
Joined: 30 Nov 2006
Posts: 614
Location: Washington DC
|
I had the same problem last month...
try to post what you message flow does.....
Actually the message flow inserts the records into a SIEBEL database.
There are several other applications that are performing some operations on the same...What i did was when my flow is not processing the messages i stopped the message flow and bounced it ....you know what happens when you start the broker it shall start with message flow stopped .. 
So turn on the debugger and start the message flow you will see where the problem is .....
For me the mystery was at the database broker tries to insert the records into database and the there are several thousands of locks acquired by other applications,it puts broker requests in queue obviously that thread hangs not allowing it to process other messages in queue which makes us to think that it is not processing the messages at all .....  |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Sun Sep 21, 2008 5:50 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
broker_new , what you have suggested to do, do you really think that it is always better to do all the things in the production environment..
ya I agree with the bouncing the message flow is ok but the other steps is it ok to follow ALWAYS |
|
| Back to top |
|
|
| broker_new |
| Posted: Sun Sep 21, 2008 5:57 am Post subject: |
|
|
Yatiri
Joined: 30 Nov 2006
Posts: 614
Location: Washington DC
|
yes I knew that is not the correct way of tackling this issue.....But
this is the worst case scenario...I was not able to figure out what the issue is until i put it in debug mode and ask the DBAs to query for the locks on the database .. 
Let me know if you have better solution for handling this. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Sun Sep 21, 2008 6:24 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Can you tell in what all scenarios the messages are not processed (offcourse it depends on the many things) but I want to know like in normal scenarios if the messages are not processed then what all things need to be checked to find the proble.
As i have already mentioned the things in my first post, but if i have missed out any thing
Also if even after all the other things I dint find any cause then what can be the issue and how can I find whether its a database issue? |
|
| Back to top |
|
|
| broker_new |
| Posted: Sun Sep 21, 2008 8:49 am Post subject: |
|
|
Yatiri
Joined: 30 Nov 2006
Posts: 614
Location: Washington DC
|
You should also check for the Input queue properties whether it is get enabled or not.
But there is no way that broker cannot process the input messages...Define a Backout queue for Input queue and start the process again if the backout count increases that means there is some problem in looping or a bug in the code
There is a possiblity where you are not handling the loopings correctly...(i.e)The loop is not terminating and leading it into a infinite loop...
Its better you explain what your flow exactly does and we shall interprete the failure scenarios |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Sun Sep 21, 2008 9:07 am Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi,
| broker_new wrote: |
| But there is no way that broker cannot process the input messages...Define a Backout queue for Input queue and start the process again if the backout count increases that means there is some problem in looping or a bug in the code |
If the msg was trying to back out and landing on the input queue every time, then you should have been able to see a 4-3-4-3 change in msg count on the queue.
| broker_new wrote: |
| There is a possiblity where you are not handling the loopings correctly...(i.e)The loop is not terminating and leading it into a infinite loop... |
If the loops were the issue, the memory usage should have shot up, which does not seem to be the case from RaviKrG's first post.
| broker_new wrote: |
| Its better you explain what your flow exactly does and we shall interprete the failure scenarios |
This is something which we require now to suggest further steps to take.
Regards |
|
| Back to top |
|
|
| broker_new |
| Posted: Sun Sep 21, 2008 9:20 am Post subject: |
|
|
Yatiri
Joined: 30 Nov 2006
Posts: 614
Location: Washington DC
|
Well, I dont see anything as of now...
It looks like this is an interesting issue let's wait and see any of our experts reply and iam curious to know what was the problem...  |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Thu Sep 25, 2008 12:04 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Well Just to update that the scenario was that the input queue had the backout queue defined and also the following things to update that the input queue had the depth as 3 (it was neither increasing nor decreasing almost for about 10 hours untill we took the messages out of the input queue and saved).
I have already mentioned that all all steps I had taken.
Still I am not clear what the real Issue Is here in this case.
Thanks |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Thu Sep 25, 2008 12:21 am Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
About the Flow :
The input queue get the message, and submit the data in to the database. After the successful upload(submit) of the data to the database we route the data to two queues according to the contents (ie some id used). If the database submit was not successful then an email is sent.
Thanks. |
|
| Back to top |
|
|
| keenlearner |
| Posted: Thu Sep 25, 2008 5:35 am Post subject: |
|
|
Acolyte
Joined: 24 Aug 2006
Posts: 62
|
How is exception is handled. I thnik message has become a poision message. The thing is that the messahe may have ecountered an exception and rolled back to the input queue. Noow this message blocks the processing of rest of the message.
Try this out.
1. Read all the message and write each message to file.
2. Put all the message back except the first one and monitor the results. |
|
| Back to top |
|
|
| RaviKrG |
| Posted: Thu Sep 25, 2008 8:18 pm Post subject: |
|
|
Master
Joined: 07 Sep 2008
Posts: 240
|
Thank you keenlearner for replying.
Well I have two questions for your update
1) When there is a backout queue defined for the input queue then why the queue cannot go the backout queue when not able to process.
2) Before I use to check for poison message in MQ as browsing the message and one of the xml tag will not have the proper format.
How will know the poison message in this case (broker).
Thanks |
|
| Back to top |
|
|
| keenlearner |
| Posted: Fri Sep 26, 2008 9:30 pm Post subject: |
|
|
Acolyte
Joined: 24 Aug 2006
Posts: 62
|
Poisons message are those messages which keep on processing and every time they get exceptioned out from the flow and rolls backs to the input queue.
What i assume is that the first message is getting exceptioned out from the flow and sitting on the Input queue of the flow.
Now answeres to your questions
| Quote: |
1) When there is a backout queue defined for the input queue then why the queue cannot go the backout queue when not able to process.
2) Before I use to check for poison message in MQ as browsing the message and one of the xml tag will not have the proper format
|
1. yes Ideally the message should get into the back out queue it there is one defined for the input queue. But this depends on the Backout Threshold property of the queue. If the Backout count in MQMD header equals the Backout Threshold of the queue then only it goes to the Backout queue. Please check the values for those two.
2. A poison message can be of proper XML format as well. |
|
| Back to top |
|
|
|
|
