| Author |
Message
|
| Kireeti84 |
 Posted: Wed Mar 11, 2015 7:45 am Post subject: Issue with a queue |
 |
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
Need suggestions. We have a queue on prod and it started en queueing all messages until we restart the enigine ? any help greatly appreciated.
Last edited by Kireeti84 on Wed Mar 11, 2015 7:50 am; edited 1 time in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Mar 11, 2015 7:50 am Post subject: Re: Issue with a queue |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Kireeti84 wrote: |
| Need suggestions. We have a queue on prod and it started de queueing all messages until we restart the enigine ? any help greatly appreciated. |
Not enough context. With the information you gave all I can say is "behaves as designed" ... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Wed Mar 11, 2015 7:56 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
| Ok.. we have a flow which drops the messages in a Audit queue for the auditing purpose. this queue started enqueuing all the messages last night until we re start the audit engine. any inputs so that i can trouble shoot ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 11, 2015 8:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Kireeti84 wrote: |
| Ok.. we have a flow which drops the messages in a Audit queue for the auditing purpose. this queue started enqueuing all the messages last night until we re start the audit engine. any inputs so that i can trouble shoot ? |
Still not a lot of context or clarity there.
SO you have a flow, which as part of it's processing drops messages onto a queue where they're processed asyncronously by an audit engine separate to the producing flow. Last night the messages backed up and were not being processed until you restarted the audit engine.
The input engine was hung up trying to process a message into a file / database / wherever it puts the audit information you've not bothered to tell us. When you restarted it this resolved the problem.
Sorted.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Mar 11, 2015 8:57 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Could it be a poison message?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 11, 2015 9:02 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
| Could it be a poison message? |
I wondered about that, but it should have still been on the queue when they restarted.
Maybe the OP will be along with some more details. Or some details.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Wed Mar 11, 2015 10:48 am Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
| Well. The moment the flow drops a message the audit engine will pick the messages and insert them in DB. thats the regular process. last night the audit engine stops picking the message until we restart. after that everything back and processing as expected. now the team wanted me to analyze the root cause and how we avoid this kind of problems in future. now my concern is how we trouble shoot and identify the problem ? |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Mar 11, 2015 10:52 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Does the input queue that is the problem have a BackOut Queue defined?
IMHO, it should do.
Then if there is a problem with processing of the message then it should end up on the BOQ provided the flow throws and exception.
Was anything reported in the System Error Log?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 11, 2015 10:59 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Kireeti84 wrote: |
| now my concern is how we trouble shoot and identify the problem ? |
Well what do the logs from the audit engine say? If that stopped picking up the messages, then the obvious first move is to see what it says about why it stopped. Possible answers include but are not limited to
"when it tried to insert the last record it read from the queue the database connection froze"
"when I tried to work out what to insert I ran out of Java heap & sat waiting for the garbage collection to happen"
"when something bad happened (like the database connection freezing) I tried to log the error in the database through the frozen connection and started looping"
Another possibility is the one my worthy associate points out - a poison (bad) audit message, which causes it to read the message, fail, roll the message back and so on. This would show up in the audit engine log and IMHO is unlikely as the message should have been on the top of the queue when you restarted. Unless you flushed the queue first, or the audit engine transaction handling sucks & you blew the message our when you restarted.
Here's another angle: what prompted you to post here? What led you to think it's a problem with the MQ software? What clues pointed you in this direction?
Better information, better advice.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 11, 2015 11:00 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| smdavies99 wrote: |
Does the input queue that is the problem have a BackOut Queue defined?
IMHO, it should do. |
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 11, 2015 11:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| Kireeti84 wrote: |
| now my concern is how we trouble shoot and identify the problem ? |
Well what do the logs from the audit engine say? If that stopped picking up the messages, then the obvious first move is to see what it says about why it stopped. |
This is the only troubleshooting you need to do. "Why did the audit engine stop working?"
You shouldn't waste any time on figuring out "Why did MQ let messages pile up on some queue". You shouldn't waste any time figuring out "Why is my message flow putting messages to some queue". |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Mar 11, 2015 11:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| You shouldn't waste any time on figuring out "Why did MQ let messages pile up on some queue". You shouldn't waste any time figuring out "Why is my message flow putting messages to some queue". |
It's to this my questions surrounding "what led you here" were obliquely referring.
If the answer is something along the lines of "the audit engine reported 2033 errors" it's at that point we move to the MQ layer and it is to be hoped that the OP would have mentioned that before now.
If the answer is something along the lines of "there were messages in the queue, we restarted the engine that processes the messages and the messages went away so it's obviously a message problem", then
The Trout Of Teaching Will Return
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Kireeti84 |
| Posted: Wed Mar 11, 2015 12:14 pm Post subject: |
|
|
Apprentice
Joined: 18 Oct 2013
Posts: 40
|
The Audit engine stops de queueing messages from ABC Queue on gateway queue manager QMGR123.
checked the logs. not much info.
Vitor . Yes im guessing its a message problem. but there are more than 50K unprocessed messages sitting in the queue. the audit engine restart should not convert the bad message into good one and process it right ?
Any thoughts sir ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Mar 11, 2015 12:23 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Loss of connectivity with the DB where you put the log records?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Mar 11, 2015 12:25 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's the audit engine that is the problem.
If it's not logging much, then that is *also* a problem.
There's almost nothing a sending application can do to prevent a consuming application from reading a message. The only thing that can be done is to fail to commit the PUT.
You would know that the put had not been committed, though, because you tried to browse the messages and couldn't see them. |
|
| Back to top |
|
|
|
|
