| Author |
Message
|
| vivica12 |
 Posted: Thu Jul 08, 2010 6:54 am Post subject: MQ behavior when Q fills up |
 |
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
This is just a search for feedback from others. We have had som weird behavior and just wanted to see if others have seen this, or thoughts on if we have something incorrect.
When a Q fills up to it's max-depth, parts of the Qmgr seem to halt or hang. We actually had this today. 1 specific Q filled up, and our Cluster Receiver channel on that Qmgr paused - so we lost our communication within the cluster.
So basically my question, is why would this happen - have others experienced weird behavior like this? there was nothing in the error log (except that messages were going to DLQ), and no FDC files etc.
We've had similar 'weird' behavior preivously when a Q file becomes large, or Q's fill up. It just seems like the Qmgr should handle this better and only that App using that Q should fail.
Looking for feedback. I have not opened a problem with IBM as i have no error logs or FDC files..so nothing to show really.
thanks
_________________
Vivica - signing off |
|
| Back to top |
|
 |
| Gaya3 |
| Posted: Thu Jul 08, 2010 6:56 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
Post your MQ Version and Enviroment details too, it will help experts to answer it properly
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| sumit |
| Posted: Thu Jul 08, 2010 7:07 am Post subject: Re: MQ behavior when Q fills up |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| vivica12 wrote: |
| 1 specific Q filled up, and our Cluster Receiver channel on that Qmgr paused - so we lost our communication within the cluster. |
Which queue is this?
Was there any channel related entry in the error log?
| vivica12 wrote: |
| (except that messages were going to DLQ) |
Is there a dead letter queue?
Did messages go to the dlq?
_________________
Regards
Sumit |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 08, 2010 7:13 am Post subject: Re: MQ behavior when Q fills up |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vivica12 wrote: |
| When a Q fills up to it's max-depth, parts of the Qmgr seem to halt or hang. We actually had this today. 1 specific Q filled up, and our Cluster Receiver channel on that Qmgr paused - so we lost our communication within the cluster. |
Depending on what's full, messages may back up. Did you lose business communication within the cluster or did the queue manager actually disappear from the cluster?
| vivica12 wrote: |
| there was nothing in the error log (except that messages were going to DLQ), and no FDC files etc. |
Working as designed in that respect.
| vivica12 wrote: |
| only that App using that Q should fail. |
If the app putting messages is not connected to that queue manager it shouldn't notice the queue is full. Or any other problem on the receiving end.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Jul 08, 2010 7:15 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
MQ V7.0.1.1
3 qmgrs in a cluster, 1 is basically a routing Qmgr, the other 2 are broker qmgrs. The routing Qmgr is where this happened.
Application Q -- not a system Q, it was a random Q that an app was putting to/getting from.
Yes the DLQ did exist, and 'some' messages went to it, but not all that were failing, only a very small percentage actually ended up on DLQ, others are gone
No channel entry in the log. I can see the channel that is connected to the Q - 24 instances doing PUT, 2 instances doing a Get, I have channel name and connections, but none of this was in the error log.
_________________
Vivica - signing off |
|
| Back to top |
|
|
| sumit |
| Posted: Thu Jul 08, 2010 7:20 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
|
| Back to top |
|
|
| sumit |
| Posted: Thu Jul 08, 2010 7:24 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Piece of advice - Refer MQ info center for any MQ related error. (Find the link at the top of the page.)
_________________
Regards
Sumit |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Thu Jul 08, 2010 7:28 am Post subject: Re: MQ behavior when Q fills up |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
| vivica12 wrote: |
When a Q fills up to it's max-depth, parts of the Qmgr seem to halt or hang. We actually had this today. 1 specific Q filled up, and our Cluster Receiver channel on that Qmgr paused - so we lost our communication within the cluster.
|
my concern is on this part, how a q fill make parts of the qmgr to halt or hang.
this is not desired or expected,as it should throw FDCs, thats what i feel.
is it a bug ? 
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Jul 08, 2010 7:37 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
Sumit,
I appreciate your reponse.
I did look in infocenter, and sometimes it's really hard to find info, when you don't know what you are looking for -- which is exactly why i posted here to look for a thought on what may be happening. I never even thought of Message Retry because the Cluster channels do not show up as connecting to the Q in channel connections -- so it did not seem as if a cluster issue.
I will research further to see if this is part of my issue.
If it is my issue, what would you do about it? The info center article just says 'make your app better', well duh, the app should be pickign up messages - obviously there is a different problem on the app side and we are dealing with that. So during an issue like this, how would you keep your cluster in tact? Do you tell your cluster channels to NOT do retry?
_________________
Vivica - signing off |
|
| Back to top |
|
|
| sumit |
| Posted: Thu Jul 08, 2010 7:49 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
When I compare the details that you have provided in this post and checked the details given in infocenter (same link that I provided here), I felt like they are related.
What I understood is that your broker cluster qmgr is sending msgs to a queue Q1 on routing cluster qmgr. This Q1 is full and hence a batch of msg is waiting at broker qmgr.
The link says in such case
| Quote: |
| This can cause the batch to slow down. |
. This may be a condition which can give an impression like the qmgr is in halt/hung state.
This is basically my assumption. You can very well elaborate what do you mean by hung and how did you determine it.
Were the commands not responding during hung state or some other symptoms? 
_________________
Regards
Sumit |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Jul 08, 2010 7:58 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
In our cluster we have 1 qmgr that bsically routes messages, and 2 qmgrs for brokers.
The Q in question is being used by a Java app on the route qmgr.
When this issue happened the brokers could no longer communciate to/from the routing qmgr.
Sumit, i think you are correct in what happened, I just didn't see the connection from broker to the Q on the routing qmgr -- probably cause it was paused. Since there are lots more apps that needed to be able to funcion in this cluster, that paused channel really messed us up. Is there a way to tell the system to not do this, not do a retry if a Q is full and the broker is trying to talk to it?
This is why i love this site  you guys see a different view. I think Sumit is right, and i missed what it was doing, and didn't search for the right thing.
_________________
Vivica - signing off |
|
| Back to top |
|
|
| Gaya3 |
| Posted: Thu Jul 08, 2010 8:00 am Post subject: |
|
|
Jedi
Joined: 12 Sep 2006
Posts: 2493
Location: Boston, US
|
| sumit wrote: |
This can cause the batch to slow down. |
. This may be a condition which can give an impression like the qmgr is in halt/hung state.
This is basically my assumption. You can very well elaborate what do you mean by hung and how did you determine it.
Were the commands not responding during hung state or some other symptoms? [/quote]
there is no assumptions on this part, if there is any abnormal things are happening MQ Series should fire FDC, that we all know about it.
Its not easy for me to digest about why MQ is getting halted or getting hung in this case and not even firing FDC. Ideally speaking it should not happen.
_________________
Regards
Gayathri
-----------------------------------------------
Do Something Before you Die |
|
| Back to top |
|
|
| sumit |
| Posted: Thu Jul 08, 2010 8:07 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| Gaya3 wrote: |
| if there is any abnormal things are happening MQ Series should fire FDC |
Ture. But as the OP said, there were no FDCs recorded. It might be differnent problem which OP took as hung.
vivica12, FYI - we say a qmgr is in hanging state when it does not respond to any command (and more).
_________________
Regards
Sumit |
|
| Back to top |
|
|
| vivica12 |
| Posted: Thu Jul 08, 2010 8:15 am Post subject: |
|
|
Acolyte
Joined: 13 Jul 2007
Posts: 58
|
You are correct, it appeared to be hung, because traffic wasn't flowing, but the qmgr did respond to commands. So not 'hung'. This would be why no FDC files.
So based on Sumit's info, he is right. It was an issue with a broker app trying to put to a Full Q on the route qmgr -- we just replicated. Good news:)
So now what? This doesn't seem like a good way to work becuase all our other broker apps stop communicating as well - not good?? So can the cluster options change so it doesn't go into retry on cluster receiver?
_________________
Vivica - signing off |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Jul 08, 2010 9:05 am Post subject: Re: MQ behavior when Q fills up |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| vivica12 wrote: |
| ...When a Q fills up to it's max-depth, parts of the Qmgr seem to halt or hang. We actually had this today. 1 specific Q filled up, and our Cluster Receiver channel on that Qmgr paused - so we lost our communication within the cluster... |
Sounds to me like a case of queue full, DLQ full, and receiving MCA unable to put the messages anywhere so rejecting the batch (persistent messages assumed). Is this queue the only target in the cluster?
| vivica12 wrote: |
| ...So basically my question, is why would this happen - have others experienced weird behavior like this? there was nothing in the error log (except that messages were going to DLQ), and no FDC files etc... |
If the previous is correct, then what you're seeing is normal (and expected) behaviour.
| vivica12 wrote: |
| ...We've had similar 'weird' behavior preivously when a Q file becomes large, or Q's fill up. It just seems like the Qmgr should handle this better and only that App using that Q should fail... |
Are these queues being put to across the cluster, or locally, when this happens?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
|
|
