| Author |
Message
|
| guest468 |
 Posted: Mon Nov 03, 2008 7:50 am Post subject: Message accumulation in xmitq |
 |
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
Hi,
We recently saw messages accumulating is close to 10 xmitqs. And the channel status looked running and was not indoubt. I didn't see any error log entries. And FS, cpu, io all looked good. This QMGR has distr channel connectivity to several qmgrs but only 10 seemed to have problems.
Have you guys seen this behavior? Actually this problem lasted for about an hour and got resolved on it's own.
Thanks. |
|
| Back to top |
|
 |
| Mr Butcher |
| Posted: Mon Nov 03, 2008 8:58 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
sounds like a network - performance issue to me. something that runs in parallel to MQ ?!?
stop doing filesharing at work 
_________________
Regards, Butcher |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 03, 2008 9:39 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Were there uncomitted messages in the xmitqs in question?
Were there long-running applications putting messages to these xmitqs?
Is there something in-common about these xmitqs? That is, are they somehow related to each other by application or location or qmgr?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| guest468 |
| Posted: Mon Nov 03, 2008 9:44 am Post subject: |
|
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
This is a dedicated server for MQ/Broker only. And all other channels were running fine(some of them are heavily used).
Could it be the n/w issue in the receiving side? (but again possibility of n/w problem in 10 receiving servers at the same time makes me think otherwise) |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 03, 2008 10:18 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Chat with your network folks.
Is there something in-common about the 10 sender channels that differentiates them from others? For example: do they share the same router? Go through the same firewall? Same telecom service-provider?
Are the channel attributes of these channels different from the others?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| guest468 |
| Posted: Mon Nov 03, 2008 10:33 am Post subject: |
|
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
Good point Bruce. Possibility of all of them belonging to same router/isp makes sense. I will have this checked with n/w team.
And to answer your earlier questions, no there were no uncommitted messages in the xmitq. And all channels have the same definition (they all send functionally same messages) |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 03, 2008 10:37 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| And all channels have the same definition ... |
Specifically: heart-beat interval and disconnect interval?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Nov 03, 2008 12:05 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Odds are against it for 10 different channels to 10 different QMs at the same time (correct assumption?), but if the destination queues filled up, those RCVR channels would drop into their Message Retry Interval x Message Retry Count, during which time no messages would flow over those channels.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Nov 03, 2008 12:07 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| bruce2359 wrote: |
| Quote: |
| And all channels have the same definition ... |
Specifically: heart-beat interval and disconnect interval? |
I don't know that I could set either of these 2 to a value that would cause message throughput problems to the point where XMITQs were backing up.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 03, 2008 12:58 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm pondering why the channels were in running state, and not retry or stopped. The original post stated (oddly worded) the channel status looked running . Either they were running or they were not.
| Quote: |
| I don't know that I could set either of these 2 to a value that would cause message throughput problems to the point where XMITQs were backing up |
.
I agree. I looked at this the other way: what channel attributes might result in a channel end not recognizing that its partner had failed. Heartbeat interval.
What kind of channels? Sender-receiver? Server-requester? Server-receiver?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Nov 03, 2008 1:17 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| bruce2359 wrote: |
| Quote: |
| I don't know that I could set either of these 2 to a value that would cause message throughput problems to the point where XMITQs were backing up |
.
I agree. I looked at this the other way: what channel attributes might result in a channel end not recognizing that its partner had failed. Heartbeat interval.
|
True, but as soon as there were 1 or more messages in the triggered XMITQ it wouldn't matter if misconfigured HBs had failed to notice a problem. At that point the channel would start up. Or it wouldn't.
Looking thru the list of SNDR channel attributes, two jump out at me that could effect throughput if set differently on 2 otherwise identical communication links - SSL cipher spec and the Message Commpression attributes.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 03, 2008 1:30 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| ...1 or more messages in the triggered XMITQ... |
Poster didn't specify if the xmit queue was triggered. I didn't make that assumption. If the xmit queue is triggered, and the init queue was put and get enabled, and if the channel initiator was running, and if...
Are the xmit queue attributes for these failed channels the same for the others?
Any trigger messages in the DLQ? Any automation software (like Tivoli) lurking here that may have intervened, and subsequently corrected?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| xhaxk |
| Posted: Mon Nov 03, 2008 10:51 pm Post subject: |
|
|
Apprentice
Joined: 30 Oct 2008
Posts: 31
|
if the channels had not stopped to INACTIVE, it does not matter whether they are triggered or not
if the queues at the RCVR had filled up, the status wouldbe PAUSED during the MRRTY
maybe there were more putting apps while the depth increased
the most likely cause is that there was network delay |
|
| Back to top |
|
|
| guest468 |
| Posted: Tue Nov 04, 2008 7:38 am Post subject: |
|
|
Centurion
Joined: 30 May 2006
Posts: 146
Location: NY
|
Hi Guys,
The channel was in running state(indoubt:no, status:running, mcastat:running). There was no increase in CURMSGS during that time. And xmitqs are triggered. Channel type is sender. No SSL or message compression is used (no exit either). And there were no error log entries.
The problem got resolved on it's own after an hour. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Nov 04, 2008 8:11 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| There was no increase in CURMSGS during that time. |
I am further confused.
From the MQSC manual:CURMSGS - For a sending channel, this is the number of messages that have been sent in the current batch. CURMSGS is a value returned from a DIS CHS command. I would not be surprised if CURMSGS remains constant. A batch is a batch.
What else did the DIS CHS command tell you? On the sender side, did the number of messages sent increase? On the receiver side, did the number of messages received increase? When was the last batch sent? Received? An hour ago?
Your original post suggested that messages were in the xmitqs. Did you verify this by displaying the xmitq? DIS QL(xmitq) CURDEPTH.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
