| Author |
Message
|
| moonwalker |
 Posted: Wed May 20, 2015 12:09 pm Post subject: Bottle neck at Gateway queue manager |
 |
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
Hello All,
We have a flow of traffic from WAS to gateway queue manager where it load balances to hit the broker queue managers. Gateway QM and broker QMs are all clustered.
When we attempted to go live we noticed the gateway QM to be a bottle neck as the SYSTEM.CLUSTER.TRANSMIT.QUEUE was stacking messages.
Then we increased the batch size on the cluster sender and receiver channels but no improvement. We are at go live and are running out of options to resolving this thing. Please share your thoughts, thanks in advance. |
|
| Back to top |
|
 |
| exerk |
| Posted: Wed May 20, 2015 12:15 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Was this apparent in the load-test environment? Can the behaviour be reproduced in a non-live environment? And are the messages stacking because of the application's commit cycle being too high?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| moonwalker |
| Posted: Wed May 20, 2015 12:18 pm Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
well most of production problems aren't reproducible. This was figured on the day of go live. So its on production boxes.
Please elaborate commit cycle thing. The producer is a Websphere application server application, dropping messages using JMS queue connection factories.
Appreciate your response! |
|
| Back to top |
|
|
| exerk |
| Posted: Wed May 20, 2015 2:38 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Check the S.C.T.Q for uncommitted messages - your producer may be dropping a lot of messages on the queue but not committing them, hence at least one reason why the change in batch size would make no difference.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| moonwalker |
| Posted: Wed May 20, 2015 10:03 pm Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
appreciate your suggestion. i will check that thing first and set it to NO incase it reads YES.
one other point i got to know is the gateway queue manager is on MQ version 6 while broker queue managers are version 8. Any possibility of interfacing issues between MQ version 6 and 8?
Thanks. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu May 21, 2015 12:32 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| moonwalker wrote: |
appreciate your suggestion. i will check that thing first and set it to NO incase it reads YES.
one other point i got to know is the gateway queue manager is on MQ version 6 while broker queue managers are version 8. Any possibility of interfacing issues between MQ version 6 and 8?
Thanks. |
As far as I am aware there should be no interoperability issues although I personally would migrate the gateway ASAP to a supported version.
Questions: Is your lower-level test environment a 'true' copy of the Production environment, i.e. MQ versions etc., and are the FR queue managers V8.0 level?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu May 21, 2015 5:12 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Also check the messages accumulating on the SCTQ. What is their destination? Do you have a queue full situation on any of the destination queue managers? A destination queue manager with a queue full will seriously slow down cluster traffic... and depending on batch size, the queue needs not be at 100%... To experience this set a max queue depth of 25 with a batch size of 50 for instance and watch where your messages are landing (DLQ ?) 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| moonwalker |
| Posted: Thu May 21, 2015 7:56 am Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
| the destination queue has a max depth of 1000000. The destination queue is clustered and there are 4 instances. So the traffic is load balanced across four boxes. So no problem there. |
|
| Back to top |
|
|
| moonwalker |
| Posted: Thu May 21, 2015 11:23 am Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
| we are replacing the cluster with just remote definition queues to see if it improves things. if it does then SCTQ is a true bottleneck and the design of gateway QM to load balance sucks. if not we have more analysis to be done. I will keep them all posted here. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu May 21, 2015 11:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You should take normal steps to optimize your channels and optimize your logs (don't want lots of log rotation).
You can, also, create more than one CLUSRCVR for a given qmgr, which will create additional clussdrs. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu May 21, 2015 11:55 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| moonwalker wrote: |
| we are replacing the cluster with just remote definition queues to see if it improves things... |
Which it won't if you have an application that is creating a lot of messages and not committing them.
| moonwalker wrote: |
| ...if it does then SCTQ is a true bottleneck and the design of gateway QM to load balance sucks... |
Not necessarily - any XMITQ can be a bottleneck especially as only one channel can service a vanilla XMITQ (see mqjeff's comment).
| moonwalker wrote: |
| ...if not we have more analysis to be done. I will keep them all posted here. |
Again, check what the app is doing because there is little you can do with MQ to mitigate a badly written app.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| moonwalker |
| Posted: Thu May 21, 2015 1:28 pm Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
to comment on the commit, the source application places the request on one queue, closes the queue and opens a different queue and listens to receive a response.
are we good to rule out the problems with commit?
Since at a point in time we noticed 212 messages stacked up on gateway's (which is a partial repository) SCTQ may we run multiple cluster sender and an equal number of cluster receiver channels to have those messages diffused off the SCTQ of the gateway?
I was told that having multiple cluster sender and receiver might not help as they might not be used by the cluster. not sure if its true. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu May 21, 2015 1:52 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| moonwalker wrote: |
to comment on the commit, the source application places the request on one queue, closes the queue and opens a different queue and listens to receive a response.
are we good to rule out the problems with commit? |
Yes, provided you checked for, and eliminated as a possibility, large numbers of uncommitted messages.
| moonwalker wrote: |
| Since at a point in time we noticed 212 messages stacked up on gateway's (which is a partial repository) SCTQ may we run multiple cluster sender and an equal number of cluster receiver channels to have those messages diffused off the SCTQ of the gateway? |
I would not describe 212 messages as being a large number.
| moonwalker wrote: |
| I was told that having multiple cluster sender and receiver might not help as they might not be used by the cluster. not sure if its true. |
Some empirical testing will prove/disprove the hypothesis.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| moonwalker |
| Posted: Thu May 21, 2015 2:14 pm Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
| 212 messages per second while the true load on production was close to 350 messages per second. that's beyond 60% |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu May 21, 2015 5:33 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
| moonwalker wrote: |
| 212 messages per second while the true load on production was close to 350 messages per second. that's beyond 60% |
Eh? You are sending 350 per second and you see the queue depth at 212. Always? second after second, minute after minute, hour after hour, 350 messages per second are being sent, but the q depth stays at exactly 212?
How do you know its 212?
How often do you check? How do you check?
Do you know if its the same 212 messages always, second after second, minute after minute, hour after hour?
Are you checking Q depth and/or Hi Q depth?
212 is awfuly close to 4 x 50. The default batch size on channels is 50. I beginning to suspect there is no problem here.Just the channels doing there job, moving 350 messages a second, and every time you check, you happen to see the depth of a batch of messages that haven't been committed yet.
If you are sending 350 messages a second, what are the 4 queues receiving per second? If its close to 350/4, and there are no messages "missing", nothing is stuck. Everything is working normally.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
