| Author |
Message
|
| pandeg |
 Posted: Thu Apr 14, 2016 6:33 am Post subject: Performance degraded from local MQ to Remote MQ Connection |
 |
|
Disciple
Joined: 21 Oct 2014
Posts: 195
|
Hi, We have lots of JMS applications which were using local MQ and then we migrated them to Remote Central MQ inorder to save the licensing cost and ease of support and maintenance ,but we found that the transaction time of the messages increased to three folds reducing the performance.
For example - previously it was taking 10 minutes to send 500 messages , after centralization it takes 30 minutes to send 500 messages. Every things remains the same , network, bandwidth, JMS connection configuration, MQ objects.
I can understand the performance cannot be same when application connecting locally vs connecting centrally. But how much performance difference can be expected. Since , its asynchronous communication in our case , its not creating any issue at this time but we want to know if we going below the expected performance level. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Apr 14, 2016 6:52 am Post subject: Re: Performance degraded from local MQ to Remote MQ Connecti |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| pandeg wrote: |
| Every things remains the same , network, bandwidth, JMS connection configuration, MQ objects. |
That can't be true. You must be traversing network routes and network components that you're not traversing when you connect locally.
(I'm assuming that when you say "using a local MQ" you mean a client connection to a local queue manager not a binding connection. If you've gone from binding to client I've found your problem.)
What do the channel & network statistics say is happening over these 30 minutes?
_________________
Honesty is the best policy.
Insanity is the best defence.
Last edited by Vitor on Thu Apr 14, 2016 6:53 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Apr 14, 2016 6:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
10 minutes to send 500 messages is an extreme performance problem.
That suggests either significant issues with the config of your app server (or queue manager) or significant issues with your application.
The fact that it goes up by a factor of 3 when you move to network location makes it more likely that it's an application issue. Although your queue manager could be undersized for the transactions you're doing.
Even if you're sending asynchronous messages, those can still sit in uncommitted transactions until your application does the right thing.
Start by reviewing how the application handles transactions. Follow up by how the app server is configured for transactions. Then work with your MQ admins to review their config and setup for transaction size and handling.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 14, 2016 7:29 am Post subject: Re: Performance degraded from local MQ to Remote MQ Connecti |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| pandeg wrote: |
| Every things remains the same , network, bandwidth, JMS connection configuration, MQ objects. |
This statement contradicts what surrounds it. Many things have changed.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| pandeg |
| Posted: Thu Apr 14, 2016 7:34 am Post subject: Re: Performance degraded from local MQ to Remote MQ Connecti |
|
|
Disciple
Joined: 21 Oct 2014
Posts: 195
|
| Quote: |
That can't be true. You must be traversing network routes and network components that you're not traversing when you connect locally. |
I meant its same in the sense from the remote Queue Manager end. When the application were connecting locally, its still the same network configuration between these local and remote queue manager.
| Quote: |
(I'm assuming that when you say "using a local MQ" you mean a client connection to a local queue manager not a binding connection. If you've gone from binding to client I've found your problem.) |
yeah, application is connecting in client mode both in case of local MQ and remote Central MQ.
| Quote: |
| What do the channel & network statistics say is happening over these 30 minutes?[ |
we haven't captured any channel and network statistics. Can you please let me know what channel and network information should i look for ? |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Apr 14, 2016 7:38 am Post subject: Re: Performance degraded from local MQ to Remote MQ Connecti |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| pandeg wrote: |
| yeah, application is connecting in client mode both in case of local MQ and remote Central MQ |
So, localhost as far as the network was concerned - and maybe never would have gone across the network?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 14, 2016 7:59 am Post subject: Re: Performance degraded from local MQ to Remote MQ Connecti |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| exerk wrote: |
| pandeg wrote: |
| yeah, application is connecting in client mode both in case of local MQ and remote Central MQ |
So, localhost as far as the network was concerned - and maybe never would have gone across the network? |
The network people should be able to produce traffic statistics including (but not limited to):
- traffic pattern (are the 500 messages being transmitted continuously but very slowly, or is there a significant pause between each message)
- route taken (are there a significant number of hops degrading performance)
- packet loss and retry (is the connection unreliable)
The channel will tell you how quickly messages are being committed from the other end and other statistics.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Apr 14, 2016 8:25 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The change in network routes only altered the performance by a factor of 3.
This is huge, but the performance was already incredibly bad to begin with.
In this rare case, it's better to start with troubleshooting the application, move to troubleshooting the queue manager, and *then* blame it all on the network team - instead of the opposite order.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Apr 14, 2016 8:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| mqjeff wrote: |
The change in network routes only altered the performance by a factor of 3.
This is huge, but the performance was already incredibly bad to begin with.
In this rare case, it's better to start with troubleshooting the application, move to troubleshooting the queue manager, and *then* blame it all on the network team - instead of the opposite order. |
If the only thing that changed is "the network," then the cause is most likely "the network."
So, no line or lines of code changed? No o/s changes? No MQ changes? No hardware changes?
Really? What exactly do you mean by "migrated?" When you migrated to a different center, did you physically move the server hardware? Or did you copy the server file system, and restore it on a new server hardware?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Apr 14, 2016 9:20 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bruce2359 wrote: |
| mqjeff wrote: |
The change in network routes only altered the performance by a factor of 3.
This is huge, but the performance was already incredibly bad to begin with.
In this rare case, it's better to start with troubleshooting the application, move to troubleshooting the queue manager, and *then* blame it all on the network team - instead of the opposite order. |
If the only thing that changed is "the network," then the cause is most likely "the network." |
Sure. That explains why the performance degraded by a factor or 3.
It doesn't explain why the performance was so horrible in the first place.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
|
|
