| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Performance tuning for Network Latency |
« View previous topic :: View next topic » |
| Author |
Message
|
| fjb_saper |
 Posted: Thu Jul 04, 2019 10:58 pm Post subject: |
 |
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Bandwidth utilization has very little to do with MQ. A lot more is dictated by TCP/IP. Especially when long distances, net lag of 140ms or thereabouts and some packet loss are involved.
At this point TCP/IP bandwidth utilization becomes quite poor and you'd be well advised to look for alternative solutions. 
_________________
MQ & Broker admin |
|
| Back to top |
|
 |
| exerk |
| Posted: Thu Jul 04, 2019 11:35 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| saurabh25281 wrote: |
...We could have packed a lot more of messages on our network, and improved our performance, if such feature was available with MQ.
TBH, this is a very basic feature that one expect from a mature product like MQ. |
That's like expecting your postman to 'compress' more letters into his sack so more can be delivered in one round - but guess what? The more letters, the heavier the sack, and the heavier the sack, the slower the postman!
It bears repeating again, no amount of tuning of MQ will mitigate poor underlying transport issues.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Jul 05, 2019 12:23 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| saurabh25281 wrote: |
This is evidence that our network bandwidth is not utilized fully. We could have packed a lot more of messages on our network, and improved our performance, if such feature was available with MQ.
TBH, this is a very basic feature that one expect from a mature product like MQ. |
Can you just confirm whether you are talking about client channels or sender/receiver channels here?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 05, 2019 4:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| saurabh25281 wrote: |
| The DCs in question are located across continents (4600 miles apart) and the ping represents realistic response times, looking at sources in the public domain I would assume. Correct me If I am wrong on this. |
That's a lot further apart than most of us are used to. DR DCs are more commonly are as close as possible given the disasters they're protecting against (for example, 300 miles away the other side of a mountain range protects from flooding; repeat for all other disasters in your list after "flooding".
But yes, that's a good response for 4,600 miles. Explain to your business users that if you go from using the local DC to the DC 4,600 miles away because there's been a disaster, SLA will be impacted. 4,600 miles is a long way for the electricity to run.
| saurabh25281 wrote: |
| I should be able to choose the buffer size as per my application needs. We can do that for TCP buffer for Sender/Receiver channels. |
But the MCA determines how much buffer to use. In this instance, your application would need that code.
| saurabh25281 wrote: |
| We are the application owner and know the average message size and hence can approximate the buffer size we want. |
No you're not. The point of decoupled messaging is that 2 separate applications are taking to each other so you're not the owner of the sending application and have no control over what they're doing. If you do own both applications, remove MQ from the picture and use a replicated database!
| saurabh25281 wrote: |
| We can decide an acceptable number of messages which does not overrun the RAM by any measure. And the messages in RAM would still be there in Disk until there is a commit request from the client application, so there is no risk of loosing messages. |
You're again assuming message size, and if you pull an acceptable number of messages (like the 2,000 in my example) then my comments about the 2001st message apply.
| saurabh25281 wrote: |
| It's our responsibility on how we handle failure. Let the application decide for themselves. |
MQ controls this with MQGET. Message control (especially for persistent messages) is a feature of MQ. Are you saying IBM can abdicate this responsibility for this new verb?
| saurabh25281 wrote: |
| Like I mentioned earlier, the application can decide how much buffer it chooses and what is the optimum no. of messages that generates a good enough throughput and acceptable response time. Probable batch size can be 50 for e.g. for my application needs, same as Sender channel batch size. |
So you want read ahead with this, so the application can be processing one filled buffer while MQ is filling the next one? Otherwise you're still waiting for the buffer to fill with all that latency.
| saurabh25281 wrote: |
| Doesn't the batch feature apply to Sender/Receiver channels? Do we provide a lower QoS for persistent messages in the case of Sender/Receiver channels? |
Not a valid comparison. The MCAs on a sender/receiver channel are in an asynchronous partnership controlled by handshakes. The application client connection is synchronous.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 05, 2019 4:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hughson wrote: |
| Can you just confirm whether you are talking about client channels or sender/receiver channels here? |
The OP is talking about it taking longer to do an MQGET from his DR queue manager than the local one. I'm thinking client.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| saurabh25281 |
| Posted: Fri Jul 05, 2019 2:18 pm Post subject: |
|
|
Centurion
Joined: 05 Nov 2006
Posts: 108
Location: Bangalore
|
Small correction, I never mentioned that the DCs are used for DR.
| Quote: |
vitor wrote:
No you're not. The point of decoupled messaging is that 2 separate applications are taking to each other so you're not the owner of the sending application and have no control over what they're doing. If you do own both applications, remove MQ from the picture and use a replicated database! |
The sending application is MQ Queue Manager and it publishes Configuration events onto it local Queue, which is read by a Java based tool, that is seperated by such distance. We dont own Qmgr app, but we know the approx size of Configuration event messages.
| Quote: |
vitor wrote:
MQ controls this with MQGET. Message control (especially for persistent messages) is a feature of MQ. Are you saying IBM can abdicate this responsibility for this new verb? |
No I am not suggesting abdication of responsibility. But there can be many solution for e.g. responding back with a rollback with the msgID that caused the problem, or, there can be GMO that send such error prone messages to designated backout queues or DLQs. I would assume rollback not to take much time for its just a question of clearing the buffer or not.
| Quote: |
vitor wrote:
So you want read ahead with this, so the application can be processing one filled buffer while MQ is filling the next one? Otherwise you're still waiting for the buffer to fill with all that latency. |
I did not think of it as a client side buffer, but server side only, that will be cleared once there is a commit message from the client.
| Quote: |
exerc wrote:
That's like expecting your postman to 'compress' more letters into his sack so more can be delivered in one round - but guess what? The more letters, the heavier the sack, and the heavier the sack, the slower the postman! |
Didn't the Ping demonstrate, that the heavier sack did not slow the postman. 
| Quote: |
hughson wrote:
Can you just confirm whether you are talking about client channels or sender/receiver channels here? |
SVRCONNs to be precise.
| Quote: |
fjb_saper wrote:
Bandwidth utilization has very little to do with MQ. A lot more is dictated by TCP/IP. Especially when long distances, net lag of 140ms or thereabouts and some packet loss are involved.
At this point TCP/IP bandwidth utilization becomes quite poor and you'd be well advised to look for alternative solutions. |
My messages are 250 bytes approx.
Ping response shows 655,00 bytes takes 128 ms.
For 200 such messages, size would have been 500,00 bytes.
What you are saying that with even such feature, delivering 200 such msgs will take around 30-40 secs or more although ping responds within 128 ms? I mention 30-40 secs for comparing it with fetching each message individually.
Regards
Saurabh |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Jul 05, 2019 4:48 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| saurabh25281 wrote: |
| hughson wrote: |
| Can you just confirm whether you are talking about client channels or sender/receiver channels here? |
SVRCONNs to be precise. |
Indeed - so I would reiterate my earlier comment - have you looked into Read Ahead? Or are you only using persistent messages in which case it doesn't help?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| saurabh25281 |
| Posted: Sat Jul 06, 2019 10:51 pm Post subject: |
|
|
Centurion
Joined: 05 Nov 2006
Posts: 108
Location: Bangalore
|
| hughson wrote: |
| saurabh25281 wrote: |
| hughson wrote: |
| Can you just confirm whether you are talking about client channels or sender/receiver channels here? |
SVRCONNs to be precise. |
Indeed - so I would reiterate my earlier comment - have you looked into Read Ahead? Or are you only using persistent messages in which case it doesn't help?
|
We are using persistent messages. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Jul 07, 2019 1:52 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
A quick reminder of how client-binding differ from server-bindings apps.
For server-bindings apps, MQI calls are shipped cross-memory (within the same o/s instance). Memory speeds are much, much faster than network speeds. You can research google for your RAM type vs. clock-speed vs memory bandwidth.
For client-bindings apps, each MQI call (MQCONN, MQOPEN, MQPUT, ...) is shipped cross-network to the server, which executes the call, then returns ReasonCode/CompletionCode to the client. For MQPUTs and MQGETs, the call also ships your message content. In short, two network flows per MQI call.
If your client-binding app runs very close to the server, response time improves.
Move the server 5,000 miles away, response time degrades.
If the app you describe is the one and only instance of the app, I'd strongly recommend that you install MQ, and create a qmgr, on the client platform, and let MQ qmgr-to-qmgr channels transmit your messages.
You might want to do this as a proof-of-concept.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| saurabh25281 |
| Posted: Sun Jul 07, 2019 10:09 pm Post subject: |
|
|
Centurion
Joined: 05 Nov 2006
Posts: 108
Location: Bangalore
|
| bruce2359 wrote: |
If the app you describe is the one and only instance of the app, I'd strongly recommend that you install MQ, and create a qmgr, on the client platform, and let MQ qmgr-to-qmgr channels transmit your messages.
|
It would be too expensive to have a MQ server just for transmitting configuration events for monitoring and auditing purpose.
We are considering using multi-threaded application to retrieve multiple message concurrently. Is it something that could help?
Regards
Saurabh |
|
| Back to top |
|
|
| exerk |
| Posted: Sun Jul 07, 2019 11:01 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| saurabh25281 wrote: |
| ...We are considering using multi-threaded application to retrieve multiple message concurrently. Is it something that could help? |
More threads waiting on the same slow network? What do you think?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| saurabh25281 |
| Posted: Mon Jul 08, 2019 12:19 am Post subject: |
|
|
Centurion
Joined: 05 Nov 2006
Posts: 108
Location: Bangalore
|
| exerk wrote: |
| saurabh25281 wrote: |
| ...We are considering using multi-threaded application to retrieve multiple message concurrently. Is it something that could help? |
More threads waiting on the same slow network? What do you think? |
We cannot beat the latency, but can improve the throughput by parallelism. Does every thread performing a MQGET, wait for the previous thread to complete? |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 08, 2019 12:57 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| saurabh25281 wrote: |
| ...We cannot beat the latency, but can improve the throughput by parallelism... |
A good idea...
| saurabh25281 wrote: |
| ...Does every thread performing a MQGET, wait for the previous thread to complete? |
Depends on how you code it...
Think of your messages and threads as the bulbs of an hour-glass, and the network as the thin neck connecting those bulbs - even if you increase the size of the bulbs, if the neck remains the same size the rate of flow through the neck will not increase. I'm sure parallelism will bring some benefit, but not necessarily the amount you seek.
Have you considered having a 'local' application read off the messages to file, and transfer the file?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Jul 08, 2019 4:22 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| exerk wrote: |
| ...Have you considered having a 'local' application read off the messages to file, and transfer the file? |
Or have a local app that combines all the data into into one MQ message, and the MQ client app does one MQGET across the n/w. This will avoid the two-way TCP handshake for each message, and avoid sending the MQMD and MQI control structures for each message (a few hundred bytes).
_________________
Glenn |
|
| Back to top |
|
|
| saurabh25281 |
| Posted: Wed Jul 10, 2019 8:19 am Post subject: |
|
|
Centurion
Joined: 05 Nov 2006
Posts: 108
Location: Bangalore
|
| gbaddeley wrote: |
| exerk wrote: |
| ...Have you considered having a 'local' application read off the messages to file, and transfer the file? |
Or have a local app that combines all the data into into one MQ message, and the MQ client app does one MQGET across the n/w. This will avoid the two-way TCP handshake for each message, and avoid sending the MQMD and MQI control structures for each message (a few hundred bytes). |
We are initially trying to multi-thread our application to read multiple messages concurrently and see how much improvement it brings. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
