| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| How does MQ take full advantage of all available bandwidth |
View previous topic :: View next topic |
| Author |
Message
|
| PeterPotkay |
 Posted: Thu May 12, 2016 11:36 am Post subject: How does MQ take full advantage of all available bandwidth |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
So, when I started MQ 15 years ago, all our servers were generally running at 100 megabit. I assume the latency for server to server within the datacenter was about the same as it is today, < 1ms as seen in a ping test.
A 1 MB message was consider a monster back then, and a 10 MB message ooooozzzzzzzzzzing across a SENDER/RCVR channel was a big concern, for its potential impact to cause little messages in the XMITQ to wait for that big fatty to make it to the other end of the channel.
Now the network is primarily 1 gigabit.
I assume MQ takes full advantage of that, because I assume the Operating System of the server that supports that queue manager is also taking full advantage of 1 gigabit. But how do I prove this to myself? What about 10 gig? Would MQ just automatically take advantage of that?
Will a 10 MB message arrive at the destination queue manager approximately 10 times faster over a 1 gigabit link versus 100 megabit. I get that the network is not any faster (physics!) but with added bandwidth, will MQ automatically squeeze more concurrent bits and bytes down the pipe? Or are there secret knobs and dials to tweak?
Right or wrong, you can yell at developers and architects all you want, average MQ message size is trending up. I want to make sure I’m not a dinosaur afraid of 1 MB messages (as it relates to QM to QM communication) when the network is 10 times fatter (not “faster”) than what is was when I had MQ burned into my brain.
I’m not asking about the impact large messages have to disk space, this is a question related to network and QM to QM channels for those big messages. And its not about understanding that smaller messages are generally better than bigger message when it comes to MQ, I get that, and will continue to fight the good fight for smaller messages. I just want to make sure my attitude has kept up with the fact that the network has 10X more bandwidth than it did a decade ago, and potentially 100X more if 10gbit connectivity is an option.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu May 12, 2016 12:33 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
my 2p worth (a bit more than 2c)
Just because the bit of wet string between two system as got a bit better at passing signals it does not mean that the overall throughput has got better.
There are a number of things to consider when looking at the transmission of data between two computers.
firstly, is your link running full or half duplex.
In days gone by, you could dictate how a connection was setup. I've found that recent releases of some server OS's are missing this somewhat basic (IMHO) option.
Next, is how the connectino between the two nodes is made. How many switches? Routers? Gateways? Firewalls? etc etc. In days gone by everything was probably on one subnet. Now with VLANS and everything, things are a tad more complicated.
Next, what other traffic is going over that link?
Does your server only have one NIC that is wired up? I've seen an essential server in a network configured just like this. Doh!
Using bonding? How efficient is it? Some early Linux implementations were awful.
This whole think is a veritable minefield. A few years ago I spent three months fine tuning a system that was using MQ and Broker in order to get the desired throughput.
We even had to update the SAN subsystem!
Are you prepared to open up a veritable can of worms?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu May 12, 2016 2:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
The question about throughput is not how big is the pipe, but much more how reliable is it?
The thing with TCP/IP is that it uses a "window" and each successful transmission allows it to grow the window size. But get a single dropped or bad packet and the window is torn down back to its minimum size.
This is where products like ASPERA have made their "beef". The next piece of that pie is latency and length of the leg...
So imagine a TCP/IP connection doing transports of high volume and long distances. Due to the number of dropped packets the bandwidth has lost its significance. Only products that use the bandwidth differently can take advantage of it. This has nothing to do with MQ. Just the way the underlying TCP/IP transport works... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun May 15, 2016 5:57 pm Post subject: |
|
|
Jedi
Joined: 25 Mar 2003
Posts: 2495
Location: Melbourne, Australia
|
Hi Peter, you probably know this...
The MQ paradigm is "delivery all msgs as soon as possible". MQ will therefore transport all msgs across channels using all resources and bandwidth at its disposal, such as CPU, memory, disk, TCP network stack, physical network. Obviously this will result in contention if there are other apps using these resources.
There is some overhead in transporting each message and batch of messages, so its not a linear relationship. To get an idea of scalability, look at the MQ performance report supportpacs.
15 years ago I used to be concerned about 100KB messages, now we have apps that routinely use 50-80MB. Network latency is the biggest factor, but they seem to go through quickly enough to keep them happy.
_________________
Glenn |
|
| Back to top |
|
|
| zpat |
| Posted: Mon May 16, 2016 12:09 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
There is a compression feature on MQ channels, which can be enabled.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 16, 2016 12:42 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| zpat wrote: |
| There is a compression feature on MQ channels, which can be enabled. |
I used to use ZLIBFAST compression between NY/Pennsylvania and Florida.
Messages were usually about 100K and it worked great (round trip with CICS call about 125 ms). This was more to minimize bandwidth usage (we had to pay for it) than anything else. It will certainly help if you send messages across continents...
Again the question is not so much how does MQ take advantage of the available bandwidth than how does TCP/IP take advantage of the full bandwidth...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue May 17, 2016 2:38 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7717
|
|
| Back to top |
|
|
|
|
|
|
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
