| Author |
Message
|
| anveshita |
 Posted: Tue Jul 14, 2009 12:16 pm Post subject: MQ Performance Message Max Size |
 |
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
My appliation connects to a queue in client mode to read the messages out. The channel I use is defined with 4MB default for the max message length. If it is changed to 100 MB,How will the performance of my client application gets affected? I do not get 100 MB messages all the time, 99.9% of the time it is less than 4 MB.
I perosnally find it should not, but do not know how to state this for sure without testing the hardway. Please let me know your input |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Jul 14, 2009 12:37 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
The 100meg maximum message length attribute has no impact on the size of the message. So there will be no performance issues for 4k mesages.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue Jul 14, 2009 12:58 pm Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| Quote: |
| The 100meg maximum message length attribute has no impact on the size of the message. |
Hmm. I thought the message size varibale of the channel dictates how big the message read by a client could be. Other wise the client would get MQRC_DATA_LENGTH_ERROR 2010. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Jul 14, 2009 1:04 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
If the channel has a max message size of 100MB, then anything < 100MB is going to sail up/down it with no issues - but you can't fit a quart into a pint pot! In other words, if you try and get a 100MB message down a 4MB pipe, it aint gonnna happen, but if you try and get a 4MB message down a 100MB pipe, it aint gonna touch the sides.
There is of course the aspect that if you do not allocate a big enough buffer for the larger messages it'll fail, but that can always be got round by the old trick of get with 0 length buffer, read the message length returned in the fail, and then do another get with the correct length buffer. Personally I wouldn't do that these days, memory's cheap.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Jul 14, 2009 1:09 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
Max message length specifies only the maximum size of a message the channel can send/recieve, not the size of a message.
Similarly, the max msg length queue attribute specifies the maximum length of a message that can be successfully put to a queue, not the length of a message.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hildobiersma |
| Posted: Sun Jul 19, 2009 9:16 am Post subject: |
|
|
Newbie
Joined: 17 Jun 2009
Posts: 6
|
| IBM Hursley have informed me (this was several years ago) that there's no performance impact of the configured maximum message size for queues and channels, but there *is* a performance impact for the buffer size specified when doing an MQGET to read messages. So specify a size that's large enough, and optionally retry with a larger buffer, but don't specify an MQGET buffer size of 10MB if your messages are going to be 20K. |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Mon Jul 20, 2009 5:30 am Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
| hildobiersma wrote: |
| So specify a size that's large enough, and optionally retry with a larger buffer, but don't specify an MQGET buffer size of 10MB if your messages are going to be 20K. |
I must be missing something here.
If your buffer is 10Mb and you get a 20K message only 20K is retrieved so why would that lead to a performance issue?
Of course if you clear the 10Mb buffer everytime you have to go get a message then performance will be impacted but that really isn't an MQ issue, that is just lazy programming |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 20, 2009 5:57 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| kevinf2349 wrote: |
| ...If your buffer is 10Mb and you get a 20K message only 20K is retrieved so why would that lead to a performance issue?... |
Not being a developer, so this is stab in the dark, but if you have reserved a large amount of memory that will be unused, that is not going to be available to anything else, and so could impact [b]total[/i] performance on the box. Agreed it's not a WMQ issue, but could be if it's sub-system affecting.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jul 20, 2009 6:05 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
You can do a MQGET with a buffer size that is OK for the majority of messages, and if you get a MQRC of 2080 (Truncated Message Failure), you examine the MQMD's message length and allocate a buffer large enough for the entire message, then repeat the MQGET (since it won't have been removed from the queue).
You should really use match on MQMD.msgid for the second attempt to make sure you are looking at the same message. To avoid the need for this you can MQINQ on the queue's max message length but then you are back to using a largest buffer all the time.
Some people prefer to do a MQGET with a zero length buffer then repeat the MQGET (as described above) to make the code path the same each time. |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Mon Jul 20, 2009 6:48 am Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
it always depends what you are trying to optimize. if you are looking for storage usage, then, yes, this double mqget may be an option. there are optimizes algorithms, e.g. start with 0k mqget, allocate storage what is needed for first message and mqget that message. then mqget second message with buffer already alocated for first message. if it fits - ok. if not, get bigger buffer and so on. just a sample.
if you are trying to optimize for throughput and / or cpu, that double get may not be an option, e.g. if you have a high message load and need to process several hundreds of messages per second.
so it always depends.
_________________
Regards, Butcher |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Jul 20, 2009 6:52 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
In-flight sizing of buffers by applications is possible, but imposes additional MQI calls for every MQGET.
This, like most other things in life, is a trade-off.
Two other options, depending...
If you have only 1k messages and 100meg messages destined for the same application, create two versions of the app - one big buffer, one small buffer.
If the messages vary from 1k through 100meg, then live with the overallocation of memory in one big buffer.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jul 20, 2009 7:05 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
| bruce2359 wrote: |
| In-flight sizing of buffers by applications is possible, but imposes additional MQI calls for every MQGET. |
Not every MQGET if the first MQGET uses a buffer size designed to cover the most commonly occuring message sizes.
4 MB would seem a good choice since so many queues end up with this as their max message size by default.
Or this could be an external referenced parameter to allow tuning without creating multiple code versions. |
|
| Back to top |
|
|
| sumit |
| Posted: Mon Jul 20, 2009 7:47 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| zpat wrote: |
| 4 MB would seem a good choice since so many queues end up with this as their max message size by default. |
But not when...
| bruce2359 wrote: |
| If the messages vary from 1k through 100meg, then live with the overallocation of memory in one big buffer. |
It higly depends the way application has been designed and the expected length of mesage it is going to generate.
Different scenario -> different approach.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 20, 2009 7:55 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| sumit wrote: |
| zpat wrote: |
| 4 MB would seem a good choice since so many queues end up with this as their max message size by default. |
But not when...
| bruce2359 wrote: |
| If the messages vary from 1k through 100meg, then live with the overallocation of memory in one big buffer. |
It higly depends the way application has been designed and the expected length of mesage it is going to generate.
Different scenario -> different approach. |
Circular argument...if 99% of your messages will be 4KB, and 1% will be 4MB, but you don't know when the larger messages will arrive, allocate the smaller buffer by default, but include the necessary trap logic to catch it and do another get with the increased buffer size. If 99% of messages are 4MB, and only 1% will be 4KB, but you don't know when the smaller messages will arrive, allocate the larger buffer and don't bother allocating a smaller buffer. If your messages are all sizes in between, then find one of the methods that is suitable for the frequency of the messages, and their size.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Jul 20, 2009 8:07 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9396
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
It higly depends the way application has been designed and the expected length of mesage it is going to generate.
Different scenario -> different approach. |
Exactly. 
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
