| Author |
Message
|
| FFSparky |
 Posted: Fri Aug 10, 2007 7:19 am Post subject: Large Messages through MQ |
 |
|
Novice
Joined: 23 Dec 2003
Posts: 16
Location: Des Moines, Iowa
|
Has anyone done some bench marks with large messages (50MB +) through queue managers. Where your messages are put on QMA and read from QMB thus the messages also have to cross a channel.
Such as:
1) put 1000 50MB messages through.
2) put the same 50MB message through but break them into 4MB messages
3) same as above but 64K messages this time?
I'm hoping someone will save me some research time ;^).
We're having performance issues and I'm having a hard time trying to convince a developer to break up his 50+ MB messages. |
|
| Back to top |
|
 |
| Avikal Jain |
| Posted: Fri Aug 10, 2007 7:59 am Post subject: |
|
|
Apprentice
Joined: 25 Jun 2007
Posts: 39
Location: Pune
|
y you have to change length of messages every time you send them
once 50 MQ then 4 MB and then again 64 Kb
havent you asked for your application side about this before setting up your environment
Last edited by Avikal Jain on Fri Aug 10, 2007 8:24 am; edited 1 time in total |
|
| Back to top |
|
|
| FFSparky |
| Posted: Fri Aug 10, 2007 8:16 am Post subject: |
|
|
Novice
Joined: 23 Dec 2003
Posts: 16
Location: Des Moines, Iowa
|
Hey Dude,
I'm well aware that the default message size is 4MB and the max message size can be 100MB.
Carefully read my post again. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 10, 2007 8:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Check out the IBM site. There are some redbooks on performance & tuning that may help.
I wouldn't break the messages up myself.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| FFSparky |
| Posted: Fri Aug 10, 2007 8:49 am Post subject: |
|
|
Novice
Joined: 23 Dec 2003
Posts: 16
Location: Des Moines, Iowa
|
Thanks, I'll look into the Red Book.
And normally I would try to avoid breaking up the message as well. However when you have 500+ clients trying to push 50MB messages through a Queue Manager cluster at the same time it tends to bring MQ to it's knees. Hence my suggestion of breaking the messages at the source.
Regardless of my final direction in this will probably remain a mess....
Thanks for the reference.
-Greg |
|
| Back to top |
|
|
| Nigelg |
| Posted: Mon Aug 13, 2007 12:29 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
The size of the msgs being sent over a channel does not really matter. The msg is split into 32k chunks anyway, so there is no real difference between a 4Mb msg and a 50Mb msg. The msg size might make a difference at the receiving qmgr, where each msg has to be put to its destination queue; the more msgs there are the more MQPUTs there have to be. However, since a put of a persistent msg takes no more than a max of 5 ms, and usually much less than that for msgs put in syncpoint, i doubt whether the (max) 60ms saved from using 50Mb msgs would be much use. Channel batch size would probably have a greater effect, sice at the end of a batch there are 2 line turnrounds, and a commit at each end. The most efficient use of the channel is for each channel batch to be close to its maximum size, i.e. the apps putting to the remoteq/xmitq, should run fast enough to maintain a depth in the xmitq of BATCHSZ, so the channel batch is always full, minimising line turnrounds and commits.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 13, 2007 4:09 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If you are running in linear logs the log size could well end up being the limiting factor... Remember that the batch size needs to be under the max logsize... Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Mon Aug 13, 2007 4:16 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
| fjb_saper wrote: |
| If you are running in linear logs the log size could well end up being the limiting factor... Remember that the batch size needs to be under the max logsize... Enjoy |
don't you mean circular logging 
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 13, 2007 4:19 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Michael Dag wrote: |
| fjb_saper wrote: |
| If you are running in linear logs the log size could well end up being the limiting factor... Remember that the batch size needs to be under the max logsize... Enjoy |
don't you mean circular logging |
The assumption, however misguided is that the messages are persistent...
Circular logging will only limit the size of the current stuff in a UOW.
Linear logging will also hold what is on a queue. So the limiting effect is stronger in linear logging especially if you don't have something to retrieve the messages as they arrive...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Mon Aug 13, 2007 4:26 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
huh? linear logging is only limited by your disk space...
am I missing something???
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 13, 2007 4:30 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Michael Dag wrote: |
huh? linear logging is only limited by your disk space...
am I missing something??? |
Sure that AND the number of active logs AND the max siye of said active logs...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| yves.nicole |
| Posted: Mon Aug 13, 2007 6:14 am Post subject: |
|
|
Newbie
Joined: 01 Aug 2007
Posts: 2
Location: Lyon (France)
|
i don't know if it helps you, but i did some tests using spazio application to send files over MQ. I enqueue/dequeue the file on windows clients. The QMGR runs under Solaris. Here is the result :
Size->Put Time (sec)->Get Time (sec)
1,5 Mb-> 2-> 1
40 Mb-> 7 ->7
200 Mb->56->32
Hope it helps ! (but of course these results highly depends on your network bandwidth, your application, etc...)
_________________
MQ & Broker Administrator http://yves.nicole.netcv.org/ |
|
| Back to top |
|
|
|
|
