| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Question on queue design for high volumne |
« View previous topic :: View next topic » |
| Author |
Message
|
| fredlee |
 Posted: Sun Aug 14, 2005 11:09 pm Post subject: Question on queue design for high volumne |
 |
|
Newbie
Joined: 02 Aug 2004
Posts: 6
Location: Hong Kong
|
Hi all,
how you design the queue for high volume, e.g. upto 100 message every millisecond?
Will multiple request queue help? or any suggestion?
The case is there is multiple Java client connect to the iSeries MQ server.
Thank in advance,
Fred |
|
| Back to top |
|
 |
| Nigelg |
| Posted: Mon Aug 15, 2005 12:14 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
If I wanted maximum performance I would certainly not use:
1. Apps written in Java
2. Client apps
3. An iSeries server
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Mon Aug 15, 2005 11:38 am Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
| Quote: |
| upto 100 message every millisecond |
- That's 100,000 messages per second.
- That's 6,000,000 messages per minute.
- That's 360,000,000 messages per hour
- That's 8,640,000,000 messages per day. 
And you haven't even quoted us the size of each message!!! Let alone if it is a request & reply scenario!!
Are you sure you got your numbers right?? If so, you better go buy the biggest mainframe that IBM is currently selling.
Regards,
Roger Lacroix
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| JohnRodey |
| Posted: Mon Aug 15, 2005 1:22 pm Post subject: |
|
|
Centurion
Joined: 13 Apr 2005
Posts: 103
|
I agree with roger that throughput of 100 messages per millisecond might be wishful thinking to say the least.
But some ideas to keep in mind when developing a high throughput system.
Try to avoid:
filtering
persistence
You might not want to rule out Java, these days Java is quickly approaching c++, as crazy as that sounds. Although this would depend on which compilers are used.
Not sure, but possibly avoiding priority queueing and using strictly fifo, I would imagine that would speed things up a little??? |
|
| Back to top |
|
|
| JohnRodey |
| Posted: Tue Aug 16, 2005 4:04 am Post subject: |
|
|
Centurion
Joined: 13 Apr 2005
Posts: 103
|
| Also wanted to add that connections are very expensive so if possible you want to retain connections, pool them, etc.... |
|
| Back to top |
|
|
| hopsala |
| Posted: Tue Aug 16, 2005 5:11 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
fredlee - your avatar is right on. If you have to design a system to work with such workloads that's indeed should be the proper mq admin reaction.
Though I have not worked with so many msgs, I have worked with loads up to 50MB per second (belive it!) so I know mq can definitely take it. All you need is good mq design and some heavy duty servers.
Anyway, this is what you should take into consideration:
1. MQGET destructively, and by order. If possible avoid getting by corrlid or msgid, if you have to, use indexing (z/OS only).
2. Take to heart such channel parms as BATCHSZ, BATCHINT, BATCHHB.
3. As said, avoid java.
4. Again as said, avoid clients.
5. Work with syncpoint and commit only, say, 10 msgs, this will increase throughput by 300% at least.
6. If possible use non-presistent msgs.
7. Important - MQ, like any other software, has many more malfunctions and crashes when working with such high loads, this is an unavoidable but little known fact of the industry. Plan ahead to have major problems - allocate time for it, plan an active/active cluster (MSCS or such) to quickly recover.
8. In z/OS there are many more considerations - archiving, STGCLASSes and more. If this is relevant post so and I will try to advise.
9. If sending the same msg to a few locations, use distribution lists.
10. In my experience no need to use more than one queue, it will take it. Just note its MAXDEPTH and such parms.
There are some wonderous redbooks and support pacs with performance tests and tuning advice, you should definitely take a good look at them. Use MQ5.3 not 5.2, it has better throughput.
All this said, I am also wondering about such figures. Why so many msgs?
Hope this helps. |
|
| Back to top |
|
|
| KeeferG |
| Posted: Tue Aug 16, 2005 5:29 am Post subject: |
|
|
Master
Joined: 15 Oct 2004
Posts: 215
Location: Basingstoke, UK
|
One of the best performance increase I have seen is to have multiple getting applications per queue. I have used this technique several times and typically see a 30% performance improvement over queue heavy solutions.
The figures you are loooking are not that much higher then some of the projects I have been on in the distributed world however be prepared for performing benchmarks on each server so that you can scale appropriately and relaise that a lot of hardware might be needed.
_________________
Keith Guttridge
-----------------
Using MQ since 1995 |
|
| Back to top |
|
|
| techno |
| Posted: Fri Sep 30, 2005 11:14 pm Post subject: |
|
|
Chevalier
Joined: 22 Jan 2003
Posts: 429
|
How should batchint and batchsz be manipulated?
I guess, batchint is 0, by default. If this is the case, suppose, batchsz is fixed at 100, shall the sender wait till it gets 100 msgs? Could somebody explain this? |
|
| Back to top |
|
|
| hopsala |
| Posted: Sat Oct 01, 2005 12:48 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
The manuals, friend, are a wonderful thing:
| About BATCHSZ the manual wrote: |
| The maximum number of messages that can be sent through a channel before taking a checkpoint |
| About BATCHINT the manual wrote: |
| The minimum amount of time, in milliseconds, that a channel will keep a batch open. |
For further explanations, please turn to the Intercommunication manual, Chapter 6. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
