| Author |
Message
|
| krishna1228 |
 Posted: Tue Jul 23, 2013 1:54 am Post subject: What will be the highest dequeue rate that we can achieve? |
 |
|
Novice
Joined: 28 May 2013
Posts: 18
|
What will be the highest dequeue rate that we can achieve through a java client that accessing a queue present in a remote queue manager
persistent messages of 4 kb length
Please share your experiments that you may done in past
Thanks in advance.. |
|
| Back to top |
|
 |
| exerk |
| Posted: Tue Jul 23, 2013 2:01 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
Any experiments that I have done in the past will be pretty much irrelevant to your situation. There are any number of factors that will influence the outcome, for example (but not limited to):
* Network bandwidth, contention, and throttling
* Server resources (RAM, CPU, Storage)
* Message persistency
* Coding of the client
* Number of client instances
There are many other considerations...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jul 23, 2013 3:59 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
There are client performance options such as Read Ahead available in the API. Also the use of syncpointing may have an effect.
But as Exert says - many variable factors can play a part - and the queue manager options can also be tuned in various ways.
In tests over the years, I have found that MQ adds little delay over and above the network latency. The effect of the latency can be mitigated to some extent with various techniques.
When network latency is not the limiting factor - probably disk i/o speed will be if messages are persistent.
Experiment and find out. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jul 23, 2013 4:51 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
The number one thing you can do for "MQ performance" is to make sure you don't connect and disconnect to the queue manager for each message. And that you don't open and close the queue for each message. Connect and open once, then loop on your messages till done.
The number two thing you can do is make the messages non persistent if business requirements allow it.
There are dozens of things you can do after that, but I think the realized benefits from them are all minor compared to the impact realized from properly handling the first 2 points.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 23, 2013 8:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| There's a reasonably large, and counterintuitive, performance gain in using syncpoint for persistent messages instead of NOT using syncpoint for persistent messages. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Jul 23, 2013 8:41 am Post subject: Re: What will be the highest dequeue rate that we can achiev |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| krishna1228 wrote: |
| What will be the highest dequeue rate that we can achieve through a java client that accessing a queue present in a remote queue manager? |
It depends.
Is the remote queue manager on the same physical box (in the same LPAR)? In the same box, but a different LPAR?
Is the qmgr on a box physically distant? How far? What kind of network?
What else is the java app doing besides consuming messages? DB updates?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Jul 23, 2013 8:46 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
You might want to search google for 'mq+java+dequeue'.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jul 23, 2013 9:40 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| mqjeff wrote: |
| There's a reasonably large, and counterintuitive, performance gain in using syncpoint for persistent messages instead of NOT using syncpoint for persistent messages. |
Jeff,
Even for single puts? Or only when dealing with multiple messages that can be grouped into a single Unit Of Work?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jul 23, 2013 8:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PeterPotkay wrote: |
| mqjeff wrote: |
| There's a reasonably large, and counterintuitive, performance gain in using syncpoint for persistent messages instead of NOT using syncpoint for persistent messages. |
Jeff,
Even for single puts? Or only when dealing with multiple messages that can be grouped into a single Unit Of Work? |
Never checked it for puts. Do all my operations under syncpoint. But definitely for gets. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Tue Jul 23, 2013 9:18 pm Post subject: Re: What will be the highest dequeue rate that we can achiev |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
| krishna1228 wrote: |
| What will be the highest dequeue rate that we can achieve .... |
did you already check those "performance white papers" on the supportpac site ?!?
_________________
Regards, Butcher |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jul 23, 2013 11:09 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| mqjeff wrote: |
| There's a reasonably large, and counterintuitive, performance gain in using syncpoint for persistent messages instead of NOT using syncpoint for persistent messages. |
Jeff,
Even for single puts? Or only when dealing with multiple messages that can be grouped into a single Unit Of Work? |
Yes, I believe even for a single put.
Although if you're doing performance testing for an application that only ever puts a single message during it's application lifetime, then I suspect that the performance of MQ is not what you're interested in. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Wed Jul 24, 2013 6:47 pm Post subject: Re: What will be the highest dequeue rate that we can achiev |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| krishna1228 wrote: |
| What will be the highest dequeue rate that we can achieve through a java client that accessing a queue present in a remote queue manager |
I think the constraints will be "Java" (limited by JVM performance) and "Client" (limited by TCP network performance and MQ Client synchronous protocol), not the MQ queue manager on the remote server.
Read-ahead in MQ Client in v7.0+ gives a big performance boost if your design and message type fits this feature.
If you are worried about high volume performance, don't use Java 
_________________
Glenn |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 24, 2013 10:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Just receiving the message in JMS, and not accessing the contents, with a commit every 1000 messages, I have achieved impressive rates, (better than the clear queue with destructive get from the MQExplorer from my desktop, but then I ran the JMS stand alone pgm from the same network segment the qmgr was on... not from my desktop.... like hitting up to 2,000 messages per second... (logged time before and after and number of messages dequeued...)
Now if you access the message payload / content, you will take a considerable performance hit compared to the above numbers... after all you are doing a lot more work...
MDBs in WAS when configured correctly should easily get you up to 100 tps on small messages and that is with reading the message content and processing it (with a reasonably small processing time <100 ms?)
You did not say what you were attempting to do... nor what your purpose was... so remember your mileage will vary, depending on a lot of outside factors that you have no to little control over...
Have fun
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| krishna1228 |
| Posted: Fri Jul 26, 2013 3:05 am Post subject: |
|
|
Novice
Joined: 28 May 2013
Posts: 18
|
Hi Grand,
thanks for sharing your experiance..
In our project we are reading the messages from the queue and we need to persist it in hadoop and hbase. main intension is to clear the queue ASAP. currently we are getting 1600-2000 msgs/min but expectation is 1,00,000 msgs/min as some of experts suggested that we need to increase the log buffer pages to 4096 but i.e not possible because we dont have access to queue manager properties as it residing in remote machine
i am commting the 100 msgs at one time i tried with 1000 as well but there is no improvement in performance and one more thing iam commiting these messages under sync point with out sync point iam getting double the performance but as my messages are critical transactions i cant do that.
i have follwing parameters to connenct to the remote queue manager
host name,queue manager name,port number,channel name..
using a java client program i am connecting to remote queue manager and reading the messages so plz suggest me the ways that i can able to do from my side to increase the performance
Thanks in advance... |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jul 26, 2013 4:25 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| krishna1228 wrote: |
| as some of experts suggested that we need to increase the log buffer pages to 4096 but i.e not possible because we dont have access to queue manager properties as it residing in remote machine |
Yes you should do that.
You'll probably find this article interesting:
http://www.ibm.com/developerworks/websphere/library/techarticles/0712_dunn/0712_dunn.html
I doubt you will get 500 times the throughput you have now by changing anything on the queue manager.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
