| Author |
Message
|
| stenix |
 Posted: Thu Dec 09, 2010 6:37 am Post subject: Does explicit segmentation increase throughput? |
 |
|
Novice
Joined: 08 Dec 2010
Posts: 12
|
Hi,
This concerns the standard java api to Websphere MQ.
In the api it is possible to do message segmentation explicitly as a complement of trusting this to the queue manager. We are facing quite heavy loads of large binary messages so my question is if it increases throughput if I do the segmentation explicitly in the code instead of letting the queue manager do this?
My thought was if the message is sent in smaller chunks, it may start sooner to transmit the message so that it does not have to wait until the last piece of data is written to the message. I guess this drills down to if the api starts to send the data upon the first call to message.write() or if it waits until the destination.put() call?
Best regards
stenix |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Dec 10, 2010 7:37 am Post subject: Re: Does explicit segmentation increase throughput? |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| stenix wrote: |
Hi,
This concerns the standard java api to Websphere MQ.
In the api it is possible to do message segmentation explicitly as a complement of trusting this to the queue manager. We are facing quite heavy loads of large binary messages so my question is if it increases throughput if I do the segmentation explicitly in the code instead of letting the queue manager do this?
My thought was if the message is sent in smaller chunks, it may start sooner to transmit the message so that it does not have to wait until the last piece of data is written to the message. I guess this drills down to if the api starts to send the data upon the first call to message.write() or if it waits until the destination.put() call?
Best regards
stenix |
The message is sent on the destination.put call if not under syncpoint. If under syncpoint it is sent once the commit call has been issued on the message (after the put of course).
Segmenting the message will slow the throughput if your network is fast and stable. However it will also slow down the other smaller messages.
This is why I recommend creating a specific channel to use for big messages only, leaving your smaller message traffic unaffected.
If you suffer from network problems, and possibly small log files etc... segmenting will help. The redo unit in the channel will be smaller, the commit unit on the channel will be smaller (using less log space) etc ...
I believe you will have to experiment and keep it sensible and down to earth.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 10, 2010 7:43 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The other question is, where do you start measuring throughput?
From the point at which the first segment of the message is put to the queue? From the point at which the last segment of the message is put to the queue? From the point at which the first message is read off the transmit queue? From the point at which the sending application receives an RC 0 for it's commit? |
|
| Back to top |
|
|
| stenix |
| Posted: Fri Dec 10, 2010 7:56 am Post subject: |
|
|
Novice
Joined: 08 Dec 2010
Posts: 12
|
I would say that throughput is the number of large messages that can be processed in a defined period of time regardless of when I start. Hmm, I guess I put the question wrong since I can see now that throughput is not increased by slicing messages. On the other hand it might affect the latency.
But if you say that the message is not written to the queue until the syncpoint is committed then it does not matter anyhow.
I will let the queue manager handle the segmentation until someone complains, then I may investigate for improvements.
Thank you for your answers. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Dec 10, 2010 8:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| The message is *written* to the queue before syncpoint is complete, it's just not *available*. |
|
| Back to top |
|
|
| stenix |
| Posted: Mon Dec 13, 2010 12:15 am Post subject: |
|
|
Novice
Joined: 08 Dec 2010
Posts: 12
|
| Quote: |
The message is *written* to the queue before syncpoint is complete, it's just not *available*. |
So when is the data actually written to the queue? Is it on the message.put() or on the message.write()?
If the message is of several GB of data, this can make quite a large difference, especially if the client is not on the same host as the queue. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 13, 2010 2:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| stenix wrote: |
| Quote: |
The message is *written* to the queue before syncpoint is complete, it's just not *available*. |
So when is the data actually written to the queue? Is it on the message.put() or on the message.write()?
|
The put.
The message.write is just adding data to the local buffer. the put sends the buffer to the queue manager, which stores the data in the q file but marked as unavailable if in syncpoint.
How and where the buffer is segmented when you are using a client connection and automatic segmentation, I don't know (and it may not be documented). But it presumably has to happen before data is written to the queue. |
|
| Back to top |
|
|
| stenix |
| Posted: Mon Dec 13, 2010 4:40 am Post subject: |
|
|
Novice
Joined: 08 Dec 2010
Posts: 12
|
| Thank you for your answers. |
|
| Back to top |
|
|
|
|
