| Author |
Message
|
| carfield |
 Posted: Wed Jun 27, 2007 10:00 am Post subject: Performance tuning of sending a lot of small message to MQ |
 |
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
| I need to send a lof of small message from HK to a MQQueue in London, each message take 3 seconds, which is slow for my use case. Is there any performance tuning technique about that? Can I tell MQ to send message by batch?? |
|
| Back to top |
|
 |
| marcin.kasinski |
| Posted: Wed Jun 27, 2007 10:20 am Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Sentinel
Joined: 21 Dec 2004
Posts: 850
Location: Poland / Warsaw
|
| carfield wrote: |
| I need to send a lof of small message from HK to a MQQueue in London, each message take 3 seconds, which is slow for my use case. Is there any performance tuning technique about that? Can I tell MQ to send message by batch?? |
Can you describe your architecture ?
Is these messages persistent or not ?
Is this 3 second only network or application workload ?
Do you use compression ?
How many channels do you use to send these messages ?
Next questions soon...
_________________
Marcin |
|
| Back to top |
|
|
| carfield |
| Posted: Wed Jun 27, 2007 10:28 am Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
Sure!
>> Can you describe your architecture ?
simple send a message where client at HK and queue at London.
>> Is these messages persistent or not ?
Probably not , it is very quick if the client at London
>> Is this 3 second only network or application workload ?
Network
>> Do you use compression ?
No, how can I turn it on??
>> How many channels do you use to send these messages ?
1, can I send many?? |
|
| Back to top |
|
|
| marcin.kasinski |
| Posted: Wed Jun 27, 2007 10:44 am Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Sentinel
Joined: 21 Dec 2004
Posts: 850
Location: Poland / Warsaw
|
Is it really simple put ? Any firewall, VPN, SSL ?
How did you test this 3 seconds ? Can you describe it ?
Search docs for channel compression. It could help. Test it.
You can use many remote queues to send these messages via many channels.
How many messages would you like to send ?
_________________
Marcin |
|
| Back to top |
|
|
| carfield |
| Posted: Wed Jun 27, 2007 10:55 am Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
>> Is it really simple put ? Any firewall, VPN, SSL ?
Need to clarify about that
>> How did you test this 3 seconds ? Can you describe it ?
I send 5000 messages to the queue, and it take 4 hours to complete.
>> Search docs for channel compression. It could help. Test it.
>> You can use many remote queues to send these messages via many channels.
Thanks
>> How many messages would you like to send ?
It is really depend the users, I may need to log the the statist later |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jun 27, 2007 11:12 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Simplest thing to try is increasing the batch size of the sender channel.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Wed Jun 27, 2007 12:21 pm Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
| carfield wrote: |
| I need to send a lof of small message from HK to a MQQueue in London, each message take 3 seconds, which is slow for my use case. Is there any performance tuning technique about that? Can I tell MQ to send message by batch?? |
Hi,
Been there, decided not to do that. 
A few years ago, I was working at a client where the business people wanted to deploy desktop applications with MQ Client.
The desktop applications were going to be used in PCs located in London, UK. The MQ servers were going to be running in Toronto, Canada (other side of the pond).
No matter what I did, I could not get the network latency low enough for the desktop application to be reasonably responsive. I finally convinced senior management that they were going to have to setup and manage a MQ environment in London, UK. Management wasn't happy about the expense but the users were happier. 
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jun 27, 2007 4:29 pm Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| carfield wrote: |
Sure!
>> Can you describe your architecture ?
simple send a message where client at HK and queue at London.
|
So you have an App in Hong Kong that is an MQClient application connecting to a queue manager in London?
| carfield wrote: |
>> Is these messages persistent or not ?
Probably not , it is very quick if the client at London
|
That was a yes or no question. Persistent messages will be slower than non persistent messages usually, but that's because the QM will be writing them to disk. If your bottle neck is the network, I don't think it makes any difference over the wire for an MQClient app how fast a message moves if its persistent or not. The only difference in the message is the header says persistence = yes versus no.
Compression in MQ is available in the base product only at version 6.0 and only for QM to QM channels, not clients. You will have to compress the message in your app before you give it to the MQPUT call if you want to do this.
Don't know if a channel exit can be used on an MQ Client connection for compression???
Using multiple channels over the same network connection will probably not help since all the channels will be competing for the same bandwidth.
Exactly how big are these messages?
Do you have the bandwidth between these 2 locations to expect better throughput?
You are connecting ONCE to the QM, opening the queue ONCE, and then looping on just the 5000 puts, right? You're not connecting openeing putting closing and disconnecting for each and every message are you?
Are you doing anything in between each put? Maybe that takes 0.5 seconds and your app is doing some other non MQ related work for 2.5 seconds every time.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jun 27, 2007 7:07 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| Compression in MQ is available in the base product only at version 6.0 and only for QM to QM channels, not clients. |
Peter have you tested this? The attribute is on the SVRCONN channel.
I suspect it only works if you use a channel table (force the client part ot the channel to know about the compression)...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| carfield |
| Posted: Wed Jun 27, 2007 8:23 pm Post subject: |
|
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
| jefflowrey wrote: |
| Simplest thing to try is increasing the batch size of the sender channel. |
How can I do that? Can I set that at client side but setting properties at MQQueueConnectionFactory? |
|
| Back to top |
|
|
| carfield |
| Posted: Wed Jun 27, 2007 10:28 pm Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
>> Is it really simple put ? Any firewall, VPN, SSL ?
Only firewall , will this affected?
>> Search docs for channel compression. It could help. Test it.
I use
Vector<Integer> headerComp = new Vector<Integer>();
headerComp.add(new Integer(MQC.MQCOMPRESS_SYSTEM));
MQEnvironment.hdrCompList = headerComp;
Vector<Integer> msgComp = new Vector<Integer>();
msgComp.add(new Integer(MQC.MQCOMPRESS_RLE));
msgComp.add(new Integer(MQC.MQCOMPRESS_ZLIBHIGH));
MQEnvironment.msgCompList = msgComp;
But it doesn't help
>> You can use many remote queues to send these messages via many channels.
According to some other reply, it likely doesn't help, or you think it will help? |
|
| Back to top |
|
|
| carfield |
| Posted: Wed Jun 27, 2007 10:49 pm Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
>>So you have an App in Hong Kong that is an MQClient application connecting to a queue manager in London?
Yes
>> Exactly how big are these messages?
>> Do you have the bandwidth between these 2 locations to expect better throughput?
We believe so.
>> You are connecting ONCE to the QM, opening the queue ONCE, and
>> then looping on just the 5000 puts, right? You're not connecting
>> openeing putting closing and disconnecting for each and every
>> message are you?
This is critical, I think may be we are setting up connection for every message. How can we set MQ JMS library to make sure connection will reuse for all message??
>> Are you doing anything in between each put? Maybe that takes 0.5
>> seconds and your app is doing some other non MQ related work for
>> 2.5 seconds every time.
It is not the case, here is the debug log from spring:
2007-06-28 14:40:33.580 [pool-2-thread-1] [JmsTemplate.426] DEBUG - Executing callback on JMS Session [com.ibm.mq.jms.MQQueueSession@970c0e]
2007-06-28 14:40:35.453 [main] [JmsTemplate.426] DEBUG - Executing callback on JMS Session [com.ibm.mq.jms.MQQueueSession@754fc]
2007-06-28 14:40:37.950 [pool-2-thread-1] [JmsTemplate.426] DEBUG - Executing callback on JMS Session [com.ibm.mq.jms.MQQueueSession@1ce1bea]
2007-06-28 14:40:38.575 [main] [JmsQueueSender.103] DEBUG - sending input: test
2007-06-28 14:40:39.761 [main] [JmsTemplate.426] DEBUG - Executing callback on JMS Session [com.ibm.mq.jms.MQQueueSession@19e11a1]
2007-06-28 14:40:40.011 [main] [JmsTemplate.509] DEBUG - Sending created message [
JMS Message class: jms_text
JMSType: null
JMSDeliveryMode: 2
JMSExpiration: 0
JMSPriority: 4
JMSMessageID: null
JMSTimestamp: 0
JMSCorrelationID:null
JMSDestination: null
JMSReplyTo: null
JMSRedelivered: false
test]
2007-06-28 14:40:41.447 [main] [JmsQueueSender.103] DEBUG - sending input: test
2007-06-28 14:40:42.243 [pool-2-thread-1] [JmsTemplate.426] DEBUG - Executing callback on JMS Session [com.ibm.mq.jms.MQQueueSession@15f157b]
2007-06-28 14:40:42.617 [main] [JmsTemplate.426] DEBUG - Executing callback on JMS Session [com.ibm.mq.jms.MQQueueSession@17b40fe]
2007-06-28 14:40:42.867 [main] [JmsTemplate.509] DEBUG - Sending created message [
JMS Message class: jms_text
JMSType: null
JMSDeliveryMode: 2
JMSExpiration: 0
JMSPriority: 4
JMSMessageID: null
JMSTimestamp: 0
JMSCorrelationID:null
JMSDestination: null
JMSReplyTo: null
JMSRedelivered: false
test]
2007-06-28 14:40:44.803 [Thread-0] [JmsQueueSender.92] INFO - Send all at: Thu Jun 28 14:40:44 CST 2007
of only send text "test" using jmstemplate.send() |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jun 28, 2007 6:40 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
| Quote: |
| Compression in MQ is available in the base product only at version 6.0 and only for QM to QM channels, not clients. |
Peter have you tested this? The attribute is on the SVRCONN channel.
I suspect it only works if you use a channel table (force the client part ot the channel to know about the compression)...
|
You are right. My mistake.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jun 28, 2007 6:45 am Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| carfield wrote: |
>> You are connecting ONCE to the QM, opening the queue ONCE, and
>> then looping on just the 5000 puts, right? You're not connecting
>> openeing putting closing and disconnecting for each and every
>> message are you?
This is critical, I think may be we are setting up connection for every message. How can we set MQ JMS library to make sure connection will reuse for all message??
|
We'll, there's the problem. Connecting to a QM is very slow compared to putting a message. Even opening the q is slow compared to putting a message. Add a close and disconnect for each message on top of that and I'd say that 5000 "transactions" in 4 hours is pretty good!
There isn't anything in the JMS library to switch. Your code needs to be changed to create the QM object once, create the q once, and then start putting in a loop.
Using a C app running as an MQClient on my laptop on the east Coast of the USA connecting to a Queue Manager on the west coast of the USA, I was able to get about 6 messages a second (1K in length each).
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| carfield |
| Posted: Thu Jun 28, 2007 10:46 am Post subject: Re: Performance tuning of sending a lot of small message to |
|
|
Novice
Joined: 30 May 2007
Posts: 13
Location: Hong Kong
|
| PeterPotkay wrote: |
We'll, there's the problem. Connecting to a QM is very slow compared to putting a message. Even opening the q is slow compared to putting a message. Add a close and disconnect for each message on top of that and I'd say that 5000 "transactions" in 4 hours is pretty good!
There isn't anything in the JMS library to switch. Your code needs to be changed to create the QM object once, create the q once, and then start putting in a loop.
Using a C app running as an MQClient on my laptop on the east Coast of the USA connecting to a Queue Manager on the west coast of the USA, I was able to get about 6 messages a second (1K in length each). |
We are using spring's JmsTemplate and this library control this code. I asked there and they suggest using SingleConnectionFactory to wrap the MQQueueConnectionFactory, but look like it is not faster a lot...
http://forum.springframework.org/showthread.php?p=128962#post128962 |
|
| Back to top |
|
|
|
|
