| Author |
Message
|
| vermakov |
 Posted: Tue Sep 22, 2009 7:00 pm Post subject: Problem: QueueSender sends message in 500ms |
 |
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
Given: Windows 2003 Server or Windows XP, MQ Version 7, MQ client is java application running in JDK1.5
Expected behavior: client should be very fast sending a short point-to-point object message, under 100ms in average.
Observed behavior: every time client application takes about 500ms to send a short object message. At the same time, server application reply comes under 100ms. Client log file shows that QueueSender.send(Message) takes all the time. QueueSender is created once and reused for subsequent messages.
Could anyone suggest what could be tweaked in java client or in MQ7 queue setup to improve the send time? Thank you. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Sep 22, 2009 9:28 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you looked at the transactional behavior of the session? (true, autoack)
Is anybody else using the connection at the same time? (should not be)
Is it a cluster queue?
Is the round robin algo taking the time?
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vermakov |
| Posted: Wed Sep 23, 2009 6:21 am Post subject: |
|
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
| fjb_saper wrote: |
| Have you looked at the transactional behavior of the session? (true, autoack) |
Yes, I looked into it. This is non-transactional implementation. But changing it to transactional + commit didn't make any difference. I also tried DUPS_OK_ACKNOWLEDGE - no difference.
| fjb_saper wrote: |
| Is anybody else using the connection at the same time? (should not be) |
No.
| fjb_saper wrote: |
| Is it a cluster queue? |
No.
| fjb_saper wrote: |
| Is the round robin algo taking the time? |
What is "round robin algo"? |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Wed Sep 23, 2009 6:29 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| vermakov wrote: |
What is "round robin algo"? |
Round Robin algorithm.
I trust your question was "what does algo mean" rather than "what is the round robin algorithm". |
|
| Back to top |
|
|
| vermakov |
| Posted: Wed Sep 23, 2009 7:40 am Post subject: |
|
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
| WMBDEV1 wrote: |
| I trust your question was "what does algo mean" rather than "what is the round robin algorithm". |
I'm not sure what round-robin scheduler you are asking about...
The OS scheduler works on different time frame level. It cannot contribute in milliseconds of delay.
I don't know if MQ7 has round-robin and if it can be tweaked or configured. But I would think that MQ7's scheduler works on the same microsecond level as Windows' one. So, if "normal commit" takes up to 100ms, then the round-robin scheduler cannot make it up to 1000ms. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 23, 2009 8:26 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's a horrible idea to send an ObjectMessage. Please don't do that.
The round robin algorithm in question is the one that performs MQ Cluster Name Resolution and determines which of several instances of an available queue should receive this particular message.
If you are not using MQ Clustering, it does not apply.
There could be a very very large number of reasons why it could take a second for MQ to write a message to a queue and return an acknowlegment back to the putting application. Time spent doing transactional commits is one such area. Time spent determining which cluster queue to send to is another. Time spent appending to the q file is a third.
You should review the MQ Peformance Report SupportPacs and make sure that a) your expectations aren't unrealistic, b) your queue manager is configured to handle the volume you're giving it, c) you understand how to evaluate the performance of an MQ application. |
|
| Back to top |
|
|
| vermakov |
| Posted: Wed Sep 23, 2009 3:03 pm Post subject: |
|
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
| mqjeff wrote: |
| You should review the MQ Peformance Report SupportPacs and make sure that a) your expectations aren't unrealistic, b) your queue manager is configured to handle the volume you're giving it, c) you understand how to evaluate the performance of an MQ application. |
I'm going through IBM documentation...
In our system, client sends point-to-point message to server over Main Queue. Server processes request and sends point-to-point reply message over Reply Queue. Main and Reply queues are identical. Code used by client and server for queue communication is the same. But, client sends message within 1 sec and server sends message instantly (under 20ms).
Is there anything in MQ7 that can cause this kind of behavior? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 23, 2009 3:11 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vermakov wrote: |
| Is there anything in MQ7 that can cause this kind of behavior? |
There are a number of things in all versions of WMQ that can cause this, as my most worthy associate alludes to. If you're asking "is there a specific fast/slow switch in WMQv7?", then the answer is no.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mvic |
| Posted: Wed Sep 23, 2009 3:43 pm Post subject: Re: Problem: QueueSender sends message in 500ms |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| vermakov wrote: |
| Expected behavior: client should be very fast sending a short point-to-point object message, under 100ms in average. |
For a single message, an MQPUT or Java equivalent should be much quicker than this unless the message is extremely large.
Ideas: Possibly network transfer time is slow? Did you try MQ v6 client or server as a comparison?
| Quote: |
| Observed behavior: every time client application takes about 500ms to send a short object message. |
How are you measuring that?
| Quote: |
| At the same time, server application reply comes under 100ms. |
How are you measuring that?
| Quote: |
| Client log file shows that QueueSender.send(Message) takes all the time. |
OK but how does the client measure the time taken by QueueSender.send() before it logs to the log file?
| Quote: |
| Could anyone suggest what could be tweaked in java client or in MQ7 queue setup to improve the send time? Thank you. |
Depends where precisely the time is being lost.
For a 500 ms delay in putting a message a trace should not mask the problem. MQ trace I mean. So perhaps trace it and see what can be seen. I suggest to take a look at strmqtrc, endmqtrc instructions in the Sys Admin Guide and capture a single slow message in the trace, then have a look thru, see where there are delays occurring. 500 ms should be simple to spot I guess. |
|
| Back to top |
|
|
| vermakov |
| Posted: Wed Sep 23, 2009 4:58 pm Post subject: Re: Problem: QueueSender sends message in 500ms |
|
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
| mvic wrote: |
| For a single message, an MQPUT or Java equivalent should be much quicker than this unless the message is extremely large. |
I agree. I should have said under 20ms. This is what I observe on the other queue.
| mvic wrote: |
| Ideas: Possibly network transfer time is slow? |
Network guys say that network communication is not the factor.
| mvic wrote: |
| Did you try MQ v6 client or server as a comparison? |
I compared with MQ5. Same pattern.
| mvic wrote: |
| OK but how does the client measure the time taken by QueueSender.send() before it logs to the log file? |
long t1 = System.currentTimeMillis();
queueSender.send(message);
logger.debug("QueueSender.send: " + (System.currentTimeMillis()-t1));
Client shows times 100 - 1000 (the average is 500)
Server shows times: 0 - 20 (the average is 15)
| mvic wrote: |
| I suggest to take a look at strmqtrc, endmqtrc instructions in the Sys Admin Guide and capture a single slow message in the trace |
We are in process of doing this. I hope we'll catch and understand the delay... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Sep 23, 2009 6:00 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| In our system, client sends point-to-point message to server over Main Queue. Server processes request and sends point-to-point reply message over Reply Queue. Main and Reply queues are identical. Code used by client and server for queue communication is the same. But, client sends message within 1 sec and server sends message instantly (under 20ms). |
Have you looked at the client application and the network being the culprit?
Are you opening a client connection for each send or are you using pooled connections?
What is the speed supported by your network for the message size you are sending?
Remember that a process using bindings will work way faster than a client process over tcp/ip client connection!
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| manicminer |
| Posted: Thu Sep 24, 2009 12:15 am Post subject: Re: Problem: QueueSender sends message in 500ms |
|
|
Disciple
Joined: 11 Jul 2007
Posts: 177
|
| vermakov wrote: |
long t1 = System.currentTimeMillis();
queueSender.send(message);
logger.debug("QueueSender.send: " + (System.currentTimeMillis()-t1));
Client shows times 100 - 1000 (the average is 500)
Server shows times: 0 - 20 (the average is 15)
|
You shouldn't use currentTimeMillis for performance timing (it is inaccurate and doesn't update regularly) but besides that...
Are you 100% sure of the message persistence selected for each message? Could it be that the server is sending a non-persistent message where as the client is sending a persistent one? That coupled with a lack of disk IO bandwidth might explain the difference between the 2.
_________________
Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 24, 2009 4:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Did you ensure that your receiver was using the native form of the selector to match the reply? |
|
| Back to top |
|
|
| vermakov |
| Posted: Thu Sep 24, 2009 6:32 am Post subject: Re: Problem: QueueSender sends message in 500ms |
|
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
| mvic wrote: |
| For a 500 ms delay in putting a message a trace should not mask the problem. MQ trace I mean. So perhaps trace it and see what can be seen. I suggest to take a look at strmqtrc, endmqtrc instructions in the Sys Admin Guide and capture a single slow message in the trace, then have a look thru, see where there are delays occurring. 500 ms should be simple to spot I guess. |
The one thing is to find it, the other is to explain it 
We found 453 ms delay in MQ trace. Can you explain what it might mean?
| Code: |
17:59:24.000.3I 0001 @10ea9ba c.i.m.j.remote.internal.RemoteHconn(RemoteHconn) ----+----+-- { enterCall() [c.i.mq.jmqi.remote.internal.RemoteHconn$CallLock@1b6101e] [false] [false]
17:59:24.453.00 0008 @90832e c.i.m.j.r.internal.system.RemoteProxyQueue(RemoteProxyQueue) ----+----+- { requestMutex(RemoteTls) [c.i.mq.jmqi.remote.internal.system.RemoteTls@a522a6] |
|
|
| Back to top |
|
|
| vermakov |
| Posted: Thu Sep 24, 2009 6:35 am Post subject: |
|
|
Novice
Joined: 22 Sep 2009
Posts: 11
Location: Canada
|
| mqjeff wrote: |
| Did you ensure that your receiver was using the native form of the selector to match the reply? |
We are not using selectors. All reply messages are consumed by the client and dispatched internally to the right thread. Also, there is no delay on server reply, delay is on client send. |
|
| Back to top |
|
|
|
|
