| Author |
Message
|
| surfnit |
 Posted: Thu Sep 21, 2006 9:09 am Post subject: |
 |
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
Both putting and getting applications are on the same machine using the same client connections (MQSERVER value).
If the bottleneck is in the client connection or network related how do some of you go about troubleshooting this other than installing a qmgr on the client machine?
I think there is a network snoop or sniffer test. Any other suggestions? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Sep 21, 2006 9:20 am Post subject: |
|
|
Guest
|
Ah! Client applications behave a little differently than server applications.
As with any MQ application, each and every MQ call is sent to the server for execution. If the application is on the server, the call is memory-to-memory; but with the client, each MQ call means a network transmission for the call and the message.
Since both getting and putting applications are running concurrently on the client, network traffic is likely the significant issue.
Consider, as well, using a client connection table, instead of the MQSERVER external variable. |
|
| Back to top |
|
|
| surfnit |
| Posted: Thu Sep 21, 2006 9:37 am Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
Thanks Bruce.
If both applications are using the same client connection with the same unix id, then how do I know each application is going to use a different client connection with MQCHLLIB and the MQCHLTAB system environment variables?
If I create 2 Client Conn channels and give the same Conn Name and Qmgr Name if 1 client connection is busy then will the second application use the next client connection? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 21, 2006 12:28 pm Post subject: |
|
|
Guest
|
Client channels are synchronous, requiring two-way network traffic for every mq call; thus, they are not stellar performers.
Add to this the size and volume of messages your applications are processing, and that both putting and getting application are on the same client, you are seeing sluggish and erratic performance .
Some things to do to improve performance:
1. more bandwidth
2. replace the client with a server (queue manager) instance
Since client channels are created at MQCONN and destroyed at MQDISC,
3. make sure both of your applications are not doing an MQCONN/MQDISC for each put/get. You may chuckle at this; but I've seen this - every message put or get required the application to be launched anew. |
|
| Back to top |
|
|
| surfnit |
| Posted: Thu Sep 21, 2006 12:43 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
Thanks, I will definitely check those MQCalls and verify they are not doing something silly.
I assume when you say more bandwidth you are referring to the network?
I will see if I can get a qmgr on this machine, but I may need to provide proof. The only proof of anything so far is that when the putting application is stopped the performance is normal.
The strange behavior is that the throughput consistently goes down almost after it clears the queue, then the queue depth grows and the writing application can still put messages at the same rate where the reading application cannot get the messages at the rate it was doing before. This is consistently re-producible on this customer's environment. |
|
| Back to top |
|
|
| mvic |
| Posted: Thu Sep 21, 2006 1:02 pm Post subject: Re: MQ Performance Bottlenecks when Reading from queue |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| surfnit wrote: |
We are having problem wih an application thats reader throughput goes down after the curdepth is at 0 messages. Just to give you an idea, the performance starts at 800 or msgs/second (On the Reader) and eventually goes down to 100 msgs to 50 msgs/second.
I do not believe this has anything to do with the application. Any ideas on how one can this issue since it was not able to be re-produced? |
With this and all performance problems, the question that must be asked (and answered) is: where is the time being spent?
Assuming the machine (client and server are on the same machine, right? - in which case it is difficult to see how this is a relevant factor) is not busy with other work - has this been checked? - the time is being spent either (a) in app code, or (b) in MQ code. I suggest you take application traces, and MQ traces to find out where the time is being spent.
To analyse where time is being spent in the app, you might consider running the app under profiling tools The only tool I know for this is Rational Quantify. But that - I think - has to be bought/licensed at a cost, and is unlikely to be acceptable in a production environment.
I hope this helps.
EDIT: edited twice for clarity
Last edited by mvic on Thu Sep 21, 2006 1:05 pm; edited 2 times in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 21, 2006 1:03 pm Post subject: |
|
|
Guest
|
Your last post confirms it! When the putting application stops putting messages to the queue, the getting application runs just fine.
Both the putting application and getting application are competing for the same network connection; and both are competing for the same queue.
As compared with a server, the client imposes additional network flows for each and every mq call your program makes. That this means is that a single put will take two network flows (one for the put and one for the completion code/reason code to be returned). This is double the work of an application connected directly to a server - not using the client.
Yes, bandwidth means network. If you can't replace the client with a server, and you've tweaked the program code to make it as efficient as possible, then a bigger network pipe may be your only option.
Client application performance due to network overhead (I hate that term) is what it is, namely: a adequately performing application with network lag imposed on it.
Or have I completely misunderstood the problem?... |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Sep 21, 2006 2:06 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I don't think network utilization has been proven to be the problem.
Sender app is putting messages in UOWs of 500 messages each.
When the queue is empty, the receiving application has to wait for 500 messages to be sent and then committed.
As long as the queue is backed up by more messages than the receiver can get in the same time it takes the sender to put 500 messages, then the receiver will get messages as fast as it can.
Without enqueue, dequeue rates I don't think we can prove anything.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Fri Sep 22, 2006 1:43 am Post subject: Re: MQ Performance Bottlenecks when Reading from queue |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| surfnit wrote: |
| I do not believe this has anything to do with the application. |
I heard this last year from a collegue (MQ and WBI expert and IBM employee). We had a short look and found the bottleneck in the application!
Do read/write with syncpoint? Do you read with MsgId or CorrelId? Check your code, my collegue made a typo error, so he wrote with but read without syncpoint!
_________________
Regards
Hubert |
|
| Back to top |
|
|
| surfnit |
| Posted: Fri Sep 22, 2006 12:35 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
When the customer ran the application i used to re-produce this behavior it worked fine with no bottleneck. I think this proves the bottleneck is most likely not in MQ. It looks like you guys got me! I should no better than to say "it does not appear the application" on an mq forum.
i am narrowing this down one by one and need to compare all the differences. We are Almost there! |
|
| Back to top |
|
|
|
|
