| Author |
Message
|
| surfnit |
 Posted: Wed Sep 20, 2006 4:04 pm Post subject: MQ Performance Bottlenecks when Reading from queue |
 |
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
We are having problem wih an application thats reader throughput goes down after the curdepth is at 0 messages. Just to give you an idea, the performance starts at 800 or msgs/second (On the Reader) and eventually goes down to 100 msgs to 50 msgs/second.
I do not believe this has anything to do with the application. Any ideas on how one can this issue since it was not able to be re-produced?
Your help is appreciated. This is with MQ Client 6.0 and MQ Server 6.0 on Linux.
Are there any supportpacs the experts are aware that help me troubleshoot this problem? |
|
| Back to top |
|
 |
| surfnit |
| Posted: Wed Sep 20, 2006 4:06 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
| I forgot to mention there are no errors in any mq logs. |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Sep 20, 2006 4:06 pm Post subject: Re: MQ Performance Bottlenecks when Reading from queue |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| surfnit wrote: |
| I do not believe this has anything to do with the application. |
Why not? You can look at the performance supportpacs, where thousands upon thousands of messages are tested without any degradation.
_________________
-wayne |
|
| Back to top |
|
|
| surfnit |
| Posted: Wed Sep 20, 2006 4:30 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
By the way, I work for an IBM partner that has a connector to MQ.
The reason I think it is not the application is that I have tried to re-produce this in-house and have been unsuccessful. Based on this it appears environmental.
Are there any supportpacs that can do a non-destructive read based on a period of time or number of messages whichever one comes first?
I am trying to prove the same behavior exists in a sample application. |
|
| Back to top |
|
|
| surfnit |
| Posted: Wed Sep 20, 2006 4:33 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
Or is there a supportpac that can trace the data between the MQServer and the MQ Client so that I can see the amount of time data is transferred between the 2. If it is during this communication, then I have proof that this is an MQ or network related problem.
Thanks for any feedback! |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Sep 20, 2006 4:55 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The performance reports are Support Packs.
The presupplied sample program amqsbcg will do a "non-destructive" read on a queue.
The presupplied sample program amqsbcgc will do a "non-destructive" read on a queue over a client connection.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| surfnit |
| Posted: Wed Sep 20, 2006 5:00 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
From my experience using this sample program, amqsbcgc will browse ALL messages.
I want to browse the number of messages over a period of time or 500 messages whichever comes first as this is what my application is doing. |
|
| Back to top |
|
|
| csmith28 |
| Posted: Wed Sep 20, 2006 5:01 pm Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
Oh no, couldn't possibly be the application. It must be a problem with MQSeries.
Lets get this straight, once the depth of the queue is 0 you start to have problems.....
Hmm...
Think about what you are saying.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| surfnit |
| Posted: Wed Sep 20, 2006 5:13 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
Sorry if I am not clear.
The queue depth starts at a very high rate around 200,000 messages where there is another application writing to the queue at 500 messages per commit point.
The reading application reads from this queue at a very high rate 800 msgs/second. Once the queue is back to 0, the queue depth then increases and eventually gets back to 100,000 messages to 200,000 messages to even higher. All these messages have been commited to the queue.
The reading application is then only processing 200 msgs/second and then down to 100 msgs/second and down to 50 msgs/second. So the source queue is then not at a depth of 0. The degragation in performance starts after the queue depth is cleared. But once the writing application puts more messages on the queue the reader is not able to process them as fast anymore.
Last edited by surfnit on Thu Sep 21, 2006 9:12 am; edited 1 time in total |
|
| Back to top |
|
|
| surfnit |
| Posted: Wed Sep 20, 2006 5:22 pm Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
So when there are 200,000 messages on the queue the application is only processing less than 100 msgs/second where it was processing 800 msgs/second.
If this is an application issue, how can i prove that as well? That would be using some sample program that reads from a queue every second and report the statistics.
Has anyone had to do this? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Sep 20, 2006 11:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| surfnit wrote: |
So when there are 200,000 messages on the queue the application is only processing less than 100 msgs/second where it was processing 800 msgs/second.
If this is an application issue, how can i prove that as well? That would be using some sample program that reads from a queue every second and report the statistics.
Has anyone had to do this? |
At the risk of being condescending, what differences have you noticed in your application stats when it's processing this heavily laden queue? Have you noticed machine problems (i.e. I/O bottlenecks as it tries to pull the messages off disc)? By what logical process have you determined MQ to be the likely cause of the problem?
Check the SupportPacks as previously suggested for pointers as to how they performed the monitoring.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Sep 21, 2006 1:45 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You shouldn't actually be using another queue to read messages and count how fast it can do that.
You should be using a program that runs the PCF command RESET QUEUE STATISTICS every minute or so, and reports and saves the ENQUEUE and DEQUEUE rate in the last interval.
Then you will know how fast messages are being PUT onto the queue and how fast they are being TAKEN OFF the queue.
Then you can see if messages are being taken off as fast as they are being put on, or significantly slower. Then you'll know that the bottleneck is the sending application or the receiving application.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 21, 2006 6:30 am Post subject: |
|
|
Guest
|
"... it appears environmental"
A few questions:
what platform?
what MQ version?
what programming language?
Are both the putting program and the getting program run concurrently?
Does the getting program do a get with a wait? Or, does it end itself and need to be re-launched for every get?
Messages all in syncpoint (unit of work)?
Are the putting and getting programs across a network? |
|
| Back to top |
|
|
| surfnit |
| Posted: Thu Sep 21, 2006 8:02 am Post subject: |
|
|
Novice
Joined: 24 Jul 2005
Posts: 15
Location: US - CA
|
MQ client version 6.0.1 MQ Server 6.0.1 on a Linux OS. Porgramming Language in C.
The putting and getting applications are running concurrently over a network. I will check on the actual MQ Calls being done and update this thread.
The putting application puts 500 messages per commit point. The getting application reads x number of messages in 1 second.
One interesting point I did not mention is that only when the application is re-started the throughput of the messages being read off the queue goes back up again and it starts to clear the messages off the queue.
The only reason I felt this was environmental (i do not know if it is the application of mq or network), was because i could not re-produce this behavior with a similar environment.
I checked MQ Server File System Space, MQ Server logs, did not check any other system level bottlenecks. I can monitor perhaps with the PCSF commands.
Thanks for the feedback! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 21, 2006 8:50 am Post subject: |
|
|
Guest
|
Which application (putting or getting) is on the client? which on the server? are both on the client?
client connection channels are more network intensive and network sensitivie than qmgr to qmgr connections.
Last edited by bruce2359 on Thu Sep 21, 2006 9:10 am; edited 1 time in total |
|
| Back to top |
|
|
|
|
