| Author |
Message
|
| novice |
 Posted: Thu Aug 03, 2006 5:35 am Post subject: performance issue |
 |
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
hi everybody!
we did some tests to measure the maximum throughput of a message-flow. the message-flow as one input queue. two compute nodes are used. the first does a transfomation, the second one a simple db lookup. then the message is written to three output queues.
the message flow should be rather i/o-intensiv: persistent messages are used, three output queues, db lookup.
the tests were run on a mainframe with 4 processors (z990 2084-304).
the queues were placed on different page sets and different disks.
according the system admin the logs are placed on fast disks.
wmq 5.3, wmb 5.0
4 instances of the same message flow have been deployed (4 exec groups).
first results:
cpu busy: 40%
no i/o delays
because of these results we deployed four more instances of the message flow (total 8 ) hoping to increase the throughput.
but the results were still the same. same usage of the cpu's.
what could be the bottleneck?
it can't be the i/o, because there are almost no delays.
it can't be the cpu's, because they are just used to 40%
any idea?
could it be some internal broker process as the transaction manager?
is it because only one input queue is used? what is actually the impact of just using one input queue? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Aug 03, 2006 5:40 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You're only doing selects on the database, right? Then you shouldn't be contending there, right?
Also, how are you measuring the throughput? You might not be measuring what you think you're measuring.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| novice |
| Posted: Thu Aug 03, 2006 6:05 am Post subject: |
|
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
yes, the same select of a char(5) for every message.
on the output-node i use the option pass identity. on this way the actual put time is written in the mqmd. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 03, 2006 6:08 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
How does the dequeue rate of the flow input queue compare to the enqueue rates of the flow output queues?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| novice |
| Posted: Thu Aug 03, 2006 6:31 am Post subject: |
|
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
i) how can i measure that?
ii) i just measure the enqueue rate of one output queue.
iii) overall the whole test time the average dequeue and enqueue rate have to be the same. otherwise this would mean that some messages would disapear  one instance of the message flow reads the next message (n+1) from the input queue after finishing the actual transaction (writing to the output queue message n).
did i misunderstood your question? |
|
| Back to top |
|
|
| novice |
| Posted: Thu Aug 03, 2006 6:37 am Post subject: |
|
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
i've to mention the way we did the tests:
i) the get of the input queue is on inhibited.
ii) filling up the input queue with thousands of msg.
iii) allow gets from the input queue.
iv) measure the enqueue rate (first and last msg) of one of the output queues |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 03, 2006 6:39 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The number of messages taken off the Input queue do indeed have to match the number of mesages put on the Output queue.
The rate at which messages are taken off the input queue does NOT have to match the rate at which messages are put ON the output queue.
In fact, you really should be measuring three numbers: enqueue rate on Input queue, dequeue rate on Input queue, enqueue rate on output queues.
This is done with a monitoring application that can issue the PCF command RESET QUEUE STATISTICS.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| novice |
| Posted: Thu Aug 03, 2006 6:48 am Post subject: |
|
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
can you be more specific with your last post.
what could be the different scenarios?
how can the dequeue be different to the enqueue rate? where are the messages?
thanks a lot |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 03, 2006 7:06 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Sorry, I didn't see your second comment about how you run your test.
You still want to see how fast broker is pulling messages off the input queue. It will help you determine whether broker is spending more time getting messages or processing them.
You can try and calculate this by making sure you know exactly when the first message is retrieved (the exact time you get-enable the queue) and the exact time the queue depth goes to zero.
But you will get much more reliable numbers using RESET QUEUE STATISTICS.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| novice |
| Posted: Thu Aug 03, 2006 7:36 am Post subject: |
|
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
hmmm... i still don't get it.
the dequeue rate will be the same as the enqueue rate.
e.g. for one instance of a message flow:
the time between to gets corresponds to the time the broker takes to get one message, process it and write it to the output queue. after that the next get is done.
what about are you thinking? i guess we are misunterstandig? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 03, 2006 8:26 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The count will be the same.
The rate may not be the same.
If you put 10,000 messages, and it takes Five minutes for broker to get all of them so that the Input Queue depth is 0, but it takes TEN minutes for the Output queues to have all 10,000 messages... then obviously the enqueue rate is slower than the dequeue rate. And the problem is in your flow.
And if you actually measure these numbers at a much smaller time scale than "the lifetime of the test" like every minute, AND you see differences over the lifetime of the test, then either the messages are getting processed at different rates, or something else is interfering with broker.
You should read the performance support pack. It discusses a good bit about how IBM conducted measurements.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Bill.Matthews |
| Posted: Thu Aug 03, 2006 12:37 pm Post subject: |
|
|
Master
Joined: 23 Sep 2003
Posts: 232
Location: IBM (Retired)
|
| novice wrote: |
i've to mention the way we did the tests:
i) the get of the input queue is on inhibited.
ii) filling up the input queue with thousands of msg.
iii) allow gets from the input queue.
iv) measure the enqueue rate (first and last msg) of one of the output queues |
Your technique is causing a lot of extra work by MQ. When you fill up the input queue with thousands of messages, unless these are very samll the odds are high that MQ is having to write many of these to the queue on disk rather than being able to retain them in memory - as would be the case in a normal environment.
You would be much better off in having the sending application maintain a steady state of messages, perhaps by sending a "batch", checking the current depth (in a loop - with a pause) until the depth goes down and send another batch.
You also do not indicate if there is a receiving application pulling the messages off of the output queue so that that queue does not build up where messages have to be pushed out of the queue buffer in order to hold the new message.
_________________
Bill Matthews |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Aug 03, 2006 1:23 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Actually, the performance support packs recommend pre-loading the queue. This prevents broker from having to wait around for another message to be put, so you know that you are only timing the broker itself and are not bound by the performance of your putting application.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| novice |
| Posted: Mon Aug 07, 2006 1:29 am Post subject: |
|
|
Apprentice
Joined: 20 Jun 2005
Posts: 37
|
the fact that preloading a queue can cause a performance degradation is in my opinion correct.
messages are always hold in memory. once the memory is not enough the messages are written to disk (virtual memory).
is this correct?
apart of that, the problem with the tests is still the same:
no I/O delays
cpu busy 40%
more instances don't use the cpu -> somewhere is bottleneck, where?
what's actually the difference of using one or more input queues? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 07, 2006 2:38 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I assume your select is on DB2. Do you do it with UR (Uncommitted Read)?
Are you sure your bottle neck is not the DB? What kind of locking do you have (page /vs record level)? What happens if you skip (comment out ) the DB lookup alltogether?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
