| Author |
Message
|
| JohnMetcalfe |
 Posted: Thu Jun 16, 2005 5:34 am Post subject: MQPut performance |
 |
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
Hi
We have a WMQI flow that takes in a XML message that contains multiple repeating rows, then spins through this message outputing a new MQ msg per row. Some of our messages have a large number of repeating rows (10,000+).
We are finding a performance problem with the mq put - the first 2-3000 messages are put to the target Q very quickly, then the rate of msg puts starts to slow down and it gets get slower and slower the more messages that are put.
To give you an idea, this is the sort of performance we are getting
No of msgs,Total Elapsed Time (sec), Time to put last 1000 msgs (sec)
1000, 0:00:02, 0:00:02
2000, 0:00:07, 0:00:05
3000, 0:00:15, 0:00:08
4000, 0:00:25, 0:00:10
5000, 0:00:38, 0:00:13
6000, 0:00:53, 0:00:15
7000, 0:01:10, 0:00:17
8000, 0:01:31, 0:00:21
9000, 0:01:53, 0:00:22
10000, 0:02:18, 0:00:25
As you can see, the first 1000 messages are put in 2 sec, the last 1000 message take 25secs to put.
Do anyone know what is happening here? - I would guess I'm hitting some limitation in MQ. Is there any tuning I can do to MQ (or WMQI!) to speed up the puts?
Regardless, I must find a way to speed things up - the flows are time critical, I might have 20 - 30 secs window to process but not over 2mins!
FYI, it seems to make little difference if the messages are put as part of a MQ transaction or not - they are marginally quicker when not under sync point but this is only by a total of 10 secs or so in the test above.
Thanks for any assistance in advance..... |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Jun 16, 2005 5:41 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I would suspect not the MQPut but the loop you are using to build the messages getting put.
This is really an WMQI question, so hopefully a mod will move it there.
Are you using references or direct indexes?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| bower5932 |
| Posted: Thu Jun 16, 2005 5:54 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
Are the messages persistent or non-persistent? Are you doing anything to get messages while you are putting them? You may be spilling over the queue memory buffer and being forced to write your messages to disk.
What operating system is this on? |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Thu Jun 16, 2005 8:16 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
Messages are non-persistent.
The intended production achitecture is that the messages will be written to a remote Q, which then goes via a channel to a local Q on a different Q manager. An application is reading from this local Q, but normally (!) more slowly than we can write them, so this local Q will back up.
In the test that produced the figures, it was just writing to a local Q, with no app reading the the messages off. However, its observed behaviour was similar to that we are seeing in the intended production setup - here we see the messages piling up on the XMIT Q, then when they are committed they are flushed across the channel.
Re the loop construct we are using to write the messages out in WMQI, it was reasonably complex, so I've retested with a really simple loop contruct in ESQL, as below:
DECLARE counter integer;
SET counter = 0;
WHILE (counter <= 10000) DO
SET OutputRoot = InputRoot;
PROPAGATE;
SET counter = counter + 1;
END WHILE;
RETURN FALSE;
The retest result showed it was slightly faster, but it still took 1min 51 secs to write out 10000 1K messages.
Does this mean I can rule out the WMQI loop construct as the problem, and turn my focus back to MQ? |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Thu Jun 16, 2005 8:17 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
| FYI, we are running on AIX Unix server (IBM P690 with 3CPUs) |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jun 16, 2005 8:33 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I don't currently have access to a broker installation to run my own tests, sorry.
Maybe you've already done this, but try reducing it to a completely simple flow: MQInput -> Compute -> MQOutput, where the compute node has the code you posted.
If this evidences the same performance problems, then open a PMR with IBM.
If not, dig into the rest of your flow.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Thu Jun 16, 2005 8:55 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
Thanks Jeff,
yep already tried this - my retest ws exactly as you described - MQInput > Compute Node with simple loop construct > MQOutput.
I'll get a PMR raised with IBM, unless anyone else comes up with any flashes of inspiration..... |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jun 16, 2005 9:00 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Ron Bower's comment about the queue memory buffer may be something to look into.
If I recall, there were some "not quite documented" environment variables that would allow you to increase this.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Thu Jun 16, 2005 9:38 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
Thanks Jeff, I'll do some digging tomorrow. I'm slightly nervous though about your 'not quite documented' comment !  |
|
| Back to top |
|
|
| bower5932 |
| Posted: Thu Jun 16, 2005 9:52 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| Not quite documented means that you can find them in the performance related SupportPacs. Basically, non-persistent messages are stored in a memory buffer. If this buffer fills, then messages start to get written to disk so they aren't lost. If this is your problem, you could try a quick test by running a program to get messages while you are putting them. This should keep the buffer empty. |
|
| Back to top |
|
|
| EddieA |
| Posted: Thu Jun 16, 2005 10:33 am Post subject: |
|
|
Jedi
Joined: 28 Jun 2001
Posts: 2453
Location: Los Angeles
|
How are you iterating through the input message to generate the individual messages from the input. If you are using indexing, then for every output message, you will be starting from the top of the input again, and so takes longer and longer for each.
Try using Reference pointers and Move, as these are much more efficient on large mesages and should give the same processing time for the 1,000th iteration as the 1st.
Cheers,
_________________
Eddie Atherton
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Jun 16, 2005 11:02 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| EddieA wrote: |
How are you iterating through the input message to generate the individual messages from the input. If you are using indexing, then for every output message, you will be starting from the top of the input again, and so takes longer and longer for each.
Try using Reference pointers and Move, as these are much more efficient on large mesages and should give the same processing time for the 1,000th iteration as the 1st. |
Eddie -
We covered that... 
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Fri Jun 17, 2005 9:48 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
After some more digging today, our current theory is that it's a WMQI performance problem, possibly related to the use of ESQL PROPAGATE and WHILE loops.
Why?...if I load a Q up with 30000 messages, then let a simple WMQI flow read the messages of the input Q and put to an output Q, we are seeing message put rates of 3000/sec i.e it takes 10secs to put 30,000 messages to the output Q - (the simple WMQI contains nothing more than a MQInput Node > Compute node that copies InputRoot to OutPutRoot > MQOutput node). Hence it has to be overhead within WMQI while loop structure or propagate function that is limiting put rates. Has anybody else out there used propagate in WMQI to output 10,000 + msgs in a while loop?
As this is almost certainly a WMQI problem, how do I get this thread moved to the WMQI Support forum? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Jun 17, 2005 11:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It's probably a performance issue with PROPAGATE.
Hopefully, a moderator will move this thread.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| malammik |
| Posted: Fri Jun 17, 2005 11:42 am Post subject: |
|
|
Partisan
Joined: 27 Jan 2005
Posts: 397
Location: Philadelphia, PA
|
What's your memory consumption during this process? Does it remain the same or does it keep growing? Where does your cpu spend more time during slower and faster interval: system mode or user mode. My guess is that your process causes exec group memory pages to get paged out.
_________________
Mikhail Malamud
http://www.netflexity.com
http://groups.google.com/group/qflex |
|
| Back to top |
|
|
|
|
