| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Performance problems putting large nos of messages to a MQ Q |
« View previous topic :: View next topic » |
| Author |
Message
|
| JohnMetcalfe |
 Posted: Mon Nov 22, 2004 6:42 am Post subject: Performance problems putting large nos of messages to a MQ Q |
 |
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
We have a WMQI flow that takes in a single XML message that contains 20,000 row items, then splits it up and writes out to 4 outputs Qs an individual message per row - hence it is writing out a total of 80,000 messages onto 4 output Qs. We are using propagate to write out the messages. I can post the code that is doing this if anyone is interested - it's pretty simple.
We are observing that the first 8,00 - 10,000 messages are written to the output Qs very quickly, but then the rate decreases markedly - initially the rate would be perhaps 200 msgs/sec, but it then declines to perhaps 10 - 20 msgs/sec. Obvious at this slower rate, it takes too long to write out the remaining messages.
We were told that it might be to do with a MQ memory buffer size setting, which is defaulted to 64K. We had the MQ ops guys change this to 1MB, but it seemed to make no difference.
We also wondered if it was down to MQ unit of work size, initially all 80,000+ messages were written out as a single unit of work. However, changing the code so that each message was in its own unit of work made no noticiable difference.
The only option we have come up with is to have a 'pre-processsor' flow split up the large message into a number of smaller ones, each one with, say, 5000 row items in it.
We are running WMQI 2.1 CSD3.
Has anyone seen this behaviour before? Can anyone give us a suggestion as to a soln? Is this an MQ problem or a WMQI problem? |
|
| Back to top |
|
 |
| EddieA |
| Posted: Mon Nov 22, 2004 10:03 am Post subject: |
|
|
Jedi
Joined: 28 Jun 2001
Posts: 2453
Location: Los Angeles
|
How are you "looping" through the input in order to Propogate the outputs. I'll guess that you are just using an Index. This is very inefficient as it has to start from the start of the tree every time you use it. As the numnber of iterations increases, so does the time taken.
I'd suggest you look into using References.
Cheers,
_________________
Eddie Atherton
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Tue Nov 23, 2004 2:08 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
Eddie,
many thanks for the response.....
We are already using references and also the delete field function to 'clean-up' memory as we move through the large input message.
Our basic loop construct approach and memory cleanup approach is shown in the following sample code (I've not just pasted in the live code as this obviously does quite a bit more on top)
| Code: |
-- Create a special folder in the output message to hold the input message tree
CREATE LASTCHILD OF Environment.Variables DOMAIN 'XML' NAME 'InputMessage';
-- Copy the input message to this just created special folder.
-- This is a root to root copy which will therefore not build trees....
SET Environment.Variables.InputMessage = InputRoot.XML;
DECLARE i integer;
DECLARE endOfList boolean;
SET i = 1;
DECLARE MyPointer REFERENCE TO Environment.Variables.InputMessage.CSL.Message.TxnGeneralList.Txn[1];
WHILE endOfList = FALSE
DO
-- copy headers
SET OutputRoot.Properties = InputRoot.Properties;
SET OutputRoot.MQMD = InputRoot.MQMD;
-- create the output tree
SET OutputRoot.XML.CSL.Message.List.Txn[i].InstCd = MyPointer.InstCd;
SET OutputRoot.XML.CSL.Message.List.Txn[i].SeqNum = MyPointer.SeqNum;
DECLARE lastProcessedRow REFERENCE TO MyPointer;
SET i = i + 1;
-- move to next row
MOVE MyPointer TO Environment.Variables.InputMessage.CSL.Message.TxnGeneralList.Txn[2];
--did the move of the pointer fail ?
IF LASTMOVE(MyPointer) = FALSE THEN
-- yes, so at end of list structure SET endOfList = TRUE;
END IF;
--delete the input message row just processed, and free memory used...
DELETE FIELD lastProcessedRow;
-- send out an output message.
PROPAGATE;
END WHILE;
-- finally delete the temporary folder and free up any memory it is using...
DELETE FIELD Environment.Variables.InputMessage; |
We are seriously thinking that this is a MQ or WMQI configuration setup, as we can construct a test harness that simply spits out messages onto a Q and we are see the same behaviour. 
For example, the simple code as below shows the noted behaviour
| Code: |
DECLARE counter integer;
SET counter = 1;
WHILE (counter <= 20000) DO
SET OutputRoot.Properties = InputRoot.Properties;
SET OutputRoot.MQMD = InputRoot.MQMD;
-- create the fragment of the output tree
SET OutputRoot.XML.Message.Field1 = 'Test Message:' || CAST(counter AS CHAR);
SET OutputRoot.XML.Message.Field2 = 'Test Field 2';
SET counter = counter + 1;
PROPAGATE;
END WHILE;
RETURN FALSE; |
We are scratching our heads on this one - any more ideas anyone... ? |
|
| Back to top |
|
|
| RAJESHRAMAKRISHNAN |
| Posted: Tue Nov 23, 2004 3:34 am Post subject: |
|
|
Voyager
Joined: 01 May 2004
Posts: 96
|
Hi John,
My question might sound bit elementary but still ...
Is your output queue a local queue or a remote queue? If it is a remote queue then can you please try with a local queue and see whether you are facing the same problem. If it is a remote queue then there could be network and other issues. |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Tue Nov 23, 2004 3:51 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
| Hi, yes we thought of this, so we tried with the test harness and the real code against both local and remote Qs. It made no noticable difference. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
