| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Trying to Parse a 70 meg message in WMQI |
« View previous topic :: View next topic » |
| Author |
Message
|
| MQ_MQSI_User |
 Posted: Wed Jul 30, 2003 1:48 pm Post subject: Trying to Parse a 70 meg message in WMQI |
 |
|
Novice
Joined: 11 Jul 2002
Posts: 20
Location: Charlotte, NC
|
I tried to parse a 70 meg message in WMQI and WMQI chocked on me. I was able to do a 14 meg file fine, are there any parameters I can increase to enable WMQI to process this 70 meg file.
_________________
MQSeries, MQSI |
|
| Back to top |
|
 |
| Craig B |
| Posted: Fri Aug 01, 2003 5:26 am Post subject: |
|
|
Partisan
Joined: 18 Jun 2003
Posts: 316
Location: UK
|
There are no specific parameters that need to be configured to process a message of this size. You should make sure that the Execution Group that contains the Message flow has enough storage to process such a request. The MQInput node will need at least 70MB to pick the message buffer up and then the storage required for the parsed message tree will depend on the number of fields in a message this size. The XML and MRM parsers support partial parsing so you should only need to parse up to the fields that you are referencing. Could you answer the following questions :
1) What message domain are you processing the message in? XML, MRM, New Era??
2) Roughly how many fields are there in the input message?
3) Do you reference every field in the message, or reference fields toward the end of the message?
4) Do you intend to perform any transformations on the message? If so, will these affect most records in the message?
5) How many compute nodes and MQOutput nodes are in your message flow?
6) Are you running WMQI V2.1 and if so, which CSD do you have installed.
Just some general points :
i) If you are running on AIX then you many need to increase the number of memory segments the Execution groups have available for storage.
ii) If you are processing a New Era message with a large number of repeating records you may need to increase the thread stack size variable for WMQI.
iii) If you are processing generic XML and manipulating a large number of repeating records, then there is functionality to process these records, without ever maintaining the whole input tree and output tree in memory.
If you could provide further information, then I could provide more detail for your specific case.
_________________
Regards
Craig |
|
| Back to top |
|
|
| mqhelp |
| Posted: Sat Aug 02, 2003 8:31 pm Post subject: |
|
|
Newbie
Joined: 20 Jun 2003
Posts: 6
|
What exactly happening? Failing at QMgr/Queue level? Taking a lot of time to process?
Some ideas:
If your MQseries version is 5.2 or later it should support the messge size of upto 100Meg. But as Craig pointed out processing a large message in a message flow involves lot of factors to be considered such as number of nodes, persistance, as each node makes a copy (if you select persistent) before processing and hence require more system memory.
Also check the MAXMSGL (max message length) property of both QManager and Queue should be set to more than message size. Set correct values and restart QMgr to take new values effective.
Regards,
Srini |
|
| Back to top |
|
|
| gcrimy |
| Posted: Sun Aug 03, 2003 2:28 am Post subject: |
|
|
Newbie
Joined: 16 Dec 2001
Posts: 7
|
Craig B,
Any chance on elaborating on point 3 in your dialog:
| Quote: |
| iii) If you are processing generic XML and manipulating a large number of repeating records, then there is functionality to process these records, without ever maintaining the whole input tree and output tree in memory. |
I'm keen to here of what method you are thinking of using..
Thanks |
|
| Back to top |
|
|
| Craig B |
| Posted: Mon Aug 04, 2003 4:18 am Post subject: |
|
|
Partisan
Joined: 18 Jun 2003
Posts: 316
Location: UK
|
If you have WMQI V2.1 CSD04 or higher then look in the readme.txt and search for 'Handling large messages' that is part of the DELETE FIELD notes. As far as I am aware, this functionality was added in CSD03, but this readme.txt documentation was added in CSD04.
This should give you information relating to point 3. The principle behind this is that only one repeating record needs to be kept in memory at anyone time for the Input message. And for the output message, you do not have to keep parsed records, you can construct the bitstream as you go along.
_________________
Regards
Craig |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
