| Author |
Message
|
| MiLi |
 Posted: Wed Feb 14, 2007 7:19 am Post subject: Large msg, Broker dies, out of memory. |
 |
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
Hi
I'm in a project now having some issues with large messages. (Only MQ as protocol)
I got XML messages that can be 50MB and needed to be parsed by the broker.
First of all, I know its a BIG msg and MQ Series and Broker is a messaging system !
Second: No the sender cant split it into smaller transactions.
I know there is alot of this subject here on MQSeries.net but i need your input on this.
With that said I think that a 50MB message should not be a problem, in fact I want the Broker (v6 csd3) to split the message before I send it further because its "too" big for the other systems.
My problem is that when the message enter the flow (parsed with a messageSet) the DataFlowEnginge process goes up to 1,8GB memory usage and 1,8GB vm memory and Dies (works fine with a slighly smaller message and 1,7GB mem usage) !
I got a test flow that parses the 50MB message and just copy the input elemets to OutputRoot but it still uses 1GB memory. (Windows, Xeon quad, 4GB mem)
Is this normal for the Broker ?
Can I use the XMLNSC parser for the split to reduce the memory cost ?
Should I use the BLOB "parser" and then parse a little bit of the message and try to puzzle together several out messages?
Just need to start a discussion for an URGENT problem !
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker
Last edited by MiLi on Thu Feb 15, 2007 4:15 am; edited 1 time in total |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Feb 14, 2007 7:35 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There is discussion in the reference documentation for the XMLTransformation node about how to increase the memory allocations for the Broker - at least for Java.
There is a "Large message" sample in the Samples Gallery.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| kimbert |
| Posted: Wed Feb 14, 2007 3:52 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
|
| Back to top |
|
|
| SandiSan |
| Posted: Wed Feb 14, 2007 10:14 pm Post subject: |
|
|
Acolyte
Joined: 01 Apr 2006
Posts: 57
Location: Pune/India
|
It also depends on what hardware configuration you are running the Broker. I can think of follwing options that can be used to handle large messages.
1. Split the message in MB using ESQL / JCN . If you are using JCN increase the JVM Heap Size using mqsichangeproperty.
2. IF you can drop the message from MQ queue as blob to a particular folder.The you can use file extender node to pick it up and do processing, as File extender node is capable of handling large msg.
3. You can also look at DSTX Plug-in Node that uses that execute Datastage tranformation extender maps . This is also capable for handling large and complex messages.
Regards
Sandisan |
|
| Back to top |
|
|
| MiLi |
| Posted: Thu Feb 15, 2007 12:48 am Post subject: |
|
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
Thanks for the input.
Kimbert, the link describes what I'm trying to do and I have tested it... still with "to much memory peaks" when using MRM.
The message coming in is like
<ROOT>
<info/>
<header>
<numberOfTran/>
<totalAmountofTrans/>
</header>
<trans/>
<trans/>
...
...
</ROOT>
as you can see, if I'm going to split it I need to parse the entire message to be able to produce messages with correct header info.
I did this with the XMLNSC parser and the memory peak is ONLY 500MB, 
So I got a solution for splitting bif messages.
I just want to know if other people have processed messages over 50MB in the Message broker. What is the limit ? Can the Broker read a segmented mq message over 100MB ?
Tanks MQSeries.net you are the best ![/quote]
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Feb 15, 2007 3:32 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| I need to parse the entire message to be able to produce messages with correct header info. |
Who designs these schemas?
Anyway, I'm glad XMLNSC did the job for you. |
|
| Back to top |
|
|
| MiLi |
| Posted: Thu Feb 15, 2007 3:57 am Post subject: |
|
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
| Quote: |
| Who designs these schemas? |
Hehe
Its not some random system its Euro Banking Association !
http://www.abe.org/
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker |
|
| Back to top |
|
|
| MiLi |
| Posted: Thu Feb 15, 2007 4:13 am Post subject: |
|
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
For the record:
I now did a test with a 150MB XML message segmented to 15*10MB messages on the input queue.
Parsing it by XMLNSC Domain.
Looping every <trans> tag to calculate value x, every 1000 trans the loop breakes and propagates !
<ROOT>
<info/>
<header>
<numberOfTran/>
<totalAmountOfX/>
</header>
<trans><value>x</value></tans>
<trans><value>x</value></tans>
...
...
</ROOT>
Works FINE, takes 30 sec and the memory peaks at 750MB.
WebSphere Message Broker is incredible
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker |
|
| Back to top |
|
|
| 4integration |
| Posted: Thu May 10, 2007 6:56 am Post subject: |
|
|
Disciple
Joined: 04 Sep 2006
Posts: 197
Location: Gothenburg, Sweden
|
| jefflowrey wrote: |
....
There is a "Large message" sample in the Samples Gallery. |
Have anyone implemented the Sample or similar with Java (JCN) and would like to share that?
EDIT: I have problems with large messages sent from IBM WBIA for JDBC and must split them up before transformation.
_________________
Best regards
4 Integration |
|
| Back to top |
|
|
|
|
