| Author |
Message
|
| 4integration |
 Posted: Mon Jun 18, 2007 1:10 pm Post subject: Large messages, approx 90 MB |
 |
|
Disciple
Joined: 04 Sep 2006
Posts: 197
Location: Gothenburg, Sweden
|
Hello,
I have a MRM/CWF message containing:
1 header, 80 chars/8 fields
x items, 400 chars/22 fields
When running this without any split the broker (v6) dies and therefore I have started to look into a split function (which have solved similar problems before).
Instead of parse the complete message into CWF field I have created a msgdef that have a 80 char header part and x number of 400 char items.
Then my thought was to split the message into maybe a header + 10 items per propagate and Reset Content Descriptor to the "real" msgdef.
The problem is that the broker still dies on the input node...
Any idea how to get this solved??
_________________
Best regards
4 Integration |
|
| Back to top |
|
 |
| Michael Dag |
| Posted: Mon Jun 18, 2007 1:25 pm Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Jun 18, 2007 2:23 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
The broker should not die, but you could explore this alternative...v6 CWF supports 'repeat until end of bitstream'.
- Model your repeating structure as an element with maxOccurs=-1
- Set 'Parse Timing' to 'On demand'
- Write your ESQL to access one instance of the structure at a time.
There's a good description of this kind of technique here. I think the examples are for XML, but they can be applied just as well to CWF:
http://www-128.ibm.com/developerworks/websphere/library/techarticles/0505_storey/0505_storey.html |
|
| Back to top |
|
|
| 4integration |
| Posted: Mon Jun 18, 2007 11:11 pm Post subject: |
|
|
Disciple
Joined: 04 Sep 2006
Posts: 197
Location: Gothenburg, Sweden
|
Thanks, I will look into that and also try to make use of URL with CWF.
A design question...
The incoming message are based on fixed length fields and currently the message sets are using CWF.
Comparing to TDS (which should also be able to handle this, or?) which format is best from a performance point of view??
_________________
Best regards
4 Integration |
|
| Back to top |
|
|
| 4integration |
| Posted: Mon Jun 18, 2007 11:44 pm Post subject: |
|
|
Disciple
Joined: 04 Sep 2006
Posts: 197
Location: Gothenburg, Sweden
|
| kimbert wrote: |
The broker should not die, but you could explore this alternative...v6 CWF supports 'repeat until end of bitstream'.
- Model your repeating structure as an element with maxOccurs=-1
- Set 'Parse Timing' to 'On demand'
- Write your ESQL to access one instance of the structure at a time.
|
Now I have the "LargeMessageMsgDef" which looks like:
| Code: |
LargeMessage (LargeMessageType)
- Header (xsd:string)
- Logical properties:
- Min: 1
- Max: 1
- Physical Properties:
- Length count: 80 bytes
- Msg (xsd:string)
- Logical properties:
- Min: 1
- Max: -1
- Physical Properties:
- Length count: 400 bytes
|
On the MQ input node "Parse Timing" = "On Demand"
I have increased the JVM HeapSize to 512 MB:
| Code: |
C:\Program Files\IBM\MQSI\6.0>mqsireportproperties WBRK6_DEFAULT_BROKER -e DEFAULT_EG -o ComIbmJVMManager -n jvmMaxHeapSize
536870912
BIP8071I: Successful command completion. |
When running with the ~90 MB message I get:
| Code: |
JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
......JVMDG318: Heap dump file written to C:\Program Files\IBM\MQSI\6.0\bin\heapdump.20070619.092616.4152.phd
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to C:\Documents and Settings\All Users\Application Data\IBM\MQSI/common/errors/\javacore.20070619.092622.4152.txt
JVMDG274: Dump Handler has Processed OutOfMemory.
JVMST109: Insufficient space in Javaheap to satisfy allocation request
JVMDG217: Dump Handler is Processing OutOfMemory - Please Wait.
JVMDG315: JVM Requesting Heap dump file
......JVMDG318: Heap dump file written to C:\Program Files\IBM\MQSI\6.0\bin\heapdump.20070619.092627.4152.phd
JVMDG303: JVM Requesting Java core file
JVMDG304: Java core file written to C:\Documents and Settings\All Users\Application Data\IBM\MQSI/common/errors/\javacore.20070619.092633.4152.txt
JVMDG274: Dump Handler has Processed OutOfMemory.
JVMST109: Insufficient space in Javaheap to satisfy allocation request
Exception in thread "Thread-12" java.lang.OutOfMemoryError: JVMCI156:OutOfMemoryError, IBMJVM_ResizeArray - stAllocArray for new array failed
at com.ibm.jvm.ExtendedSystem.resizeArray(Native Method)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:112)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1733)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1642)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1089)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:309)
at com.ibm.etools.vfd.comm.command.SerializerDeserializer.serialize(SerializerDeserializer.java:57)
at com.ibm.etools.vfd.comm.command.SerializerDeserializer.serialize(SerializerDeserializer.java:42) |
I have a break point after the input node and does not reach that, very frustrating...
have I missed something or do I need to increase the max JVM HeapSize even more?
Any input is appreciated!!
_________________
Best regards
4 Integration |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jun 19, 2007 12:23 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
How did you determine that 512Mb was sufficient?
Did you read the article that was linked?
How many repeats do you actually have in the message that is causing your broker to fall over?
There won't be any noticable improvements using TDS instead of CWF. Simplifying the model - so that it treats the repeatable bits as large blocks of data rather than individual fields - will help some.
Did you open a PMR about the heap/core dump? What did IBM Support have to say?
Are you running at least Broker Runtime 6.0.0.3?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| 4integration |
| Posted: Tue Jun 19, 2007 12:46 pm Post subject: |
|
|
Disciple
Joined: 04 Sep 2006
Posts: 197
Location: Gothenburg, Sweden
|
| jefflowrey wrote: |
| How did you determine that 512Mb was sufficient? |
Actually I didn't have any good figure, just tried to increase it a little
| Quote: |
| Did you read the article that was linked? |
Yes, I have also used it in another scenario where we have repeatable elements in XML.
I am a "java guy" and not that good in ESQL + that we (the company) have as policy to use Java as the primary language in the broker, therefore I am trying to do the similar thing in Java but not very successful yet.
If you have any hints in how to do the similar thing in Java I would be very happy.
Currently I have only Message Domain (=MRM) set on the input node and it now passes the input node.
| Quote: |
| How many repeats do you actually have in the message that is causing your broker to fall over? |
The number of repeatable items is 250.968 (400 bytes each)
| Quote: |
| There won't be any noticable improvements using TDS instead of CWF. Simplifying the model - so that it treats the repeatable bits as large blocks of data rather than individual fields - will help some. |
Thanks. When I used the LargeMessage on MQ Input node I had each item as repeatable (400 bytes) but will try to increase that to maybe 40.000 or 400.000 bytes and see if it behaves better.
| Quote: |
| Did you open a PMR about the heap/core dump? What did IBM Support have to say? |
I have not yet opened a PMR (have not got the time for it  )
| Quote: |
| Are you running at least Broker Runtime 6.0.0.3? |
Yes
_________________
Best regards
4 Integration |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jun 19, 2007 12:53 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There's not a lot different in Java than in ESQL - in terms of working with the message tree - that will improve the memory usage.
The thing that is different in Java, though, is that you should reuse objects as much as possible, and limit the number of new objects that you create. Which is really not that different from doing the same kinds of performance tuning in Java outside of Broker.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| 4integration |
| Posted: Tue Jun 19, 2007 1:45 pm Post subject: |
|
|
Disciple
Joined: 04 Sep 2006
Posts: 197
Location: Gothenburg, Sweden
|
| 4integration wrote: |
| Quote: |
| There won't be any noticable improvements using TDS instead of CWF. Simplifying the model - so that it treats the repeatable bits as large blocks of data rather than individual fields - will help some. |
Thanks. When I used the LargeMessage on MQ Input node I had each item as repeatable (400 bytes) but will try to increase that to maybe 40.000 or 400.000 bytes and see if it behaves better.
|
I tried to include several items per block; 40.000, 400.000 and 4.000.000 bytes per "item block" but it didn't help. The broker runs out of memory...
_________________
Best regards
4 Integration |
|
| Back to top |
|
|
|
|
