| Author |
Message
|
| sat_in_field |
 Posted: Wed Jun 20, 2007 5:03 am Post subject: Large messages >30Mb |
 |
|
Novice
Joined: 23 Jan 2003
Posts: 20
|
Hi,
We are processing large messages through WMB. We are handling these messages as a blob. We do not interrogete the content or need to split these messages.
We store the message to a database, add an RFH2 header and put to an output queue.
Several distinct message flows are involved in the process and are deployed to the same exection group
During the process our Execution group continues to grab memory as each flow performs its work, this memory is not released and subsequent messages add to the memory that the execution group has taken, eventually after processing a small number of these 30Mb messages the execution group falls over.
After processing each message why does the Execution group not return the memory used to the operating system? This is potentially a big issue for us
I believe this may be a memory leak
we are running WMB 6.0 (No CSD's have been applied) |
|
| Back to top |
|
 |
| wschutz |
| Posted: Wed Jun 20, 2007 5:08 am Post subject: Re: Large messages >30Mb |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| sat_in_field wrote: |
we are running WMB 6.0 (No CSD's have been applied) |
Why not? That would be the first thing I'd do.
_________________
-wayne |
|
| Back to top |
|
|
| sat_in_field |
| Posted: Wed Jun 20, 2007 5:32 am Post subject: |
|
|
Novice
Joined: 23 Jan 2003
Posts: 20
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jun 20, 2007 5:54 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
How is "the execution group falls over" not a good reason?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sat_in_field |
| Posted: Wed Jun 20, 2007 6:08 am Post subject: |
|
|
Novice
Joined: 23 Jan 2003
Posts: 20
|
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jun 20, 2007 7:02 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
PK31749 EVAL WITH DATABASE INSERT CAUSES MEMORY LEAK
IY85680 - MEMORY LEAK IN CAST WITH NUMERIC WITH FORMAT KEYWORD
PK25994 - EXTENDED MEMORY USE ON ERRORING LARGE MESSAGE
IY77856 MEMORY LEAK FROM TIMEOUTNOTIFICATION NODE
PQ93688 COMPUTE NODE USES A LOT MORE MEMORY AT V501
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sat_in_field |
| Posted: Wed Jun 20, 2007 7:45 am Post subject: |
|
|
Novice
Joined: 23 Jan 2003
Posts: 20
|
As I said - paging through the APAR's fixed up to wmb6 csd 04 - nothing seems relevant
PK31749 EVAL WITH DATABASE INSERT CAUSES MEMORY LEAK
->Not using an EVAL statements
IY85680 - MEMORY LEAK IN CAST WITH NUMERIC WITH FORMAT KEYWORD
->Not used
PK25994 - EXTENDED MEMORY USE ON ERRORING LARGE MESSAGE
->These messages are not failing
IY77856 MEMORY LEAK FROM TIMEOUTNOTIFICATION NODE
->Not used
PQ93688 COMPUTE NODE USES A LOT MORE MEMORY AT V501
->we have 2 -3 compute nodes in our flows |
|
| Back to top |
|
|
| MiLi |
| Posted: Tue Jun 26, 2007 10:33 pm Post subject: Re: Large messages >30Mb |
|
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
| sat_in_field wrote: |
I believe this may be a memory leak
|
Hmm, what do you want to hear? Yes !?
You dont post any code so we have no clue whats going on !
However, 600MB of memory usage to read 30MB BLOB. No way !
I think the Doc states somewhere that the executiongroups never returns their memory alloc.
Have fun !
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker |
|
| Back to top |
|
|
| Paul D |
| Posted: Wed Jun 27, 2007 9:57 am Post subject: |
|
|
Master
Joined: 16 May 2001
Posts: 200
Location: Green Bay Packer Country
|
We have done 30 MB messages without issue under v5, but we needed some configuations. Things that caught us were the MQ channel thread memory size. This must be adjusted to be larger (less concurrency in the MQ pipe, but it can handle larger messages). If you started from MQ and got into the flow you might be past this one. We were doing this over HTTP requests so we needed to up the heap in the HTTPListener and the exe group's JVM as we were also hitting that before we reduced the message size. We were also on AIX and needed to run some commands against the DataFlowEngine.exe to increase the memory capacity. This is typical, there was a tech note on that one. I'd suggest working with IBM through a PMR to achieve success.
_________________
Thanks!!!
Paul D |
|
| Back to top |
|
|
| sat_in_field |
| Posted: Tue Jul 03, 2007 2:16 am Post subject: |
|
|
Novice
Joined: 23 Jan 2003
Posts: 20
|
Hi, we recieved a document from IBM detailing WMB handling of large messages and efficient processing of them.
The document is "Memory and performance considerations when handling large messages in the broker."
We had many code issues but it seems that large messages need large amounts of memory. We are now succesfully processing the 30 Mb messages and our dataflow engine is only using an extra @100mb of memory.
I can pass the doc on. |
|
| Back to top |
|
|
| mymq |
| Posted: Tue Jul 17, 2007 6:35 am Post subject: |
|
|
Centurion
Joined: 01 Mar 2007
Posts: 101
Location: US-Greenwille
|
Can u place the document somewhere where we can have access. That would be great.
_________________
--SRK-- |
|
| Back to top |
|
|
|
|
