| Author |
Message
|
| chrisgclark |
 Posted: Fri May 08, 2009 6:38 am Post subject: large maxMsg size, java.lang.OutOfMemoryError |
 |
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Hi,
I'm using the Java MQ Base API to get messages from a queue. The messages can be quite large (up to 300mb, using message segmentation) so i'm setting max message size when i do a get from the queue:
| Code: |
queue.get(message, getMsgOptions, 314572800) |
Message segmentation is working, but when i try to retrieve a large message I get the following Java error:
| Quote: |
| java.lang.outofmemoryError: requested 314572800 bytes for jbyte in .... Out of swap space? |
I've tried increasing max heap size using -Xms500m -XMx1100M but still get above error.
A bit of googling suggested increase max perm size, but if I set -XX:MaxPermSize=700M then i get
| Quote: |
Error occurred during initialization of VM
Could not reserve enough space for object heap
|
Any less then 700M then I get the outofmemory error.
Any suggestions for fixing this problem would be appreciated.
Thanks |
|
| Back to top |
|
 |
| WMBDEV1 |
| Posted: Fri May 08, 2009 6:59 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Dont load it all into memory at once.....
use the readfully(byte[]) method to read it in bit by bit. |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Mon May 11, 2009 2:19 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Hi WMBDEV1,
Thanks for your reply.
Unless I'm missing something I don't think using the readfully(byte[]) method will solve this problem.
I'm getting Java memory errors even when running the following code:
| Code: |
MQMessage message = new MQMessage();
queue.get(message, getMsgOptions, 314572800)
// do nothing else |
So the problem is when getting the message off the queue, not doing something with the message once it has been read. The readfully method seems to only help with processing a message once it has been read off a queue.
If there any way to read a message off a queue in 50MB sections?
If not, any other ideas.
Thanks |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon May 11, 2009 3:07 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you tried reading the message byte by byte and passing it to a temp file using FileOutputStream? Then for processing the message use a FileInputStream and finally when done clean up by deleting the temp file.
Have fun... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Mon May 11, 2009 3:37 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Hi fjb_saper,
Thanks for your suggestion, however the problem is when getting a message off the queue, not reading a message I have taken off a queue and have in memory. My program is trying to get a message of a queue then write this to file. The error occurs when issuing the MQ get method, so writing the message to file then reading back in will not help.
The get method, with 314572800 specified (300mb) seems to ask the JVM for 300MB of memory. This then errors as can not get 300MB of memory, even though my dev PC has plenty of free RAM and the java heap size is far more than my program is using.
I can not find a method to get a message off a queue byte by byte like you have suggested, only process byte by byte a message I have in memory. |
|
| Back to top |
|
|
| gunter |
| Posted: Mon May 11, 2009 4:22 am Post subject: |
|
|
Partisan
Joined: 21 Jan 2004
Posts: 307
Location: Germany, Frankfurt
|
use this MQQueue::get method:
| Code: |
| void get(MQMessage message, MQGetMessageOptions getMessageOptions) |
_________________
Gunter Jeschawitz
IBM Certified System Administrator - Websphere MQ, 5.3 |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Mon May 11, 2009 5:30 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Hi Gunter,
thanks but I started off with just
| Code: |
void get(MQMessage message, MQGetMessageOptions getMessageOptions) |
Unfortunately using the above get method causes an error when trying to get a large message (e.g. 150MB). The error was
| Quote: |
| mqrc 2079: MQRC_TRUNCATED_MSG_ACCEPTED error. |
This is with MQSEG_ALLOWED and MQGMO_COMPLETE_MSG as the get message options.
After researching online I discovered you can set the max message size in the get method, and I am setting this to my max message (i.e. 300MB). The problem just moves to a Java out of memory error, shown above. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Mon May 11, 2009 6:12 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
If you add the get option:
| Code: |
| MQC.MQGMO_ACCEPT_TRUNCATED_MSG |
You wont get the 2079 code then. Maybe you could then read it smaller chunks  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon May 11, 2009 6:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It may not make any sense to read the message in smaller chunks.
It may, for example, be an XML file. XML is unparsable in smaller segments.
It seems like the system does not have enough of the right memory in a continuous chunk to meet the requirement being asked. "Out of swap space" for example says that maybe there needs to be more physical swap space available. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Mon May 11, 2009 6:27 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Jeff,
I agree it may not be possible but then again.... it may be and so it was just a suggestion
Also, I disagree with your comment about XML being unparseable in smaller segments. It may be possible to use a SAX parser or XMLStreamReader to parse the XML structure chunk by chunk. |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Mon May 11, 2009 6:35 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Hi,
The issue is NOT parsing/processing a large message I have retrieved from a queue. The issue is getting a large (e.g. 200MB) message from the queue manager, with segmentation and get complete message enabled. i.e. the get() method fails as JVM can not allocate 300MB in message to store the message it is trying to retrieve.
I need the whole message, so unfortunately accepting a truncated message is not an option.
I don't mind getting the message in smaller chunks as long as I can reassemble, but I do not know how to do this and can not see any API methods to achieve this. If I do a get and receive a truncated message, how would I then retrieve the rest of the message (i.e. the part after the chunk I have already received)? I do not think this will be possible as the message has been removed from the queue at this point.
I thought about not setting MQSEG_ALLOWED and MQGMO_COMPLETE_MSG as get options, then retrieving the separate messages manually and reassembling them in my program to get the large original message.. however, the whole point of MQ doing segmentation is that it avoids the programmer having to implement this in the sending and retrieving application. I do not really want to go down this route.
I have 3GB of RAM on my PC. With at least 2GB of free RAM I start my Java program and still get this error. It's only trying to allocate 300MB so this should be easy for the o/s to do. Are there any JVM settings for setting swap space? Any other Java memory settings I may be hitting.
Chris
p.s. thanks for all help so far  |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon May 11, 2009 7:44 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| WMBDEV1 wrote: |
| Also, I disagree with your comment about XML being unparseable in smaller segments. It may be possible to use a SAX parser or XMLStreamReader to parse the XML structure chunk by chunk. |
well, maybe kimbert will wander by and give us both a trout but...
how can you parse an XML document without the closing tag? |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Mon May 11, 2009 9:47 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| mqjeff wrote: |
how can you parse an XML document without the closing tag? |
The parser can Keep reading until you reach the closing tag, then raise an event about what you have found and then discard what you had read and repeat for the other tags.
Maybe we are misunderstanding each other.... All im trying to say is that it should be possible to stream the data to the parser to reduce memory usage. To answer the posters question though.... I cant see any streaming APIs for MQ (Except maybe in the undocumented MQMsg2 class). |
|
| Back to top |
|
|
| belchman |
| Posted: Mon May 11, 2009 10:01 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
Be advised that the 300 MB you have allocated is to the buffer (field) into which your message will be read. You have not changed the allocation to the app of the jvm hosting the app.
Below is one example of what I think is someone's method for increasing the allocation to the app itself (so that it can support the buffer you are creating).
http://www.rgagnon.com/javadetails/java-0131.html
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Mon May 11, 2009 10:04 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Belchman.... thats quite an old link you posted there!
Also, the poster has said....
| Quote: |
| I've tried increasing max heap size using -Xms500m -XMx1100M but |
in his opening post. Im guessing the incorrect capitalisation on -XMx110M is just a typo. |
|
| Back to top |
|
|
|
|
