| Author |
Message
|
| belchman |
 Posted: Mon May 11, 2009 10:09 am Post subject: |
 |
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
I assumed it was old because of the v1.2... 1) I am lazy and it was near the top of my search, 2) I am dangerous when it comes to java, 3) I was trying to find a way to inform poster that I think constraint is on app as opposed to MQ or JVM should be eliminated as cause
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Mon May 11, 2009 11:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20757
Location: LI,NY
|
| Quote: |
| I don't mind getting the message in smaller chunks as long as I can reassemble, but I do not know how to do this and can not see any API methods to achieve this. |
Look at the MQMD of the different messages on the queue.
If the message has been segmented there should be a common identifier for the segments (group-id?) a sequence number for each message in the sequence, and a marker for last in sequence.
This should allow you to reassemble and you can then use a temp file to store the full message and do your stuff from there...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gunter |
| Posted: Tue May 12, 2009 5:19 am Post subject: |
|
|
Partisan
Joined: 21 Jan 2004
Posts: 307
Location: Germany, Frankfurt
|
This works for me:
| Code: |
...
gmo.options = MQC.MQGMO_LOGICAL_ORDER|MQC.MQGMO_ALL_MSGS_AVAILABLE;
gmo.matchOptions = MQC.MQMO_NONE;
do
{
queue.get(message, gmo);
int len = message.getDataLength();
....
gmo.matchOptions = MQC.MQMO_MATCH_GROUP_ID;
} while (gmo.groupStatus != MQC.MQGS_LAST_MSG_IN_GROUP);
... |
_________________
Gunter Jeschawitz
IBM Certified System Administrator - Websphere MQ, 5.3 |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Thu May 14, 2009 4:49 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Gunter,
Thanks for the suggestion. I have tried your code and have the following revised lines:
| Code: |
while(queue.getCurrentDepth() > 0)
{
...
gmo.options = MQC.MQGMO_LOGICAL_ORDER|MQC.MQGMO_ALL_MSGS_AVAILABLE;
gmo.matchOptions = MQC.MQMO_NONE;
do
{
queue.get(message, gmo);
int len = message.getDataLength();
....
gmo.matchOptions = MQC.MQMO_MATCH_GROUP_ID;
if(gmo.segmentStatus== MQC.MQSS_LAST_SEGMENT)
gmo.matchOptions = MQC.MQMO_NONE;
} while (gmo.segmentStatus!= MQC.MQSS_LAST_SEGMENT);
..
} |
The check for last message should check for LAST_SEGMENT, not LAST_MSG_IN_GROUP (this is used for msg grouping, but no necessary set in segmentation). Also, adding a outer while loop allows for multiple segmented message on the queue.
This is a good workaround, so thanks gunter.
However, MQ supports message segmentation and so does the BaseMQ Java API. Implementing the above in my code is not letting MQ do the hard work as the programmer is having to implement the segmentation logic. I still feel there should be a way for my Java app to progress large message (e.g. 300MB) by only setting the MQC.MQSEG_ALLOWED and MQC.MQGMO_COMPLETE_MSG flags and asking MQ for the whole message. Its no good MQ supporting segmentation for large messages if my Java application can not handle messages over 150MB. Any ideas/comments. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu May 14, 2009 4:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You aren't having an issue with MQ, nor with the MQ API.
You're having an issue with the JVM acquiring sufficient memory space for what it needs to meet your request. |
|
| Back to top |
|
|
| belchman |
| Posted: Thu May 14, 2009 5:08 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
 In fear of getting laughed out of here, I will still ask...
If you cannot resolve the issue of your app or your JVM providing enough memory in which to read your message, is it thinkable (especially since XML is mostly whitespace) that you compress the message before putting the message on the queue and uncompressing it upon getting it off of the queue?
Other than that, using MQ grouping and send the XML across in subsets (not segmentation, grouping), then...
Oh wait... those ideas won't work either... you are still going to need 300MB to process the message.
But I concur... your problem is not during MQ transport. The crux is that you do not have a large enough buffer into which to read your message.
I ramble 
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Thu May 14, 2009 5:31 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
I agree that the problem is due to the size of Java buffer I require to read the message. Its just frustrating that MQ supports large message with segmentation and the BaseMQAPI supports segmentation, but I can not create a large enough buffer in my application, therefore not being able to fully use these features.
I was hoping someone might know what JVM memory parameter I need to change but no joy so far.
Thanks for all the suggestions. At least I have a work around for now
 |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu May 14, 2009 5:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Right, so setting -Xmx should generally be sufficient for what you need.
It appears from the errors you posted that the JVM is having difficulty getting memory *from* the OS. So either the memory is not available in the way you think it is, or something else is funny. |
|
| Back to top |
|
|
| belchman |
| Posted: Thu May 14, 2009 5:53 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
I had a IM conversation with one of our WebLogic/WebSphere admins... Here are some of the key points of the discussion... may or may not be help...
Me: if an application is getting an out of memory error... and the app is on a JVM... what thoughts do you have on how to alter env to get more alloc mem to app
Him: seems like the xmx settings was tried. that was going to be my suggestion- but since they did that...
Him: my next suggestion would be to have them enable verbose garbage collection. that way we can see what part of the heap is getting filled / not collected. from there we can target the perm gen size and such.
Me: so how much mem did he alloc to app with his xms msg
Him: min heap size of 500mb, max heap size of 1100mb
Me: so u suggest he cud have > 800mb or garbage?
Him: Yes, it is possible he is reading too many messages for that allocated heap- he may need to read them one at a time or get an even larger heap to compensate for the number of messages. if he is reading in a message- it will stay in 'reference' throughout the read and will not be 'collected'
Let me know if this is helpful and I will tell Him he is 'of use'
_________________
Make three correct guesses consecutively and you will establish a reputation as an expert. ~ Laurence J. Peter |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu May 14, 2009 6:11 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
I have been caught out by this issue before when using WAS and trying to read 80MB messages. After raising a PMR with IBM they eventually told me there was a parameter you could specify that reserved space in the heap for large objects. I applied this setting and it definatly helped.
Unfortunalty I cant remeber the parameter name (and ive since left the company where we applied it) but if you raise a PMR with IBM they should be able to help you out. |
|
| Back to top |
|
|
| belchman |
| Posted: Thu May 14, 2009 6:35 am Post subject: |
|
|
Partisan
Joined: 31 Mar 2006
Posts: 386
Location: Ohio, USA
|
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu May 14, 2009 6:40 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Thats the parameter! Thanks for putting me out of my misery! |
|
| Back to top |
|
|
| anilit99 |
| Posted: Mon Jun 01, 2009 1:43 pm Post subject: |
|
|
Voyager
Joined: 28 May 2009
Posts: 75
Location: London, UK
|
Very interesting thread ! As a complete novice to MQ, I never thought you could actually post a 300-700mb of message !
But am I missing some thing here, because why cant we just write that message in a file in some accessible location (to producer and receiver, ftp or sftp) and just send a message containing the path to that file location ?
I know I am assuming zillion things here, but if each message is of 300 mb, its a huge network hog and as a matter of fact a hog of almost everything.
I know I am not helping in any way to your immediate problem in hand, but just saying !
I also read somewhere that this out of swap space can occur if the JVM failed to allocate that 300mb in one contiguous chunk (since we are talking of one single object) |
|
| Back to top |
|
|
| chrisgclark |
| Posted: Tue Jun 02, 2009 7:10 am Post subject: |
|
|
Apprentice
Joined: 26 Mar 2009
Posts: 35
|
Hi,
We want to use MQ to send these large messages as:
1. MQ ensures reliable, guaranteed one-time delivery
2. We are using MQ as our messaging program and don't want to start manually implementing FTP/other file transfer logic into our programs. Let MQ handle the details of transferring the message. This saves implementation time and increase quality of the code.
3. sender and receiver are on separate servers and don't want to setup and support a separate FTP service.
4. as sender and receiver are on separate servers you still need to send the large message over the network, whether its over MQ or file transfer.
My JVM should have large amount of continuous memory as my dev PC has lots of RAM and very little is currently in use.
If anyone gets the same problem I had, please try belchman and WMBDEV1's suggestion of the JVM parameter (haven't had time yet). If not, use the workaround I posted on 14th May. |
|
| Back to top |
|
|
|
|
