ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » IBM MQ Java / JMS » Streaming large data to a Java Message.

Post new topic  Reply to topic
 Streaming large data to a Java Message. « View previous topic :: View next topic » 
Author Message
JFK99
PostPosted: Tue Jan 22, 2013 4:31 am    Post subject: Streaming large data to a Java Message. Reply with quote

Newbie

Joined: 22 Jan 2013
Posts: 4

In the following JBoss/HornetQ user manual page (http://docs.jboss.org/hornetq/2.2.2.Final/user-manual/en/html/large-messages.html) you can see how HornetQ provides a mechanism for streaming data to a Message for a Queue using a java.io.InputStream. A JMS version of the same code is given. Has anyone come across an equivalent using IBM MQSeries / WebsphereMQ?

Say I have a large amount of data to place in the JMS Message which to me is just a stream of bytes. In the Hornet example, the stream is only read when the message is sent, so if it is, say a FileInputStream, then we only need enough memory to buffer a chunk of the bytes. I can use a javax.jms.BytesMessage to send in chunks of bytes and use the BytesMessage to buffer them. The problem with this is that the IBM implementation of BytesMessage (com.ibm.msg.client.jms.internal.JmsBytesMessageImpl) has to cache them until the Message is sent and if that is a large amount of data it is a problem. Worse it appears that although I am only sending bytes, the IBM implementation appears to keep duplicate copies, one in a BytesArrayOutputStream the other in a DataOutputStream.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jan 22, 2013 5:02 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

An MQPUT requires the full buffer at the time of MQPUT.

Therefore, any program sending a message requires enough memory to hold at least one copy of the message.

JMS must take whatever object is passed in to the BytesMessage object and serialize that to a byte array to pass to MQPUT.

You can use the JMS classes to enable MQ message segmentation and, probably more importantly for your scenario, message grouping.

http://pic.dhe.ibm.com/infocenter/wmqv7/v7r5/topic/com.ibm.mq.doc/jm41030_.htm

This will enable you to stream a chunk of the input data of a fixed length into a single MQ message that is logically correlated with a set of other MQ messages. This group of messages can be read in order and the entire big stream rebuilt at the receiving side, or parsed in fixed-length chunks if logically possible.
Back to top
View user's profile Send private message
JFK99
PostPosted: Tue Jan 22, 2013 6:13 am    Post subject: Reply with quote

Newbie

Joined: 22 Jan 2013
Posts: 4

Thank you. I had forgotten about grouping.

Here is a useful example/quick tutorial for anyone following on grouping in JMS.

http://www.ibm.com/developerworks/websphere/library/techarticles/0602_currie/0602_currie.html

Here is an old link which discusses the problem of large messages, including segmentation.

http://www.tek-tips.com/viewthread.cfm?qid=33458

Does com.ibm.msg.client.jms.internal.JmsBytesMessageImpl store my byte[] twice. Once in a ByteArrayOutputStream the other in a DataOutputStream? Is there anyway to ask it to stop?
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jan 22, 2013 6:21 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

JFK99 wrote:
Does com.ibm.msg.client.jms.internal.JmsBytesMessageImpl store my byte[] twice. Once in a ByteArrayOutputStream the other in a DataOutputStream? Is there anyway to ask it to stop?


You see that part where it says 'internal' in the package name?

That means that it's an internal class that is not documented by IBM.

If you need this level of specific information about internal and undocumented classes and behavior, you need to open a PMR.

Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Jan 22, 2013 6:43 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

JFK99 wrote:
Does com.ibm.msg.client.jms.internal.JmsBytesMessageImpl store my byte[] twice. Once in a ByteArrayOutputStream the other in a DataOutputStream? Is there anyway to ask it to stop?


mqjeff wrote:
If you need this level of specific information about internal and undocumented classes and behavior, you need to open a PMR




I'd also recommend that you tackle the PMR from the direction of what exactly it is you're trying to do and/or the problems you're getting so that's there's the possiblty of providing you with an alternative that doesn't rely on the internal code.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Jan 22, 2013 8:52 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

I thought there existed an alternative to the JMS BytesMessage for this behavior: the JMS StreamMessage... Have you tried using it?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
JFK99
PostPosted: Wed Jan 23, 2013 5:14 am    Post subject: Reply with quote

Newbie

Joined: 22 Jan 2013
Posts: 4

Many thanks for all your replies. On stack overflow someone also suggested a ReferenceMessage. http://pic.dhe.ibm.com/infocenter/wmqv7/v7r5/index.jsp?topic=%2Fcom.ibm.mq.doc%2Ffg12740_.htm

The StreamMessage will still keep a buffer of all that is written, because of the MPUT restriction mentioned above?

With StreamMessage each write of a byte[] chunk will be treated as a separate field, so it wasn't my first choice. However it may use less memory than a BytesMessage. I will try it out.
Back to top
View user's profile Send private message
JFK99
PostPosted: Wed Jan 23, 2013 9:47 am    Post subject: Reply with quote

Newbie

Joined: 22 Jan 2013
Posts: 4

With equivalent payload under MQ 7.0 StreamMessage cannot be written on to Q (the same one which is set up for BytesMessage and works) because it is too big!
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jan 23, 2013 9:54 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

JFK99 wrote:
With equivalent payload under MQ 7.0 StreamMessage cannot be written on to Q (the same one which is set up for BytesMessage and works) because it is too big!

presumably because the stream message wraps each block that was streamed in metadata, which increases the size of the message so that it exceeds the maxmsgln of the queue.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » IBM MQ Java / JMS » Streaming large data to a Java Message.
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.