| Author |
Message
|
| JFK99 |
 Posted: Tue Jan 22, 2013 4:31 am Post subject: Streaming large data to a Java Message. |
 |
|
Newbie
Joined: 22 Jan 2013
Posts: 4
|
In the following JBoss/HornetQ user manual page (http://docs.jboss.org/hornetq/2.2.2.Final/user-manual/en/html/large-messages.html) you can see how HornetQ provides a mechanism for streaming data to a Message for a Queue using a java.io.InputStream. A JMS version of the same code is given. Has anyone come across an equivalent using IBM MQSeries / WebsphereMQ?
Say I have a large amount of data to place in the JMS Message which to me is just a stream of bytes. In the Hornet example, the stream is only read when the message is sent, so if it is, say a FileInputStream, then we only need enough memory to buffer a chunk of the bytes. I can use a javax.jms.BytesMessage to send in chunks of bytes and use the BytesMessage to buffer them. The problem with this is that the IBM implementation of BytesMessage (com.ibm.msg.client.jms.internal.JmsBytesMessageImpl) has to cache them until the Message is sent and if that is a large amount of data it is a problem. Worse it appears that although I am only sending bytes, the IBM implementation appears to keep duplicate copies, one in a BytesArrayOutputStream the other in a DataOutputStream. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Jan 22, 2013 5:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
An MQPUT requires the full buffer at the time of MQPUT.
Therefore, any program sending a message requires enough memory to hold at least one copy of the message.
JMS must take whatever object is passed in to the BytesMessage object and serialize that to a byte array to pass to MQPUT.
You can use the JMS classes to enable MQ message segmentation and, probably more importantly for your scenario, message grouping.
http://pic.dhe.ibm.com/infocenter/wmqv7/v7r5/topic/com.ibm.mq.doc/jm41030_.htm
This will enable you to stream a chunk of the input data of a fixed length into a single MQ message that is logically correlated with a set of other MQ messages. This group of messages can be read in order and the entire big stream rebuilt at the receiving side, or parsed in fixed-length chunks if logically possible. |
|
| Back to top |
|
|
| JFK99 |
| Posted: Tue Jan 22, 2013 6:13 am Post subject: |
|
|
Newbie
Joined: 22 Jan 2013
Posts: 4
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 22, 2013 6:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| JFK99 wrote: |
| Does com.ibm.msg.client.jms.internal.JmsBytesMessageImpl store my byte[] twice. Once in a ByteArrayOutputStream the other in a DataOutputStream? Is there anyway to ask it to stop? |
You see that part where it says 'internal' in the package name?
That means that it's an internal class that is not documented by IBM.
If you need this level of specific information about internal and undocumented classes and behavior, you need to open a PMR.
 |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 22, 2013 6:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| JFK99 wrote: |
| Does com.ibm.msg.client.jms.internal.JmsBytesMessageImpl store my byte[] twice. Once in a ByteArrayOutputStream the other in a DataOutputStream? Is there anyway to ask it to stop? |
| mqjeff wrote: |
| If you need this level of specific information about internal and undocumented classes and behavior, you need to open a PMR |
I'd also recommend that you tackle the PMR from the direction of what exactly it is you're trying to do and/or the problems you're getting so that's there's the possiblty of providing you with an alternative that doesn't rely on the internal code.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jan 22, 2013 8:52 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I thought there existed an alternative to the JMS BytesMessage for this behavior: the JMS StreamMessage... Have you tried using it? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JFK99 |
| Posted: Wed Jan 23, 2013 5:14 am Post subject: |
|
|
Newbie
Joined: 22 Jan 2013
Posts: 4
|
Many thanks for all your replies. On stack overflow someone also suggested a ReferenceMessage. http://pic.dhe.ibm.com/infocenter/wmqv7/v7r5/index.jsp?topic=%2Fcom.ibm.mq.doc%2Ffg12740_.htm
The StreamMessage will still keep a buffer of all that is written, because of the MPUT restriction mentioned above?
With StreamMessage each write of a byte[] chunk will be treated as a separate field, so it wasn't my first choice. However it may use less memory than a BytesMessage. I will try it out. |
|
| Back to top |
|
|
| JFK99 |
| Posted: Wed Jan 23, 2013 9:47 am Post subject: |
|
|
Newbie
Joined: 22 Jan 2013
Posts: 4
|
| With equivalent payload under MQ 7.0 StreamMessage cannot be written on to Q (the same one which is set up for BytesMessage and works) because it is too big! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 23, 2013 9:54 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| JFK99 wrote: |
| With equivalent payload under MQ 7.0 StreamMessage cannot be written on to Q (the same one which is set up for BytesMessage and works) because it is too big! |
presumably because the stream message wraps each block that was streamed in metadata, which increases the size of the message so that it exceeds the maxmsgln of the queue. |
|
| Back to top |
|
|
|
|
