| Author |
Message
|
| digulla |
 Posted: Mon Mar 07, 2005 9:05 am Post subject: Reading large messages (> 100MB) |
 |
|
Novice
Joined: 07 Mar 2005
Posts: 10
|
Hello,
I have a router application where I get huge messages and I need to process them with Java. Usually, I get a large data message to send to some app or I get a request to query an app for data which results in a large data message which goes the other way.
Currently, MQ seems to try to read all message data at once which means that I need lots of heap space: 500MB per connection (2x256MB to have some slack). That means, even with the most simple routing (2 apps), I need 1GB of RAM (each processor runs in its own thread and I cannot change that either  )
Is there a way to postpone the reading of the message data and to get this data as a stream using Java (so I can copy it in 100KB chunks)?
If not, how should I handle such messages? I'd try segmentation but I cannot change the applications which send me data. Can I tell the MQManager to join the segments before it delivers them to the other side and to chop the messages up before it delivers them to me?
_________________
Aaron Digulla |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Mar 07, 2005 9:26 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Your post title is wrong. There's no way to read messages greater than 100MBs. Messages can't be bigger than 100MBs. Segmentation doesn't change that (neither does grouping, but grouping allows you to consider multiple 100MB messages as "one").
You also can't, as far as I know, change the RAM allotment needed by the MQMessage class or the JMSMessage* classes.
If you consider the way that MQ works, you might understand better. MQGET accepts and then passes back a buffer consisting of the entire contents of the message. Any program that uses MQ has to allocate enough storage in their own memory space to hold the amount of data needed.
Any use of QMgr segmentation will require code changes on both sides - in order to set the options that allow segmentation and the options that tell the QM to join the segments.
You could reduce the memory overhead and increase the speed by switching your routing app to C... 
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| digulla |
| Posted: Mon Mar 07, 2005 9:34 am Post subject: |
|
|
Novice
Joined: 07 Mar 2005
Posts: 10
|
Thanks for the quick reply 
Can you give me an example for GETting and PUTting segmented messages in Java? I browsed the various threads in the forum but couldn't find anything useful (most code says 'this doesn't work. Why?').
_________________
Aaron Digulla |
|
| Back to top |
|
|
| MiLi |
| Posted: Thu May 24, 2007 6:28 am Post subject: |
|
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
| jefflowrey wrote: |
Your post title is wrong. There's no way to read messages greater than 100MBs. Messages can't be bigger than 100MBs. Segmentation doesn't change that (neither does grouping, but grouping allows you to consider multiple 100MB messages as "one").
|
What mq version ?
Segmentation allows you to put messages larger than 100Mb on queues ! or am I missing something ?
I have the same problem now anyway, I have a scenario where I'm reading a file stream and running some regular expressions on it (as CharBuffer). And now I whant to do that on a MQ message, Can I stream the message to a charBuffer somehow ?
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker |
|
| Back to top |
|
|
| zpat |
| Posted: Thu May 24, 2007 6:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
|
| Back to top |
|
|
| Vitor |
| Posted: Thu May 24, 2007 10:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| MiLi wrote: |
Segmentation allows you to put messages larger than 100Mb on queues ! or am I missing something ?
|
Pedantically, the largest single segment remains 100Mb.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| MiLi |
| Posted: Fri May 25, 2007 12:37 am Post subject: |
|
|
Acolyte
Joined: 07 Oct 2005
Posts: 51
|
| Vitor wrote: |
Pedantically, the largest single segment remains 100Mb. |
Sure thing.
But when dealing with messages like with the jms api you dont care how the message is represented by the message provider, only that you can put and get a message... and you can put messages larger than 100 MB (its up to MQSeries to segment it to 20 segmented messages if it whants to).
_________________
IBM Certified System Administrator - WebSphere MQ V6.0
IBM Instructor - WebSphere MQ, WebSphere Message Broker |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 25, 2007 10:22 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| MiLi wrote: |
| Vitor wrote: |
Pedantically, the largest single segment remains 100Mb. |
Sure thing.
But when dealing with messages like with the jms api you dont care how the message is represented by the message provider, only that you can put and get a message... and you can put messages larger than 100 MB (its up to MQSeries to segment it to 20 segmented messages if it whants to). |
Can you quickly remind me if the JMS API supports segmentation?? I did not think so. What you can do is split it and reconstruct it as the JMS API supports grouping...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
