| Author |
Message
|
| klamerus |
 Posted: Sat Jul 02, 2005 9:27 am Post subject: Best approach to large messages |
 |
|
Disciple
Joined: 05 Jul 2004
Posts: 199
Location: Detroit, MI
|
| What would be a good approach to handling large messages (perhaps as large as 20 MB) that we may occasionally get? |
|
| Back to top |
|
 |
| EddieA |
| Posted: Sat Jul 02, 2005 11:02 am Post subject: |
|
|
Jedi
Joined: 28 Jun 2001
Posts: 2453
Location: Los Angeles
|
Make sure all the queues, including transmit and DLQs, and all the channels have a MAXMSGLEN capable of handling the message size you expect.
Maybe, depending on your setup and needs, allocate seperate channels to handle these so small messages don't queue up behind them.
Cheers,
_________________
Eddie Atherton
IBM Certified Solution Developer - WebSphere Message Broker V6.1
IBM Certified Solution Developer - WebSphere Message Broker V7.0 |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sat Jul 02, 2005 12:26 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Also, don't forget MaxMsgLN on the queue managers.
Also, don't forget that messages are received in unit buffers - so your programs need to be able to allocate memory buffers up to 100MB in size.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Jul 02, 2005 4:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And don't forget to adjust the size of your logs for those big messages.
Adjust as well the batchsize on your channels. You do not want 50 20MB messages in one channel transaction...
The logs will need to accomodate for transactional logging of the big messages so if you run with the default (3 +2 ) * 4 MB you're doomed as you would only be able to (barely) handle a 20MB transaction if no other transaction is in flight... Check as well on filesystem space.
Remember you have to delete and recreate the qmgr to change the log file size. You do not need this to change the number of log files.
You may as well need more frequent archiving if you use linear logs.
Enjoy  |
|
| Back to top |
|
|
| klamerus |
| Posted: Sat Jul 02, 2005 4:56 pm Post subject: |
|
|
Disciple
Joined: 05 Jul 2004
Posts: 199
Location: Detroit, MI
|
| I thought there was a max size limit of 4 MB on messages. At least that's the default, and I don't think we can set it any higher. |
|
| Back to top |
|
|
| klamerus |
| Posted: Sat Jul 02, 2005 5:00 pm Post subject: |
|
|
Disciple
Joined: 05 Jul 2004
Posts: 199
Location: Detroit, MI
|
NVM.
I was relying on my guys and this is what they told me. Sheesh.
Well given we can make them bigger (which is what we want), I appreciate all the added info on how to manage those bigger messages in real life. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Jul 02, 2005 5:54 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If these fat messages are going to be transported between your various QMs, I would recomend dedicated batch channels as well. A 20 MB messages will take a little while for the channels to move. At that instant, if your other applications' little messages land on the XMITQ, they will have to wait. If they are not time sensitive, no big deal. If they are, then opt for a second set of batch channels.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| klamerus |
| Posted: Sun Jul 03, 2005 1:35 pm Post subject: |
|
|
Disciple
Joined: 05 Jul 2004
Posts: 199
Location: Detroit, MI
|
| Well, actually we seriously hope to never see these buggers and if we do to see very, very few of them. |
|
| Back to top |
|
|
| sebastianhirt |
| Posted: Sun Jul 03, 2005 3:21 pm Post subject: |
|
|
Yatiri
Joined: 07 Jun 2004
Posts: 620
Location: Germany
|
Even if the chance is less then 1%, make sure to take fjb_saper's point on board and have your logs big enough. I just ended up biting into my own ass, because having to little logspace on one of my queue managers. We were expecting some messages with 20MB (1 or 2 a day) on that queue manager. Now we are getting a lot of messages with 50 to 300 MB and are struggeling with log file space.
cheers
Sebastian |
|
| Back to top |
|
|
| klamerus |
| Posted: Sun Jul 03, 2005 3:36 pm Post subject: |
|
|
Disciple
Joined: 05 Jul 2004
Posts: 199
Location: Detroit, MI
|
| If we set the system up to allow very large messages, does anything "preallocate" space bsaed on this parameter thus consuming system resources even if we don't run into hardly any of these very large messages. In other words, we don't actually use up any specific resource unless the situation comes along where we start getting these big messages (yes/no)? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sun Jul 03, 2005 4:39 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Malloc is usually pretty consistent about allocating the space you ask it to...
That said, you can do a get with a 1 byte buffer, and then process the returned MQMD for the size you need and handle the MQRC warning about truncated message...
Then you will only every allocate exactly how much space you need and no more or no less.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Mon Jul 04, 2005 12:15 am Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
| If you are going to peek at the msgs first, as jeff suggests, use BROWSE with LOCK, so that you are sure to get the same msg when you do a destructive get. |
|
| Back to top |
|
|
| kalash |
| Posted: Mon Jul 04, 2005 12:32 am Post subject: |
|
|
Newbie
Joined: 24 Nov 2004
Posts: 6
|
| EddieA wrote: |
Make sure all the queues, including transmit and DLQs, and all the channels have a MAXMSGLEN capable of handling the message size you expect.
Maybe, depending on your setup and needs, allocate seperate channels to handle these so small messages don't queue up behind them.
Cheers, |
I tested, on Windows WMQ 6, with a message (18Mo). I just increased the MAXMSGLEN for the : QMs, XMITQ, DLQ and QL receive. Test => OK. Thanx for your information...   |
|
| Back to top |
|
|
| klamerus |
| Posted: Mon Jul 04, 2005 6:03 am Post subject: |
|
|
Disciple
Joined: 05 Jul 2004
Posts: 199
Location: Detroit, MI
|
Lucky you (getting MQ v6). I've been trying to get the time to download and test that for a couple of weeks now  .
Anyway, it's not the program memory I mean in allocation of resources, it was pre-allocation of any MQ resources. For intsance, we have maximum queue length set and we also have max message length set. The concern was whether there were any sort of internal resource management structures that MQ had that might combine those with something else and blow a limit.
MQ doesn't do the following, but imagine if it pre-allocated disk space of the size needed for messages (say 100 MB) times the max queue length (which for us is 100,000 on these particular queues), and thereby used up all the disk on the system.
We know that doesn't happen, but were thinking of some other resource. That was what we were concerened with. Seems like we're all okay on this though.
Our system is a batch system (not real-time), so if it takes a while for the messages to get across the channels to us, that'll be fine (even if it's a minute or two). |
|
| Back to top |
|
|
| malammik |
| Posted: Tue Jul 05, 2005 5:00 am Post subject: |
|
|
Partisan
Joined: 27 Jan 2005
Posts: 397
Location: Philadelphia, PA
|
In my opinion best approach for dealing with large messages over 4mb is not to have them at all. Not always, but very often when the message has to exceed the 4mb threashold, mq is either being misused or there could have been better solution to design applications not to use such large messages. That is my personal opinion, yours might be different. For me, it is a warning sign that something aint right.
_________________
Mikhail Malamud
http://www.netflexity.com
http://groups.google.com/group/qflex |
|
| Back to top |
|
|
|
|
