| Author |
Message
|
| Vin |
 Posted: Wed Oct 23, 2002 10:57 am Post subject: Message Bundling |
 |
|
Master
Joined: 25 Mar 2002
Posts: 212
Location: India
|
| We are designing an application which involves remote communication and exchange of messages of various types (Application Specific). The thing we wanted to do was to bundle the messages according to the type, so that there would be this one huge message with baby messages inside it. The recieving application would then parse this huge message and extract the smaller messages from it, that way we can avoid the header which goes out with each message. The communication is happening between the Mainframe and Unix (Solaris) machine. Any thoughts or inputs on how to do that. We're still in the design phase for this app and would highly appreciate any suggestions what so ever. Thanks so much for the help. |
|
| Back to top |
|
 |
| RogerLacroix |
| Posted: Wed Oct 23, 2002 8:42 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Yuk,
To save a few milliseconds, you would rather introduce more complexity in the receiving application (probably in the sending app. too).
KIS - Keep It Simple. There is a lot to be said for this expression.
If a little message is a unit of work or describes a business function then send it by itself.
That's my 2 cents.
later
Roger...
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| abmanesh |
| Posted: Thu Oct 24, 2002 5:01 am Post subject: |
|
|
Apprentice
Joined: 06 Nov 2001
Posts: 39
|
| You could format the message in XML . You could have multiple nodes within the XML document, where individual nodes represent the messages you wanted to bundle. |
|
| Back to top |
|
|
| Vin |
| Posted: Thu Oct 24, 2002 9:17 am Post subject: |
|
|
Master
Joined: 25 Mar 2002
Posts: 212
Location: India
|
| I agree that we need to keep it simple but when the message header becomes a significant part of the whole message and there are many messages which do not have a defined function except to be used by other messages then I guess we need to worry about things like bundling. Also the sending application is a mainframe. Is there something which we can do to have them send the messages more effeciently instead of one at a time? Thanks again. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Mon Oct 28, 2002 4:20 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
I suppose on paper it does seem silly to send a 300+ byte MQMD header for a message that is only say 10 bytes long. But is there really a problem sending messages like that? Is anyone raising red flags saying that this is going to be a problem. If so, why? Unless someone can give you a valid reason to avoid that, I say send 'em separatly.
Put it another way. If your 10 byte messages were 10,000 bytes, would anyone complain about seperate messages? If not, should anyone worry if you found a way to shrink your message by a factor of 100?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Vin |
| Posted: Mon Oct 28, 2002 4:53 pm Post subject: |
|
|
Master
Joined: 25 Mar 2002
Posts: 212
Location: India
|
Peter,
The only reason for not sending messages multiple messages is that sending more messages remotely takes more time to transmit them right? If we could bundle these small messages and send them across, would it not save time in the sense that only one message needs to be sent compared to ten? Your thoughts are appreciated. Thanks again. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Mon Oct 28, 2002 9:07 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
I have worked for various financial institutions over the last several years. Most of these companies have environments where millions of MQ messages move to/from some Unix platform to/from a mainframe per DAY (at one shop it was 10 million MQ messages per day). You are worrying an awful lot about a very insignificant amount of time.
The time difference between sending 1000 * 1KB messages or 1 * 1MB message is trivial.
Note: A poorly constructed SQL statement will waste far more time than the little bit of overhead to handle the MQMD.
later
Roger...
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| nimconsult |
| Posted: Mon Oct 28, 2002 11:31 pm Post subject: |
|
|
Master
Joined: 22 May 2002
Posts: 268
Location: NIMCONSULT - Belgium
|
Vin,
I also have had this discussion many times with other clients and they raised the same argument:
| Quote: |
| The only reason for not sending messages multiple messages is that sending more messages remotely takes more time to transmit them right? If we could bundle these small messages and send them across, would it not save time in the sense that only one message needs to be sent compared to ten? Your thoughts are appreciated. Thanks again. |
My answer is: do not implement this, because MQ already does it. Channels typically do not send messages one by one (assuming significant throughput of course), but "bundle" messages together by batches. You have control on the batch size, and on the time you are ready to wait to build up your batch (batch interval).
You will find that the highest overhead is in the TCP/IP conversation and not the small overhead of message size. MQ optimizes this overhead with batching (and even pipelining).
Of course, there is the overhead of MQMD size, but I agree with Peter and Roger that it is not significant.
Other arguments: how do you manage LUW, how do you manage poisoned sub-messages... I'm sure that there are many others...
Kind regards,
_________________
Nicolas Maréchal
Senior Architect - Partner
NIMCONSULT Software Architecture Services (Belgium)
http://www.nimconsult.be |
|
| Back to top |
|
|
| pomeroy8 |
| Posted: Sun Nov 17, 2002 12:03 pm Post subject: |
|
|
Novice
Joined: 25 Jun 2001
Posts: 21
Location: Markham, ON Canada
|
I read this thread with interest, since in fact, we just implemented an application that reads a file and bunldes the records together into large messages.
The amount of time the applcation ran was not significantly affected by the size of the message batch, so comments advising against this approach may be worth considering.
In our case, however, we were sending the messages to MQSI. In this case, the time the message flow ran was significantly affected by the size of the message batch. A 10x increase in the batch size resulted in only a 2x increase in the amount of time the message flow ran (by inspecting the timestamps on the output messages, so not very scientific). Furthermore, the relationship between batch size and message flow performance is not linear. Ultimately, if you continued to increase the batch size you would reach a point of zero/negative improvement. Lastly the records in the file were fixed length, so no need to added delimiters etc.
My point: you have to consider the target application as well and how it will be affected by batching records into messages.
Good luck |
|
| Back to top |
|
|
|
|
