| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| RouteToLabel vs MQOutput |
« View previous topic :: View next topic » |
| Author |
Message
|
| felfel |
 Posted: Fri Nov 25, 2005 6:22 am Post subject: RouteToLabel vs MQOutput |
 |
|
Apprentice
Joined: 02 Nov 2004
Posts: 26
|
Hi ,
A customer had implemented an Input Flow which routes to many destinations queues.
For each queue , there is a flow that listens for message.
The size of messages is very important (until 30 Mo).
And of course , there is a performance problem.
After optimizing ESQL , is it a good way to change MQOutput nodes with a routeTLabel node with subflows ?
What can i wait in terms of perf by making this change ?
Thanks,
_________________
"Mens sana in corpore sano" |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Nov 25, 2005 6:30 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
It's hard to say how much performance gain you will get by using subflows instead of external flows.
There is overhead from having to rebuild the bitstream from the message tree that will be avoided, though.
But you will make things more complex and dependant this way.
If possible, you can look at using additional instances of the regular flows that you have now, or additional instances of the main flow.
As with all questions of performance, I think it is best to try and identify the major points of slowdown first, and address those. And then if things aren't fast enough, look for the next set of major slowdowns, and etc.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Tibor |
| Posted: Sat Nov 26, 2005 6:55 pm Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Jeff,
This way (MQOutput) may be better in performance when you use it in a right place. Just an example from my practice:
- MQInput getting an IDOC from SAP with domain BLOB
- a Compute node analyzing the header (substring + cast  ) then sending it to the destination queue(s)
felfel,
When a flow becomes bigger (with more and more nodes), the memory consuming is growing, because all nodes get a copy of the full message tree.
HTH,
Tibor |
|
| Back to top |
|
|
| hopsala |
| Posted: Sun Nov 27, 2005 12:23 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
I think, although am not sure without more info, that in your scenario MQOutput will be preferable, due to the following points:
- As jeff said "you will make things more complex and dependant" - the larger the flow, the harder to maintain it, the more chances it crashes, the more difficult logging and exception handling becomes etc.
- In addition, any upgrade requires shutdown of the entire master flow, whereas in an atomic solution you can shutdown only that small flow with relevant code.
- And mark that if you wish to add a whole new destination+logic, no need to edit the big-ol-scary flow, just to add a destination to the list of routing flow (which usually listens on the first queue) through a DB table or whatnot, build a queue, and build a new flow.
- If some day you wish to use WMQ clusters combined with several WBIs running, then it will be easy to load-balance the smaller logics across the network.
- If these destinations have to do with different applications or teams in your organization, you'll have a big pollitical problem on your hands - who's responsible for the flow's well being? who makes upgrades? If someone makes an upgrade how do you make sure he doesn't touch the code of the other team? As any WMQ admin knows full well, political problems = technical problems.
- Any problematic message will only go back a small step backwards, to originating queue, instead of backing out through the entire logic. Much easier to know how to deal with backout queues this way.
- Say you have three atomic logics A,B and C, each take about the same time to complete processing for one message, namely X. If they all reside in the same flow, it will take about 3X. What this means is that if suddenly 3 messages came in one after the other they will complete processing after 9X. In a small-flow solution, they will also take 9X, but you will see them hopping from queue to queue much faster.
This is a big improvement: First off, when a queue is emptier, it works faster. Second, every queue has an independant memory buffer. Third, it will be easier to monitor hangups - in the former solution you will have "simulated errors" upon sudden message bursts. Fourth... well, I think this point is clear.
- WMQ performance improvements may greatly increase your throughput, see Question on queue design for high volumne (Odd, I keep posting this lately)
- There used to be an excellent redbook regarding this, called "MQSI performance tuning" or something similar; I can't seem to be able to find it, maybe it got withdrawn. It's worth looking up, though.
- Most importantly, when performance queries are at hand, you must always ask yourself "it may be that this will decrease performance, but do I care?" - You only have so-and-so messages per min, if you are capable of sending thrice as many, you can, and should, sacrifice the superfluos throughput slack for the sake of maintenance, readability and all the aforementioned articles.
The list, of course, goes on... To know what is the performance degradation that ensues, you must as jeff said, simply try it; given a common scenario, I wouldn't think it should be more than a 20% degradation, but I'd try before I speak if I were you.  |
|
| Back to top |
|
|
| felfel |
| Posted: Mon Nov 28, 2005 4:01 am Post subject: |
|
|
Apprentice
Joined: 02 Nov 2004
Posts: 26
|
Thanks all for your replies.
I agree that the solution with many external flows offers more scalability.
But I am afraid that with larges messages persistent ( 30 Mo) , the I/O cost is very important.
At first , the customer had a Linux machine with only 2 CPU and 4 Go RAM.
Now , the solution runs on an AIX 4 CPU 16 Go RAM that will provide more power for processing message.
It's why I am looking for optimisation about I/O ( MQ and Database).
I'm going to test both and compare.
I will give you a backup about the gain of this change.
Thanks,
_________________
"Mens sana in corpore sano" |
|
| Back to top |
|
|
| mgk |
| Posted: Mon Nov 28, 2005 4:29 am Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
Also, FYI, Message Broker V6 allows you to PROPAGATE to a Label node directly from ESQL, avoiding the use of a RouteToLabel node and without having to set up anything in the LocalEnvironment.
Regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| KeeferG |
| Posted: Mon Nov 28, 2005 5:25 am Post subject: |
|
|
Master
Joined: 15 Oct 2004
Posts: 215
Location: Basingstoke, UK
|
Just a small point I wish to make.
It is not strictly true that a queue runs faster the emptier it is.
If the queue is empty and there are flows waiting to read from the queue then when a message arrives it will be given straight to the flow rather than being queued. This is known as queue avoidance and is essential in high throughput systems.
There are ways to check the queue avoidance but the general rule is to have many instances of a flow so that when a message arrives there will always be one flow waiting on the queue for the message.
_________________
Keith Guttridge
-----------------
Using MQ since 1995 |
|
| Back to top |
|
|
| hopsala |
| Posted: Mon Nov 28, 2005 6:45 am Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| KeeferG wrote: |
| If the queue is empty and there are flows waiting to read from the queue then when a message arrives it will be given straight to the flow rather than being queued. This is known as queue avoidance and is essential in high throughput systems. |
Um, what?
Maybe i'm missing something here, but for a message to "arrive" it has to be on queue; if you had intended that it does not reach queue file than you are correct, otherwise you are gravely mistaken...
Regardless, I fail to see how this negates what I said about empty queues; In fact, it even strengthens my point - if a queue is empty this "queue avoidance" (which I still think doesn't exist) occurs, thus n empty queue "runs" faster. Please explain. |
|
| Back to top |
|
|
| KeeferG |
| Posted: Mon Nov 28, 2005 7:22 am Post subject: |
|
|
Master
Joined: 15 Oct 2004
Posts: 215
Location: Basingstoke, UK
|
Perhaps it is the way I explained the scenario which didnt get the point across.
Forget about broker for the time being as this is a feature of MQ. Application A performs its put to the queue. The queue manager checks to see if there are any outstanding gets on that queue. If so the queue manager will pass the message directly to the awaiting get not touching the queue buffers at all and the avoide count goes up on the queue stats. If no gets are waiting the message is then placed on the queue as normal, in memory buffers first then to queue files. There are commands to check this however they are/were IBM internal use only whilst I was there. This may have changed since I left, however 6 is now out and internals may have changed.
As for the strengthening your point about empty queues then fine. Queues that have messages in them that have to be written to disk are the slowest due to extra IO, queues with a few messages in them run faster still as messages will be stored in the in memory buffers that we all love and use. A empty queue is the fastest of the lot as in reality the queue never gets used. I usually see around 30% throughput improvment when using queue avoidance techniques. It is mentioned in at least one performance report somewhere.
Hope this clears up a few things.
_________________
Keith Guttridge
-----------------
Using MQ since 1995 |
|
| Back to top |
|
|
| chrisc |
| Posted: Sun Jul 09, 2006 5:15 pm Post subject: |
|
|
Voyager
Joined: 19 Mar 2006
Posts: 77
|
Sorry for resurrecting such an old post, but I stumbled across this queue avoidance part and was quite intrigued...
Has it been confirmed by people that this is indeed a feature of MQ? Also, does this only apply to non-persistent messages, or does it do some fancy stuff to handle persistent messages as well?
I have done some searching on this site, with Google and on the IBM site and I can't find any reference to the term, so I was wondering whether there were still any IBM insiders (or others) with knowledge about this, and experience in how much performance increase it can bring.
Cheers,
Chris |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sun Jul 09, 2006 6:42 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I think the term "Queue avoidance" is not necessarily established industry jargon.
But, yes, particularly for non-persistent messages in syncpoint, MQ will do things like skip writing to the q file if there is a program that is eligible to receive the message.
For persistent messages, MQ has to do certain things always - like write to the log files, and take other measures to ensure that the message is actually never lost. So you don't necessarily get all of the performance optimizations that are possible with non-persistent messages.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| KeeferG |
| Posted: Mon Jul 10, 2006 1:35 am Post subject: |
|
|
Master
Joined: 15 Oct 2004
Posts: 215
Location: Basingstoke, UK
|
jefflowrey is correct in saying that it is not established industry jargon. It was a term used in at least one of the 5.3 for windows performance reports though cannot remember which one. The report just said about the concept and did not state improvements though as it is hard to measure how to measure the number of messages that avoid the queue without using IBM internal commands.
It is definately worth experimenting to see for yourself though
_________________
Keith Guttridge
-----------------
Using MQ since 1995 |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
