| Author |
Message
|
| Cogito-Ergo-Sum |
 Posted: Wed Feb 02, 2011 8:54 pm Post subject: Multiple execution groups or multiple MQInput ndoes ? |
 |
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
We have a message flow that starts with a MQInput node that is populated by an upstream interface. The very next node is a Compute node. There are multiple such upstream interfaces who populate a MQ queue each. The business processing logic is same for all upstream interfaces. Therefore, my question is what parameters should I consider before deciding on one of the following approaches.
One execution group, one message flow, with multiple MQInput nodes ? As per the documentation, each input node will have its own transaction and thus multiple input nodes wouldn't 'interfere' with each other.
Multiple execution groups, one message flow in each execution group, each message flow catering to one particular MQ queue.
The answer is likely 'It depends'  But, a few pointers on this would be most help.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
 |
| zpat |
| Posted: Wed Feb 02, 2011 10:55 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You have missed the option of setting multiple instance value for a flow.
I would keep the single mqinput node. Then you have complete deploy and runtime control of how many copies run. using any mixture of multiple instances and multiple execution groups. WMB is very fast though so it's unlikely you will need very many.
Multi-threading only improves throughput if you have multiple CPUs and/or the flow waits at any time (e.g. for I/O to complete). Otherwise a single thread can use all of a particular CPU.
You might want to leave some CPU for other workloads anyway, or you can consider WLM to manage this. |
|
| Back to top |
|
|
| Cogito-Ergo-Sum |
| Posted: Wed Feb 02, 2011 11:30 pm Post subject: |
|
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
| Quote: |
| You have missed the option of setting multiple instance value for a flow. |
I did consider that option. But, the processing order of messages is very important and therefore I held back.
Documentation excerpt:
| Quote: |
Additional Instances
Specifies the number of additional threads that the broker can use to service the message flow. These additional threads are created only if there are sufficient input messages. You can use up to 256 threads. The default value is 0. Additional threads can increase the throughput of a message flow but you must consider the potential effect on message order. |
The italicised text is applicable when there is a single input node only. But, even when there are multiple input nodes, it can so happen that only one node has multiple threads.
Another excerpt from the same documentation link :
| Quote: |
| If you have multiple input nodes in your message flow, the available additional threads might not be allocated evenly between the different input nodes. In an extreme case, all the threads might be allocated to a single input node, and only one aspect of message flow throughput is improved. To avoid this problem, you can use the Additional Instances Pool property, together with the Additional Instances property, to allocate a pool of additional instance threads for each input node. |
Hence my concern about using Additional Instances - unless, there is another interpretation of the documentation excerpts above.
| Quote: |
| Then you have complete deploy and runtime control of how many copies run. using any mixture of multiple instances and multiple execution groups. |
Yes, that point is well taken.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
|
| fatherjack |
| Posted: Thu Feb 03, 2011 1:52 am Post subject: Re: Multiple execution groups or multiple MQInput ndoes ? |
|
|
Knight
Joined: 14 Apr 2010
Posts: 522
Location: Craggy Island
|
| Cogito-Ergo-Sum wrote: |
One execution group, one message flow, with multiple MQInput nodes ? As per the documentation, each input node will have its own transaction and thus multiple input nodes wouldn't 'interfere' with each other.
Multiple execution groups, one message flow in each execution group, each message flow catering to one particular MQ queue. |
Another option is multiple flows in a single execution group.
Take a look at supportpac IP04.
_________________
Never let the facts get in the way of a good theory. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Feb 03, 2011 5:00 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Cogito-Ergo-Sum wrote: |
| Quote: |
| You have missed the option of setting multiple instance value for a flow. |
I did consider that option. But, the processing order of messages is very important and therefore I held back. |
I don't understand the distinction, no matter how you execute the same flow more than once at the same time (whether in the same E.G. or not, whether multiple input nodes or not, whether using the additional instance setting or not) then you will face the same issue of sequencing. Unless perhaps you use the new sequencing node (whatever it's called) in V7. You need to understand this very important point.
There are ways to multi-task and still retain some order, for example if the order only needs to be maintained per customer - have a prior flow split the messages by the first letter of the customers last name to different queues and have only one flow executing per queue (does not have to be 26 queues, as you can assign a group of letters to a queue to give an approximate even spread).
I think the IBM documentation assumes that the multiple input nodes are referring to different input queues, and not to the same one. Having more than one input node identically configured creates exactly the same effect as having additional instances of the same flow.
Multiple EGs can exploit more than one CPU as I believe threads (the instances) under the same process (the EG) can only run on the same CPU. |
|
| Back to top |
|
|
| Cogito-Ergo-Sum |
| Posted: Thu Feb 03, 2011 7:48 am Post subject: |
|
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
| Quote: |
| I don't understand the distinction, no matter how you execute the same flow more than once at the same time (whether in the same E.G. or not, whether multiple input nodes or not, whether using the additional instance setting or not) then you will face the same issue of sequencing. Unless perhaps you use the new sequencing node (whatever it's called) in V7. |
That is true only when we have a single MQ queue serviced in multiple EG or multiple input nodes, additional instances set ?
In my case, I will be having different MQ queues populated by upstream interfaces. The processing logic, however, will not vary. And therefore, I found myself with two choices.
| Quote: |
Multiple EGs can exploit more than one CPU as I believe threads (the instances) under the same process (the EG) can only run on the same CPU. |
That is a good piece of information.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Feb 03, 2011 8:23 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I vaguely remember that there is some WMB CPU affinity option that can be controlled by an environment variable. I think the default is OFF so that multiple instances can dispatch on any CPU.
The IBM performance reports on WMB (by Tim Dunn) are worth reading (they are available as support pacs). |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 03, 2011 8:24 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| I would have multiple QALIASes that all point to the same physical queue that is read by the same flow, with multiple-instances across multiple EGs as needed to meet demand. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 03, 2011 4:49 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| zpat wrote: |
Multiple EGs can exploit more than one CPU as I believe threads (the instances) under the same process (the EG) can only run on the same CPU. |
I also believed this to be true.
Then a few weeks ago I saw one Execution Group taking 100% of each of the 4 CPUs in a test environment where the flow was looping.
RHEL 5
WMB 6.1.0.7
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Cogito-Ergo-Sum |
| Posted: Thu Feb 03, 2011 6:51 pm Post subject: |
|
|
Master
Joined: 07 Feb 2006
Posts: 293
Location: Bengaluru, India
|
| Quote: |
| I would have multiple QALIASes that all point to the same physical queue that is read by the same flow, with multiple-instances across multiple EGs as needed to meet demand. |
But, wouldn't that affect the message order ?
At any rate, in my case, I will be having different message queues each populated by a given upstream interface.
_________________
ALL opinions are welcome.
-----------------------------
Debugging tip:When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth.
---Sherlock Holmes |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Thu Feb 03, 2011 10:45 pm Post subject: Re: Multiple execution groups or multiple MQInput nodes |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| zpat wrote: |
| Multiple EGs can exploit more than one CPU as I believe threads (the instances) under the same process (the EG) can only run on the same CPU. |
That depends on the type of threads; kernel threads can exploit more than one CPU, while user threads (within a single process) cannot. A given application typically depends on the OS and/or user libraries for multi-threading support, so it's behavior can vary from one platform to another. |
|
| Back to top |
|
|
| AntonBricina |
| Posted: Fri Oct 14, 2016 1:35 am Post subject: Re: Multiple execution groups or multiple MQInput nodes |
|
|
Newbie
Joined: 29 Oct 2015
Posts: 6
|
| rekarm01 wrote: |
| kernel threads can exploit more than one CPU, while user threads (within a single process) cannot. |
| Quote: |
| kernel threads can exploit more than one CPU, while user threads (within one kernel thread) cannot |
Kernel threads = Additional Flow Instances ?!
User threads = Additonal (Input-)Node Instances ?!
BTW... I am mimicking "Additional (Input-)Node Instances" by copying the Input Node (20 Times), since the "HTTP-Input-Node" does not have the "Additional (Input Node) Instances" Propert (in my Version (IIB9))
Well there are two options:
| Quote: |
Select the pool from which additional instance threads for the input node are obtained.
If you select "Use Pool Associated with Message Flow", additional instances are obtained from the message flow pool.
If you select "Use Pool Associated with Node", additional instances are allocated from the additional instances of the node based on the component level number specified in the Additional instances property. |
_________________
Nothing beats a message which is delivered exactly once |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Oct 14, 2016 3:54 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I don't think it is documented what kind of thread a mesage flow uses, nor what kind of thread more than one input node uses.
I know you can't control the type of thread.
An execution group is a process. A message flow is a thread within a process. One of the input nodes in the message flow use the message flow thread, the rest use new threads. Likewise additional instances of a flow use new threads.
You should tune your thread pools to match *the number of simultaneous messages* that are arriving.
Or tune it for the number of messages that arrive in the time window that a single flow takes to process a single message. So if you have a flow that takes 1 second to complete, and you get 10 messages in that second, you should determine your response time, and create enough threads to handle enough of those messages to meet your response time. This is very likely less than 10- if the last message comes in at the end of the 1 second, it will get picked up by the original thread almost immediately.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Oct 14, 2016 5:16 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
I don't think it is documented what kind of thread a mesage flow uses, nor what kind of thread more than one input node uses.
I know you can't control the type of thread.
An execution group is a process. A message flow is a thread within a process. One of the input nodes in the message flow use the message flow thread, the rest use new threads. Likewise additional instances of a flow use new threads.
You should tune your thread pools to match *the number of simultaneous messages* that are arriving.
Or tune it for the number of messages that arrive in the time window that a single flow takes to process a single message. So if you have a flow that takes 1 second to complete, and you get 10 messages in that second, you should determine your response time, and create enough threads to handle enough of those messages to meet your response time. This is very likely less than 10- if the last message comes in at the end of the 1 second, it will get picked up by the original thread almost immediately. |
Don't forget that all this good stuff goes right out the door when 'Message Affinity' is needed.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| timber |
| Posted: Fri Oct 14, 2016 5:37 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Am I the only one who finds it slightly unusual to revive a 3-year-old thread without providing any reason/background? |
|
| Back to top |
|
|
|
|
