| Author |
Message
|
| andrewjhenderson |
 Posted: Mon Dec 15, 2008 4:17 am Post subject: thread starvation? - multiple input nodes |
 |
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
Hello , I have noticed an an announcement within a broker 6.1 What's new :
"Allow additional instances to be set on individual input nodes :Avoids thread starvation in multiple input node message flows."
Can I ask if anyone has seen this kind of behaviour and how it manifests itself in earlier versions of MB?
We have a situation currently in v6.0 where particular qs served by a message flow with multiple input nodes appear to hang on occasion. ( No addition instances set currently) . Other qs in the same flow are processed normally.
The broker execution group refuses to stop without mqsistop -i .
The messages are simple, they do not cause looping and resubmit normally after the broker restart. There are no application level errors
that we can see.
Could this be a thread starvation issue? How might I confirm it?
Any thoughts welcome. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Dec 15, 2008 4:27 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Thread starvation, in this case, is when one input node in a flow consumes all of the available threads in the pool - in your case if you haven't increased additional instances there is one thread per Input node.
If the flow is processing *any* new inputs, then this could be thread starvation.
If the flow is processing *no* new inputs, then it's a bug in your flow with the current message that's in flight.
Or someone attached a debugger, hit a break point in the flow, and then walked away. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Mon Dec 15, 2008 4:54 am Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
Hmm, so thread starvation does not mean that a q never gets processed?
Just that it might be processed infrequently ( disproportionately to the number of additional instances configured)?
Is that correct?
Andy |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 15, 2008 5:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
No.
It means that an input node may never get fired, if the workload on the other input nodes is sufficiently large. Imagine an MQInput that has a backlog of messages, where each message takes longer to process than the interval at which messages arrive.
But if no input is processing any messages at all, then this isn't thread starvation. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Mon Dec 15, 2008 6:08 am Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
Hi,
To be clear, other messages are being processed normally by input nodes within the same flow.
Given that all the messages process successfully once the broker is restarted, then there is no particular intrinsic problem with the messages themselves.
So I guess I repeat my original query, is there any way in particular to ascertain whether thread starvation might be the issue?
As we have no additional instances set, is there no chance of thread starvation?
I am looking for a clear definition of when thread starvation can occur in MB. Is it only a relative condition? ( e.g. processing slows down temporarily?) Or can it be absolute ? Processing appears to stop entirely?
If nobody knows offhand, I'll PMR it.
Thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Dec 15, 2008 7:01 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
I thought that Jeff was quite clear on the symptoms.
Even if you have no additional instances, but if you have multiple input nodes (and you do) imagine having just one input node processing while the other 2 are waiting for threads...being hogged by the processing node....
So what you might see is one queue processing and the 2 others getting a backlog...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Mon Dec 15, 2008 7:36 am Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
He may well have been quite clear, but the point is for me to understand, and I am not there yet.
So for example I have 1 message flow with 3 input nodes all assigned to read different queues.
No additional instances.
Are there 3 threads with 1 thread servicing each input node exclusively?
( I guess not or thread starvation would not occur?)
Or 3 threads servicing the 3 input nodes in some fashion, such that at least one of the input nodes will never be serviced? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 15, 2008 8:39 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The only real way to determine if you have thread starvation going on is to observe the pattern of what input nodes are processing and what input nodes are not processing.
If you have two input nodes, then one could consume both threads and leave the other hanging. This is relatively straight forward.
If you have *three* input nodes, then any individual input node could be starved for threads temporarily, while the other two nodes are using up the (three) available threads. And which node is starved could change at any interval.
Thread starvation can be entirely temporary, or entirely "permanent".
For example, if you wire up a JavaCompute node that executes a sleep wait indefinitely. Each input connected to that JCN will fire once, and the thread it created is then locked "forever". This is, in theory, thread starvation because each thread is in use processing a message, and there are no threads available for new input.
If you sit down and think about this for a few minutes, and maybe draw a diagram, it's really very straight forward. A message flow is given one thread plus one thread for each additional input node plus one thread for each additional instance. Those threads are, unless you use the new features in 6.1, shared by every input node on a first-come first-served basis. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Mon Dec 15, 2008 9:26 am Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
Thanks Jeff.
So basically there is a pool of threads serving ALL the input nodes whose size = number of MQInput nodes in the flow + additional instances, and the distribution of those threads serving particular queues at any one time is irrespective of the number of MQinput nodes with a particular q assigned to them.
e.g.
if I have 1 input node assigned to Q1 and 2 input nodes assigned to Q2, it makes no odds, Q1 could still get 3 threads and Q2 none at all.
Sorry if I am labouring this point somewhat, but I have not seen this clearly documented anywhere. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Mon Dec 15, 2008 9:39 am Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 15, 2008 10:35 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| andrewjhenderson wrote: |
| So basically there is a pool of threads serving ALL the input nodes whose size = number of MQInput nodes in the flow + additional instances, and the distribution of those threads serving particular queues at any one time is irrespective of the number of MQinput nodes with a particular q assigned to them. |
This is correct as far as I understand it, and has been true since version 2.1 of Broker at least.
This is exactly *why* the feature of a node-specific thread pool was introduced in version 6.1 - to allow a specific Input node to always have threads available even if the other Input nodes are processing at full speed.
For example, if you have a "control" queue in which you want to instruct the flow to stop processing - you don't want the input node reading that queue to be starved for threads because the flow is running at full speed.
I'm not clear why you would configure more than one MQInput node to read from the same queue, though. It seems like you'd never know which one would process a message - and adding additional instances is a better choice to increase processing speed. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Mon Dec 15, 2008 1:28 pm Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Dec 15, 2008 1:39 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
That should have been specifically addressed in one of the FPs of WMB v6, and should not exist in 6.1.
You may notice that page is two yeards old... " Modified date: 2006-06-07"
It's worth either the ten minutes it would take to confirm this by experimentation, or (even better from some respects) opening a PMR to get clarification and suggest that the technote gets updated based on the results of that clarification. |
|
| Back to top |
|
|
| andrewjhenderson |
| Posted: Tue Dec 16, 2008 7:26 am Post subject: |
|
|
Novice
Joined: 06 Sep 2007
Posts: 15
|
Here's an experiment.
Take a message flow, add an MQinput node assigned to read Q1.
Deploy it and look at open input count on MQ Q1. This shows 1.
Add 2 more MQInput nodes with the same properties assigned to read Q1 to the same flow.
Deploy it and look at open input count on MQ Q1. This shows 3.
Configure additional instances on the bar file to have say 7 additional instances.
Deploy it and look at open input count on MQ Q1. This shows 3.
Now put some load on the queue say 10k messages. Open input count on MQ Q1. This now shows 10 as additional instances kick in.
Go back to the flow and add 3 more MQInput nodes assigned to Q2 this time.
Set additional instances to zero on bar file.
Deploy look at open input count on MQ Q1 and MQ Q2, 3 each.
Now put 10000 messages on just Q1, leave Q2 empty.
How many threads process Q1? 3 or 6.
Remember the contention is that :
"So basically there is a pool of threads serving ALL the input nodes whose size = number of MQInput nodes in the flow + additional instances, and the distribution of those threads serving particular queues at any one time is irrespective of the number of MQinput nodes with a particular q assigned to them"
Well if we accept open input counts as a proxy for threads doing work against the queue manager, the answer is 3 not 6.
Therefore assigning more than 1 input node to a given queue does seem to allow you to control how many threads work against a given queue, irrespective of loads applied to other queues served by the same flow, provided you use MQInput nodes explicitly, and not additional instances.
So the statement in quotes above does not seem to be true.
Or is the number of open input handles on the queues misleading? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Dec 16, 2008 7:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Additional instances are created and quiesced on demand.
This is a *good* thing.
Using additional MQ input nodes means that you are keeping thread memory space in use and MQ connections in use that are sitting idle and consuming EG resources that are not available to any other flows in your EG. |
|
| Back to top |
|
|
|
|
