| Author |
Message
|
| x061294 |
 Posted: Fri Feb 24, 2006 5:41 am Post subject: Get order for multi threaded app's |
 |
|
Acolyte
Joined: 05 Apr 2005
Posts: 62
|
I'd like to know if someone know's the order MQGET's are satisfied with multi threaded applications, running as client and local app's.
The situation is that I have two multi threaded applications, one running on a Windows box that has a 5.3 queue manager on it, and one running as a client to that box. Both open up 5 threads, doing connections and MQGET with wait against a queue. So now there are 10 open handles to the messages on that queue. For the sake of argument, let's say the server app started up first, then the client app.
My belief is that MQ would "queue" up the get requests, so the first 5 get handles would be for the server, and the next 5 would be for the client app. Therefore, if a message would arrive on that queue, the first get request in the "queue" would be handed the message and go off and do it's processing. When it is done it would issue another MQGET, and would be added to the end of the queue of get's waiting to be fulfilled. Following this logic, if messages arrived in a drip fashion and all backend processing finished before the next message arrived, the processing would be handled 5 by the server app, 5 by the client app, 5 by the server app, ...
However, based on what we're seeing when running tests, that isn't what is happening. So I thought I'd ask and see if someone know's the way that these get requests are serviced.
What we're encountering is that the local application (when running full out, not slowly in this drip fashion as I used as the example) is getting 90% of the messages while the client app is getting only 10%. Or even 95/5. Now obviously network latency comes into play as to how many messages the local app can process in the same amount of time that the client app can, but since the backend processing of the messages can take several minutes before finishing and coming back for another message, the latency issue is not in play here.
So what we appear to be seeing is that if there are outstanding MQGET's against a queue, for client and local, the queue manager is giving preference to the local requests. Which goes against how I thought MQ would "queue" up the MQGET requests. Which makes our load balancing not work out to 50/50, which is the net result of what we're trying to do.
Hence my qeuestion, does anyone know how the queue manager makes it's determination as to which MQGET to hand the message off to? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Feb 24, 2006 6:04 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
What happens if you start the client app before you start the server app?
I don't think you have a lot of control over this in general. There's nothing documented that I know of on how this is handled , so it works however it works at the discretion of the MQ development team.
In general, the first handle that is available will get the message. The question is "which one is 'first'?" And I suspect that the answer to this doesn't in any way depend on whether or not there are ten threads from one application, five threads from two, or one thread from ten applications.
_________________
I am *not* the model of the modern major general.
Last edited by jefflowrey on Fri Feb 24, 2006 6:51 am; edited 1 time in total |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Feb 24, 2006 6:06 am Post subject: Re: Get order for multi threaded app's |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| x061294 wrote: |
My belief is that MQ would "queue" up the get requests, so the first 5 get handles would be for the server, and the next 5 would be for the client app. Therefore, if a message would arrive on that queue, the first get request in the "queue" would be handed the message and go off and do it's processing. When it is done it would issue another MQGET, and would be added to the end of the queue of get's waiting to be fulfilled. Following this logic, if messages arrived in a drip fashion and all backend processing finished before the next message arrived, the processing would be handled 5 by the server app, 5 by the client app, 5 by the server app, ... |
You should not make this assumption. There is no documentation that supports it.
In other words, MQ is working as designed.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Feb 24, 2006 6:27 am Post subject: Re: Get order for multi threaded app's |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| x061294 wrote: |
| I'd like to know if someone know's the order MQGET's are satisfied with multi threaded applications, running as client and local app's. |
API calls get serviced in the same order in which they arrive at the queue manager. |
|
| Back to top |
|
|
| markt |
| Posted: Fri Feb 24, 2006 6:43 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
>> API calls get serviced in the same order in which they arrive at the queue manager.
You CANNOT make such an assumption, and it is clearly untrue in a number of cases. There are NO guarantees about which waiting MQGET retrieves the message (other than the specific vs non-specific MQGET(WAIT) behaviour).
The product documentation is explicit - rather than being quiet on the question it states that the behaviour is undefined. So you should never build anything that relies on, for example, an even distribution. |
|
| Back to top |
|
|
| wschutz |
| Posted: Fri Feb 24, 2006 6:55 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Can you tell us why you want this behavior? It seems odd to worry about evenly distributing workload between a server and client bound set of applications .... perhaps there is a solution that uses documented features of MQ. (clustering?)
_________________
-wayne |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Feb 24, 2006 9:25 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| x061294 |
| Posted: Fri Feb 24, 2006 11:52 am Post subject: |
|
|
Acolyte
Joined: 05 Apr 2005
Posts: 62
|
What we had was a local based applicaton that was running into capacity issues, so, we thought we'd add another server to help process some of the load. However, when adding this server into the mix, it only took 5% of the load off the server, the rest stayed on the main server.
We certainly could introduce another server, make that the queue manager server, and client both the app's into it. And that might make the distribution more equal. But without knowing the reason for not servicing the MQGET's equally, we didn't know if that would solve the problem.
I'm not worried about exactly 50/50, and depending on network and other factors I certainly wouldn't have been surprised to see a 60/40, or worst case 70/30. But I certainly didn't expect 95/5.
As for
| Quote: |
You CANNOT make such an assumption, and it is clearly untrue in a number of cases. There are NO guarantees about which waiting MQGET retrieves the message (other than the specific vs non-specific MQGET(WAIT) behaviour).
The product documentation is explicit - rather than being quiet on the question it states that the behaviour is undefined. So you should never build anything that relies on, for example, an even distribution. |
I guess you learn something new every day. I wasn't building directly on that thought, it just seemed reasonable.
So, is anyone distibuting, through a single queue manager/queue, load across multiple app servers in a pretty equal level?
Thanks. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Feb 24, 2006 11:57 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| x061294 wrote: |
So, is anyone distibuting, through a single queue manager/queue, load across multiple app servers in a pretty equal level?
|
We have a few apps that have 3 MQ Client threads pulling from a local queue. Anytime we bothered to check to see what the distribution was, it fluctuated every day, but was never 33/33/33. The one thing that we could count on was that if there was a thread with an outstanding Get with wait, and a message arrived, it would get picked up instantly.
Since the queue depth is staying at zero, we are happy, even though on any one day any one of the three threads may have processed significantly more messages than its brothers.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Feb 24, 2006 12:00 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If you increase the number of threads on the client app, you will increase the likelyhood that it will process more work. Similarly with decreasing the threads on the server app.
Your best bet may actually be to modify the MIRRORQ api exit to do loadbalancing for you between two queues, and point the server at one and the client at the other.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wschutz |
| Posted: Fri Feb 24, 2006 12:12 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
| What we're encountering is that the local application (when running full out, not slowly in this drip fashion as I used as the example) is getting 90% of the messages while the client app is getting only 10%. Or even 95/5. |
But isn't this saying that you can handle the workload alone with the server application? If message were coming in faster than the server applications threads could handle, you'd see the client application getting more of the messages. (Because then, when the message come into the queue, none of the server threads would likely have an outstanding mqget where-as the client application would.)
_________________
-wayne |
|
| Back to top |
|
|
| x061294 |
| Posted: Fri Feb 24, 2006 12:19 pm Post subject: |
|
|
Acolyte
Joined: 05 Apr 2005
Posts: 62
|
There is concern by having one server running at 80 - 90% utilization while the other is at 5 - 10% utilization. Technically there is room on the first server to continue processing, and we could take the approach that wait until that server goes 100% before processing is failing over to the second server, but that just worries me, and others.
I can take a look at the api and see about that balancing. Again, logically it just made sense to me that a MQGET thread is an MQGET thread is an MQGET thread, regardless of client versus local. But it doesn't appear that that is the case. I'm sure that there are good reasons for that, but, then I would think that there should be a way to cause this result, and I don't know what that would be. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Feb 24, 2006 12:23 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
If you want to change the distribution of work in your situation, then change the number of threads on the server and change the number of threads on the client.
As has already been said, there isn't a way to control this in MQ - because that's the way it's been written.
You can raise a feature request to have this added to a future release of the product - but that won't help you in the short term.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wschutz |
| Posted: Fri Feb 24, 2006 12:24 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
Well, you could put a 2nd qmgr on the "client" machine and then just use normal MQ clustering to distribute the messages round-robin to the queues (even, as Peter says, it might no be "perfect", but it sounds like you don't need perfect distribution).
_________________
-wayne |
|
| Back to top |
|
|
| x061294 |
| Posted: Fri Feb 24, 2006 12:27 pm Post subject: |
|
|
Acolyte
Joined: 05 Apr 2005
Posts: 62
|
|
| Back to top |
|
|
|
|
