| Author |
Message
|
| mattb |
 Posted: Wed Oct 17, 2007 8:51 am Post subject: Contention on queue ? |
 |
|
Newbie
Joined: 15 Oct 2007
Posts: 8
|
Platform MQ5.3 CSD 13 on Windows. C/C++ API.
We have ~2000 open input connections to a single queue each doing blocking MQGETs (servers looking for jobs)
At the same time we have 1..20 clients doing MQPUT (clients posting jobs)
Each server periodically (every 2 mins) opens an additional connection, does MQGET on same queue and closes connection.
We have experienced sporadic episodes of slow posting (eg 4 jobs/second compared with usual speed of 40 jobs/sec). Job size 500k - 1MB.
Has anyone out there got a similar setup and experienced similar problems ?
Could there be contention on the queue ?
How expensive is opening/closing connections and queues vs keeping them open (in terms of load on the queue manager)
Many thanks for your help
Matt |
|
| Back to top |
|
 |
| Bill.Matthews |
| Posted: Wed Oct 17, 2007 9:12 am Post subject: |
|
|
Master
Joined: 23 Sep 2003
Posts: 232
Location: IBM (Retired)
|
While I cannot quote you a number - if you look at this logically - it has to be much more unnecessary work - when a Get with Wait would accomplish the same thing. A\of course, the server apps should also include the "fail if quiseing" option so that the queue manager can be shut down gracefully - when needed.
Then, when the wait is over and there's no work to do, just repeat the request.
I am assuming that these are server based applications. I am also assuming that the applications are not doing other useful and necessary work in the two minute intervals.
_________________
Bill Matthews |
|
| Back to top |
|
|
| mattb |
| Posted: Thu Oct 18, 2007 4:05 am Post subject: |
|
|
Newbie
Joined: 15 Oct 2007
Posts: 8
|
Thanks for your reply Bill. I agree we could keep the connection open for the additional 2 min check rather than opening/closing it each time, however this increases the total number of open input connections.
I am concerned that we are stretching MQ by having so many open input connections to a single queue. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 18, 2007 4:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mattb wrote: |
| I am concerned that we are stretching MQ by having so many open input connections to a single queue. |
Making a connection is a resource-hungry activity. It's a much better use of resources to establish a connection and leave it open until all the work is complete than open it, use it, close it and then repeat the cycle.
Subject to the advise on fail if quiescing given earlier.
Also, slightly pedantically, connections are established with a queue manager, not a queue.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jsware |
| Posted: Thu Oct 18, 2007 5:01 am Post subject: Re: Contention on queue ? |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| mattb wrote: |
| Each server periodically (every 2 mins) opens an additional connection, does MQGET on same queue and closes connection. |
In trying to understand your design - why is a process that is already doing a get with wait on a queue opening the queue again, doing a separate MQGET and then closing it?
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| bower5932 |
| Posted: Thu Oct 18, 2007 6:09 am Post subject: Re: Contention on queue ? |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| mattb wrote: |
Platform MQ5.3 CSD 13 on Windows. C/C++ API.
We have ~2000 open input connections to a single queue each doing blocking MQGETs (servers looking for jobs) |
2000 open input connections on a single queue on a Windows system also seems like a lot to me. Does this server really need this many connections to look for jobs? |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Oct 18, 2007 6:23 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| Can you really run 2000 programs at the same time (if they were not waiting)? If not then reduce the number to a value that could run concurrently. |
|
| Back to top |
|
|
| mattb |
| Posted: Thu Oct 18, 2007 6:36 am Post subject: |
|
|
Newbie
Joined: 15 Oct 2007
Posts: 8
|
Hi Scott
We have a concept of Home job and Away jobs. A server attempts to find Home jobs in preference to Away jobs.
If a Home job arrives while an Away job being processed, we pre-empt the Away job.
Each server's calc thread does something like:
loop
if GETMSG(MSGID='Home')
process Home job
else if GETMSG(MSGID='Away')
process Away job
endif
sleep
endloop
Each server has an additional thread which periodically (2 mins) does:
if processing an Away job
if GETMSG(MSGID='Home')
Abandon Away job - allowing server to pick up new Home job
endif
endif |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 18, 2007 6:40 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
When you say "MSGID" you don't mean MQ Msg Id do you? 
Why not use queue priorities to schedule the work?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mattb |
| Posted: Thu Oct 18, 2007 7:07 am Post subject: |
|
|
Newbie
Joined: 15 Oct 2007
Posts: 8
|
| Quote: |
| When you say "MSGID" you don't mean MQ Msg Id do you? |
Yes... (now you've got me worried)
I agree we should use queue priorities (next version...), but we still want to be able to pre-empt a running job if a HOME (future high priority) job arrives. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 18, 2007 7:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mattb wrote: |
| Quote: |
| When you say "MSGID" you don't mean MQ Msg Id do you? |
Yes... (now you've got me worried) |
There are a number of threads on the forum about why setting msg id, especially to a character value, is a bad idea.
| mattb wrote: |
| I agree we should use queue priorities (next version...), but we still want to be able to pre-empt a running job if a HOME (future high priority) job arrives. |
What would change? However it detects that the job is a HOME job now is modified to detect the higher priority job and it does what it does now.
Priorities also allows more granularity than HOME or AWAY.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bower5932 |
| Posted: Thu Oct 18, 2007 7:41 am Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| mattb wrote: |
Each server has an additional thread which periodically (2 mins) does:
if processing an Away job
if GETMSG(MSGID='Home')
Abandon Away job - allowing server to pick up new Home job
endif
endif |
So, how long do these jobs run? Does it make sense to abandon all the work that you've done on an Away to go pick up a Home? Why not just finish what you are doing and call an MQGET that is priority based? |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 18, 2007 7:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bower5932 wrote: |
| Does it make sense to abandon all the work that you've done on an Away to go pick up a Home? |
I was wondering that, but didn't want to get into a detailed disection of the requirements 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| tleichen |
| Posted: Thu Oct 18, 2007 9:30 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
| Vitor wrote: |
| ...There are a number of threads on the forum about why setting msg id, especially to a character value, is a bad idea. |
I concur wholeheartedly. Understanding how the queue manager uses msgid and correlid will serve you better in the future.
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| jsware |
| Posted: Fri Oct 19, 2007 4:08 am Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| mattb wrote: |
if GETMSG(MSGID='Home')
process Home job
else if GETMSG(MSGID='Away')
process Away job
endif |
How deep does the queue get? We experienced performance problems when we were caching large numbers of messages on a queue (containing reply-to details) and then matching on correlid when the response was returned to an intermediate process (which then got the cache message containing the original reply-to details, did its work and used the original reply-to details from the cache queue to send the reply back to the initial process).
If you are doing a "GETMSG(MSGID='Home')" and then a "GETMSG(MSGID='Away')" these will cause a scan of every message on the queue until a match is found. If there are no Home or Away jobs, then every message will be scanned (i.e. if no away jobs then GETMSG(MSGID='Away') will scan every message.
I speculate on whether this queue scanning is causing MQPUTs to pause until the scan is complete (somewhere deep inside MQ internals I am guessing).
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
|
|
