| Author |
Message
|
| sebastia |
 Posted: Tue Oct 09, 2007 9:09 am Post subject: 1 queue with 3.000 messages or 3.000 queues with 1 msg each |
 |
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Our design sends 3.000 large (4 MB) messages,
one for each destination, selected (today) by MSG-ID.
The message is read asynchronously by MQ Client programs.
Question : iIs it better to have :
a) 3.000 messages in a queue, and let the clients select their own (by msg-id)
or
b) 3.000 queues, each having 1 or 2 (maybe 3, max) messages, but
assigned by name to the client.
Another way to ask the question :
DOES ANYBODY SEE ANY PROBLEM ON DEFINING 3.000 QUEUES ?
( How many disk does this require ? )
Seb. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Tue Oct 09, 2007 9:16 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Use dynamic queues.
Each client will have it's own queue.
This will potentially require the same amount of disk space as 3,000 queues, but not all the time.
Q files will not generally be significantly larger than the messages that they hold.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Nigelg |
| Posted: Tue Oct 09, 2007 12:56 pm Post subject: |
|
|
Grand Master
Joined: 02 Aug 2004
Posts: 1046
|
The performance of selecting a msg by MsgId (non zOS systems) will degrade as the number of msgs in a queue increases. 3000 should not be too bad, but the time taken will become more noticeable as the queue depth increases.
_________________
MQSeries.net helps those who help themselves.. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 09, 2007 2:19 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And don't forget that if not using the Provider native twist(ID:...) a JMS selector will kill your performance even at such a low volume... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| JuulV |
| Posted: Wed Oct 10, 2007 1:42 am Post subject: |
|
|
Apprentice
Joined: 05 Sep 2007
Posts: 28
Location: Belgium
|
One thing that may have to be considered here is the performance hit of openening/closing multiple queues repeatedly at the sender side, versus the gain at the receiver side when it has to handle less messages to find the applicable ones...
If your choice for 1 or for 3000 queues doesn't (significantly) change the number of times queues have to opened and closed, I don't think that the performance will change significantly (nor the diskspace required; as already said before, a queue file normally isn't using much more diskspace than needed to store the current messages)... so in this case I would choose what's easiest to implement...
But, I've done some testing (on Win/XP SP2 with WMQ 6.0), and at least in this environment MQOPEN/MQCLOSE is extremely expensive: in one test case, I had an application write 100,000 (non-persistent) messages as fast as it coud to one of 30 output queues selected at random. When these 30 queues were only openened/closed only once, this test took about 100 seconds elapse time, and the application consumed less than 1 second CPU time. When I opened/closed the queue before/after every PUT, the elapse time went up to about 3000 seconds, and CPU time to 5 seconds!!!
_________________
Juul Vanparijs
Senior Developer
Cressida Technology Ltd |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 10, 2007 2:06 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Dynamic queues are good for request/reply, not sure how you would know what the queue name to use was on a one-way design?
We have 3000+ clients sharing queues for request/reply processing on z/OS and performance is OK (with indexed by correlid set). The queue depth rarely builds up because messages are being removed as fast as they arrive anyway.
3000 clients will not mean a depth of 3000 unless all the messages arrive before any of the clients attempt to remove them.
You don't say whether the clients are waiting all the time or connect only once, or connect at intervals.
If sending from the same system, MQ optimises the code path for waiting MQGETs by "joining up" the MQPUT and MQGET and avoiding the queueing entirely. |
|
| Back to top |
|
|
| sebastia |
| Posted: Wed Oct 10, 2007 3:41 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Thanks for all the opinions !!!
We think we will go for 3.000 static queues and I will tell you why.
The MQ Clients are PC's spread all around the world, one in each site.
The messages we sent to them (once a day) is specific for every destination, and has to be there, in the queue, dont know how many time.
( my question is : how long does a dynamic queue exist ?
who decides it is not required anymore ?)
The client opens the Qmgr and the Q once every hour and looks if there is a message there. If there is, it removes it. If not, nothing to do.
I would like to understand a little bit more any advantadge a DYNAMIC queue does provide, and will try to read THE BOOKS this weekend ...
Seb. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Oct 10, 2007 3:55 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sebastia wrote: |
( my question is : how long does a dynamic queue exist ?
who decides it is not required anymore ?)
|
A temporary dynamic queue is deleted when the connection handle is closed or goes out of scope (typically because the programme's ended).
A permanent dynamic queue must be specifically deleted, ideally by the app that created it or some luckless administrator cleaning up.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 10, 2007 6:32 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
If the clients used MQGET with WAIT, then they would receive the message as soon as it arrived, rather than delaying for up to one hour.
Also this immediate processing means the queue depth would never get that high, so they can all use the same queue without issues.
Admin of 3000 queues for one app is a nightmare, that's about as many as we have for all of our applications put together on a mainframe!
Maybe you could compromise and use regional queues (about 10 would seem ideal). But why not try one queue and see how it goes (there may not be a problem to solve here at all). |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Oct 10, 2007 8:25 am Post subject: |
|
|
Guest
|
As you've learned from this post, the answer to your question is: it depends.
It depends on lots of variables, including (but not limited to) the current and anticipated application usage, how the application was coded (get with wait, use of MQPUT1, database accesses, for examples). We should not write applications around a hardware configuration, because it will change.
A client (not a program) asked a similar question. Which is better, having every app dynamically create a reply-to-queue (enduring the cpu cycles for each mqopen and mqclose); or having one all-purpose-reply-to-queue that all applications use.
The all-purpose-reply-to-queue would dramatically reduce the number of queues created real-time. But applications would need to search this queue to find their replies. (The more applications using this queue, the more erratic the response time.)
Beginning analysis with impact on hardware configuration will inevitably lead to architecting applications to satisfy physical resources FIRST, and business requirments SECOND.
Since servers have a really short life-span (canaries and hampsters live longer), focus on the business requirements FIRST. |
|
| Back to top |
|
|
| sebastia |
| Posted: Wed Oct 10, 2007 3:05 pm Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Can I ask for a bit more of light, please ?
Vitor : you say
"Temporary dynamic queue is deleted when the connection handle is closed"
In our environment, on the MQ Server, the source of messages creates the message and places it in the queue -> (a)
The message is read by a MQ Client, the destination of the data, some time later -> (b)
Which "connection handle" are you talking about ?
"a" or "b" ?
If it is "a", we made it clear : we can NOT use dynamic queues.
Because the source of messages places them in the queue and the disconnects from the queue manager.
Is it this way ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 10, 2007 3:16 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sebastia wrote: |
Can I ask for a bit more of light, please ?
Vitor : you say
"Temporary dynamic queue is deleted when the connection handle is closed"
In our environment, on the MQ Server, the source of messages creates the message and places it in the queue -> (a)
The message is read by a MQ Client, the destination of the data, some time later -> (b)
Which "connection handle" are you talking about ?
"a" or "b" ?
If it is "a", we made it clear : we can NOT use dynamic queues.
Because the source of messages places them in the queue and the disconnects from the queue manager.
Is it this way ? |
I believe Vitor was referring to the typical request reply model where the queue is dynamically created by the requester who then listens on it for a reply...
As your model is clearly different you may well need 3000 queues. You could use the same model (dynamic reply to queue) with a server application waiting for a request message from the client and serving up the messages from a DB. Be aware however that a dynamic replyto queue will not accept persistent messages. For this you need a permanent queue. You could create it out of a model queue and delete it when you are done reading the messages... You should request for an end of messages notification being sent by the server program before deleting that queue. The server would then wait for a new request with a reply to queue before serving other messages to that destination...
Theese all define a polling mechanism with clients that are initiating the connection. In other words you have a message store, messages to deliver and no way of knowing when you can deliver them as the clients will come calling for their messages...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sebastia |
| Posted: Wed Oct 10, 2007 3:40 pm Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Thanks, mr Saper - this is the environment I do understand for the dynamic queues = where you wait for a reply. And you keep Connected ...
Will use 3.000 static queues then !
I shall read the books for the DYMANIC queues and the PERSISTENT messages - I did NOT notice that detail ....
Thanks !!! |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Oct 10, 2007 11:31 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I disagree that many clients on one queue causes erratic response times.
We have 3350 clients using one reply queue and the queue depth is ZERO.
Because they are using MQGET with WAIT. Messages are removed as soon as they arrive.
If clients are "polling" (I think this is almost always a bad thing) then the polling interval can be tuned for best effect.
I would rather have one queue to worry about than 3000. |
|
| Back to top |
|
|
| sebastia |
| Posted: Thu Oct 11, 2007 3:15 am Post subject: |
|
|
Grand Master
Joined: 07 Oct 2004
Posts: 1003
|
Yes, mr Zpat, I agree that administering so many queues can be ... complicated ...
I want to sugest a "mixed" solution :
We have 1 physical queue and 3.000 alias for it.
================================
The clients TODAY connect to the same queue (using "alias")
and maybe in the future we can provide physical queue(s)
for all or few of the Clients, if we decide so.
And this can be done in steps - we shall have 3.000 clients,
but can "migrate" from the "lonely queue" to the "nightmare 3k queues"
in steps ...
... or GO BACK !!!
I'd say it is quite an elegant solution, don't you agree ?
( ) |
|
| Back to top |
|
|
|
|
