ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » 80% Queue Depth

Post new topic  Reply to topic Goto page 1, 2  Next
 80% Queue Depth « View previous topic :: View next topic » 
Author Message
objectthinker06
PostPosted: Mon Jul 28, 2008 10:22 am    Post subject: 80% Queue Depth Reply with quote

Novice

Joined: 15 Aug 2006
Posts: 15

Is it a good practice to have a queue's depth always 80% (i.e. approx. 50K messages) of its maximum depth? What MQ system level implications would there be under such a situation ? assuming that the client application using this queue connects to the Queue Manager using a sever connection channel.

Thank you,
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Jul 28, 2008 10:54 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

Max message length and max queue depth effectively constrains the amount of disk space an abusive program can use.

These are application design issues. Your application architects will dictate both of these queue attributes.

A queue should have maxdepth set to accomodate the maximum number of messages that the application demands. For example, a payroll application that pays 1000 employees requires a timecard queue. So, maxdepth could be set at 1000.

But what if each employee incorrectly submits one timecard each - in which case 2000 would be a better maxdepth - or 3000.

There is automation that can increase maxdepth when a threshold queue depth is reached.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sami.stormrage
PostPosted: Mon Jul 28, 2008 10:58 am    Post subject: Reply with quote

Disciple

Joined: 25 Jun 2008
Posts: 186
Location: Bangalore/Singapore

Depends whether the Q is holding messages for Batch or real time messaging scenario. Its always bad to hold messages on Q's for a longer period of time.. it simply increases the overhead on the system.
_________________
*forgetting everything *
Back to top
View user's profile Send private message Yahoo Messenger
rxm8778
PostPosted: Mon Jul 28, 2008 10:59 am    Post subject: Reply with quote

Apprentice

Joined: 15 Sep 2005
Posts: 37

Messages sitting in a queue take up take up queue file space and Log file space (if persistent). From an operating system point of view, this transaltes to file system space. Ultimately, MQ is not designed to store message over long period of time. While the queuing capability exist and messages can stack up in a queue, you should design your application to consume them as soon as possible rather than letting messages stack up in the queue up to 80%. Start multiple message consumers (if you business application allows it) to prevent this type of build up. With that said, if you do have the file system storage to allow your queues to build up to 80% of their queue depth then nothing is stopping you. But just know that you will run into other undesired issues that could cause slow down (especially if the messages are persistent)... i.e iowrite..ect...

The best performing queues are nearly empty queues so it is always best to consume messages as they arrive to prevent the type of build up you are refering to.

Hope this helps
Back to top
View user's profile Send private message AIM Address
ipmqadm
PostPosted: Mon Jul 28, 2008 11:31 am    Post subject: Reply with quote

Acolyte

Joined: 18 Apr 2007
Posts: 68

The function of message queueing is to allow data to pass from point A to point B. If you are experiencing constant data depths of 50% to say 80% of the max queue depth then you are either seeing a stressed application where the enqueue rate exceeds the dequeue rate or you are using messaging where you should possibly consider another approach.

You may want to look into the mq clustering option with some workload balancing to keep the queue depths at a minimum.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Jul 28, 2008 11:43 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

Quote:
If you are experiencing constant data depths of 50% to say 80% of the max queue depth then you are either seeing a stressed application where the enqueue rate exceeds the dequeue rate or you are using messaging where you should possibly consider another approach.

Or you are just seeing normal application behavior.

There is nothing good or bad about messages being in a queue, or a queueu being at some percentage of maxdepth. There is no additional overhead for having messages in a queue. These are just events.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sami.stormrage
PostPosted: Mon Jul 28, 2008 12:05 pm    Post subject: Reply with quote

Disciple

Joined: 25 Jun 2008
Posts: 186
Location: Bangalore/Singapore

Bruce: what do u intend to tell by that?

Its always a bad idea to have ur batch jobs to run on a parellel sysplex since it considerably increaces your useage of CF structures. On bad day u could get additional messages and end up crashing ur CF structures.

And on a AIX server the Qmgr can simply hang if a file system used by the Q's fills up because there are a number of such Q's with similar depths.

Please apply some thought before u reply back. I know might have a lot of aquired MQ knowledge but, people who have worked on large environments know what works and what not!
_________________
*forgetting everything *
Back to top
View user's profile Send private message Yahoo Messenger
bruce2359
PostPosted: Mon Jul 28, 2008 1:08 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

Quote:
Bruce: what do u intend to tell by that?

(feeling the hostility)

I'm suggesting: analyze, plan. Resources are finite.

Yes, messages take disk space. Most newer midrange servers have lots of disk space. If you don't, get more. If you can't, set some constraints.

The same general rules apply at the parallel sysplex level; but in addition to disk space, the coupling facility must be addressed.

The original post pondered if having queue depth at 50% (or any other %) of maxdepth represented any system-level issues. Yes, No, and It depends, are the answers here.

Quote:
Please apply some thought before u reply back. I know might have a lot of aquired MQ knowledge but, people who have worked on large environments know what works and what not!

Hmmmm.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Jul 28, 2008 1:55 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

Quote:
Please apply some thought before u reply back. I know might have a lot of aquired MQ knowledge but, people who have worked on large environments know what works and what not!

Sincerest appologies if you don't like my posts.

I would appreciate it if you would focus on factual content of the posts, and omit the personal attacks.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sami.stormrage
PostPosted: Mon Jul 28, 2008 1:57 pm    Post subject: Reply with quote

Disciple

Joined: 25 Jun 2008
Posts: 186
Location: Bangalore/Singapore

Ahhh.. u made me go through my first post again and even u wrote the same thing now.. But have a look at one of ur previous posts. Think u are contradicting something here.

.. And lets call it day.. got to go to sleeeeeeeeeeeeeeeepppppp
_________________
*forgetting everything *
Back to top
View user's profile Send private message Yahoo Messenger
kevinf2349
PostPosted: Mon Jul 28, 2008 6:29 pm    Post subject: Re: 80% Queue Depth Reply with quote

Grand Master

Joined: 28 Feb 2003
Posts: 1311
Location: USA

objectthinker06 wrote:
Is it a good practice to have a queue's depth always 80% (i.e. approx. 50K messages) of its maximum depth? What MQ system level implications would there be under such a situation ? assuming that the client application using this queue connects to the Queue Manager using a sever connection channel.

Thank you,


It really depends.

I think thast in order to give a beneficial answer we would need more details. For example is the queue used to fill up during the day and then emptied by an overnight process? Is the queue supposed to be a serviced during the day? What platforms are involved?

...and where do the coupling facility and parallel sysplex come into it?
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Jul 28, 2008 7:39 pm    Post subject: Re: 80% Queue Depth Reply with quote

Jedi Knight

Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia

objectthinker06 wrote:
Is it a good practice to have a queue's depth always 80% (i.e. approx. 50K messages) of its maximum depth? What MQ system level implications would there be under such a situation ? assuming that the client application using this queue connects to the Queue Manager using a sever connection channel.


As one poster said already "it depends".

In an idea world, the long term average of all queue depths should be zero. MQ provides store-and-forward asynchronous messaging, but it should not be used to hold application data for long periods (days...weeks).

If the receiving application cannot consume the messages at the rate at which they are being produced then you have an application problem or a system performance problem.

The maxdepth is an arbitrary limit. Its setting should be based on the maximum number of messages that would normally expected to be queued, and the length of those messages, and regard to the available disk space to store the messages.

A general rule of thumb is to set maxdepth to double the peak expected number of queue messages. This allows an 'abnormal' peak to be handled ok, but it will generate a 'queue full' condition when there is very abnormal application behaviour, so that the situation is called out before MQ totally runs out of disk space. However MQ monitoring of queue depth high events (you do that, don't you?) should pick up abnormal conditions before this occurs.
_________________
Glenn
Back to top
View user's profile Send private message
ipmqadm
PostPosted: Tue Jul 29, 2008 5:48 am    Post subject: Reply with quote

Acolyte

Joined: 18 Apr 2007
Posts: 68

While it may be true that there is nothing 'good or bad' about data residing in a local queue, what I was intending to state was that the presence of a relatively large amount of data remaining in a local queue over extended periods of time is indicative of either a stressed application or the use messaging when other data processing techniques would be a better logical solution.

I've spent over 15 years writing both batch and online MQ applications for numerous large corporations and we always ensured that the data flowed through the queues as quickly as possible. The best performing services/applications/jobs are those whose local queues maintain low queue depths.

As an administrator, I also monitor for high queue depths and now work with the business units to resolve those situations. So I just think that it is best to try to design the MQ architecture to keep all local queue depths as low as possible...
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Jul 29, 2008 6:28 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

Quote:
What MQ system level implications would there be under such a situation ?

As I read the original post, I noted that objectthinker06 asked about system-level implications. System-level seemed to me to be all other than application-level; leaving the o/s, processor hardware, disk, network, and such.

I responded with this limited definition in mind. The presence or absence of messages in a queue, or a queue at 50% of maxdepth, has little/no impact on the o/s, processors, network. I further assumed that filesystem was a given, that messages consume disk space.

I believe we're all in agreement that MQ is not the ideal place to store messages (data) long-term (hours, days, ...); but even this is application-dependant. A queue that stores messages until midnight processing doesn't violate any rules or have system-level implications. Batch work exists. It's a choice, perhaps a poor one, but a choice nonetheless.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Jul 29, 2008 6:09 pm    Post subject: Reply with quote

Jedi Knight

Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia

FYI, IBM Developerworks has a new article "The top 15 WebSphere MQ best practices" at http://www.ibm.com/developerworks/websphere/library/techarticles/0807_hsieh/0807_hsieh.html
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » 80% Queue Depth
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.