ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportLarge queue sizes

Post new topicReply to topic
Large queue sizes View previous topic :: View next topic
Author Message
Gideon
PostPosted: Thu Apr 05, 2012 10:35 am Post subject: Large queue sizes Reply with quote

Chevalier

Joined: 18 Aug 2009
Posts: 403

I have a large non-persistent queue that spilled into disk

Code:
   find /var -size +200000c -ls| awk ' { print $7,$NF } ' | sort -rn | more

   515163136 /var/mqm/qmgrs/BBB/queues/INPUT/q


Is it possible to shrink the physical size of the queue after the messages have been drained by an application

If so how would I do it
Back to top
View user's profile Send private message Send e-mail
PeterPotkay
PostPosted: Thu Apr 05, 2012 10:45 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7535

Add one dummy message to the queue, then issue the clear q command.

Or restart the QM.

Or wait and eventually the QM will shrink the size when its good and ready.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Apr 05, 2012 10:48 am Post subject: Re: Large queue sizes Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25668
Location: Ohio, USA

Gideon wrote:
Is it possible to shrink the physical size of the queue after the messages have been drained by an application


Yes.

Gideon wrote:
If so how would I do it


Grab a cup of coffee & wait. After a non-configurable interval, the queue manager will reclaim the space.

If this queue is filling so much disk it's causing a problem, you need to buy more disk.

Seriously. If you're that close to the edge a slightly larger number of non-persistent messages spilling out could run the queue out of space & cause a problem. Likewise, if there was some kind of queue compressing utility how would you determine when to run it? Squeezing the file at the wrong time could seriously degrade performance, which is why currently the queue manager picks its moment.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Apr 05, 2012 10:50 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25668
Location: Ohio, USA

PeterPotkay wrote:
Add one dummy message to the queue, then issue the clear q command.


Would this actually shrink the file size? I always thought that it just emptied the queue.

Things you learn.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Apr 05, 2012 10:52 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25668
Location: Ohio, USA

PeterPotkay wrote:
Or restart the QM.




I didn't mention it as typically uptime is more important than file size. But this would do it.

Or possibly I'm just to idle to go to the trouble of a stop/start when all I need to do is sit back...
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Gideon
PostPosted: Thu Apr 05, 2012 12:44 pm Post subject: Reply with quote

Chevalier

Joined: 18 Aug 2009
Posts: 403

The restart is a good iea

Would that shrink the queue file size of an empty persistent queue as well (restarting the qmgr) ?
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Fri Apr 06, 2012 4:14 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25668
Location: Ohio, USA

Gideon wrote:
Would that shrink the queue file size of an empty persistent queue as well (restarting the qmgr) ?


For the upteenth time on this forum:

There is not such thing as a persistent queue!

There is a queue where the default persistence is persistent, but persistence is a message attribute. On any queue except a temporary dynamic one you can freely mix persistent & non-persistent messages.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri Apr 06, 2012 5:58 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7535

Yeah, that Clear Q command will shrink the file size, but only if there is at least one message for it to clear. Issuing it against an empty queue doesn't force the file shrinkage. At least it didn't when I last tested this on MQ 6.

Th eproblem with waiting for the QM to do it on its own time is sometimes you don't have that luxury. For whatever reason the q loaded up with a ton of messages pushed you dangerously close to a disk full condition. The queue has now been drained by the app. Yet you are stuck with an almost full file system. You can't restart the QM. You can't clear the queue. You can't wait. Sure wish we had an admin command to tell the QM to go clean up.

I stocked my QM dirs with 4 or 5 dummy files, each about 500 MB is size, in a folder called DummyFillerFiles, with a read me that explained one or more of these files could be safely deleted in a space saving emergency, just please put them back once the real space hog (Piiiigs...Innnn....Spaaaace!) was identified and cleaned up.



So here's a question I've always wanted to debate. When you size all your queues's Max Message Size x Max Queue Depth, do you size them all assuming one day you will have the worst case scenario where all queues fill up concurrently? That odds of that happening are slim, and it would seem you would have to plan for a ton of space that you would likely never come even close to using. Well, I gues depending on the app topology it might be more than a slim change. One QM, one hundred queues, 100 sending apps, one receiving app. If that one receiving app dies, its possible all the queues could eventually fill up.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Apr 06, 2012 6:53 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25668
Location: Ohio, USA

PeterPotkay wrote:
Sure wish we had an admin command to tell the QM to go clean up.


Yes, but it would be as abused as the REFRESH CLUSTER command.

PeterPotkay wrote:
(Piiiigs...Innnn....Spaaaace!)




PeterPotkay wrote:
So here's a question I've always wanted to debate. When you size all your queues's Max Message Size x Max Queue Depth, do you size them all assuming one day you will have the worst case scenario where all queues fill up concurrently?


Yes, balanced with the business impact of a given queue filling. Also max out xmitqs & DLQs. Disc tends to be the cheapest of the sizing resources.

Another aspect is how the site views this. It's within my experience that the queues are sized on "how long it takes us to notice a problem and shut down the sending systems".
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Fri Apr 06, 2012 7:43 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8379
Location: US: west coast, almost. Otherwise, enroute.

Or, this can be viewed as a business process rule: if you are processing payroll time cards, one per employee each payroll period, then max depth should allow for each employee to submit one time card. Reality sets in here, as sometimes it takes two or three tries to get it right.

So, for example: 1000 employees x 3 tries each = maxdepth 3000 for the time card queue.

Max msg length, like max depth, comes from developers. In the case of message length, take what the developers tell you (100 bytes each message, then multiply by some fudge-factor - like 100%. Set maxmsglength to 200. This will allow for message length expansion over time.

There is no logical or practical reason to set max depth to 999,999,999; or to set max msg length to 100MB. This would just entice app developers to create programs that include infinite loops that fill queues and filesystems.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportLarge queue sizes
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.