| Author |
Message
|
| dotaneli |
 Posted: Thu Feb 19, 2009 12:49 am Post subject: Automatic mechanism to "inject" expiry to existing |
 |
|
Voyager
Joined: 19 Oct 2005
Posts: 99
Location: Israel
|
Hi all.
A client of mine requested to have an automatic mechanism which deletets message older than one week.
Well - we used a message broker flow to do that..
We read a certain queue, setting the expiry to '7 days' and injecting to a new queue.
Problems are:
A. without initiating an MQBROWSE or MQGET, the message count is not updated even if the messages are already expired.
B. Even after issuing an MQBROWSE and messages are deleted, the file size does not shrink back.
We are talking about very big queue sizes (few gigs). As I understand, only "clear q" and restarting the qmgr shrinks the file size..
P.S. - messages on that queue are said to be all persistant. Queue is defined as non-persistent but the client's flow makes them persistent.
So - i want to know if there's a better solution to clearing old messages than a flow. and - if i do use this flow - how do i shrink the file size and update the message count without initiating external commands?
thanks,
Eli.
[edit]corrected spelling in title[/edit] |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Feb 19, 2009 1:34 am Post subject: Re: Automatic macansim to "inject" expiry to exist |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dotaneli wrote: |
| B. Even after issuing an MQBROWSE and messages are deleted, the file size does not shrink back. |
Not strictly true. The file will shrink after an undisclosed period if the queue manager decides (in an undisclosed way) it doesn't need all that space.
| dotaneli wrote: |
| We are talking about very big queue sizes (few gigs). As I understand, only "clear q" and restarting the qmgr shrinks the file size.. |
I agree that there's no command to trigger this process
| dotaneli wrote: |
| P.S. - messages on that queue are said to be all persistant. Queue is defined as non-persistent but the client's flow makes them persistent. |
Pedantically queues have no persistance attribute. Messages have a persistence attribute. The setting on the queue is only a default and there has been much discussion of the pros & cons of applications not following it.
| dotaneli wrote: |
| So - i want to know if there's a better solution to clearing old messages than a flow. and - if i do use this flow - how do i shrink the file size and update the message count without initiating external commands? |
No. 
I wait for one of my more imaginative associates to post a brilliant idea... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Feb 19, 2009 2:50 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'm confused that messages are getting to be 7 days old in the first place.
Don't leave them on the queue. 
Stick them in a database. Have a stored procedure run every time you insert that deletes everything older than 7 days. |
|
| Back to top |
|
|
| dotaneli |
| Posted: Thu Feb 19, 2009 10:46 am Post subject: |
|
|
Voyager
Joined: 19 Oct 2005
Posts: 99
Location: Israel
|
Well, the poor design is the client's design. Believe, i suggested moving the data to a DB. Mq is no DB, it's a messaging mechanism.
It's so wierd that you cannot shrink the file's size immediatly, and that the queue count is left unchanged when messages expire, until you make a Browse or Get from the queue.
The only stupid solution i found is using another queue; Every day, a process will copy content to the tomporary queue. The source queue will be cleared with "clear ql" MQSC command.
Messages will be copied back, and therefor both count issue and file size issue would be resolved.
I am desperatly looking for a more elegant solution... |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Feb 19, 2009 11:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dotaneli wrote: |
| It's so wierd that you cannot shrink the file's size immediatly, and that the queue count is left unchanged when messages expire, until you make a Browse or Get from the queue. |
As mqjeff and yourself point out, WMQ is not designed for long term storage. Or medium term storage. So the design principle is that if there are messages on a queue, something should be trying to read them. Remember what message expiry is actually intended for.
| dotaneli wrote: |
| I am desperatly looking for a more elegant solution... |
The voice in the back of my head says that v7 for distributed platforms includes the cleaning process that's been in z/OS for ages, where the queue manager periodically scans for and deletes expired messages. It could be lying to me; it does that. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| exerk |
| Posted: Thu Feb 19, 2009 11:17 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Vitor wrote: |
| The voice in the back of my head says that v7 for distributed platforms includes the cleaning process that's been in z/OS for ages, where the queue manager periodically scans for and deletes expired messages. It could be lying to me; it does that. |
I thought V6.0 did that too, or is that only if 'something' has the queue open? Or is this yet another of my misapprehensions about to be put right?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Feb 19, 2009 3:42 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| exerk wrote: |
| Vitor wrote: |
| The voice in the back of my head says that v7 for distributed platforms includes the cleaning process that's been in z/OS for ages, where the queue manager periodically scans for and deletes expired messages. It could be lying to me; it does that. |
I thought V6.0 did that too, or is that only if 'something' has the queue open? Or is this yet another of my misapprehensions about to be put right? |
IIRC there is a background process purging expired messages in V6, but you have no control over it. So it will get to them when it will get to them... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Feb 19, 2009 7:07 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If the queue is open, even just for MQINQ, its eligible for the built in Expoiry Scavenger to consider the queue for clean up of expired messages.
You cannot rely on this. IBM makes no guarantees on this behaviour. The only thing you can count on is never being given an expired message.
Issuing the Clear Q command on a q with 1 or more messages on it has shrunk the q file size for me in the past, without needing a QM restart. Issuing the clear q commmand on an empty did not shrink the file size. This is not documented behaviour, just something I discovered on my own, so again, don't bank on it.
The best thing I can think of that you can rely on is to have a dummy app browse your queue for a message with Message ID = ThisMessageIdIsBogus every few minutes. It will scan the entire queue each time searching for the message ID that won't be there, cleaning up expired messages in the process.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Mr Butcher |
| Posted: Thu Feb 19, 2009 8:58 pm Post subject: |
|
|
Padawan
Joined: 23 May 2005
Posts: 1716
|
why not have a set of queues and work without expiry? have a queue for every day of the week, then go like this
/* get rid of messages that are 7 days old */
clear qlocal "day-of-week"
/* move new messages that should be purged in 7 days */
move messages-that-should-be-purged-in-7-days to queue "day-of-week"
in that case you do not need to browse to get rid of expired messages, and storage usage is spread over 7 queues... this may help in freeing up reserved, but unused space (not sure about this).
this has another advantage... if somebody drops in saying "hey, can we keep the messages for 1 more week we need to investigate something" then you can just move them away to a different queue. If you used expiry you need to change that again so they do not get purged.....
_________________
Regards, Butcher |
|
| Back to top |
|
|
| dotaneli |
| Posted: Sat Feb 21, 2009 12:28 pm Post subject: |
|
|
Voyager
Joined: 19 Oct 2005
Posts: 99
Location: Israel
|
Thanks guys. I can't see how the new solution will make things better. Keep 7 queues? why not a queue for every hour?
about the app that does browse with UUID, well - it's a huge queue. Can get to 74 gigs. That would realy consume resources... no? |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Feb 22, 2009 10:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| dotaneli wrote: |
| about the app that does browse with UUID, well - it's a huge queue. Can get to 74 gigs. That would realy consume resources... no? |
You have 74 Gb of data on a queue? You have much more serious issues than setting expiry!
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gunter |
| Posted: Mon Feb 23, 2009 12:41 am Post subject: |
|
|
Partisan
Joined: 21 Jan 2004
Posts: 307
Location: Germany, Frankfurt
|
Hi,
we have a similar problem, not with gigs of data but millons messages.
We implemented a function it in the application for purging the queue
This function is called if the day changes or the application starts.
Messages are read withe buffer length 0, accept truncation and in a local unit of work, not to big for the system. This is fast and doesn't need so much ressources.
Why do you concerned about big files, you need this space an it should be not important if it is free or allocated by the queue. Maybe it is better never to shrink the files, because this results in more fragmented files on the harddisk.
_________________
Gunter Jeschawitz
IBM Certified System Administrator - Websphere MQ, 5.3 |
|
| Back to top |
|
|
| dotaneli |
| Posted: Mon Feb 23, 2009 9:01 am Post subject: |
|
|
Voyager
Joined: 19 Oct 2005
Posts: 99
Location: Israel
|
About the 74 gigs of messages - yeah, it sucks. Not my doing...
About the solution you offered - sounds interesting, could work.
Reason for shrinking files is that messages are always removed, and so - i don't want to take extra space. Fragmentation issues are for the OS guys to resolve, and is really OS dependant... |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Feb 23, 2009 10:25 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You are using a broker flow - so you do have a database to use. This is not pure MQ.
WMB is ideal for message and database interaction. |
|
| Back to top |
|
|
|
|
