ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportGetting Expired messages

Post new topicReply to topic
Getting Expired messages View previous topic :: View next topic
Author Message
PeterPotkay
PostPosted: Fri May 18, 2001 2:56 pm Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7546

A quote from the online MQ reference manual:

After a message's expiry time has elapsed, it becomes eligible to be discarded by the queue manager. In the current implementations, the message is discarded when a browse or nonbrowse MQGET call occurs that would have returned the message had it not already expired. For example, a nonbrowse MQGET call with the MatchOptions field in MQGMO set to MQMO_NONE reading from a FIFO ordered queue will cause all the expired messages to be discarded up to the first unexpired message. With a priority ordered queue, the same call will discard expired messages of higher priority and messages of an equal priority that arrived on the queue before the first unexpired message.


My test:
1. I put a message to a Q with a CorrelID of JOESHMOE and an Expiry of 120 seconds.

2. For the next couple of minutes, I can look at the message in QPASA and if I look at the properties of the Q, the QDEPTH of the Q is one.

3.During these 2 minutes I also do GETS with a CORRELID of PETERPOTKAY. I keep getting back 2033s, as I should, because the only message on the Q has a CORRELID of JOESHMOE. After each of these GET attempts, the Q Depth remains at 1. (As a seperate test I did GETS with a JOESHMOE CORRELID just to make sure I can get the message.)

4. We wait a bit and now the message has expired. If I look at the Q thru QPASA, there are no messages to look at, BUT the depth is still 1. This is expected and is acting as documented, i.e. expired messages count towards Q depth.

5. I repeat step 3. I get a 2033. I go back to QPASA and look at the Q Depth, it is now ZERO. A GET that would not have returned the message had it not already expired has cleaned up the Q, it direct contrast to what the manual says above!

So the next thing I did was put 2 messages with 60 second EXPIRY times and JOESHMOE CORRELIDs, an UNLIMITED EXPIRY message with PETERPOTKAY, and 2 more JOESHMOE messages with 60 second EXPIRY times. The order of the messages in the FIFI Q would look like the following...........

JOESHMOE
JOESHMOE
PETERPOTKAY
JOESHMOE
JOESHMOE

After 2 minutes all the JOESHMOE messages are expired and PETERPOTKAY is still unlimited. I check the Q DEPTH and it is 5. I BROWSE the Q in QPASA and only see PETERPOTKAY. I do a get with CORRELID of PETERPOTKAY and get back PETERPOTKAY message. Go look at the Q, and it looks empty. Check the Q DEPTH and it says 2! The first 2 JOESHMOE messages got cleaned up as MQ went thru the FIFO Q looking for PETER. The last 2 were ignored (until I did another GET for PETER and then they went away).

Is the manual wrong? Is QPASA wrong?



_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Fri May 18, 2001 3:09 pm Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7546

OS/390 Version 2.1, if it makes any difference.....

_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bduncan
PostPosted: Fri May 18, 2001 3:48 pm Post subject: Reply with quote

Padawan

Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley

Peter,
I may be wrong, but I believe the documentation AND your observations are correct, but the documentation could have been a little more specific when describing the rules about discarding expired messages. I think an example may most easily illustrate what I have observed in the past:
1. queue has 100 messages on it.
2. all messages have the same priority.
3. the first 50 messages are expired, the second 50 have not.
4. all 100 messages have a unique correlid.
5. now, I do an MQGET matching against correlid=BRANDON, and there is one message on the queue with this correlid. More specifically, this is one of the 50 that are NOT expired. In other words, it is in the second group of 50.
6. Since all the messages on the queue have the same priority, the FIFO rules should apply, and MQSeries will begin matching against each message starting with the first looking for a correlid of BRANDON.
7. Assuming the correct message is message #75, then MQ will have to "touch" the first 74 messages on the queue before it comes to the message with the proper correlid.
8. As each expired message is "touched" it is discarded - in accordance with what you observed.
9. When the manual explains the conditions for discarding an expired message: "an MQGET that would otherwise give you the message were it not expired" it is explaining the most simplistic situation where there are no matching options. In the same way that message priorities can affect the discarding algorithm, so can matching options. In the simplest sense, MQSeries is searching through an unsorted array, element by element, looking for a specific value. By searching in this fashion, it must "touch" every element in the array until it arrives at the desired element. Part of this touching function says that because we are doing an MQGET (that COULD yield this message were we not using any matching options), and this particular message is expired, it should be discarded.

This also explains the results of your second test. Because the PETER message was #3, MQSeries had to match against 1 and 2 before it arrived at 3. 1 and 2 had correlids of JOE, and were expired, so in touching them it removed them. Then the third message yielded the correct correlid, and it was returned to you. Messages 4 and 5 were never "touched" so even though they are expired, they are still there.


_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator
Back to top
View user's profile Send private message Visit poster's website AIM Address
PeterPotkay
PostPosted: Mon May 21, 2001 4:25 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7546

Thanks Brandon, I suppose if you read between the lines in the online manual instead of taking it literally, then it all makes sense. The real reason all this came about was because of a discussion on nightly clean up jobs. In other words, should a shop have a job that runs once every 24 hours that simply does a plain vanilla GET to flush the Queue of any Expired messsages.
With my previous interpetation of the manual, I would say yes. After my tests and your explanation of the manual, I would now say no. Do you agree?


_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
bduncan
PostPosted: Mon May 21, 2001 9:22 am Post subject: Reply with quote

Padawan

Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley

Peter,
We have had similar issues with queues filling up with expired messages over time. If you just start putting messages to a queue, and they expire, but you never do a get against that queue, then yes, eventually the queue will fill up, or the filesystem will fill up. If you are worried about either of these two scenarios, then you are on the right track with a "getter" program, but keep in mind that if you do a vanilla get, you will not only "touch" the expired messages, but pull off any non-expired messages. A better solution may be to do a get against the queue matching for a correllid that you know won't exist. This way you will touch every message without actually removing any non-expired ones...


_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator
Back to top
View user's profile Send private message Visit poster's website AIM Address
PeterPotkay
PostPosted: Mon May 21, 2001 9:26 am Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7546

Ah, but wouldn't the next "real" GET, done the next morning, accomplish the same thing, negating the need for these clean-up type utilities?

_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
HarveyHarrison
PostPosted: Wed May 23, 2001 7:09 am Post subject: Reply with quote

Newbie

Joined: 22 May 2001
Posts: 1

Peter is correct - the get would discard the expired messages.

The expired message discarding behavior is different on some platforms - for example MQ 5.1 for Compaq NSK, the physical removal of messages is based on a housekeeping timed event rather than when GETs occur.

Harvey Harrison
MQSeries Kernel Architect
Back to top
View user's profile Send private message Yahoo Messenger
bduncan
PostPosted: Wed May 23, 2001 9:36 am Post subject: Reply with quote

Padawan

Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley

But Peter, the next "real" GET might not run through the entire queue. In other words, if the message you eventually pull off is halfway through the FIFO queue, then all the messages that come after it were never touched, hence if any are expired they won't be discarded. The advantage of a "getter" program is that it is guaranteed to "touch" every message on the queue.


_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator
Back to top
View user's profile Send private message Visit poster's website AIM Address
zpat
PostPosted: Sat Jun 02, 2001 1:14 pm Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5669
Location: UK

Either do a MQGET for a MSGID and CORRELID combination that will not exist, or loop around doing MQGETs under syncpoint until "no msg available" then BACKOUT the entire UOW.

This will work for all platforms. See the May or June issue (I forget which) of MQ Update, see "Journals" at http://www.xephon.com

[ This Message was edited by: zpat on 2001-06-02 14:16 ]
Back to top
View user's profile Send private message
RichArnfield
PostPosted: Mon Jun 04, 2001 4:22 am Post subject: Reply with quote

Newbie

Joined: 24 May 2001
Posts: 4
Location: Currently London, soon Frankfurt

Hi Folks,

How about this for a possible solution, for cleaning a queue of expired messages.
Put a dummy message with a specific CorrelId or even MsgId.
Make sure it is committedto the queue.
Then get it off again.
It's gone on the end of the queue, so every single message on the queue will be 'touched' in the process of getting it off again. If you make the Correl/MsgId specific or unique enough, the chances of any of the other messages matching it should be pretty low.

Reckon that would work?

Cheers,
Rich.

_________________
Rich Arnfield, Castlemay Ltd.
Certified MQSeries Solutions Expert
Back to top
View user's profile Send private message Send e-mail
Cliff
PostPosted: Mon Jul 02, 2001 4:13 am Post subject: Reply with quote

Centurion

Joined: 27 Jun 2001
Posts: 145
Location: Wiltshire

Why not simply browse the whole queue? The manual states 'In the current implementations, the message is discarded when a browse or nonbrowse MQGET call occurs that would have returned the message had it not already expired. ' so surely a browse program accepting truncated messages will work for all?
Back to top
View user's profile Send private message Send e-mail
zpat
PostPosted: Mon Jul 02, 2001 6:38 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5669
Location: UK

The browse method does not work on OS/390 (or so I understand).
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportGetting Expired messages
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.