ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » User Exits » Exit event for expired message clean up task

Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next
 Exit event for expired message clean up task « View previous topic :: View next topic » 
Author Message
bruce2359
PostPosted: Sat Oct 18, 2014 8:03 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

fjb_saper wrote:
bruce2359 wrote:
I'm still missing the point of what kind of process would need to be instantiated by a message expiring in a queue - any queue. What is there to clean up? How can a non-consumable message be correlated to something in need of cleaning up?

The point of the get with a non existent correlId is to "effectively" expire messages beyond their expiry time and not waiting for the reaper process.

Buy why force expired message cleanup? Are MQGETs being delayed by apps creating excessively large quantities of messages that expire? If so, set expiry cleanup to run more frequently. What other purpose?

Clean up what? A unit of work?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sat Oct 18, 2014 11:14 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

By running cleanup more frequently you will affect the whole qmgr. This may take minutes if you have a few legitimate deep queues... Remember loading the catalog??

Instead having a long running process doing get with an inexisting correlId should get the messages expired as soon as possible.

What we are trying to cover here is the target application being down and all messages expiring but not in a timely manner...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
PeterPotkay
PostPosted: Sun Oct 19, 2014 4:51 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

So the OP is nowhere to be found and this thread has taken on a life of its own. Oh well.


You may want to force the purge of expired messages because the queue might be reporting a q depth over some threshhold that matters to you and that depth is partially or wholly made up of expired messages. The threshold might even be Q Full.

rekarm01, distributed platforms do have an expiry scavenger. If you look at the MQ Erorr Log when the QM starts up you see it as one of the tasks that starts up. It cleans up, "periodically", expired messages from loaded queues. Loaded queues are ones that have an active MQOPEN handle on them by some app, any app, for any reason (gets, puts, MQINQ). When this first came out and I played with it, I came to the conclusion it runs about every 5 minutes, but there is no documentation to that point, no evidence in the logs when it runs, and we base no designs on that assumption.

I see the quote you refer to in the KC, but MQ does purge expired messages if you go a get using a non existent Corrrel ID. Whether the expired message is at the front, middle or end of a queue of non expired messages, doing an MQGET for a non existent Correl ID does seem to purge the queue of ALL expired messages. Based on andyh's info, perhaps this would not work on a queue with millions of messages where that hash table is huge. Nowadays this 'trick' is no longer needed with the background Expiry Scavenger running. Even if the queue is only opened by the putting app, that queue is loaded and thus on the Expiry Scavenger's hit list. We have not encountered inflated queue depths of expired messages since this was added by IBM, back in MQ 6 I believe. Indeed, I assumed the IBM scavenger was simply doing what we used to do with that get against a non existent correl ID, but that was based on zero evidence and no idea how else they may have done it.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
tczielke
PostPosted: Sun Oct 19, 2014 7:29 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

PeterPotkay wrote:
I see the quote you refer to in the KC, but MQ does purge expired messages if you go a get using a non existent Corrrel ID. Whether the expired message is at the front, middle or end of a queue of non expired messages, doing an MQGET for a non existent Correl ID does seem to purge the queue of ALL expired messages.


I tested this out on my 8.0 Linux queue manager, and I don't see this behavior. I PUT 5 messages, each with a unique CorrelId and an expiry of 60 seconds on a queue. I then waited more than 60 seconds and did one GET with a CorrelId that did not match any of the CorrelIds on the 5 messages on this queue. The result was that the MQGET came back with 2033 and the queue still had the 5 expired messages on the queue.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sun Oct 19, 2014 8:35 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

There continues to be a general caution from IBM not to depend on exactly and precisely how an internal process works. The only guarantee is that the process will (should?) meet the documentation.

In the instance of expired messages, the doc is pretty clear that expired messages will not be delivered to a consuming application; and that an expiry cleanup process (garbage collector) will purge expired messages from queues.
Over time, exactly and precisely how this took place has changed.

In early versions/releases, the next MQGET launched the garbage collector; and in later releases, it was launched as a background task. Now, you can specify (approximately) how often garbage collection is to take place.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
tczielke
PostPosted: Sun Oct 19, 2014 8:46 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

I also don't see any evidence of this background Expiry process running on this v8 Linux queue manager. According to this doc, it should be running by default every 5 minutes -> http://www-01.ibm.com/support/docview.wss?uid=swg21288579

However, I have left expired messages on a queue for over an hour and have never seen them removed. I also tried explicitly setting the ExpiryInterval on the qm.ini file, and also leaving a process having the queue open for > 5 minutes. I do see the queue manager error log reporting that the EXPIRER process is starting with the queue manager, but it does not appear to be doing anything, from what I can tell.

On a side note, I find it a borderline product defect that there is no way to echo back the qm.ini attributes that you have set as an administrator. This should be a basic configuration requirement. It is also a significant encumberance to being able to be doing your job effectively as an MQ distributed administrator.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sun Oct 19, 2014 9:30 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

From the URL you cited:

Code:
Beginning in WMQ V6.0, functionality has been added to internally discard expired messages silently. When a message expires, it will be discarded when: an MQGET which matches the message is issued, or It is internally removed by the queue manager's expirer task.


...which matches...

Matching may be explicit - a specific MsgiD/CorrelId/GroupId; or it may be none of these, and may be the next available message in the queue.

As to why garbage collection never runs seems to be a PMR question. If I had to speculate, I'd speculate that the qmgr sees no reason to launch the garbage collector if little/no work is taking place on the qmgr.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.


Last edited by bruce2359 on Sun Oct 19, 2014 9:33 am; edited 1 time in total
Back to top
View user's profile Send private message
smdavies99
PostPosted: Sun Oct 19, 2014 9:31 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

In my current position 99% of our messages have expiry set. The times range from 30 minutes to 30 days. We have never experienced issues with expired messages hanging around on queues.
We even have applications that do selective deletes using MQGET using MsgId (Not MQ Generated and I know that this is not best practive). The messages that this application use have a 24 hour expiry. New ones (again with another 24hour expiry are loaded every day at 00:02:00.
Come messages are read by an MQGET, have data changed and are re-written using the same MsgID.
While I know that a number of people here will be contact admin their teeth at the obvious bad practices used here, this was something that I inherited when I picked up this project. As I said, this has been working in production for more that 2 years without any occurence of expired messages not being contact admin by the system even when the reading application is down.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sun Oct 19, 2014 9:43 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

smdavies99 wrote:
We have never experienced issues with expired messages hanging around on queues.

What problems do you/your applications have with expired messages hanging around queues? Please be specific with your answer.

Do your apps MQINQuire on queue-depth before MQGETting a message?

The mere existence of expired messages should pose no significant problem to applications.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Sun Oct 19, 2014 11:16 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

tczielke wrote:
On a side note, I find it a borderline product defect that there is no way to echo back the qm.ini attributes that you have set as an administrator. This should be a basic configuration requirement. It is also a significant encumberance to being able to be doing your job effectively as an MQ distributed administrator.


I'm with ya - please vote for this RFE:
http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=42528

It's got 34 votes already with some good comments to make the RFE better. Seems like this is something they could sneak into a Fix Pack and open up in all supported version - hope they don't hold off on implementing it until the next Refresh Pack or Major Release.



The only time expired messages have messed with us is for out monitoring tools, where a false positive will fire (Q Depth > x) and you then find out that a good portion of x messages were expired. But again, since MQ 6.0 this has not been a problem.

Tim, all I can say is I'm surprised by your Expiry Test results.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
smdavies99
PostPosted: Sun Oct 19, 2014 11:38 am    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

bruce2359 wrote:
smdavies99 wrote:
We have never experienced issues with expired messages hanging around on queues.

What problems do you/your applications have with expired messages hanging around queues? Please be specific with your answer.

Do your apps MQINQuire on queue-depth before MQGETting a message?

The mere existence of expired messages should pose no significant problem to applications.


The application (actually a Broker Flow) does not do an inquire first. There may not be a message that matched the MQMD.MsgId on the queue. If there is, it gets removed by the MQGET node. The flow then writes a message with the same MsgId back to the queue. In two years of production operation and some 10-20,000 operations a day on the queue we have never had an error thrown due to problems with messages on the queue.
There are no expired messages hanging around on queues. Actually the flow sets the expiry time to be 00:02:00 the next day for all messages. This coincides with the next time a batch of messages are written for the next days schedules.
In all other cases, the messages with expiry get removed without causing any issues to the running system.

I hope that this is sufficient information for you. As I said, I inherited the design. A design that (sadly) uses queues as a data store.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
rekarm01
PostPosted: Sun Oct 19, 2014 2:52 pm    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1415

PeterPotkay wrote:
I see the quote you refer to in the KC, but MQ does purge expired messages if you do a get using a non existent Corrrel ID.

If it does, then that's undocumented behavior, not guaranteed to work the same way from one day to the next. Users should not design applications that rely on undocumented behavior.

tczielke wrote:
I also don't see any evidence of this background Expiry process running on this v8 Linux queue manager. According to this doc, it should be running by default every 5 minutes -> http://www-01.ibm.com/support/docview.wss?uid=swg21288579

That technote refers to WMQ Software versions v6.0 through v7.5, but not v8. Maybe it doesn't apply to v8?

PeterPotkay wrote:
distributed platforms do have an expiry scavenger ..., but there is no documentation to that point, no evidence in the logs when it runs, and we base no designs on that assumption.

Ok, one or more versions of one or more of the distributed platforms may have an undocumented (?) expiry scavenger, that works in an unspecified manner, at least some of the time ... is there any other supporting documentation for this feature, (beyond the one technote)?

PeterPotkay wrote:
The only time expired messages have messed with us is for out monitoring tools, where a false positive will fire (Q Depth > x) and you then find out that a good portion of x messages were expired. But again, since MQ 6.0 this has not been a problem.

If the monitoring tools themselves are browsing queues, (for example, to determine message age), then that would also help to clean up expired messages.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Sun Oct 19, 2014 4:29 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

smdavies99 wrote:
The application (actually a Broker Flow) does not do an inquire first. There may not be a message that matched the MQMD.MsgId on the queue. If there is, it gets removed by the MQGET node. The flow then writes a message with the same MsgId back to the queue. In two years of production operation and some 10-20,000 operations a day on the queue we have never had an error thrown due to problems with messages on the queue.
There are no expired messages hanging around on queues. Actually the flow sets the expiry time to be 00:02:00 the next day for all messages. This coincides with the next time a batch of messages are written for the next days schedules.
In all other cases, the messages with expiry get removed without causing any issues to the running system.

I hope that this is sufficient information for you. As I said, I inherited the design. A design that (sadly) uses queues as a data store.

No, it doesn't explain at all. Is your only perceived issue the presence of expired messages in a queue? And, further, that if there are no expired messages, then the app runs flawlessly?

From your narrative above, what error do you see if there are expired messages in the queue? What misbehavior occurs because there are expired messages in the queue? Will just one expired message in the queue cause the app to misbehave? Does it take 10 expired messages?

If the application is behaving differently now, what has changed in two years? Absolutely nothing? No broker changes software changes? No MQ CSDs? Nothing?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
smdavies99
PostPosted: Sun Oct 19, 2014 9:49 pm    Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.

bruce2359 wrote:


If the application is behaving differently now, what has changed in two years? Absolutely nothing? No broker changes software changes? No MQ CSDs? Nothing?


Ah! I see where you are coming from.

There is no issue with our setup. I merely posted as an example of where we have NOT seen expired messages hanging around on the queues. To be honest, without deliberately injecting some carefully crafted 'rogue' messages onto the queues I don't think that I'd see this problem at all.

The system was developed for MQ 7.0.1.4/WMB 7.0.0.4 and is now undergoing SAT for MQ 8.0.0.1/IIB 9.0.0.3. So far everything is working as expected.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
Andyh
PostPosted: Mon Oct 20, 2014 12:21 am    Post subject: Reply with quote

Master

Joined: 29 Jul 2010
Posts: 237

The expiry task was first introduced in MQ V6 and hasn't changed much since that time. It is still present in MQ V8.
The most likely issue with the expiry task is the decision to only operate on "loaded" queues, as discussed earlier in this thread.
The expiry task works off the queue index, the queue index is built at first reference (a.k.a. queue loading) and remains in memory until the queue becomes unreferenced for some period of activity (deliberately vague!). Rather than scan messages on disk the expiry task intentionally ignores queues that haven't recently been referenced.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page Previous  1, 2, 3  Next Page 2 of 3

MQSeries.net Forum Index » User Exits » Exit event for expired message clean up task
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.