| Author |
Message
|
| StefanSievert |
 Posted: Thu Mar 18, 2004 10:54 am Post subject: Dequeue at - Anybody implemented this? |
 |
|
Partisan
Joined: 28 Oct 2001
Posts: 333
Location: San Francisco
|
Hi,
I was wondering if anybody has thought about or implemented a "dequeue-at" mechansim.
If you have a queue that holds a number of messages and you want to have them delivered based on a timestamp, instead of FIFO or priority, you currently have no way (that I know of) of achieving this with base MQ and JMS native facilities.
The only solution I could come up with only works for JMS applications, which could use an IntegerProperty and a selector. I haven't tried this though, so I don't know, for example, what happens to messages that arrive on the queue after the QueueReceiver has started to process messages.
Is anybody playing with something like that and would want to share comments/ideas/solutions?
Any input is welcome and appreciated!
Stefan
_________________
Stefan Sievert
IBM Certified * WebSphere MQ |
|
| Back to top |
|
 |
| Michael Dag |
| Posted: Fri Mar 19, 2004 3:30 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2602
Location: The Netherlands (Amsterdam)
|
I have seen this on IBM requirements list a couple of years ago, it was known as the "Do not open until Christmas" requirement.
Basically what it meant was to put a datetime parameter on the message
whereby the QueueManager would only make the message available to
a 'getting' application past the specified datetime. It would allow to pre-
feed information whereby all parties should get access to the same
information at the same datetime.
It never made it into the product.
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Mar 19, 2004 6:38 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I'm kind of glad it never made it into the product.
Queues are not really long-term storage mechanisms.
If you need to hold data for X time or until time X - use a database.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| StefanSievert |
| Posted: Fri Mar 19, 2004 4:26 pm Post subject: |
|
|
Partisan
Joined: 28 Oct 2001
Posts: 333
Location: San Francisco
|
| jefflowrey wrote: |
I'm kind of glad it never made it into the product.
Queues are not really long-term storage mechanisms.
If you need to hold data for X time or until time X - use a database. |
Well, at the same token, databases make a really lousy queuing mechanism.  I view the dequeue-at option as the inverse of the Expiry option, with a handful of really good use cases. I do agree with you though that queues shouldn't be used for long-term storage, but having a dequeue-at time doesn't imply that this is days or even hours away.
There are two good things about options: 1) the fact that you have them and 2) you don't have to use them. 
Have a great weekend!
_________________
Stefan Sievert
IBM Certified * WebSphere MQ |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Mar 19, 2004 5:39 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
Stefan, what if you put them all under syncpoint, and then only issue the MQCMIT when the clock strikes dequeue time?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Sat Mar 20, 2004 5:35 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| PeterPotkay wrote: |
| Stefan, what if you put them all under syncpoint, and then only issue the MQCMIT when the clock strikes dequeue time? |
I try to shy away from any solutions that involve long running transactions, myself... There are probably other things I'd like to do with the space in the log files than hold messages until a later time...
I can see Stefan's points, too.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| StefanSievert |
| Posted: Wed Mar 24, 2004 5:40 pm Post subject: |
|
|
Partisan
Joined: 28 Oct 2001
Posts: 333
Location: San Francisco
|
| PeterPotkay wrote: |
| Stefan, what if you put them all under syncpoint, and then only issue the MQCMIT when the clock strikes dequeue time? |
Peter,
that's not quite what I thought. I would want to have the messages safely in MQ custody, but just not delivered to an application until the dequeue-at time. The sender should be able to process its UOW as quickly as possible instead of locking resources for extended periods of time (where 'extended periods of time' in the context of a UOW is actually more in the seconds range...).
Anyhoo, looks like nobody here ever came across the need to build something like that.
Thanks to all for your comments!
Stefan
_________________
Stefan Sievert
IBM Certified * WebSphere MQ |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Mar 24, 2004 8:19 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
How about the putting application dumps them all to a STAGING.QUEUE as persistent messages, and immediatly commits them.
At dequeue time, intermediate app is started by O/S scheduling, and moves the messages from STAGING.QUEUE to REAL.APPLICATION.QUEUE?
I know this is not exactly what you are looking for, but it may at least give you the behaviour you are looking for. And it is better than tying up the logs as was the case with my first suggestion.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqonnet |
| Posted: Tue Apr 27, 2004 12:35 pm Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Another option that might be of interst would be to use "expiry" itself. Why not use "expiry" itself to reverse-engineer expiry.
Say you have 100's of messages being put to a queue, say DESTQ. Put with a very small expiry, may be 1 sec, and mark this queue as noshare. When your app believes its, de-queue time, issue a "dummy" get/browse on DESTQ.
All Puts to DESTQ should be made with small expiry and MQRO_EXPIRATION_WITH_FULL_DATA, of course specifying the replytoq and qmgr name. Open the replytoq with your app(that believes its de-qeuue time) and reads them just fine.
The drawback to this approach would be "all or none". You cannot select one or a bunch of messages and say i want to process them.
Just a thought.
Cheers
Kumar |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 27, 2004 12:39 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
Unfortunatly, all those messages in the final queue will be of type REPORT, with no valid ReplyToQueue. If they are one way messages, no big deal. But if they each require a reply, we are out of luck, or we have to hard code it.
Slick idea though Kumar.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqonnet |
| Posted: Tue Apr 27, 2004 12:54 pm Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Peter, i would think all the messages ending up on the replytoq where the expiration report would be sent be regular messages with format MQSTR. And that you could read the message data without any issues since you requested full_data.
Cheers
Kumar |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 27, 2004 12:59 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
The message Format would indeed be MQSTR.
But the message TYPE would now be REPORT, instead of REQUEST. A message type of REQUEST has a valid Reply2Queue and Reply2QueueManager. But the REPORT message does not, so the processing app would not be dynamically able to determine where to send the replies to, if that was what was required. It could of course hard code it, or bury it in the message buffer.
I've always been of the opinion that there should be two Reply2Q / Reply2QM fields. One for actually reply messages, and one for report messages (which could default to the first pair if not explicitly set.) I suggested this to IBM formally a while back. No luck. To much involved to expand the MQMD.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqonnet |
| Posted: Tue Apr 27, 2004 1:15 pm Post subject: |
|
|
Grand Master
Joined: 18 Feb 2002
Posts: 1114
Location: Boston, Ma, Usa.
|
Peter, i think we are not really on the same page here. Let me explain again.
App1 puts messages to queue DESTQ specifying the following.
-MQRO_EXPIRATION_WITH_FULL_DATA
-ReplyToQ, say REPLYQ(for example sake)
-ReplyToQMgr, say QM2(lets consider we are doing all this on only one qm)
-Expiry set to a very low value, say 1 sec.
App1 puts 100 messages to DESTQ. All of them expired.
App2 believes its now that time of the day and starts to "dequeue". Until this time, assumption is that no other app has issued a browse/get against DESTQ(in fact you can manage that). App2 opens the queue and issues a browse/get which comes back with 2033, and physically removes all of the expired messages and generates a report that is sent to REPLYQ as specified in replytoq field on QM2 as specified in replytoqmgr field.
If you issue amqsbcg/amqsget on REPLY, you would be able to retrieve the message(data) just fine, without having to bother about the message type. Of course, messages here are considered small for you to run bcg/get sample apps. :)
Assumption here is that you are *not* replying for this message and just consuming it. Because if you had to reply, then this whole theory would go down the drain, which is what you are pointing at, i believe. :)
Cheers
Kumar |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 27, 2004 1:21 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7716
|
| Quote: |
App2 opens the queue and issues a browse/get which comes back with 2033, and physically removes all of the expired messages and generates a report that is sent to REPLYQ as specified in replytoq field on QM2 as specified in replytoqmgr field.
|
So now there are all these messages on REPLYQ on QM2. App3 can now process them. App3 picks up the first message, process it, and sends the reply where? These REPORT messages no longer have Reply To info in the MQMD.
If App3 needs to send back a reply, it is stuck. Unless we bury it in the payload, or App3 hard codes it.
I agree there is no issue with App3 reading the messages. The buffer indeed will be a perfect copy of the original message. I am just taking it to the next logical step and wondering what App3 will do when it comes time to reply to all these messages that it is processing at the correct time.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Apr 30, 2004 2:10 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20697
Location: LI,NY
|
Very easy implementation for dequeue at:
It is a two pronged approach:
You need 2 queues
Post messages into queue 1 with unique group id for expiration
Post trigger message into queue 2 with group id after countdown complete.
Have queue 2 trigger at first
When the trigger is run the application reads the data in the trigger message (queue 2) (determines how many msgs) groupid etc..., then it goes and reads queue 1 with a filter on group id and gets all msgs from the same group.
There you go.
Hope it helps.
F.J. |
|
| Back to top |
|
|
|
|
