Author |
Message
|
GLudgate |
Posted: Mon Dec 18, 2006 4:13 pm Post subject: Get messages in LIFO order |
|
|
Newbie
Joined: 18 Dec 2006 Posts: 6 Location: Vancouver BC
|
I need to be able to read messages off an MQ queue in LIFO order. There is a single application PUTing to the queue but many applications GETing off the queue. MQ supports only FIFO and priority order reading of messages from a queue. Does anyone know of any technique that I could use to simulate LIFO order GETs?? It does not have to be exactly LIFO ie it could sometimes get the last but one message. But in the main it must read from the tail of the queue. Any algorithms will be appreciated. _________________ George Ludgate |
|
Back to top |
|
 |
jefflowrey |
Posted: Mon Dec 18, 2006 4:22 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
No, you don't need to get messages in LIFO order.
No, there's no way to GET messages in LIFO order, without tying yourself into knots. _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
fjb_saper |
Posted: Mon Dec 18, 2006 4:23 pm Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
Dequeue into a stack and than work with the stack... Of course this might give you problems with the memory....
Dequeue into a DB and process from the DB...
Care to share a little bit more about the requirement? Why would you need LIFO dequeueing?
Enjoy  _________________ MQ & Broker admin |
|
Back to top |
|
 |
wschutz |
Posted: Mon Dec 18, 2006 5:22 pm Post subject: |
|
|
 Jedi Knight
Joined: 02 Jun 2005 Posts: 3316 Location: IBM (retired)
|
fjb_saper wrote: |
Care to share a little bit more about the requirement? Why would you need LIFO dequeueing?
Enjoy  |
Maybe he's writing an "infix" to RPN system  _________________ -wayne |
|
Back to top |
|
 |
GLudgate |
Posted: Tue Dec 19, 2006 12:23 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006 Posts: 6 Location: Vancouver BC
|
The messages contain time sensitive data. ALL messages must arrive at the destination, eventually, but the latest message contains the most current information and the rest are of historical interest only. Hence LIFO - the last one onto the queue is the most recent data and MUST be processed first. If an application reads the second (or N-th) most current data, that is allowed but an affective action may occur too late to be of any use.
There are many reader applications, all on different distributed hosts, so I cannot read all the messages into any one of the memories and reverse the order. _________________ George Ludgate |
|
Back to top |
|
 |
Vitor |
Posted: Tue Dec 19, 2006 12:31 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
The "standard" method of dealing with time-sensitive material is to use the expiry property to prevent old data being used. For example, if the data held in the message is replaced by another message sent a minute later, each message will expire after 59 seconds and thus the most recent is always at the top of the queue.
You say you have many readers - is this a pub/sub environment? And how does the current system square the requirement to only use the most recent information while processing all the messages for "historical interest"? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
dp111443 |
Posted: Tue Dec 19, 2006 1:05 am Post subject: |
|
|
Voyager
Joined: 25 Feb 2004 Posts: 82
|
Hi GLudate,
You mentioned that "ALL historical data must arrive at the destination". In this case I guess Message Expiry is out of the question since you don't want to lose any messages.
You mentioned "many readers" off one queue. Do you have an issue with the rate of the messages that are being put onto the queue is greater than the readers pulling them off the queue? i.e. a back log?
thanks,
Dharmesh _________________ Integration Design/Developer
IBM Certified System Administrator -
WebSphere MQ V5.3 |
|
Back to top |
|
 |
GLudgate |
Posted: Tue Dec 19, 2006 1:08 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006 Posts: 6 Location: Vancouver BC
|
ALL messages MUST be processed. None can be lost. They just lose their value over time but COULD be of use. An application decides how useful a message is after reading the message. So if "expiring" a mesage removes it from the queue that would not work for me.
There are indeed many readers. It is not a pub/sub environment, though the number of readers may vary. As long as any ONE of them processes the most recent message then our requirements are met. Readers of old messages MAY find them to be of some use. But they must process it as best they can and never ignore it.
I am looking for any MQ mechanisms to simulate LIFO reads. _________________ George Ludgate |
|
Back to top |
|
 |
GLudgate |
Posted: Tue Dec 19, 2006 1:13 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006 Posts: 6 Location: Vancouver BC
|
Dharmesh
A backlog will occur if the dequeue rate is less than the enqueue rate. This will not be a problem if I can read in LIFO order. The backlog is usually temporary, by design (we know our rates of production and consumption). Such a backlog can occur if there is heavy input, the TCP/IP links go down, reader applications go down etc etc. We have seen it all happen at one time or another. _________________ George Ludgate |
|
Back to top |
|
 |
Vitor |
Posted: Tue Dec 19, 2006 1:16 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
Okay, that's new. A message that might be useful; just when you think you've seen everything....
I think you're going to struggle a bit with this. One way you might achieve this is to get the queue depth (n), browse through the queue to the nth record and read the nth record.
That's just contact admin code. In a multi reader environment there's no guarantee the nth record will still be there by the time the read happens & if it works it will eat CPU for breakfast. And thrash the disc - if all messages must be read I'm guessing they're persistent so the logging will go crackers.
No chance you can dequeue them into a database and do a SELECT MIN(PUT_TIME) sort of thing? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
dp111443 |
Posted: Tue Dec 19, 2006 1:33 am Post subject: |
|
|
Voyager
Joined: 25 Feb 2004 Posts: 82
|
I agree with Vitor.
The only options I can see is if you dequeue them to an external persistance store and read the messages that you are interested in based on time.
Alternitively, have you thought about using a number of queues rather than just one? This may allow you to spread the load. It may also allow your readers to scan the queues more easily depending on how you do this (i.e. Vitor's suggestion).
I guess the key here is to not have a backlog so your always processing the latest messages.
cheers, _________________ Integration Design/Developer
IBM Certified System Administrator -
WebSphere MQ V5.3 |
|
Back to top |
|
 |
GLudgate |
Posted: Tue Dec 19, 2006 1:34 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006 Posts: 6 Location: Vancouver BC
|
Sorry for the "might" but requirements are requirements and I have to meet them.
I cannot read the messages into a central database as all inter-application messaging is via MQ. This is just a new (perceived to be "small") requirement from the Business. It is not a technical requirement. And no-one will believe it cannot be done - quickly.
If I were to read the messages into a database I could process them at the same time. Reading in LIFO order was intended to get to the most current message as FAST as possible. _________________ George Ludgate |
|
Back to top |
|
 |
Vitor |
Posted: Tue Dec 19, 2006 1:44 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
GLudgate wrote: |
Reading in LIFO order was intended to get to the most current message as FAST as possible. |
Use my suggestion & it'll run like a dog. With 3 legs. And general health issues.
A better way would be to go with dp111443 and use multiple queues with optimised, tuned readers so the queues have a very limited backlog and you're probably getting the most recent message.
The downside is that this method, explained to the business, clearly does not guarantee the latest message. And if a backlog does build up you're stuffed!  _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
Vitor |
Posted: Tue Dec 19, 2006 2:19 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
GLudgate wrote: |
This is just a new (perceived to be "small") requirement from the Business. It is not a technical requirement. And no-one will believe it cannot be done - quickly.
|
It's the "small", "quick" business requirements that will get you every time.  _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
fjb_saper |
Posted: Tue Dec 19, 2006 4:05 am Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
GLudgate wrote: |
ALL messages MUST be processed. None can be lost. They just lose their value over time but COULD be of use. An application decides how useful a message is after reading the message. So if "expiring" a mesage removes it from the queue that would not work for me.
There are indeed many readers. It is not a pub/sub environment, though the number of readers may vary. As long as any ONE of them processes the most recent message then our requirements are met. Readers of old messages MAY find them to be of some use. But they must process it as best they can and never ignore it.
I am looking for any MQ mechanisms to simulate LIFO reads. |
So how about using 2 queues. Each with a different SLA.- On the first queue the messages will expire if not consumed. So you will always have the latest value.
- On the second queue the messages do not expire and are read to be put into the archive (DB). It does not matter how old they are and it does not matter whether they are dequeued FIFO or LIFO as the first queue with expiry will take care of that.
You said that it did not matter who is dequeuing the message but that somebody is (first come/first serve) and I'll hold you to that. So the 2 queue concept should satisfy your requirement...
Enjoy  _________________ MQ & Broker admin |
|
Back to top |
|
 |
|