| Author |
Message
|
| GLudgate |
 Posted: Mon Dec 18, 2006 4:13 pm Post subject: Get messages in LIFO order |
 |
|
Newbie
Joined: 18 Dec 2006
Posts: 6
Location: Vancouver BC
|
I need to be able to read messages off an MQ queue in LIFO order. There is a single application PUTing to the queue but many applications GETing off the queue. MQ supports only FIFO and priority order reading of messages from a queue. Does anyone know of any technique that I could use to simulate LIFO order GETs?? It does not have to be exactly LIFO ie it could sometimes get the last but one message. But in the main it must read from the tail of the queue. Any algorithms will be appreciated.
_________________
George Ludgate |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Dec 18, 2006 4:22 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
No, you don't need to get messages in LIFO order.
No, there's no way to GET messages in LIFO order, without tying yourself into knots.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Dec 18, 2006 4:23 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Dequeue into a stack and than work with the stack... Of course this might give you problems with the memory....
Dequeue into a DB and process from the DB...
Care to share a little bit more about the requirement? Why would you need LIFO dequeueing?
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| wschutz |
| Posted: Mon Dec 18, 2006 5:22 pm Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| fjb_saper wrote: |
Care to share a little bit more about the requirement? Why would you need LIFO dequeueing?
Enjoy |
Maybe he's writing an "infix" to RPN system 
_________________
-wayne |
|
| Back to top |
|
|
| GLudgate |
| Posted: Tue Dec 19, 2006 12:23 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006
Posts: 6
Location: Vancouver BC
|
The messages contain time sensitive data. ALL messages must arrive at the destination, eventually, but the latest message contains the most current information and the rest are of historical interest only. Hence LIFO - the last one onto the queue is the most recent data and MUST be processed first. If an application reads the second (or N-th) most current data, that is allowed but an affective action may occur too late to be of any use.
There are many reader applications, all on different distributed hosts, so I cannot read all the messages into any one of the memories and reverse the order.
_________________
George Ludgate |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Dec 19, 2006 12:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
The "standard" method of dealing with time-sensitive material is to use the expiry property to prevent old data being used. For example, if the data held in the message is replaced by another message sent a minute later, each message will expire after 59 seconds and thus the most recent is always at the top of the queue.
You say you have many readers - is this a pub/sub environment? And how does the current system square the requirement to only use the most recent information while processing all the messages for "historical interest"?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| dp111443 |
| Posted: Tue Dec 19, 2006 1:05 am Post subject: |
|
|
Voyager
Joined: 25 Feb 2004
Posts: 82
|
Hi GLudate,
You mentioned that "ALL historical data must arrive at the destination". In this case I guess Message Expiry is out of the question since you don't want to lose any messages.
You mentioned "many readers" off one queue. Do you have an issue with the rate of the messages that are being put onto the queue is greater than the readers pulling them off the queue? i.e. a back log?
thanks,
Dharmesh
_________________
Integration Design/Developer
IBM Certified System Administrator -
WebSphere MQ V5.3 |
|
| Back to top |
|
|
| GLudgate |
| Posted: Tue Dec 19, 2006 1:08 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006
Posts: 6
Location: Vancouver BC
|
ALL messages MUST be processed. None can be lost. They just lose their value over time but COULD be of use. An application decides how useful a message is after reading the message. So if "expiring" a mesage removes it from the queue that would not work for me.
There are indeed many readers. It is not a pub/sub environment, though the number of readers may vary. As long as any ONE of them processes the most recent message then our requirements are met. Readers of old messages MAY find them to be of some use. But they must process it as best they can and never ignore it.
I am looking for any MQ mechanisms to simulate LIFO reads.
_________________
George Ludgate |
|
| Back to top |
|
|
| GLudgate |
| Posted: Tue Dec 19, 2006 1:13 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006
Posts: 6
Location: Vancouver BC
|
Dharmesh
A backlog will occur if the dequeue rate is less than the enqueue rate. This will not be a problem if I can read in LIFO order. The backlog is usually temporary, by design (we know our rates of production and consumption). Such a backlog can occur if there is heavy input, the TCP/IP links go down, reader applications go down etc etc. We have seen it all happen at one time or another.
_________________
George Ludgate |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Dec 19, 2006 1:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Okay, that's new. A message that might be useful; just when you think you've seen everything.... 
I think you're going to struggle a bit with this. One way you might achieve this is to get the queue depth (n), browse through the queue to the nth record and read the nth record.
That's just contact admin code. In a multi reader environment there's no guarantee the nth record will still be there by the time the read happens & if it works it will eat CPU for breakfast. And thrash the disc - if all messages must be read I'm guessing they're persistent so the logging will go crackers.
No chance you can dequeue them into a database and do a SELECT MIN(PUT_TIME) sort of thing?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| dp111443 |
| Posted: Tue Dec 19, 2006 1:33 am Post subject: |
|
|
Voyager
Joined: 25 Feb 2004
Posts: 82
|
I agree with Vitor.
The only options I can see is if you dequeue them to an external persistance store and read the messages that you are interested in based on time.
Alternitively, have you thought about using a number of queues rather than just one? This may allow you to spread the load. It may also allow your readers to scan the queues more easily depending on how you do this (i.e. Vitor's suggestion).
I guess the key here is to not have a backlog so your always processing the latest messages.
cheers,
_________________
Integration Design/Developer
IBM Certified System Administrator -
WebSphere MQ V5.3 |
|
| Back to top |
|
|
| GLudgate |
| Posted: Tue Dec 19, 2006 1:34 am Post subject: |
|
|
Newbie
Joined: 18 Dec 2006
Posts: 6
Location: Vancouver BC
|
Sorry for the "might" but requirements are requirements and I have to meet them.
I cannot read the messages into a central database as all inter-application messaging is via MQ. This is just a new (perceived to be "small") requirement from the Business. It is not a technical requirement. And no-one will believe it cannot be done - quickly.
If I were to read the messages into a database I could process them at the same time. Reading in LIFO order was intended to get to the most current message as FAST as possible.
_________________
George Ludgate |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Dec 19, 2006 1:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| GLudgate wrote: |
| Reading in LIFO order was intended to get to the most current message as FAST as possible. |
Use my suggestion & it'll run like a dog. With 3 legs. And general health issues. 
A better way would be to go with dp111443 and use multiple queues with optimised, tuned readers so the queues have a very limited backlog and you're probably getting the most recent message.
The downside is that this method, explained to the business, clearly does not guarantee the latest message. And if a backlog does build up you're stuffed! 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Dec 19, 2006 2:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| GLudgate wrote: |
This is just a new (perceived to be "small") requirement from the Business. It is not a technical requirement. And no-one will believe it cannot be done - quickly.
|
It's the "small", "quick" business requirements that will get you every time. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Dec 19, 2006 4:05 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| GLudgate wrote: |
ALL messages MUST be processed. None can be lost. They just lose their value over time but COULD be of use. An application decides how useful a message is after reading the message. So if "expiring" a mesage removes it from the queue that would not work for me.
There are indeed many readers. It is not a pub/sub environment, though the number of readers may vary. As long as any ONE of them processes the most recent message then our requirements are met. Readers of old messages MAY find them to be of some use. But they must process it as best they can and never ignore it.
I am looking for any MQ mechanisms to simulate LIFO reads. |
So how about using 2 queues. Each with a different SLA.- On the first queue the messages will expire if not consumed. So you will always have the latest value.
- On the second queue the messages do not expire and are read to be put into the archive (DB). It does not matter how old they are and it does not matter whether they are dequeued FIFO or LIFO as the first queue with expiry will take care of that.
You said that it did not matter who is dequeuing the message but that somebody is (first come/first serve) and I'll hold you to that. So the 2 queue concept should satisfy your requirement...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
