| Author |
Message
|
| MABeatty1978 |
 Posted: Fri May 22, 2015 6:23 am Post subject: Releaseing message at a specified time |
 |
|
Acolyte
Joined: 17 Jul 2014
Posts: 54
|
| I'm writing an application that receives a message that contains an xml document and a blob of data. I'm writing the blob to disk, the xml doc contains instructions on how to process the data. One of the parameters is a date/time when the data should be processed. My initial thought was to dump that xml data into a new message and stage it in another queue which has another application looping through the messages, checking the date/time value and triggering the load if appropriate. I'm worried, though, that since it could be days, even weeks before it should be processed that the continuously looping browse would tax the cpu much more than necessary. Is this a legitimate worry or does the browse not tax the cpu enough to even worry about it? Has anyone done anything similar a different way, maybe by using a database instead of a queue? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Fri May 22, 2015 6:41 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
How often would your app need to browse the queue? Once an hour? Once a minute? Every microsecond?
How many messages will be in the queue? 10, 100, 1000, 1000000?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 22, 2015 6:42 am Post subject: Re: Releaseing message at a specified time |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| MABeatty1978 wrote: |
| Is this a legitimate worry |
Yes
| MABeatty1978 wrote: |
| does the browse not tax the cpu enough to even worry about it? |
Browse is the most resource intensive of the MQ verbs. Never mind the CPU, think of the I/O you'll be eating doing what is effectively a table scan of the queue; a queue which (we assume given the timeframe involved) will build to a significant depth.
| MABeatty1978 wrote: |
| Has anyone done anything similar a different way, maybe by using a database instead of a queue? |
Yes. What you're describing is an ideal use case for a database.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| MABeatty1978 |
| Posted: Fri May 22, 2015 6:51 am Post subject: |
|
|
Acolyte
Joined: 17 Jul 2014
Posts: 54
|
| bruce2359 wrote: |
| How often would your app need to browse the queue? Once an hour? Once a minute? Every microsecond? |
I guess there is some thought that would need to go into this to balance out the "wants" with the "needs". I my head I'm envisioning the precision down to the second... but in reality, I don't think it would be problematic to check every hour.
| bruce2359 wrote: |
How many messages will be in the queue? 10, 100, 1000, 1000000? |
I'm comfortable saying < 10., definitly not 100. Typically, there would probably only be 1 or 2. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 22, 2015 6:58 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you given thought about using the delayed delivery capability of JMS in MQ 8 ? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri May 22, 2015 6:59 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| MABeatty1978 wrote: |
| bruce2359 wrote: |
| How often would your app need to browse the queue? Once an hour? Once a minute? Every microsecond? |
I guess there is some thought that would need to go into this to balance out the "wants" with the "needs". I my head I'm envisioning the precision down to the second... but in reality, I don't think it would be problematic to check every hour.
| bruce2359 wrote: |
How many messages will be in the queue? 10, 100, 1000, 1000000? |
I'm comfortable saying < 10., definitly not 100. Typically, there would probably only be 1 or 2. |
Assuming that the messages are relatively small, at 1 or 2 or 10 or 100, the messages would likely be in a queue buffer (not read from disk at each MQGET) - easy for mq to browse.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri May 22, 2015 7:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| bruce2359 wrote: |
| Assuming that the messages are relatively small, at 1 or 2 or 10 or 100, the messages would likely be in a queue buffer (not read from disk at each MQGET) - easy for mq to browse. |
For weeks? Especially if persistent (as they'd need to be to guarantee their life for that length of time).
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri May 22, 2015 7:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Queues are not databases.
Use cron or etc. to start a program that scans a database table for rows that are ready to be processed. dump them to a queue, or simple process them in the same program.
You could, in theory, use Expiry and Expiry reports to get notified when a specific message expires. I don't remember how long the maximum interval is.
And even then, that's not necessarily accurate to very fine time intervals. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri May 22, 2015 7:07 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Vitor wrote: |
| bruce2359 wrote: |
| Assuming that the messages are relatively small, at 1 or 2 or 10 or 100, the messages would likely be in a queue buffer (not read from disk at each MQGET) - easy for mq to browse. |
For weeks? Especially if persistent (as they'd need to be to guarantee their life for that length of time). |
Perhaps non-persistent messages with NPMCLASS(HIGH) queue attribute. IMS, persistent messages are not logged for browse, only for destructive MQGET.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| MABeatty1978 |
| Posted: Fri May 22, 2015 7:10 am Post subject: |
|
|
Acolyte
Joined: 17 Jul 2014
Posts: 54
|
| mqjeff wrote: |
Queues are not databases.
Use cron or etc. to start a program that scans a database table for rows that are ready to be processed. dump them to a queue, or simple process them in the same program.
You could, in theory, use Expiry and Expiry reports to get notified when a specific message expires. I don't remember how long the maximum interval is.
And even then, that's not necessarily accurate to very fine time intervals. |
In reading the responses, I'm going with this as my answer. A database is the appropriate approach.
Thank you all. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri May 22, 2015 8:10 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| fjb_saper wrote: |
| Have you given thought about using the delayed delivery capability of JMS in MQ 8 ? |
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun May 24, 2015 4:26 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| MABeatty1978 wrote: |
| mqjeff wrote: |
Queues are not databases.
Use cron or etc. to start a program that scans a database table for rows that are ready to be processed. dump them to a queue, or simple process them in the same program.
You could, in theory, use Expiry and Expiry reports to get notified when a specific message expires. I don't remember how long the maximum interval is.
And even then, that's not necessarily accurate to very fine time intervals. |
In reading the responses, I'm going with this as my answer. A database is the appropriate approach.
Thank you all. |
Or, MQ is not an appropriate approach. MQ is a reliable store-and-forward messaging system. It is not designed for use as a DB for app data that is browsed over and over. Even if MQ features can be shoe-horned to meet application design requirements, it doesn't mean they should be used that way. In fact, it will quite often lead to trouble down the track.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun May 24, 2015 6:10 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
This is not say that MQ is the wrong solution, just not a solution contemplated by the IBM developer team. Like most other middle-ware, MQ evolves because customers demand new functionality.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon May 25, 2015 5:54 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| bruce2359 wrote: |
| This is not say that MQ is the wrong solution, just not a solution contemplated by the IBM developer team. Like most other middle-ware, MQ evolves because customers demand new functionality. |
How about MQ as a Database? Message = Row. Schema is a queue property. Support SQL INSERT, UPDATE, DELETE. I don't think I'd be game to raise that as an RFE. 
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue May 26, 2015 5:06 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| gbaddeley wrote: |
| bruce2359 wrote: |
| This is not say that MQ is the wrong solution, just not a solution contemplated by the IBM developer team. Like most other middle-ware, MQ evolves because customers demand new functionality. |
How about MQ as a Database? Message = Row. Schema is a queue property. Support SQL INSERT, UPDATE, DELETE. I don't think I'd be game to raise that as an RFE. |
How about using SQL92 subset to select messages based on Message Properties?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
