| Author |
Message
|
| wilsonjohn24 |
 Posted: Thu Jul 02, 2009 12:42 am Post subject: Queue = Deferred Work Item Database? |
 |
|
Voyager
Joined: 02 Feb 2007
Posts: 93
Location: Scotland
|
Context:
Some batch process sends 1000’s of FF messages to queues for say Sending Letters/invoices ( low priority), but I don’t want this traffic to interfere with other traffic on the ESB (normal business transactions).
So my solution is to tune the flow to process message slower – perhaps timer nodes, low no of threads etc. Not really worried about this lots of ways of tackling this.
Concern:
1) Is it acceptable working principle to allow message build up on queues for slow processing?
2) This quickly materializes into the question of using MQ as a database for deferring working items?
Queue = Deferred Work Item Database?
Would be interested in the forums thoughts on this one, where to draw the line both theoretically and practically?
ps: I know the MQ/WMB can do all this, but what I want is to know is this correct usage of Base MQ. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jul 02, 2009 1:08 am Post subject: Re: Queue = Deferred Work Item Database? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| wilsonjohn24 wrote: |
| 1) Is it acceptable working principle to allow message build up on queues for slow processing? |
IMHO it's valid to use a queue to decouple a sending application from a receiving one.
| wilsonjohn24 wrote: |
2) This quickly materializes into the question of using MQ as a database for deferring working items?
Queue = Deferred Work Item Database?
|
I'd draw a distinction between a queue, where work items are drawn from a FIFO stack, and a database such as you describe where items are drawn in a random access manner .
Other opinions may be valid of course.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 02, 2009 3:40 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If you have multiple brokers and an MQ Cluster with load sharing you can define qaliases only to certain brokers that would carry the bulk load of batch processing where others would carry the bulk load of online processing....
Perhaps make sure the batch work gets processed in a different queue/eg from online processing...
As well for the traffic not to interfere have an online cluster and a batch cluster with different channels...
Make sure your Input Queue Maxdepth is sized correctly for the batch...
Make sure you message priority for batch is lower than for online... although with a big qdepth and subsecond SLAs your are better off separating the queues or brokers...
I would just let the eg process normally. No need for a timer node...
Finally MOVE the batch process to hours with low online volume...
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| telecoda |
| Posted: Thu Jul 02, 2009 4:28 am Post subject: |
|
|
Novice
Joined: 05 Feb 2008
Posts: 15
|
| Also make sure your queue is persistent if you are storing valuable data in there. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Jul 02, 2009 4:35 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
What a little gem this is.....
| telecoda wrote: |
| Also make sure your queue is persistent if you are storing valuable data in there. |
Unrelated to the question and a little basic / general, but a gem all the same. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Jul 02, 2009 6:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| WMBDEV1 wrote: |
| Unrelated to the question and a little basic / general, but a gem all the same. |
Especially as it's inaccurate. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 02, 2009 6:23 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| WMBDEV1 wrote: |
| Unrelated to the question and a little basic / general, but a gem all the same. |
Especially as it's inaccurate. |
Because queue's are not persistent, messages are?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Jul 02, 2009 6:26 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| Vitor wrote: |
| mqjeff wrote: |
| WMBDEV1 wrote: |
| Unrelated to the question and a little basic / general, but a gem all the same. |
Especially as it's inaccurate. |
Because queue's are not persistent, messages are? |
Absolutly, one of the most common mistakes people make. Surprises me how many people dont get this..... |
|
| Back to top |
|
|
| wilsonjohn24 |
| Posted: Thu Jul 02, 2009 10:11 pm Post subject: |
|
|
Voyager
Joined: 02 Feb 2007
Posts: 93
Location: Scotland
|
Thanks for the responses to question.
So based on the responses I am gathering it is an fully acceptiable use of MQ to allow build up of messages - for trickle/batch processing.
Easy design justification then I will just link off to this forum URL in my design rational section 
cheers
John
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Jul 02, 2009 10:46 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| wilsonjohn24 wrote: |
Thanks for the responses to question.
So based on the responses I am gathering it is an fully acceptiable use of MQ to allow build up of messages - for trickle/batch processing.
Easy design justification then I will just link off to this forum URL in my design rational section
cheers
John
|
Don't remember if we told you yet but if you are running in circular logging and even more so if you are running in linear logging you better make sure you have also enough log space to accommodate your batch.
We have queues that get drained 3 times a day and can have up to 200,000 small messages on it. The schedule drains them usually when they hit between 50,000 and 100K messages.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| telecoda |
| Posted: Fri Jul 03, 2009 3:14 am Post subject: |
|
|
Novice
Joined: 05 Feb 2008
Posts: 15
|
| WMBDEV1 wrote: |
| Vitor wrote: |
| mqjeff wrote: |
| WMBDEV1 wrote: |
| Unrelated to the question and a little basic / general, but a gem all the same. |
Especially as it's inaccurate. |
Because queue's are not persistent, messages are? |
Absolutly, one of the most common mistakes people make. Surprises me how many people dont get this..... |
*hangs head in shame*
Well that's me told off then! |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jul 03, 2009 3:42 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| telecoda wrote: |
| Well that's me told off then! |
Better you take the hit from us than you discover a load of business-critical messages have been lost despite the queue being set to persistent. Discovering too late that the actual queue attribute is default persistence, and the developer chose to set it directly. 
You'll find a large number of discussions in here on the advantages of applications using default persistence & the mq admin determining persistence over applications setting it directly. One of those questions with no right answer.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
