| Author |
Message
|
| deepak_paul |
 Posted: Thu Jan 23, 2014 10:42 am Post subject: Detect low traffic in MQ and process the msgs |
 |
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
All,
We are working on a solution where we need to detect the traffic in the flow and if not much, we need to get messages from DB and send to the app flow.
Scenario:
1. If the data flow traffic is less and the application queue(MQ) is loaded with less number of inbound messages(only 500 or less), we need to take more messages(maybe 10000) from DB and pump them in to the application queue.
2. If the data flow traffic is more and the application Queue(MQ) is loaded with more number of messages(~10000 or more), we need to take less messages(maybe 200) from DB and pump them in to the application queue.
One solution we are thinking of is to use the WMB Timer nodes in a message flow that wakes up for every 5-10 mins and check the depth of the app queue and if way less, take out messages from DB and process them to the app queue.
Any other suggestions...
_________________
Regards
Paul |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Jan 23, 2014 10:55 am Post subject: Re: Detect low traffic in MQ and process the msgs |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| deepak_paul wrote: |
| Any other suggestions... |
Query all the rows you need to process from the database. Dump all of these messages into the queue (creating a much higher depth than you indicate). Allow the flow to process the messages.
Why do you need to stage them in this way? What's the requirement?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Thu Jan 23, 2014 11:35 am Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
| Vitor wrote: |
| Why do you need to stage them in this way?... |
App Queue is always flooded with real time messages. The messages which are in DB are reprocess messages. So basically we want to process all reprocess messages when there is not much of real time traffic.
One other solution i think of at this moment is MQ Event triggering. Need to investigate more on how it can satisfy our scenarios.
_________________
Regards
Paul |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 23, 2014 11:44 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| deepak_paul wrote: |
| Vitor wrote: |
| Why do you need to stage them in this way?... |
App Queue is always flooded with real time messages. The messages which are in DB are reprocess messages. So basically we want to process all reprocess messages when there is not much of real time traffic. |
And using message priority doesn't help because.....?
| deepak_paul wrote: |
| One other solution i think of at this moment is MQ Event triggering. Need to investigate more on how it can satisfy our scenarios. |
You could use a threshold to detect when the queue depth drops below a given point.
What do you do when the queue's mostly empty, you drop 10,000 DB messages on it and then 10,000 realtime messages show up? How do you plan to manage that scenario? Say at the start of the online day, when you're happily pushing DB messages onto the queue and that 8am rush shows up. All the realtime traffic will be stuck behind the DB messages.
Unless you're using message priority. When they're not. And you can just dump all the DB messages onto the queue and let them wait their turn.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| deepak_paul |
| Posted: Thu Jan 23, 2014 12:32 pm Post subject: |
|
|
Centurion
Joined: 04 Oct 2008
Posts: 147
Location: US
|
| Vitor wrote: |
And using message priority doesn't help because.....?
|
Very good point. Thanks Vitor. So setting the priority to lower value for the reprocess messages will solve it without any staging in DB?
| Vitor wrote: |
You could use a threshold to detect when the queue depth drops below a given point.
|
Wonder if MQ event trigger supports to detect if the queue depth drops below a given point since infocenter docs say that it is always same or above the given point. Vitor, Could you help me how?
| Vitor wrote: |
What do you do when the queue's mostly empty, you drop 10,000 DB messages on it and then 10,000 realtime messages show up? How do you plan to manage that scenario? Say at the start of the online day, when you're happily pushing DB messages onto the queue and that 8am rush shows up. |
Good thought. I think we can solve it by sending a small chunk of messages(~500/1000 msgs) every 5 mins during the time where traffic is low.
_________________
Regards
Paul |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jan 23, 2014 12:40 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| deepak_paul wrote: |
| Vitor wrote: |
And using message priority doesn't help because.....?
|
Very good point. Thanks Vitor. So setting the priority to lower value for the reprocess messages will solve it without any staging in DB? |
Yes, unless you want to stage them in a database to start overnight / at a weekend / during known quiet times / to allow users to query & resolve issues before reprocessing.
But if you put messages to be reprocessed back on the queue at the lowest priority, they'll only be reprocessed when all the higher priority messages are gone. So:
- you need to ensure the real time messages have a higher priority
- the queue is using priority not FIFO
| deepak_paul wrote: |
| Vitor wrote: |
You could use a threshold to detect when the queue depth drops below a given point.
|
Wonder if MQ event trigger supports to detect if the queue depth drops below a given point since infocenter docs say that it is always same or above the given point. Vitor, Could you help me how? |
Not sure I understand the question, and I was speculating. If I was doing this, I'd use message priority.
| deepak_paul wrote: |
| Vitor wrote: |
What do you do when the queue's mostly empty, you drop 10,000 DB messages on it and then 10,000 realtime messages show up? How do you plan to manage that scenario? Say at the start of the online day, when you're happily pushing DB messages onto the queue and that 8am rush shows up. |
Good thought. I think we can solve it by sending a small chunk of messages(~500/1000 msgs) every 5 mins during the time where traffic is low. |
If you're going to use message priority, you don't need to solve this. Because the 10,000 reatime messages will have a higher priority and be immediately processed. The 10,000 DB messages (or however many of them are left) will wait.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Jan 23, 2014 1:49 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
IIB (WMB v9) has workload management built in.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
|
|
