| Author |
Message
|
| anveshita |
 Posted: Fri Apr 22, 2011 9:20 am Post subject: workload management question |
 |
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
My knowledge of Datapower is limited, but I would like to know how DP can be used in this particular situation.
I have a queue Q where I get messages which are a mix of small and large. The small messages (Msg_Small) constitutes say 80% . Eash Msg_Small gets processed in 1 sec and large messages (Msg_Large) constitues 20%. Eash Msg_Large gets processed in 5-6 sec.
For the time being let us assume I could identify the type of the message(MSG_large vs MSg_small)before feeding to the queue Q.
The messages from Q are sent to queues Q1,Q2,Q3,Q4 in a round robin fashion.
We have consumers P1,P2,P3 and P4 that consume the messages from queues Q1,Q2,Q3,Q4.
Now the issue is since I have a mix of Msg_Large and Msg_Small messages with different processing times.
By having a round-robin logic in feeding the queues Q1,Q2,Q3 and Q4, I am finding most of the times Msg_Smalls are getting stuck behing Msg_Larges and resulting in timeouts.
Now instead of using a round-robin routing logic, I would like to some type of intelligent router say (IR).
I want to allocate Q1 and Q2 for processing Msg_Small and allocate Q3 and Q4 for processing msg_Large. Msg_small can be routed to Q3 and Q4 if the queues are empty
IR will send messages to Q1,Q2, Q3 and Q4 based on the queue depth instead of round robin. In case of
Msg_small, since these are of high volume, we will route the messages to Q3 and Q4 if the queue depth is zero, so that the process P3 and P4 can process these small messages. But Msg_Large will always go to queues Q3 and Q4.
Now my questions:
1. Is this a right approach for work load balancing or are there any established patterns for such sort of situations?
2. Can the IR router I indicated above can be done on Datapower? Can the configuration changes can be changed on the fly. Think of as property file changes for a running java program. For example I may wish to revert to round-robin instead of IR logic.
3. If #2 is yes, then can you indicate the steps in general one has to follow to set up the router? ( I can go read the manuals but need the steps so I know what to read.)
Please let me know your comments |
|
| Back to top |
|
 |
| anveshita |
| Posted: Tue Apr 26, 2011 7:52 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Tue Apr 26, 2011 7:59 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Have you considered using message priorities?
It may give you a simpler interface.
How quickly do the smaller messages need to be processed? |
|
| Back to top |
|
|
| anveshita |
| Posted: Tue Apr 26, 2011 9:39 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
Thanks.
| Quote: |
Have you considered using message priorities?
|
Could you please explain. I was told assigning priorities and polling the messages based on priority would slow down the process and is not a best practice. ( Not sure where I read).
The smaller ones need to be processed in 0.6 sec |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 26, 2011 2:15 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
You don't poll. MQ will put the messages with the higher priorities into the queues in front of the lower priorities. Your consumers just process the messages as they are on the queue. Small messages with higher priorities will be always in front of large messages with lower priorities. But if a consumer is already working on a large message, small messages will wait on that queue until the consumer is done with the large. Either have multiple instances of the consuming app watching each queue, so there is always an available thread waiting for a new small message, or split the messages into 2 separate sets of queues - 1 set for small and 1 set for big, and use something like WMB to route them to the correct queues.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Apr 26, 2011 8:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PeterPotkay wrote: |
| You don't poll. MQ will put the messages with the higher priorities into the queues in front of the lower priorities. Your consumers just process the messages as they are on the queue. Small messages with higher priorities will be always in front of large messages with lower priorities. But if a consumer is already working on a large message, small messages will wait on that queue until the consumer is done with the large. Either have multiple instances of the consuming app watching each queue, so there is always an available thread waiting for a new small message, or split the messages into 2 separate sets of queues - 1 set for small and 1 set for big, and use something like WMB to route them to the correct queues. |
The assumption here is that your message delivery method for the queue is the default (priority) and not set to FIFO.
Also having a consumer with a selector on high priority can assure you that only messages with high priority will be processed by this consumer. You can then have an additional consumer (either without selector or with a selector on low priority) will allow you to consume the other messages.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| anveshita |
| Posted: Wed Apr 27, 2011 11:49 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
PeterPotkay:
Message priorties may help, but I am leaning towards seperating the messages to different queues. But the downside will be what to do if I need to classify messages further down the line, for ex: small,large and batch/online. Having separate queues for each classification may not scale well in such situations.
fjb_saper:
| Quote: |
| Also having a consumer with a selector on high priority can assure you that only messages with high priority will be processed by this consumer. You can then have an additional consumer (either without selector or with a selector on low priority) will allow you to consume the other messages. |
The consumer is a packaged application I do not have much control. It is a .NET app. I believe selectors are applicable to JMS clients in MQV6.0 which we are using. If I understand it is provided on MQ server side with MQOPEN calls going forward with MQ V 7.0. Hence this may not work for me.
As a follow up, indicated in OP, is it possible to implement "least queue depth" pattern efficiently. Means the PUT application would post to the queue with least number of messages contrary to the regular "round-robin pattern" we use. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Apr 27, 2011 4:28 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| anveshita wrote: |
As a follow up, indicated in OP, is it possible to implement "least queue depth" pattern efficiently. Means the PUT application would post to the queue with least number of messages contrary to the regular "round-robin pattern" we use. |
So you will be happy sending your next small message to a queue with a Q depth of 3 instead of a q depth of 4? What if the q with a Q depth of 3 has 3 large messages, and the 4 q depth is 4 small messages. (Don't propose to then browse each queue to see how big the messages are!)
Not only can you not rely on the q depth to tell you exactly how many available messages are really there, it will not perform well if you have to constantly poll each q depth to see what the # is before each and every put.
Polling for queue depth to make decisions on where to put messages is a bad design. If it was a good idea, IBM would have built that type of option into basic MQ clustering.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| anveshita |
| Posted: Thu Apr 28, 2011 4:57 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| Quote: |
| So you will be happy sending your next small message to a queue with a Q depth of 3 instead of a q depth of 4? What if the q with a Q depth of 3 has 3 large messages, and the 4 q depth is 4 small messages. (Don't propose to then browse each queue to see how big the messages are!) |
Not necessarily. I still need to have queues separated out for small and large. In some of the queuing theory simulations, I have seen an options to route messages to a queue with least depth. Instead of round-robin way. In my case I will have small messages sent to separate queues. With in the queues meant for small messages I would like to follow "least queue depth".
You are right IBM could have done this easily. But did not, I would like to know a reason if there is any. This is my idea, you are welcome to comment on it. Set up PCF events to be generated on a QM for queue depth and route the messages to a pre-defined queue say pcfQ. A trigger on PCF queue will invoke a program that reads the message and updates a table or file with the latest message depth. The router reads the table to know the queue with least depth and sends the message to it. I know I am missing lot of details, but that is just an idea. |
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Apr 28, 2011 5:17 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
| anveshita wrote: |
| This is my idea, you are welcome to comment on it. |
Ok, I will..... I think you're now trying to implement a solution for a problem that doesnt really exist and wasnt the original problem you presented.
In the process of doing this you are creating a support nightmare. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 28, 2011 5:29 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anveshita wrote: |
| You are right IBM could have done this easily. But did not, I would like to know a reason if there is any. |
Because it's not a good idea and not as easy as it sounds.
| anveshita wrote: |
| This is my idea, you are welcome to comment on it. Set up PCF events to be generated on a QM for queue depth and route the messages to a pre-defined queue say pcfQ. A trigger on PCF queue will invoke a program that reads the message and updates a table or file with the latest message depth. The router reads the table to know the queue with least depth and sends the message to it. I know I am missing lot of details, but that is just an idea. |
Let's see now:
- so you make a PCF call for each inbound message.
- once this call has executed and the message interpreted a table's updated
- this table is read and the message sent
All this takes how long? For each message?
What happens when you get high message volume and the table updates either single thread or deadlock?
What happens when the database is unavailable? At best you need to pay for sufficient hardware/software to make it HA, and in the event of a ham-fisted DBA it'll still go down.
As my associate says, this is a nightmare. It could probably be made to work, but it'll be a rod for your own back.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| anveshita |
| Posted: Thu Apr 28, 2011 5:30 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| WMBDEV1 wrote: |
| anveshita wrote: |
| This is my idea, you are welcome to comment on it. |
Ok, I will..... I think you're now trying to implement a solution for a problem that doesnt really exist and wasnt the original problem you presented.
In the process of doing this you are creating a support nightmare. |
Thanks.
But that I stated in my OP.
| Quote: |
IR will send messages to Q1,Q2, Q3 and Q4 based on the queue depth instead of round robin. In case of
Msg_small, since these are of high volume, we will route the messages to Q3 and Q4 if the queue depth is zero, so that the process P3 and P4 can process these small messages |
|
|
| Back to top |
|
|
| WMBDEV1 |
| Posted: Thu Apr 28, 2011 6:14 am Post subject: |
|
|
Sentinel
Joined: 05 Mar 2009
Posts: 888
Location: UK
|
Ok. It had been a while since I had read your OP but to me the real problem here is:
| Quote: |
| I am finding most of the times Msg_Smalls are getting stuck behing Msg_Larges and resulting in timeouts. |
We are really debating your solution to this issue now and trying to help you make a solution that is supportable going forward. |
|
| Back to top |
|
|
| anveshita |
| Posted: Thu Apr 28, 2011 8:31 am Post subject: |
|
|
Master
Joined: 27 Sep 2004
Posts: 254
Location: Jambudweepam
|
| WMBDEV1 wrote: |
| We are really debating your solution to this issue now and trying to help you make a solution that is supportable going forward. |
Thank you. I appreciate your input.
Taking that "least queue depth" is not a choice I am sticking to the round robin pattern.
Now referring back to my OP: The solution so far I have is:
--------------
Group Q1 and Q2 to handle small messages and Q3 and Q4 to handle large messages. Process messages from Q1 with consumer P1, Q2 with P2, Q3->P3,Q4->p4
-----------
I see a problem with the above. I can not configure P3 to process messages from other queues if there are no messages in Q3 as it is a packaged application. Now if Q3 is empty then I will not be using P3 at all. Ideally I would like small messages to go to Q3 and Q4 in case the queues are empty. Is there a better way to achieve this? or you have different approach to this problem. Please share |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 28, 2011 9:07 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| anveshita wrote: |
| Is there a better way to achieve this? or you have different approach to this problem. |
What proportion of small messages to large do you have? What is the difference in SLA?
Following from this:
- How did you arrive at 4 queues? Is that just for illustrative purposes in this thread?
- Why are there the same number of queues for large and small messages? As you talked about timeouts. I'm assuming here the small messages have a short SLA? Why do you not have 3 queues (using this example) working the short messages & 1 chewing through the long?
- Why is there only 1 processor per queue? Is this a restriction of the package or just how you've written this example? It's unusual for any application, even a package, to open a queue exclusively.
- Why are you so keen to avoid idle queues (by routing different message types)?
Consider this:
You have 2 service classes of message; large & small. We assume you can route them by size using WMB or similar (I don't know DataPower especially). One possible pattern is this
SQA (Small Queue A) serviced by Process 1, Process 2, Process 3
SQB with P4, P5, P6
SQC with P7, P8, P9
Then one Large Queue with enough processes that the large messages are always processed within SLA.
Clearly you could have many more queues with many more processes, probably split over a number of machines to maximize resources.
This is a basic illustration only, other solutions are undoubtably possible & may be better, no warranty express or implied is supplied with this solution and no responsibility is accepted for any loss or damage resulting from it's use howsoever caused.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
