| Author |
Message
|
| abhyyy |
 Posted: Mon Nov 28, 2011 8:18 am Post subject: Limit no. of messages/sec (throughput) to a specific value. |
 |
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
Hi Friends,
I have a WMB flow that throws 100 BLOB messages/ second to a TCPIP port (actually to SMS gateway) but the requirement is that if this flow throws more than 5 messages/sec to the SMS Gateway then SMS Gateway will only accept 5 messages in that second and discard all other messages..
Is there a way that I can assure that No. of messages Leaving from my message flow are 4-5 messages/second and not more than that.
_________________
----------------------
NeVeR StOp LeaRnInG. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Mon Nov 28, 2011 8:23 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
What will you do with the other 95, while you are waiting for the next second to occur?
what will happen if another 100 come in during that second, such that you previously had 95 unsent messages but now you have 195 unsent messages?
You can certainly do a number of different things to provide this kind of staggered output - the simplest being an ESQL sleep statement that will pause for 1 second or a 20th of a second...
But that doesn't actually address the problem, which is that you have a mismatch between the rate of input messages and the rate of output messages. |
|
| Back to top |
|
|
| kimbert |
| Posted: Mon Nov 28, 2011 8:30 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
I assume that your input is 'bursty' - the average rate of messages arriving in your flow is < 5 per second, but you get short-term bursts of high message rates.
I would put a counter into a shared variable, and increment it every the flow emits a message. After every 5 messages, store a time stamp, and check that at least one second has elapsed since the last time you did it. If not, SLEEP for the remainder of the one-second period. |
|
| Back to top |
|
|
| abhyyy |
| Posted: Mon Nov 28, 2011 8:34 am Post subject: |
|
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
Scenario is like :
Every morning I'll receive a file with 50,000 records and I have to process them and send them to SMS Gateway at a rate of 5 messages/second and not faster than that. What I'm doing is get all the messages from file to a Queue at once and then myflow reads from the queue one by one. Since flow is capable of processing 100 messages/ sec so it is throwing 100 messages/sec to SMS gateway which is not acceptable by gateway.
So can I use Sleep statement with suitable interval as u said ? Do you recommend that? or if wmb has anything which is well suited for this job, my roadblock will be removed 
_________________
----------------------
NeVeR StOp LeaRnInG. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Nov 28, 2011 9:54 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
- Process 5 messages
- Set Timeout Notification for n seconds
- End thread
- New thread wakes up
- Processes 5 messages
- Goes to sleep.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rbicheno |
| Posted: Tue Nov 29, 2011 3:00 am Post subject: |
|
|
Apprentice
Joined: 07 Jul 2009
Posts: 43
|
| You have decoupled the input and output using a queue to act as a buffer which is a good design. As suggested just use SLEEP function in ESQL or Thread.sleep in a Java compute node to pace your output flow. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 29, 2011 3:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If you have 50,000 input records at time 0, and you can send at most 5 of them per second, it will take you 10,000 seconds to send them all.
There are only 86,400 seconds in a day. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Nov 29, 2011 4:00 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
To avoid the WMB transactioanal issues inherent with partially processed files. I recommend that you process all the file records immediately and create one MQ message per record (to an intermediate queue), with no delays.
THEN you can process these messages through in a (separate) throttling flow to the final MQ output queue.
This way if someone wants to stop the flows or the broker, you won't have a massive delay, or transactional issues. Same applies to any failures.
Both flows run all the time, but it does solve a number of potential issues. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 29, 2011 4:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
To avoid the WMB transactioanal issues inherent with partially processed files. I recommend that you process all the file records immediately and create one MQ message per record (to an intermediate queue), with no delays.
THEN you can process these messages through in a (separate) throttling flow to the final MQ output queue.
This way if someone wants to stop the flows or the broker, you won't have a massive delay, or transactional issues. Same applies to any failures.
Both flows run all the time, but it does solve a number of potential issues. |
This appears to be what is already being done. |
|
| Back to top |
|
|
| abhyyy |
| Posted: Tue Nov 29, 2011 6:01 am Post subject: |
|
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
| mqjeff wrote: |
If you have 50,000 input records at time 0, and you can send at most 5 of them per second, it will take you 10,000 seconds to send them all.
There are only 86,400 seconds in a day. |
I know that's lousy, but that's what the telecom client wants.
_________________
----------------------
NeVeR StOp LeaRnInG. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 29, 2011 6:03 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| abhyyy wrote: |
| mqjeff wrote: |
If you have 50,000 input records at time 0, and you can send at most 5 of them per second, it will take you 10,000 seconds to send them all.
There are only 86,400 seconds in a day. |
I know that's lousy, but that's what the telecom client wants. |
I just hope that the last record in the file was not supposed to have been sent within the first 30 minutes...  |
|
| Back to top |
|
|
| abhyyy |
| Posted: Tue Nov 29, 2011 6:04 am Post subject: |
|
|
Voyager
Joined: 29 Sep 2011
Posts: 83
|
| zpat wrote: |
To avoid the WMB transactioanal issues inherent with partially processed files. I recommend that you process all the file records immediately and create one MQ message per record (to an intermediate queue), with no delays.
THEN you can process these messages through in a (separate) throttling flow to the final MQ output queue.
This way if someone wants to stop the flows or the broker, you won't have a massive delay, or transactional issues. Same applies to any failures.
Both flows run all the time, but it does solve a number of potential issues. |
Thanks.But output of my flow is directly to TCPIPOUTPUT node and messages are manipulated realtime depending on the reply received synchronously. So cant do that 
_________________
----------------------
NeVeR StOp LeaRnInG. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Tue Nov 29, 2011 6:50 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| abhyyy wrote: |
| zpat wrote: |
To avoid the WMB transactioanal issues inherent with partially processed files. I recommend that you process all the file records immediately and create one MQ message per record (to an intermediate queue), with no delays.
THEN you can process these messages through in a (separate) throttling flow to the final MQ output queue.
This way if someone wants to stop the flows or the broker, you won't have a massive delay, or transactional issues. Same applies to any failures.
Both flows run all the time, but it does solve a number of potential issues. |
Thanks.But output of my flow is directly to TCPIPOUTPUT node and messages are manipulated realtime depending on the reply received synchronously. So cant do that |
Seems like a design oversight problem. Rethink. Its really not that hard.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 29, 2011 6:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| lancelotlinc wrote: |
| abhyyy wrote: |
| zpat wrote: |
To avoid the WMB transactioanal issues inherent with partially processed files. I recommend that you process all the file records immediately and create one MQ message per record (to an intermediate queue), with no delays.
THEN you can process these messages through in a (separate) throttling flow to the final MQ output queue.
This way if someone wants to stop the flows or the broker, you won't have a massive delay, or transactional issues. Same applies to any failures.
Both flows run all the time, but it does solve a number of potential issues. |
Thanks.But output of my flow is directly to TCPIPOUTPUT node and messages are manipulated realtime depending on the reply received synchronously. So cant do that |
Seems like a design oversight problem. Rethink. Its really not that hard. |
I think both you and zpat are missing some understanding about what was said in the second post in this thread from abhyyy. The solution as described seems reasonable for the requirements given. The situation as described seems a bit unreasonable - but as long as the number of messages to send out does not grow by an order of magnitude and none of the messages are time sensitive, it's a workable solution to an ugly situation. |
|
| Back to top |
|
|
|
|
