| Author |
Message
|
| hunghip95 |
 Posted: Mon Nov 16, 2020 11:54 pm Post subject: Delay a MQ processing flow with ESQL SLEEP |
 |
|
Apprentice
Joined: 12 Oct 2017
Posts: 30
|
Hi all,
I want to share you all a process and expect that whether there is a better suggestion.
Requirement:
- Develop an asynchronous service using MQ Input. This service gets message by a delay time, this is for purpose of throttling messages come to "Business Process" sector. The "Business Process" sector is so weak for a huge throughput. For example: This service reads message after each 10s. This delay time is supposed to be adjusted during runtime.
Solution:
- There are 2 services that I develop for this requirement.
- Service 1:
MQInput(queueA.in) -> MQOutput(queueB.in) -> ESLQ( sleep 10s) -> MQGet (queueB.out) -> MQOutput(queueA.out).
- Service 2:
MQInput(queueB.in)-> Business Process -> MQOutput(queueB.out).
Description:
1. Client put request message to queueA.in. The transaction begins from MQInput at Service 1.
2. Service 1 put message to queueB.in and then sleep 10s which is as same as the delay time. I configure this parameter as a user-defined property. Then this service wait message(by MQMD.messageID) in MQGet(queueB.out) node. Once it gets the message from MQGet node, it response to queueA.out.
3. Service 2 get message from queueB.in and process message, then response it which remains the MQMD.messageID to queueB.out.
Explanation:
- Service 1 allows multiple processing. It sleeps 10s before get response from Service 2.
- There are 3 case of getting response: ( assume that Service 2 takes k serconds for processing, timeout setting at MQGet node in Service 1 is p).
+ If k<10s, Service 1 will get next request after10s.
+ If k>10s, Service 1 will get next request after k seconds.
+ If k>p, Service 1 will get next request after p seconds.
My question is:
1. Is it a good idea when using sleep for multiple processing ?
2. Is there a better solution for delaying time ?
Thank you all
Last edited by hunghip95 on Mon Nov 23, 2020 6:44 am; edited 1 time in total |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Nov 17, 2020 5:59 am Post subject: Re: Delay a MQ processing flow with ESQL SLEEP |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hunghip95 wrote: |
| 1. Is it a good idea when using sleep for multiple processing ? |
No. It stops a thread which isn't doing anything from being paged out and wastes resources, as well as slowing throughput.
| hunghip95 wrote: |
| 2. Is there a better solution for delaying time ? |
I'm having a hard time understanding the problem you're trying to solve, and you probably need to take another run up at explaining it. Age has weakened my already diminished mental powers you see....
….so stop me when I go wrong.
So you've got multiple instances of Service 1, which reads a message, sends a request to Service 2, gets a response from Service 2 by id and then replies itself. The reply from Service 2 is retrieved by an MQGet node with a timeout set.
I don't see why you sleep for 10 seconds before the MQGet. If Service 2 takes at least 10 seconds to respond, add 10 seconds to the MQGet wait period.
That's the bit you need to explain. Because an MQGet in a wait state can be paged out, unlike a SLEEP.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Nov 17, 2020 2:51 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| hunghip95 wrote: |
Hi all,
I want to share you all a process and expect that whether there is a better suggestion.
Requirement:
- Develop an asynchronous service using MQ Input. This service gets message by a delay time. For example: This service reads message after each 10s. |
You haven't explained the requirement for 10 seconds. Why not 1? Why not 30?
Sleep is generally a bad idea in any messaging or asynchronous solution. There is always a better way.
_________________
Glenn |
|
| Back to top |
|
|
| hunghip95 |
| Posted: Mon Nov 23, 2020 6:48 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2017
Posts: 30
|
Sorry for lacking some information.
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput.
So that's why I sleeping before MQGet node. Besides, I configure this delaytime as a user-defined property, then I can adjust that during runtime. |
|
| Back to top |
|
|
| hunghip95 |
| Posted: Mon Nov 23, 2020 6:50 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2017
Posts: 30
|
| Quote: |
| You haven't explained the requirement for 10 seconds. Why not 1? Why not 30? |
This parameter can be adjusted during the runtime. 10s is my example. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Nov 23, 2020 10:55 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hunghip95 wrote: |
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput. |
If you want to stop the flow overloading a descendent process, IBM provides Workload Management policies which are intended for that very purpose.
Much better than a SLEEP statement.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Nov 23, 2020 11:45 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Vitor wrote: |
| hunghip95 wrote: |
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput. |
If you want to stop the flow overloading a descendent process, IBM provides Workload Management policies which are intended for that very purpose.
Much better than a SLEEP statement.
|
Sometimes even better than that, have the downstream process consume from a queue. This way it can consume at it's maximum speed and the messages accumulate on the queue while waiting... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Nov 23, 2020 12:45 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
| Sometimes even better than that, have the downstream process consume from a queue. This way it can consume at it's maximum speed and the messages accumulate on the queue while waiting... |
According to the OP, it already is. My assumption is the downstream service is so tragic it can't keep up, you get a full queue leading to exponential SLA increase and people bleating like you can't just put more disc in the queue & let the poor thing finish. So you've got 300k messages in a queue and it will take hours for the application to chew through them; has it got a doctor's appointment later? Has to pick its mother up from the station? Is a member of an application software union that won't let its members process for more than an hour at a time??
Seen the same thing myself first hand at this very site. Makes you weep.
Well it made me sarcastic but you get the idea.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Nov 23, 2020 2:04 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| hunghip95 wrote: |
Hi all,
I want to share you all a process and expect that whether there is a better suggestion... |
I don't understand why a delay is required.
Why don't you use
MQInput(queueA.in)-> Business Process -> MQOutput(queueA.out)
and run multiple instances of the flow?
If the BP can't keep up, the messages will queue up on queueA.in until BP capacity is available.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Mon Nov 23, 2020 2:55 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| hunghip95 |
| Posted: Mon Nov 23, 2020 6:46 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP |
|
|
Apprentice
Joined: 12 Oct 2017
Posts: 30
|
| gbaddeley wrote: |
| hunghip95 wrote: |
Hi all,
I want to share you all a process and expect that whether there is a better suggestion... |
I don't understand why a delay is required.
Why don't you use
MQInput(queueA.in)-> Business Process -> MQOutput(queueA.out)
and run multiple instances of the flow?
If the BP can't keep up, the messages will queue up on queueA.in until BP capacity is available. |
The delay time allows the "Business Process" sector to keep handling messages at a slow speed. For example: with delay time = 10s, Service 1 has 1 instance, so maximal speed of "Business Process" sector is just 0.1message/s ( 10s for 1 message).
And you can speed it up by decreasing this delay time to be lower or to be 0. |
|
| Back to top |
|
|
| hunghip95 |
| Posted: Mon Nov 23, 2020 6:53 pm Post subject: |
|
|
Apprentice
Joined: 12 Oct 2017
Posts: 30
|
Do you mean I should integrate ACE with MQ by using JMS nodes ?
I've not really worked with JMS nodes, is it better than MQ nodes ? |
|
| Back to top |
|
|
| hunghip95 |
| Posted: Tue Nov 24, 2020 12:44 am Post subject: |
|
|
Apprentice
Joined: 12 Oct 2017
Posts: 30
|
| Vitor wrote: |
| hunghip95 wrote: |
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput. |
If you want to stop the flow overloading a descendent process, IBM provides Workload Management policies which are intended for that very purpose.
Much better than a SLEEP statement.
|
Do you mean that I should use maximumRateMsgsPerSec properties ?
Hmm, I've not tested about this. I'm not sure that it works with Message flows which has MQInput node. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 24, 2020 5:30 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
[quote="hunghip95"]Do you mean that I should use maximumRateMsgsPerSec properties ?]/quote]
Yes.
| hunghip95 wrote: |
| Hmm, I've not tested about this. |
Good time to start.
| hunghip95 wrote: |
| I'm not sure that it works with Message flows which has MQInput node. |
Given that by your own admission you've not tested this, point me to where you are getting that uncertainty from.
Also, as a general point, explain to me why IBM would release a feature that has a very large limitation (given how many flows in the ecosystem use MQInput). Then fail to mention that fairly clearly in the documentation.
If that's a left handed way of asking if I've tested it in a flow with an MQInput node to save yourself some effort, yes I have.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Nov 24, 2020 1:57 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| hunghip95 wrote: |
| gbaddeley wrote: |
| hunghip95 wrote: |
Hi all,
I want to share you all a process and expect that whether there is a better suggestion... |
I don't understand why a delay is required.
Why don't you use
MQInput(queueA.in)-> Business Process -> MQOutput(queueA.out)
and run multiple instances of the flow?
If the BP can't keep up, the messages will queue up on queueA.in until BP capacity is available. |
The delay time allows the "Business Process" sector to keep handling messages at a slow speed. For example: with delay time = 10s, Service 1 has 1 instance, so maximal speed of "Business Process" sector is just 0.1message/s ( 10s for 1 message).
And you can speed it up by decreasing this delay time to be lower or to be 0. |
Can you provide more information about how the Business Process interface works? Is it a web service call, and the message flow waits for a response? Is the web service single threaded (can only handle 1 request at a time) ?
We should try to find a solution using MQ queueing that does not require dodgy rate throttling or sleeping.
_________________
Glenn |
|
| Back to top |
|
|
|
|
