ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportDelay a MQ processing flow with ESQL SLEEP

Post new topicReply to topic Goto page 1, 2  Next
Delay a MQ processing flow with ESQL SLEEP View previous topic :: View next topic
Author Message
hunghip95
PostPosted: Mon Nov 16, 2020 11:54 pm Post subject: Delay a MQ processing flow with ESQL SLEEP Reply with quote

Apprentice

Joined: 12 Oct 2017
Posts: 29

Hi all,
I want to share you all a process and expect that whether there is a better suggestion.
Requirement:
- Develop an asynchronous service using MQ Input. This service gets message by a delay time, this is for purpose of throttling messages come to "Business Process" sector. The "Business Process" sector is so weak for a huge throughput. For example: This service reads message after each 10s. This delay time is supposed to be adjusted during runtime.

Solution:
- There are 2 services that I develop for this requirement.
- Service 1:
MQInput(queueA.in) -> MQOutput(queueB.in) -> ESLQ( sleep 10s) -> MQGet (queueB.out) -> MQOutput(queueA.out).
- Service 2:
MQInput(queueB.in)-> Business Process -> MQOutput(queueB.out).

Description:
1. Client put request message to queueA.in. The transaction begins from MQInput at Service 1.
2. Service 1 put message to queueB.in and then sleep 10s which is as same as the delay time. I configure this parameter as a user-defined property. Then this service wait message(by MQMD.messageID) in MQGet(queueB.out) node. Once it gets the message from MQGet node, it response to queueA.out.
3. Service 2 get message from queueB.in and process message, then response it which remains the MQMD.messageID to queueB.out.

Explanation:
- Service 1 allows multiple processing. It sleeps 10s before get response from Service 2.
- There are 3 case of getting response: ( assume that Service 2 takes k serconds for processing, timeout setting at MQGet node in Service 1 is p).
+ If k<10s, Service 1 will get next request after10s.
+ If k>10s, Service 1 will get next request after k seconds.
+ If k>p, Service 1 will get next request after p seconds.

My question is:
1. Is it a good idea when using sleep for multiple processing ?
2. Is there a better solution for delaying time ?

Thank you all


Last edited by hunghip95 on Mon Nov 23, 2020 6:44 am; edited 1 time in total
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Nov 17, 2020 5:59 am Post subject: Re: Delay a MQ processing flow with ESQL SLEEP Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26077
Location: Texas, USA

hunghip95 wrote:
1. Is it a good idea when using sleep for multiple processing ?


No. It stops a thread which isn't doing anything from being paged out and wastes resources, as well as slowing throughput.

hunghip95 wrote:
2. Is there a better solution for delaying time ?


I'm having a hard time understanding the problem you're trying to solve, and you probably need to take another run up at explaining it. Age has weakened my already diminished mental powers you see....

.so stop me when I go wrong.

So you've got multiple instances of Service 1, which reads a message, sends a request to Service 2, gets a response from Service 2 by id and then replies itself. The reply from Service 2 is retrieved by an MQGet node with a timeout set.

I don't see why you sleep for 10 seconds before the MQGet. If Service 2 takes at least 10 seconds to respond, add 10 seconds to the MQGet wait period.

That's the bit you need to explain. Because an MQGet in a wait state can be paged out, unlike a SLEEP.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Nov 17, 2020 2:51 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

hunghip95 wrote:
Hi all,
I want to share you all a process and expect that whether there is a better suggestion.
Requirement:
- Develop an asynchronous service using MQ Input. This service gets message by a delay time. For example: This service reads message after each 10s.

You haven't explained the requirement for 10 seconds. Why not 1? Why not 30?

Sleep is generally a bad idea in any messaging or asynchronous solution. There is always a better way.
_________________
Glenn
Back to top
View user's profile Send private message
hunghip95
PostPosted: Mon Nov 23, 2020 6:48 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2017
Posts: 29

Sorry for lacking some information.
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput.
So that's why I sleeping before MQGet node. Besides, I configure this delaytime as a user-defined property, then I can adjust that during runtime.
Back to top
View user's profile Send private message
hunghip95
PostPosted: Mon Nov 23, 2020 6:50 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2017
Posts: 29

Quote:
You haven't explained the requirement for 10 seconds. Why not 1? Why not 30?


This parameter can be adjusted during the runtime. 10s is my example.
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Nov 23, 2020 10:55 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26077
Location: Texas, USA

hunghip95 wrote:
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput.


If you want to stop the flow overloading a descendent process, IBM provides Workload Management policies which are intended for that very purpose.

Much better than a SLEEP statement.


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Nov 23, 2020 11:45 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20423
Location: LI,NY

Vitor wrote:
hunghip95 wrote:
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput.


If you want to stop the flow overloading a descendent process, IBM provides Workload Management policies which are intended for that very purpose.

Much better than a SLEEP statement.



Sometimes even better than that, have the downstream process consume from a queue. This way it can consume at it's maximum speed and the messages accumulate on the queue while waiting...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Mon Nov 23, 2020 12:45 pm Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26077
Location: Texas, USA

fjb_saper wrote:
Sometimes even better than that, have the downstream process consume from a queue. This way it can consume at it's maximum speed and the messages accumulate on the queue while waiting...


According to the OP, it already is. My assumption is the downstream service is so tragic it can't keep up, you get a full queue leading to exponential SLA increase and people bleating like you can't just put more disc in the queue & let the poor thing finish. So you've got 300k messages in a queue and it will take hours for the application to chew through them; has it got a doctor's appointment later? Has to pick its mother up from the station? Is a member of an application software union that won't let its members process for more than an hour at a time??



Seen the same thing myself first hand at this very site. Makes you weep.

Well it made me sarcastic but you get the idea.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Nov 23, 2020 2:04 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

hunghip95 wrote:
Hi all,
I want to share you all a process and expect that whether there is a better suggestion...

I don't understand why a delay is required.
Why don't you use
MQInput(queueA.in)-> Business Process -> MQOutput(queueA.out)
and run multiple instances of the flow?
If the BP can't keep up, the messages will queue up on queueA.in until BP capacity is available.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Mon Nov 23, 2020 2:55 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8822
Location: US: west coast, almost. Otherwise, enroute.

JMS2.0 implements a delivery delay. Could this be used at the front end?
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q119200_.htm
_________________
My life flows on in endless song;
How can I keep from singing?
Back to top
View user's profile Send private message
hunghip95
PostPosted: Mon Nov 23, 2020 6:46 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP Reply with quote

Apprentice

Joined: 12 Oct 2017
Posts: 29

gbaddeley wrote:
hunghip95 wrote:
Hi all,
I want to share you all a process and expect that whether there is a better suggestion...

I don't understand why a delay is required.
Why don't you use
MQInput(queueA.in)-> Business Process -> MQOutput(queueA.out)
and run multiple instances of the flow?
If the BP can't keep up, the messages will queue up on queueA.in until BP capacity is available.


The delay time allows the "Business Process" sector to keep handling messages at a slow speed. For example: with delay time = 10s, Service 1 has 1 instance, so maximal speed of "Business Process" sector is just 0.1message/s ( 10s for 1 message).

And you can speed it up by decreasing this delay time to be lower or to be 0.
Back to top
View user's profile Send private message
hunghip95
PostPosted: Mon Nov 23, 2020 6:53 pm Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2017
Posts: 29

bruce2359 wrote:
JMS2.0 implements a delivery delay. Could this be used at the front end?
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.dev.doc/q119200_.htm


Do you mean I should integrate ACE with MQ by using JMS nodes ?
I've not really worked with JMS nodes, is it better than MQ nodes ?
Back to top
View user's profile Send private message
hunghip95
PostPosted: Tue Nov 24, 2020 12:44 am Post subject: Reply with quote

Apprentice

Joined: 12 Oct 2017
Posts: 29

Vitor wrote:
hunghip95 wrote:
The purpose of sleeping before MQGet node is throttling messages come to Service 2 ( particularly is the "Business Process" sector).
Imagine that "Business Process" sector is a Core system which just can handle a small throughput.


If you want to stop the flow overloading a descendent process, IBM provides Workload Management policies which are intended for that very purpose.

Much better than a SLEEP statement.



Do you mean that I should use maximumRateMsgsPerSec properties ?
Hmm, I've not tested about this. I'm not sure that it works with Message flows which has MQInput node.
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Nov 24, 2020 5:30 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26077
Location: Texas, USA

[quote="hunghip95"]Do you mean that I should use maximumRateMsgsPerSec properties ?]/quote]

Yes.

hunghip95 wrote:
Hmm, I've not tested about this.


Good time to start.

hunghip95 wrote:
I'm not sure that it works with Message flows which has MQInput node.


Given that by your own admission you've not tested this, point me to where you are getting that uncertainty from.

Also, as a general point, explain to me why IBM would release a feature that has a very large limitation (given how many flows in the ecosystem use MQInput). Then fail to mention that fairly clearly in the documentation.

If that's a left handed way of asking if I've tested it in a flow with an MQInput node to save yourself some effort, yes I have.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Nov 24, 2020 1:57 pm Post subject: Re: Delay a MQ processing flow with ESQL SLEEP Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

hunghip95 wrote:
gbaddeley wrote:
hunghip95 wrote:
Hi all,
I want to share you all a process and expect that whether there is a better suggestion...

I don't understand why a delay is required.
Why don't you use
MQInput(queueA.in)-> Business Process -> MQOutput(queueA.out)
and run multiple instances of the flow?
If the BP can't keep up, the messages will queue up on queueA.in until BP capacity is available.


The delay time allows the "Business Process" sector to keep handling messages at a slow speed. For example: with delay time = 10s, Service 1 has 1 instance, so maximal speed of "Business Process" sector is just 0.1message/s ( 10s for 1 message).

And you can speed it up by decreasing this delay time to be lower or to be 0.

Can you provide more information about how the Business Process interface works? Is it a web service call, and the message flow waits for a response? Is the web service single threaded (can only handle 1 request at a time) ?

We should try to find a solution using MQ queueing that does not require dodgy rate throttling or sleeping.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexWebSphere Message Broker SupportDelay a MQ processing flow with ESQL SLEEP
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.