Author |
Message
|
akulavijay |
Posted: Mon Sep 22, 2014 2:59 pm Post subject: MQ Input Browse one message at a Time |
|
|
Newbie
Joined: 17 Jul 2014 Posts: 9
|
Is there a way by which I could ensure the MQ Input node to browse one message at a time. For the failed message retry I have a browse only MQ Input with browse timeout, a Filter, a MQ get, a Compute and a MQ Output. After browsing, in the filter if the delay is met then MQ get will get it and put it back onto either the original queue or failed queue. This is working fine. But there is a requirement to delay all of the retry message for about 5 minutes during the restart of the Broker, using the ESQL caching I have implemented a restart delay and is working fine but the browse only MQ Input is browing a chunk of the messages and testing for the restart delay this might consume the broker resources during the start up, is there a way I can force the Browse only MQ Input node to browse one message at a time? ie keep browsing the first message till an MQ Get picks up the first message.
Thanks |
|
Back to top |
|
 |
smdavies99 |
Posted: Mon Sep 22, 2014 9:58 pm Post subject: |
|
|
 Jedi Council
Joined: 10 Feb 2003 Posts: 6076 Location: Somewhere over the Rainbow this side of Never-never land.
|
Quote: |
But there is a requirement to delay all of the retry message for about 5 minutes during the restart of the Broker, using the ESQL caching I have implemented a restart delay |
I get a horrible feeling that this statement is either wrong or that you don't fully understand what happens when a broker is stopped. (It might be the time of day and the lack of though)
Can you give us more details of what you mean by ESQL Caching and how it works over a broker restart. _________________ WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
Back to top |
|
 |
Vitor |
Posted: Tue Sep 23, 2014 4:42 am Post subject: Re: MQ Input Browse one message at a Time |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
akulavijay wrote: |
Is there a way by which I could ensure the MQ Input node to browse one message at a time. For the failed message retry I have a browse only MQ Input with browse timeout, a Filter, a MQ get, a Compute and a MQ Output. After browsing, in the filter if the delay is met then MQ get will get it and put it back onto either the original queue or failed queue. This is working fine. |
Define "fine"!
Really? With all the built-in capabilities for handling failed message retry this complex collection of nodes is the best you could find to meet your requirements? If it works then clearly it's meeting a key requirement, but I would doubt that something with so many moving parts works reliably under all circumstances. As well as my innate suspicision of an MQInput node that browses. But if you want to control message retry timing, what's your aversion to a Timer node? Why this complex mix of stuff?
akulavijay wrote: |
But there is a requirement to delay all of the retry message for about 5 minutes during the restart of the Broker |
Why? What's the reason for this requirement? Acceptable answers do not include "it's the requirement I've been given" - what is this intended to achieve?
akulavijay wrote: |
using the ESQL caching |
I agree with my worthy associate - what's ESQL caching?
akulavijay wrote: |
I have implemented a restart delay and is working fine but the browse only MQ Input is browing a chunk of the messages and testing for the restart delay this might consume the broker resources during the start up, is there a way I can force the Browse only MQ Input node to browse one message at a time? ie keep browsing the first message till an MQ Get picks up the first message. |
Of course not. The MQInput node is going to keep reading (or in this instance) browsing messages as designed. This sounds a lot like an example of a circumstance where this "working fine" design isn't working that fine.
Assuming there's actually some kind of genuine reason why the flow can't start processing until 5 minutes after broker restart. you could achieve this if you used the in built facilities. I don't see how you'll achieve it with the solution you have.
This could be because I've never heard of ESQL caching. You need to explain that. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
akulavijay |
Posted: Tue Sep 23, 2014 7:14 pm Post subject: |
|
|
Newbie
Joined: 17 Jul 2014 Posts: 9
|
|
Back to top |
|
 |
akulavijay |
Posted: Tue Sep 23, 2014 8:14 pm Post subject: |
|
|
Newbie
Joined: 17 Jul 2014 Posts: 9
|
@Vitor
Quote: |
Assuming there's actually some kind of genuine reason why the flow can't start processing until 5 minutes after broker restart. you could achieve this if you used the in built facilities. I don't see how you'll achieve it with the solution you have. |
Please provide some details on the in built options that could potentially be used instead |
|
Back to top |
|
 |
smdavies99 |
Posted: Tue Sep 23, 2014 10:36 pm Post subject: |
|
|
 Jedi Council
Joined: 10 Feb 2003 Posts: 6076 Location: Somewhere over the Rainbow this side of Never-never land.
|
ESQL Caching as described in the article will not survive a broker restart. That was a point that I made before and one that you said in your original post was what you wanted to achieve. _________________ WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
Back to top |
|
 |
Vitor |
Posted: Wed Sep 24, 2014 5:05 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
smdavies99 wrote: |
ESQL Caching as described in the article will not survive a broker restart. That was a point that I made before and one that you said in your original post was what you wanted to achieve. |
I don't see how you can say:
akulavijay wrote: |
using the ESQL caching I have implemented a restart delay and is working fine |
when the "caching" solution is using shared variables. The shared variables will not exist prior to restart so the best I believe you can be doing is humping everything you read into shared variables for the first 5 minutes.
Explain your solution in more detail please. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
Vitor |
Posted: Wed Sep 24, 2014 5:11 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
That's a possible solution for a failed message retry and has it's merits; using the timing nodes is another and also has it's merits. I personally prefer the latter as it's cheaper in terms of resources, clearer how many retry messages exist and the timing survives a broker restart, where a browse cursor will not. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
Vitor |
Posted: Wed Sep 24, 2014 5:13 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
akulavijay wrote: |
@Vitor
Quote: |
Assuming there's actually some kind of genuine reason why the flow can't start processing until 5 minutes after broker restart. you could achieve this if you used the in built facilities. I don't see how you'll achieve it with the solution you have. |
Please provide some details on the in built options that could potentially be used instead |
Still assuming there's a genuine reason for this delay you can use timing events to control the release of messages.
Or upgrade to IIBv9 and configure the flow not to start automatically when the broker does.
Or just accept you don't need to wait 5 minutes before you start processing messages. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
akulavijay |
Posted: Wed Sep 24, 2014 6:56 pm Post subject: |
|
|
Newbie
Joined: 17 Jul 2014 Posts: 9
|
Yes, the SHARED variables are wiped out during the restart or flow redeploy, when the SHARED variable is NULL the assumption is broker is restarted or the flow redeployed hence the filter will return false till the SHARED variable for ex: CACHE_FLOWNAME_ RESTARTTIME is NOT NULL and difference with CURRENT_TIMESTAMP is more than the BOOT delay. Ofcourse during the first browse (after restart) CACHE_FLOWNAME_ RESTARTTIME will be set to CURRENT_TIMESTAMP. During the subsequent browses this time stamp is compared to CURRENT_TIMESTAMP. This is just as described in the article for determining the retry delay. |
|
Back to top |
|
 |
Vitor |
Posted: Thu Sep 25, 2014 4:58 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
akulavijay wrote: |
Yes, the SHARED variables are wiped out during the restart or flow redeploy, when the SHARED variable is NULL the assumption is broker is restarted or the flow redeployed hence the filter will return false till the SHARED variable for ex: CACHE_FLOWNAME_ RESTARTTIME is NOT NULL and difference with CURRENT_TIMESTAMP is more than the BOOT delay. Ofcourse during the first browse (after restart) CACHE_FLOWNAME_ RESTARTTIME will be set to CURRENT_TIMESTAMP. During the subsequent browses this time stamp is compared to CURRENT_TIMESTAMP. This is just as described in the article for determining the retry delay. |
Ok.
Moving aside from discussions of the desiability of this solution, and back to your original post, the MQInput node is going to browse the next available message from the queue each time control returns to it. Which means you're going to pile up resources as yoou describe during this start up delay. So if you want to prevent that you need to change your logic so that instead if the current timestamp is less than the restart time, it calculates the interval between the two times and sleeps for that period. Hence control never leaves the node and the next message is not browsed.
A number of people are about to post what a bad idea it is to use sleep and in good conscience I can't disagree with them. But if you're determined to introduce this 5 minute delay then you need to choose the least worst option, and accept all of them will cause some resources or another to burn during this period. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
akulavijay |
Posted: Thu Sep 25, 2014 9:40 pm Post subject: |
|
|
Newbie
Joined: 17 Jul 2014 Posts: 9
|
Thanks Vitor for the feedback and suggestion. I am looking at another option with Timeout Notification. The flow would would look like this
A Timeout Notification -> Java Compute Node (Checks the interval, current depth and take decision to stop or move forward till the depth is 0, if timer kicks in mean while using shred variable I can stop moving forward) -> MQ Get -> Compute (to check the retry count) -> MQ Put.
Not sure if there is any context switching because of timer issuing the check for the queue depth.
Not a great option but we are still on 8. Please let me know your feedback |
|
Back to top |
|
 |
Vitor |
Posted: Fri Sep 26, 2014 4:58 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
akulavijay wrote: |
Thanks Vitor for the feedback and suggestion. I am looking at another option with Timeout Notification. The flow would would look like this
A Timeout Notification -> Java Compute Node (Checks the interval, current depth and take decision to stop or move forward till the depth is 0, if timer kicks in mean while using shred variable I can stop moving forward) -> MQ Get -> Compute (to check the retry count) -> MQ Put.
Not sure if there is any context switching because of timer issuing the check for the queue depth. |
Ok, this is slightly more insane than your previous design and makes me regret I suggested timing events.
First and most important - don't access the queue manager from a Java Compute Node - using anything other than the supplied nodes endangers broker stability.
Secondly, if you're using a Timeout notification, why are you checking the interval? You know the interval because it's the interval after which the Timeout notification is produced.
Thirdly, yes there is context switching because the notification is in a different thread. Again, don't check the depth. No application that's not an administration application should ever need to know the queue depth and if the application is broker it can be fatal. Also remember that the depth you get back is not an accurate number; it can be affected by uncommitted messages and other factors.
Fourthly, you've still not explained why you need this post start up delay. What does waiting 5 minutes achieve that starting immediately does not? _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
mqjeff |
Posted: Fri Sep 26, 2014 5:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008 Posts: 17447
|
To be fair, akulavijay is not using a JCN to access the queue. There is an MQGet node in there.
But otherwise, yes, a lot of the things that the JCN are doing is likely unecessary. The most it should be doing is a propagate in a loop that ends when the MQGet node throws a 2033. |
|
Back to top |
|
 |
Vitor |
Posted: Fri Sep 26, 2014 5:22 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
mqjeff wrote: |
To be fair, akulavijay is not using a JCN to access the queue. There is an MQGet node in there. |
akulavijay wrote: |
Java Compute Node (Checks the interval, current depth and take decision to stop or move forward till the depth is 0 |
If I misinterpreted that phrase, I think it's understandable. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
|