| Author |
Message
|
| Sabina |
 Posted: Tue Sep 27, 2016 3:40 am Post subject: Is it possible to configure MQInput to process not at once? |
 |
|
Novice
Joined: 27 Sep 2016
Posts: 14
|
Hello, everyone!
I have a question concerning the capabilities of WebSphere message queues.
I know that MQInput usually reads the newly-come message instantly after it comes to the queue (let's assume that the queue was empty). But I need to have some control over it, for example, in some cases I need to process the message after a period of time, e.g. 15 minutes. Is it possible to configure this somehow? I need to control this from the program, e.g. in some cases set the queue to "wait 15 minutes before processing the message from the queue", or in the other case revert it back to the default "begin to process the message from the queue without delays until it becomes empty".
The original task is as follows: my service does some actions on each of the several entities. And in the end the entities for which those actions finished unsuccessfully (for example, due to system errors or something else) I need to automatically store somewhere (for example in queue) for further processing. The problem is if the outer system is shut down I would like to make those entities just hung up in the queue until the system becomes available.
And the other question is if it is suitable at all to use message queues for this kind of task.
Thanks. |
|
| Back to top |
|
 |
| adubya |
| Posted: Tue Sep 27, 2016 4:06 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
I love these crazy requirements 
You could use an MQInput node in "browse" mode to detect when the queue is populated and then kick off a timer to delay for a period of time before using an MQGet node to retrieve the msg. You can control how frequently the MQInput performs the browse.
Could get yourself into a whole heap of bother though.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| Sabina |
| Posted: Tue Sep 27, 2016 4:30 am Post subject: |
|
|
Novice
Joined: 27 Sep 2016
Posts: 14
|
| adubya wrote: |
I love these crazy requirements
You could use an MQInput node in "browse" mode to detect when the queue is populated and then kick off a timer to delay for a period of time before using an MQGet node to retrieve the msg. You can control how frequently the MQInput performs the browse.
Could get yourself into a whole heap of bother though. |
Thanks for your response!
The problem is, that this timer will apply for every new message that comes to MQInput. But I need to have some dynamic control over it. For example, I would need to set the timer to zero wait later in the flow, in case some specific event occcurs. |
|
| Back to top |
|
|
| adubya |
| Posted: Tue Sep 27, 2016 4:58 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
OK, another approach is to read each message as it arrives and then determine how long its processing needs to be delayed. Add content to the message representing that delay time and then add to a "delay" queue which is processed by a separate flow using a browse/interrogate/skip or process approach. Not a good model for large volumes of messages however !
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Sep 27, 2016 5:34 am Post subject: Re: Is it possible to configure MQInput to process not at on |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Sabina wrote: |
| And the other question is if it is suitable at all to use message queues for this kind of task. |
IMHO No.
This requirement (correctly described as "crazy") by one of my associates is a rabbit hole leading to pain and misery. How do you know 15 minutes is the right length of time? Suppose you set 15 minutes and the problem isn't resolved? You'll then be going round again, resetting timers and all sorts.
The actual requirement (if I'm reading your description correctly) is to store failed entities until they can be processed. This calls for a database. Now if you want to add a column with a type of datetime and a name of "Next time to have a try", you could easily develop some kind of scanning application to see if that datetime has been reached and reprocessed.
If it was me (and it isn't of course) then I'd add a "Ready for retry" column and scan for that. This allows you to process entities when the relevant source system is back as soon as it's back, rather than waiting for a timer to expire. It also allows for multiple back end failures being fixed at different times.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| psyconic |
| Posted: Mon Oct 31, 2016 6:36 pm Post subject: |
|
|
Novice
Joined: 17 Jun 2015
Posts: 11
|
You could use an MQ Input node in browse mode straight to a filter node that checks the time lapsed, when the appropriate time has lapsed the next node is an mq get that gets that message off the queue for processing. No timers involved.
The code behind the filter node could use a shared variable being say the delay, that way if you change that variable elsewhere in the flow that delay will be applied to incoming messages and store the last time a message was processing in a shared variable so you can compare the time lapsed.
If you need it at message level resolution you will have to compare the interval against a date in the message.
I use this technique for a message retry system. We have messages that come in and the are passed on to a microsoft webservice, if that service is down they get put on our retry queue which uses this functionality if the webs service is still down the message gets put back onto the queue. If the webservice stays down a while the retry delay is increased and when it comes back up the retry delay is set to 0 and all messages are processed off the queue. Thats the instance we use this technique in. We also send emails to interested users when the service goes down and at intervals to keep people updated re the processing of the backlog. It works great.
The only trick was to set the reset browse timeout to 4000ms on the input node that was in browse mode so no messages got missed after putting them back on the queue. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Oct 31, 2016 10:56 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
As my esteemed colleague (Vitor) says, using queues for this is a recepie for lots of pain, anguish and late night support calls when it stops working.
MQ is not a substiture Database.
We see this sort of requirement pop up from time to time. My guess is that there is some so called 'architect' out there who thinks that this is the (and to quote the sooty & sweep show) 'That's the way to do it'. IMHO it is not.
In a previous position I inherited a similar process where messages contained sales orders that needed to be delayed for 48hours for a cooling off period. It worked but when there was a lot of volume if fell flat on its face.
I ended up re-writing the whole thing to use a database. Easy to find the messages due for processing. AFAIK, it has been working for the past 8 years. At least I have not had any support calls about it.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 01, 2016 4:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| psyconic wrote: |
| You could use an MQ Input node in browse mode straight to a filter node that checks the time lapsed, when the appropriate time has lapsed the next node is an mq get that gets that message off the queue for processing. No timers involved. |
Not sure I'd enjoy having a flow spin round and round browsing messages burning CPU. Also not convinced about doing a browse then get.
I'd still prefer a developer.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Nov 01, 2016 5:13 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
There is a new MQ v8 (JMS 2.0) feature which allows delayed delivery of a message.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 01, 2016 5:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| There is a new MQ v8 (JMS 2.0) feature which allows delayed delivery of a message. |
JMS. Java. Bah.
You store things, you store them in databases. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Nov 01, 2016 5:51 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
All hail...
SYSTEM.DDELAY.LOCAL.QUEUE
Remember the hideous WMB registry database? I am still giving thanks for its removal. Although now rather worried about the removal of MQ from IIB.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 01, 2016 5:58 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| Remember the hideous WMB registry database? |
Ok, by "database" I mean a real database.
Not Bernard.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 01, 2016 7:10 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| Vitor wrote: |
| zpat wrote: |
| Remember the hideous WMB registry database? |
Ok, by "database" I mean a real database.
Not Bernard. |
Don't diss Bernard. It is a real DB delivered with the JDK under the name Java DB (ex Derby?). And it will use disk...
O.K. it might not be as popular as the well known industry strength ones, but it is pretty good. And let's be honest, every DB has its quirks... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 01, 2016 7:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fjb_saper wrote: |
| Don't diss Bernard. It is a real DB delivered with the JDK under the name Java DB (ex Derby?). And it will use disk... |
I'm sure Angus will slap me in due course.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 01, 2016 10:19 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| Vitor wrote: |
| fjb_saper wrote: |
| Don't diss Bernard. It is a real DB delivered with the JDK under the name Java DB (ex Derby?). And it will use disk... |
I'm sure Angus will slap me in due course. |
I'll leave that to Dumbledore 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
