| Author |
Message
|
| j1 |
 Posted: Thu Jan 19, 2012 9:00 pm Post subject: Persisiting LocalEnvironment accross broker crash/restart |
 |
|
Centurion
Joined: 23 Jun 2003
Posts: 139
|
Hi,
I have a message flow which sets up a few environment variables based on data in certain messages. Now, if the broker were to crash or restart for some reason, I want these environment variables to be made available again seamlessly, instead of having to re-run those specific messages through the flow. Is this possible in any way in ESQL or can I instrument it in some way, say using a file,queue or some other persistent storage? Im wondering how I would re-read only those specific messages, stored on (say) a state queue or file back through the i/p queue or delete them once they are no longer needed....any ideas?
Cheers
Last edited by j1 on Sun Jan 22, 2012 3:47 pm; edited 1 time in total |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Thu Jan 19, 2012 10:49 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
In essence you are looking for a node that will start something off when a flow is started. Most nodes at the start of flow typically wait for something to happen (i.e. trigger) before they burst into life.
A Timeout Node will fire when a flow is started. You could wire that to something that reads the queues or dare i say it, the data from a database table?
This is my preferred method but there are others here that disagree with this approach.
Now where's the tinfoil hat Emoticon when you need it?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jan 19, 2012 11:36 pm Post subject: Re: Persisting Local Environment accross restarts/crashes |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| j1 wrote: |
Hi,
I have a message flow which sets up a few environment variables based on data in certain messages. Now, if the broker were to crash or restart for some reason, I want these environment variables to be made available again seamlessly, instead of having to re-run those specific messages through the flow. Is this possible in any way in ESQL or can I instrument it in some way, say using a file,queue or some other persistent storage? Im wondering how I would re-read only those specific messages, stored on (say) a state queue or file back through the i/p queue or delete them once they are no longer needed....any ideas?
Cheers |
The title of your topic is quite misleading. People may think you are trying to use LocalEnvironment as a semi-persistent storage... ... and interpret it as if you haven't got a clue. In fact you are reading data from configuration messages and storing it in semi-persistent ESQL shared or java static variables, aren't you?
You don't have to trigger sending/reading the configuration messages in case the EG has restarted. The normal way is to implement it like this:
When the flow gets a real payload message, it checks if the semi-persistent storage is initialized. If not, it uses MQGet in browse mode to get the configuration messages and initializes the storage.
Your flow doesn't have to persist your configuration if you have placed it in a persisten storage - a queue - from the beginning.
You still have some administrative overhead with this solution. How to make sure that the messages stay in the queues, for example. Although very widely used, this solution is getting a bit old-fashioned. Take a look at User Defined Configurable Services. |
|
| Back to top |
|
|
| cociu_2012 |
| Posted: Fri Jan 20, 2012 12:09 am Post subject: Re: Persisting Local Environment accross restarts/crashes |
|
|
Acolyte
Joined: 06 Jan 2012
Posts: 72
|
| Esa wrote: |
You still have some administrative overhead with this solution. How to make sure that the messages stay in the queues, for example. |
I believe that setting persistence mode, and setmqaut is safe enough, am I wrong?
I totally agree with Esa, why using a solution that's not persistent when you have so many other. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Jan 20, 2012 1:23 am Post subject: Re: Persisting Local Environment accross restarts/crashes |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| cociu_2012 wrote: |
| I believe that setting persistence mode, and setmqaut is safe enough, am I wrong? |
I may remember incorrectly, but can you really browse a queue without get permission?
If you are a member of mqm group, the authority check is bypassed. Who else do you think would accidentally delete a message from a queue but an administrator? |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jan 20, 2012 4:25 am Post subject: Re: Persisting Local Environment accross restarts/crashes |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| cociu_2012 wrote: |
| I believe that setting persistence mode, and setmqaut is safe enough, am I wrong? |
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| j1 |
| Posted: Fri Jan 20, 2012 6:24 am Post subject: |
|
|
Centurion
Joined: 23 Jun 2003
Posts: 139
|
| Quote: |
When the flow gets a real payload message, it checks if the semi-persistent storage is initialized. If not, it uses MQGet in browse mode to get the configuration messages and initializes the storage
|
I like this idea, but how do I do a conditional read from an i/p node in ESQL? i.e' IF <Variable NOT Initialised> THEN Browse State Queue '
Will need to look inot the timer node idea too, but think ill give this a shot first [/code] |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jan 20, 2012 6:31 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| j1 wrote: |
| I like this idea, but how do I do a conditional read from an i/p node in ESQL? i.e' IF <Variable NOT Initialised> THEN Browse State Queue ' |
You don't do it in ESQL. When the message arrives in the flow, you use a Filter node to check to see if the storage is initialized. You can then use an MQGet node to read the message.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jan 20, 2012 6:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| j1 wrote: |
| I like this idea, but how do I do a conditional read from an i/p node in ESQL? i.e' IF <Variable NOT Initialised> THEN Browse State Queue ' |
You don't do it in ESQL. When the message arrives in the flow, you use a Filter node to check to see if the storage is initialized. You can then use an MQGet node to read the message. |
Or you do it in ESQL and use PROPAGATE to decide which path out of a compute node to take. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Jan 20, 2012 6:42 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
I have only implemented caches with java Singletons, so I'm not the right person to tell you how to do it with ESQL SHARED ROW variables. It's clumsier, and I have never bothered to take a look at it. Search mqseries.net, I think this subject must have been discussed.
It's important to write in the cache object within an ATOMIC block, at least if you ever plan to allow additional instances of the flow.
Otherwise it goes like this:
If some indicator variable in your cache is null, or the number of rows is 0 or something else that tells you that the cache hasn't been initialized, you PROPAGATE to a terminal that is wired to an MQGet. Depending on your skills level you can store the message where you like so that it's available when the control returns to your ESQL code after the propagate statement. Then initialize the cache from the message.
Last edited by Esa on Fri Jan 20, 2012 6:45 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jan 20, 2012 6:45 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| Vitor wrote: |
| j1 wrote: |
| I like this idea, but how do I do a conditional read from an i/p node in ESQL? i.e' IF <Variable NOT Initialised> THEN Browse State Queue ' |
You don't do it in ESQL. When the message arrives in the flow, you use a Filter node to check to see if the storage is initialized. You can then use an MQGet node to read the message. |
Or you do it in ESQL and use PROPAGATE to decide which path out of a compute node to take. |
Quite. The point I was trying to get at is that you don't read the message within ESQL.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Jan 20, 2012 6:57 am Post subject: Re: Persisting Local Environment accross restarts/crashes |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| j1 wrote: |
Hi,
I have a message flow which sets up a few environment variables based on data in certain messages. Now, if the broker were to crash or restart for some reason, I want these environment variables to be made available again seamlessly, instead of having to re-run those specific messages through the flow. Is this possible in any way in ESQL or can I instrument it in some way, say using a file,queue or some other persistent storage? Im wondering how I would re-read only those specific messages, stored on (say) a state queue or file back through the i/p queue or delete them once they are no longer needed....any ideas?
Cheers |
solidDB would work extremely well for your use-case. You may like to read about it.
http://www-01.ibm.com/software/data/soliddb/
The performance will be the same as storing in LocalEnvironment, but you get the benefit of advanced data management, as in the scenario you posed.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| cociu_2012 |
| Posted: Fri Jan 20, 2012 7:17 am Post subject: Re: Persisting Local Environment accross restarts/crashes |
|
|
Acolyte
Joined: 06 Jan 2012
Posts: 72
|
| lancelotlinc wrote: |
The performance will be the same as storing in LocalEnvironment |
you can't share LocalEnvironment between treads/flows.
The subject is wrong. He probably uses shared variables (those reset after crash/ restart) .
@j1 - correct the subject. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jan 20, 2012 7:39 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| You can set browse permission for a queue. |
|
| Back to top |
|
|
| Esa |
| Posted: Fri Jan 20, 2012 8:05 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| zpat wrote: |
| You can set browse permission for a queue. |
Yes. If you are referring to my comment about setmqaut above, I remembered incorectly, as I suspected. I guess I was recalling the - already corrected - bug in MQGet node, where it opened the queue handle with different options depending on if browse only was selected or not...
... regarding the discussion whether the solution to store configuration messages in an MQ queue is idiot proof. Am I the only one to think it's not? |
|
| Back to top |
|
|
|
|
