| Author |
Message
|
| pmarie |
 Posted: Tue Jul 23, 2019 10:04 am Post subject: How to load all data into the global cache at startup? |
 |
|
Novice
Joined: 04 Apr 2002
Posts: 23
|
My customer wants to have all the data needed to process requests loaded (from a database) into the global cache as soon as the integration server is started, so that, when the first requests are received, they can be processed without delay, all the data being already in the cache.
This is an ACE on premise configuration.
Any idea about how to achieve this? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Jul 23, 2019 10:15 am Post subject: Re: How to load all data into the global cache at startup? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| pmarie wrote: |
| Any idea about how to achieve this? |
- Give your customer a nice cool glass of water, sit them down and tell them that unless they have an unfeasibly large amount of data and/or a ludicrously short SLA, the extra complexity required to achieve this isn't worth the extra time it will take the first request to load the cache (this sounds a lot like a manager trying to sound clever, like trying to get a transaction down from 3 seconds to 2 seconds when the SLA is 5 seconds)
- Have a separate flow that populates the cache triggered at start up that in turn starts the request flow when the cache is loaded (because you don't want a request to start processing in parallel with the cache load).
- Push a dummy request through on start up to trigger the cache load.
I would seriously ask your customer exactly how much time they need to save on the first transaction. If I was looking for time savings in this situation, I'd optimize the SQL queries. I'd also ask the DBAs how much buffer they've got assigned, and how many queries actually go through to the spinning disc. It'll be a lot easier for you (and them) to have all this data cached on the DB side.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| timber |
| Posted: Tue Jul 23, 2019 12:58 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| Another option would be to do the loading of the cache in a JavaCompute node. I'm fairly sure that there is an option to perform once-only initialisation tasks in a JavaCompute node. If you want to do the actual work by calling out to some ESQL from the JavaCompute, that's probably do-able also. Not tried this - just throwing out ideas. |
|
| Back to top |
|
|
| pmarie |
| Posted: Wed Jul 31, 2019 8:32 am Post subject: |
|
|
Novice
Joined: 04 Apr 2002
Posts: 23
|
Thanks for your answers.
I finally recommended the customer to do the following:
- Add a script in the directory "<work_path>/config/<broker_name>/<server_name>/profiles".
- In that script, put the following actions:
- Stop all the flows which process requests and need data in the global cache for that purpose.
- Start using CURL a specific flow (to be implemented) which will load all the data into the global cache. Wait for its response (meaning all the data are ready in the cache).
- Restart all the flows which process requests and need data in the global cache for that purpose. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 31, 2019 9:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
you also want to think about your global cache and it's lifetime cycle.
What if you never have to load the global cache ever again once it is initialized because it spans multiple integration servers?
Just because one server goes down does not mean the global cache is down...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
