| Author |
Message
|
| DP1234 |
 Posted: Wed Aug 13, 2014 1:29 am Post subject: Global Cache Issue |
 |
|
Apprentice
Joined: 25 May 2014
Posts: 40
|
Am facing some serious issue related to Global Cache in multi thread scenario..i have been maintaning a Global cache which holds the count of number of input messages/events occur in the input. For every incoming message/event the count is incremented and the same is Updated in global cache as well..in an load test of 200k messages i found that count is not updated in cache properly..say for 200k messages the count would be some 30000...The instance to my flow be 1 or more than one the count is poorly reflecting in cache.
My analysis on the issue was when multi threads are approaching to update the cache in Java block the current thread which is in process is locking the procedure doesnt letting other, resulting false value of count reflecting in Cache...
Making Update Cache Java Procedure Synchronised didn't help
Am using IIB.
Badly need ur help on this issue
Thanks |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Aug 13, 2014 4:55 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
Seems to me you need to create a singleton to hold the count information. In turn the singleton would update the cache.
If the value gets changed from multiple execution groups, your singleton needs to be a rmi accessible object.
It might be judicious to have the location of the singleton in the cache...
As this gets quite complicated, I'd suggest you subscribe to the flow statistics and get the number of invocations from there... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 13, 2014 5:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fjb_saper wrote: |
Seems to me you need to create a singleton to hold the count information. In turn the singleton would update the cache.
If the value gets changed from multiple execution groups, your singleton needs to be a rmi accessible object.
It might be judicious to have the location of the singleton in the cache...
As this gets quite complicated, I'd suggest you subscribe to the flow statistics and get the number of invocations from there... |
A singleton doesn't help across multiple EGs.
But clearly the cache needs to be updated within an atomic block. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Aug 13, 2014 5:50 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
| mqjeff wrote: |
| fjb_saper wrote: |
If the value gets changed from multiple execution groups, your singleton needs to be a rmi accessible object.
It might be judicious to have the location of the singleton in the cache...
As this gets quite complicated, I'd suggest you subscribe to the flow statistics and get the number of invocations from there... |
A singleton doesn't help across multiple EGs.
But clearly the cache needs to be updated within an atomic block. |
Clearly if the singleton is accessed via RMI from other eg's it would still have the "single point of update"...
And even within an atomic block (java sychronized) it won't work across multiple egs as each JVM only synchronizes it's set and not the cache...
My understanding is that retrieving the cache object you actually retrieve a copy / clone. This means that a synchronize (cache object) would not do the trick. A synchronized method won't do it either as the different egs will just overwrite the last known value.
Using a remote object singleton as cache and having the location of it in the global cache effectively synchronizes across egs but also creates a bottleneck... and would need to synchronize with the actual global cache at intervals... The flow would need to access the singleton via RMI to get and update the value. The singleton will then need to synchronize internally to keep the count thread safe...
Like I said, quite complicated and I don't see the advantage of it over a count after the fact from the flow statistics...
_________________
MQ & Broker admin
Last edited by fjb_saper on Wed Aug 13, 2014 6:00 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 13, 2014 5:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Yes, if you go to all the trouble you mention to implement it in a straight Java means, then it gets quite complicated.
IF, instead, you think of "atomic block" as a generic notion, rather than a java implementation of a generic notion, that might lead one to either looking at available functions with GlobalCache (or at least WS XtremeScale) or otherwise for a solution within the larger Broker product.
And, yes, I agree that if this count is needed for any kind of *monitoring* purpose, then coding it at all rather than using stats is entirely the wrong solution.
But *surely* DP1234 wouldn't be looking to implement this for *monitoring* purposes. Surely there's a business requirement to include this number in the message data. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Aug 13, 2014 6:08 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
Correct me if I'm wrong, but the update to the global cache is always done in an "atomic" way. However the global cache only allows you to get the object and not to implement a (thread safe) method on the cached object.
Thus each e.g. will overwrite the last value with it's own, even if you manage to make the update in thread safe manner within the e.g. ...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| DP1234 |
| Posted: Mon Aug 18, 2014 2:09 am Post subject: |
|
|
Apprentice
Joined: 25 May 2014
Posts: 40
|
fjb_saper and mqjeff Thanks for ur replies
I tried making java update/retrive block synchronised, implemented singleton and also tried using reetrant locks it didn't help.
Making ESQL block atomic didn't help.
Count value is same or getting still lowered
One thing i wanna know is this multi thread cache update issue is limitation of Global Cache by any way..?
wat could be done to overcome any alternate...least possibilities |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 18, 2014 4:27 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
In order for it to work like you intend to, you would need an add function implemented in the global cache object, accessible through the global cache and residing in the global cache. This function would of course need to be synchronized in some way, either internally or through global cache access protocol.
What you are doing is a replace of the global cache with the new value, when in fact your intention was to do an add.
So if thread one sets the value to 5 and thread 2 sets the value to 3, which will be the value in the global cache?
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| DP1234 |
| Posted: Mon Aug 18, 2014 7:28 am Post subject: |
|
|
Apprentice
Joined: 25 May 2014
Posts: 40
|
| fjb_saper wrote: |
So if thread one sets the value to 5 and thread 2 sets the value to 3, which will be the value in the global cache?
|
its 3 literally |
|
| Back to top |
|
|
|
|
