ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportSHARED ROW (cache) and ATOMIC

Post new topicReply to topic
SHARED ROW (cache) and ATOMIC View previous topic :: View next topic
Author Message
grebenar
PostPosted: Wed Nov 04, 2020 1:31 am Post subject: SHARED ROW (cache) and ATOMIC Reply with quote

Novice

Joined: 10 Apr 2006
Posts: 22
Location: Budapest, Hungary

Hello,

I've read many topics regarding SHARED valiables and ATOMIC operations, and of course the IIB documentation, but thy still not answered my question.

We have a message router application in IIB that run in 3 EGs, each EG has 12 instances of the same flow. The flow routes messages (millions a day) based on database tables, which are cached in a SHARED ROW variable.
We have a problem that sometimes the flow marks normal valid messages as errorous. This marking is made against cached database tables. I refresh the cache every minute, and interestingly the error last exactly to 1 minute. Meanwhile some other same messages are routed properly (I suspect by another EG).

Here is how I refresh and use the cache. What I understood in the docs that I do not have to use ATOMIC blocks - the flow is not sensitive if it sees the current, or the 1-minute-before state of the cache.

Code:

-- Refresh time: every 1 minute
-- CACHE is declare in an other node
-- DECLARE CACHE SHARED ROW;

CREATE COMPUTE MODULE mod_DiszpecserMqMain_CacheTables
   CREATE FUNCTION Main() RETURNS BOOLEAN
   BEGIN
      -- CALL CopyMessageHeaders();
      CALL CopyEntireMessage();
      
      DECLARE TEMP ROW;
         DECLARE CACHE_NEW ROW;
      DECLARE I INT;
      DECLARE cacheRef REFERENCE TO CACHE; 
            
      --APPLICATION
      SET TEMP.APPLICATIONS.APPLICATION[] = SELECT T.* FROM Database.APPLICATION AS T;     
      FOR appRef AS TEMP.APPLICATIONS.APPLICATION[] DO       
           CREATE LASTCHILD OF CACHE_NEW.APPLICATION AS cacheRef NAME appRef.ID;
         FOR fieldRef AS appRef.[] DO
            IF FIELDNAME(fieldRef) <> 'DESCRIPTION' THEN CREATE LASTCHILD OF cacheRef FROM fieldRef; END IF;
         END FOR;
      END FOR;

      --MESSAGETYPE
      SET TEMP.MESSAGETYPES.MESSAGETYPE[] = SELECT T.* FROM Database.MESSAGETYPE AS T;     
      FOR msgRef AS TEMP.MESSAGETYPES.MESSAGETYPE[] DO       
           CREATE LASTCHILD OF CACHE_NEW.MESSAGETYPE AS cacheRef NAME msgRef.ID;
         FOR fieldRef AS msgRef.[] DO
            IF FIELDNAME(fieldRef) NOT IN ('DESCRIPTION','EXTERNAL_LINK') THEN CREATE LASTCHILD OF cacheRef FROM fieldRef; END IF;
         END FOR;
      END FOR;

      SET CACHE = CACHE_NEW;
      SET CACHE.REFRESH_TIMESTAMP = CURRENT_TIMESTAMP;
      
      RETURN NOT(EXISTS(InputLocalEnvironment.TimeoutRequest[]));
      
   END;


I use the cache this way:

Code:
SET Environment.Variables.IncomingMessageInfo.MessageTypeActive = CACHE.MESSAGETYPE.{C}.ACTIVE;


Sometimes this code gives wrong result (MessageTypeActive will not be 'Y' as it is in the database).

I have the idea that maybe "SET CACHE = CACHE_NEW" expression is not thread safe and I should use ATOMIC around it?

Greetings,
Thanks,
Robert
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Nov 04, 2020 6:26 am Post subject: Re: SHARED ROW (cache) and ATOMIC Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20421
Location: LI,NY

grebenar wrote:

Sometimes this code gives wrong result (MessageTypeActive will not be 'Y' as it is in the database).

I have the idea that maybe "SET CACHE = CACHE_NEW" expression is not thread safe and I should use ATOMIC around it?

Greetings,
Thanks,
Robert

You got it! So wrap the change of the cache in the Atomic block and move on
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
grebenar
PostPosted: Mon Nov 09, 2020 2:34 am Post subject: Reply with quote

Novice

Joined: 10 Apr 2006
Posts: 22
Location: Budapest, Hungary

Thanks for the answer!

I deployed the change to the prod environment last Friday, and it seems that we're getting the error again.

I modified my code this way:

Code:
      SetCache: BEGIN ATOMIC
         SET CACHE = CACHE_NEW;
         SET CACHE.REFRESH_TIMESTAMP = CURRENT_TIMESTAMP;
      END;


I increased the cache refresh time to 2 minutes, so the problematic periods last till 2 minutes - so the problem is with the cache refresh (write). That's why I think that the reads don't need to be in ATOMIC.

Somehow I have to trace out the contents of the cache to see exactly what is the "wrong data". My next step may be logging the cache in case of an error, and maybe revise the write code and not use "temp" cache (just perform an atomic write on the master global cache SHARED ROW).
Back to top
View user's profile Send private message
rekarm01
PostPosted: Tue Nov 10, 2020 10:04 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1396

grebenar wrote:
That's why I think that the reads don't need to be in ATOMIC.

Generally speaking, READS need to be in ATOMIC blocks too, to prevent one instance from trying to read a SHARED variable at the same time that some other instance is currently writing to it. And all of the ATOMIC blocks accessing the same shared variable should have the same label.
Back to top
View user's profile Send private message
grebenar
PostPosted: Thu Nov 12, 2020 6:38 am Post subject: Reply with quote

Novice

Joined: 10 Apr 2006
Posts: 22
Location: Budapest, Hungary

My problem with putting READS into atomic blocks, that also those threads (12-24 copies) will be blocked to execute the same query (from the cache) the same time, so they will not only concur with the write (which is correct) but the reads will concure with each other.

Quote:
If ATOMIC is specified, only one instance of a message flow (that is, one thread) is allowed to execute the statements of a specific BEGIN ATOMIC... END statement (identified by its schema and label), at any one time.


What I would see reasonable, that when WRITE is happening, no one else can reach the variable. But when a READ is accessing it, why should it block other READs... Unfortunately, I see no solution for it.

Anyway, IBM's explanation is a bit misleading:

Quote:
It is also unnecessary to use the BEGIN ATOMIC construct on reads and writes to shared variables. The integration node always safely writes a new value to, and safely reads the latest value from, a shared variable.


I know it was quoted many times on these forums, but I still read this that when I enter SET CACHE = CACHE_NEW, the broker "safely writes a new value to" the new variable (which is a SHARED ROW). It seems that it is not true.
Is it published anywhere, how is a ROW (CACHE_NEW) copied/moved to a SHARED ROW (CACHE)?
Back to top
View user's profile Send private message
rekarm01
PostPosted: Sun Nov 15, 2020 6:39 pm Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 1396

grebenar wrote:
My problem with putting READS into atomic blocks, that also those threads (12-24 copies) will be blocked to execute the same query (from the cache) the same time, so they will not only concur with the write (which is correct) but the reads will concure with each other.

Blocking concurrent readers is not optimal, but how big of a problem is that? What fraction of time does one thread spend inside an ATOMIC block, accessing the shared cache? If it's significant, then could a thread reduce that time some other way, such as by initially copying what it needs from the shared cache to its Environment tree, and spending the rest of its time accessing that instead?

grebenar wrote:
What I would see reasonable, that when WRITE is happening, no one else can reach the variable. But when a READ is accessing it, why should it block other READs... Unfortunately, I see no solution for it.

If the flow really needs to use readers-writer locks, then it could implement its own in ESQL, using ATOMIC blocks, auxiliary variables, and busy-wait loops. But if the performance gain is marginal, then it may not be worth the added overhead. There are also non-ESQL options, such as JavaCompute nodes, or C custom nodes.

grebenar wrote:
Anyway, IBM's explanation is a bit misleading:

Quote:
It is also unnecessary to use the BEGIN ATOMIC construct on reads and writes to shared variables. The integration node always safely writes a new value to, and safely reads the latest value from, a shared variable.

A single read or write to a scalar variable is, by itself, inherently atomic. But navigating/constructing/copying non-scalar trees, and attaching/detaching elements is more complicated, generally requiring multiple read/write operations.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportSHARED ROW (cache) and ATOMIC
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.