ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Strange problem after 7.0.0.5 fix deployment

Post new topic  Reply to topic
 Strange problem after 7.0.0.5 fix deployment « View previous topic :: View next topic » 
Author Message
raphg
PostPosted: Wed Jun 05, 2013 3:24 am    Post subject: Strange problem after 7.0.0.5 fix deployment Reply with quote

Newbie

Joined: 04 Apr 2013
Posts: 7

Hello,

I've got a very strange problem to submit you.
In our company, we've got a critical application running under WMB 7.0.0.1. We have deployed two instances (ie: 2 computers running the same code). We noticed database cursors leak for a long time, never found where it came from. We decided to apply 7.0.0.5 on one of the computer and it was magic: no more leaks on this machine !

But.... since we have deployed this fix, we have a strange problem (no other modification were done, no bar deployment, ... just 7.0.0.5). The application still run fine for several hours, and suddenly, NullPointerException in a little piece of java code. We added traces, then deployed bars, .... and after 10 or more iteration we have found a strange thing.

Here is a piece of code :

Code:
DECLARE lastLineCount SHARED INTEGER -1;
DECLARE lastUpdateDate SHARED GMTTIMESTAMP '1970-01-01 00:00:00';
DECLARE activeTimers SHARED ROW;
DECLARE lastTimerTimestamp SHARED GMTTIMESTAMP '1970-01-01 00:00:00';
-- DECLARE currentActiveTimers SHARED INTEGER 1;

CREATE DATABASE MODULE TimerManager_LogTimerSet
   CREATE FUNCTION Main() RETURNS BOOLEAN
   BEGIN
      DECLARE fileRef REFERENCE TO Root.XMLNSC.Timer.FileRotation;
      CALL logInfo('ROCADE',9999,'TimerSet: ' || fileRef.FileDirectory || ' / ' || fileRef.FileName,'','','');
      
      CALL logInfo(   'ROCADE',
                  1033,
                  CAST(joinPaths(fileRef.FileDirectory, fileRef.FileName) AS CHARACTER),
                  CAST(CAST(CAST(fileRef.Context.NextRotationTimestamp AS GMTTIMESTAMP FORMAT TIMESTAMP_FORMAT) AS TIMESTAMP) AS CHARACTER FORMAT 'yyyy-MM-dd HH:mm:ss'), '', '');

      RETURN TRUE;
   END;
END MODULE;

CREATE COMPUTE MODULE TimerManager_Check
   
   CREATE PROCEDURE DumpTimer(IN timers REFERENCE)
   BEGIN
      DECLARE SB CHARACTER;
      DECLARE I INTEGER;
      SET SB = 'DUMP DES ' || CAST(CARDINALITY(timers.Timer[]) AS CHARACTER) || ' timers :';
      SET I = 0;
      FOR timer AS timers.Timer[] DO
         SET I = I + 1;
         DECLARE timerRef REFERENCE TO timer;
         SET SB = SB || ' ' || timerRef.TimerId;
         CALL logInfo('ROCADE',9999,'Timer ' || CAST(I AS CHARACTER) || ' : ' || CAST(timerRef.TimerId AS CHARACTER),'','','');
      END FOR;
      CALL logInfo('ROCADE',9999,SB,'','','');
   END;
   
   CREATE FUNCTION Main() RETURNS BOOLEAN
   BEGIN
      -- Variable locale de timers;
      DECLARE localTimers ROW;
      
      -- Vérification de la mise à jour
      DECLARE maxUpdateDate GMTTIMESTAMP    SELECT    MAX(f.UPDATEDATE_UTC)
                                 FROM    Database.BRK01.BRKCFG.BRKQTOUTFILE AS f;

      DECLARE lineCount INTEGER          SELECT COUNT(f.OUTFILE_ID)
                                 FROM Database.BRK01.BRKCFG.BRKQTOUTFILE AS f;

      -- Chargement des timers le cas échéant
      IF (lastLineCount <> CARDINALITY(activeTimers.Table.Timer[])) THEN
         CALL logInfo('ROCADE',9999,'DEPHASAGE DETECTE','','','');
         CALL dumpTimer(localTimers.Table);
         SET lastLineCount = -1;
      END IF;
      IF maxUpdateDate <> lastUpdateDate OR lineCount <> lastLineCount THEN
         SET lastUpdateDate = maxUpdateDate;
         SET lastLineCount = lineCount;
         SET localTimers.Table.Timer[] =
                     SELECT   'file://' || joinPaths(outfile.FILE_DIRECTORY, outfile.FILE_NAME) AS TimerId,
                           outfile.ROTATION_MINUTE AS TimerMinute,
                           outfile.ROTATION_HOUR AS TimerHour,
                           outfile.ROTATION_DAY AS TimerDay,
                           outfile.ROTATION_MONTH AS TimerMonth,
                           outfile.ROTATION_DAYOFWEEK AS TimerDayOfWeek,
                           outfile.FILE_DIRECTORY AS Directory,
                           outfile.FILE_NAME AS FileName,
                           outfile.FILE_FORMAT AS Format,
                           outfile.MASTER AS Master,
                           outfile.SLAVE AS Slave,
                           outfile.GENERATE_EMPTY AS GenerateIfEmpty,
                           outfile.POSTPROCESSING_DELAY AS PostProcessingDelay,
                           outfile.EXE_COMMAND AS ExeCommand,
                           package.PACKAGE_ID AS PackageId,
                           package.NAME AS PackageName
                     FROM    Database.BRK01.BRKCFG.BRKQTOUTFILE AS outfile,
                           Database.BRK01.BRKCFG.BRKQTPACKAGE AS package
                     WHERE   outfile.ENABLED <> 0
                        AND   outfile.PACKAGE_ID = package.PACKAGE_ID
                        AND   package.ENABLED <> 0;

         CALL logInfo('ROCADE', 1007, '', '', '', '');
         CALL logInfo('ROCADE',9999,'Load des timers nb=' || CAST(CARDINALITY(localTimers.Table.Timer[]) AS CHARACTER),'','','');
         CALL dumpTimer(localTimers.Table);
      ELSE
         SET localTimers.Table = activeTimers.Table;
      END IF;
      CALL logInfo('ROCADE',9999,'RotateCheck sur ' || CAST(CARDINALITY(localTimers.Table.Timer[]) AS CHARACTER) || ' timers','','','');
      
      -- Création du message de timer
      SET OutputRoot.Properties.Persistence = TRUE;
      CREATE NEXTSIBLING OF OutputRoot.Properties DOMAIN 'MQMD';
      SET OutputRoot.MQMD.Version = MQMD_CURRENT_VERSION;
      SET OutputRoot.MQMD.Persistence = MQPER_PERSISTENT;
      SET OutputRoot.MQMD.Format = 'XML';               
      CREATE LASTCHILD OF OutputRoot DOMAIN 'XMLNSC';

      -- Copie des timers partagés dans une variable locale
      DECLARE lastTimer REFERENCE TO localTimers.Table.Timer[<];
      FOR timer AS localTimers.Table.Timer[] DO
         DECLARE timerRef REFERENCE TO timer;
         CALL logInfo('ROCADE',9999,'Checking ' || timer.FileName || ' ' || timer.TimerDay || ' baseTimestamp=' || CAST(baseTimestamp AS CHARACTER),'','','');
         DECLARE nextTimestamp TIMESTAMP computeNextTimestamp(CURRENT_TIMESTAMP, timerRef);
         DECLARE nextNextTimestamp TIMESTAMP computeNextTimestamp(nextTimestamp, timerRef);
         
         -- Le timer est reprogrammé seulement une fois l'évènement dépassé et seulement si on est master ou slave
         IF (timer.Master = BrokerName OR timer.Slave = BrokerName) AND (timer.NextEvent IS NULL OR (CURRENT_TIMESTAMP > timer.NextEvent + CAST(WaitTimeout / 2 AS INTERVAL SECOND))) THEN
            -- On ne peut pas programmer d'évènement plus fréquent que 2x WaitTimeout
            IF (nextNextTimestamp - nextTimestamp) SECOND < CAST(WaitTimeout * 2 AS INTERVAL SECOND) THEN   
               CALL logError('ROCADE', 1039, CAST(joinPaths(joinPaths(timer.Directory, 'work'), timer.FileName) AS CHARACTER), CAST(WaitTimeout * 2 AS  CHARACTER), '', '');
            ELSE         
               CALL logInfo('ROCADE', 9999, 'REPROGRAMMATION: ' || timer.Directory || ' / ' || timer.FileName, '', '', '');
               SET timer.NextEvent = nextTimestamp;
               SET timer.NextNextEvent = nextNextTimestamp;
               
               -- Création de la requete
               DECLARE msgRef REFERENCE TO OutputRoot;
               DELETE FIELD OutputRoot.XMLNSC.Timer;
               CREATE FIELD OutputRoot.XMLNSC.Timer AS msgRef;

               SET msgRef.TimeoutRequest.Action = 'SET';
               SET msgRef.TimeoutRequest.Identifier = CAST(timer.TimerId AS CHARACTER);
               SET msgRef.TimeoutRequest.StartDate = CAST(CAST(nextTimestamp AS TIMESTAMP) AS CHARACTER FORMAT 'yyyy-MM-dd');
               SET msgRef.TimeoutRequest.StartTime = CAST(CAST(nextTimestamp AS TIMESTAMP) AS CHARACTER FORMAT 'HH:mm:ss');
               SET msgRef.TimeoutRequest.Interval = '1';
               SET msgRef.TimeoutRequest.Count = '1';
               SET msgRef.TimeoutRequest.IgnoreMissed = 'TRUE';
               SET msgRef.TimeoutRequest.AllowOverwrite = 'TRUE';

               SET msgRef.FileRotation.FileDirectory = joinPaths(timer.Directory, 'work');
               SET msgRef.FileRotation.FileName = timer.FileName;
               SET msgRef.FileRotation.FileFormat = timer.Format;
               SET msgRef.FileRotation.PackageId = timer.PackageId;
               SET msgRef.FileRotation.PackageName = timer.PackageName;
               SET msgRef.FileRotation.ExeCommand = timer.ExeCommand;
               SET msgRef.FileRotation.Master = timer.Master;
               SET msgRef.FileRotation.Slave = timer.Slave;
               SET msgRef.FileRotation.GenerateIfEmpty = timer.GenerateIfEmpty;
               SET msgRef.FileRotation.PostProcessingDelay = CAST(timer.PostProcessingDelay AS INTEGER);
               SET msgRef.FileRotation.Context.NextRotationTimestamp = CAST(CAST(nextTimestamp AS GMTTIMESTAMP) AS CHARACTER FORMAT TIMESTAMP_FORMAT);
               
               -- Activation du timer
               IF SAMEFIELD(timer, lastTimer) THEN
                  PROPAGATE;
               ELSE
                  PROPAGATE DELETE NONE;
               END IF;
            END IF;
         END IF;
      END FOR;

      -- Mise à jour des timers actifs
      ActiveTimersMutex : BEGIN ATOMIC
         DELETE FIELD activeTimers.Table;
         SET activeTimers.Table = localTimers.Table;
      END;
      DELETE FIELD localTimers.Table;

      RETURN FALSE;
   END;
END MODULE;


And the little java function (used for cron evaluation) :
Code:
public static MbTimestamp getNextEvent(String pattern, MbTimestamp baseTimestamp) {
         log("Entering getNextEvent: '"+pattern+"' TS:"+new Date(baseTimestamp.getTimeInMillis()));
         Calendar nextDate = GregorianCalendar.getInstance();
         nextDate.setTime(new Predictor(pattern, baseTimestamp.getTime()).nextMatchingDate());
         log("SetTime OK: "+new Date(nextDate.getTimeInMillis()));
         return new MbTimestamp(   nextDate.get(Calendar.YEAR),
                           nextDate.get(Calendar.MONTH),
                           nextDate.get(Calendar.DAY_OF_MONTH),
                           nextDate.get(Calendar.HOUR_OF_DAY),
                           nextDate.get(Calendar.MINUTE),
                           0);
   }


The interesting part is here:
Code:
DECLARE lastTimer REFERENCE TO localTimers.Table.Timer[<];
      FOR timer AS localTimers.Table.Timer[] DO
         DECLARE timerRef REFERENCE TO timer;
         CALL logInfo('ROCADE',9999,'Checking ' || timer.FileName || ' ' || timer.TimerDay || ' baseTimestamp=' || CAST(baseTimestamp AS CHARACTER),'','','');
         DECLARE nextTimestamp TIMESTAMP computeNextTimestamp(CURRENT_TIMESTAMP, timerRef);
         DECLARE nextNextTimestamp TIMESTAMP computeNextTimestamp(nextTimestamp, timerRef);


Here is what we've got in our logs (the last correct execution with brkfi15.dat file, and the next one brkfi17.dat which crash) :
Code:
#ROC9999D
TRACE - Checking brkfi15.dat * baseTimestamp=19700101000000000000: BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 161: SqlThrowExceptionStatement::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#ROC9999D
TRACE - computeNextTimestamp baseTimestamp=TIMESTAMP '2013-06-04 15:19:11.746710' Pattern: */20 * * * *: BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 161: SqlThrowExceptionStatement::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#ROC9999D
TRACE - computeNextTimestamp baseTimestamp=TIMESTAMP '2013-06-04 15:20:00' Pattern: */20 * * * *: BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 161: SqlThrowExceptionStatement::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11

#ROC9999D
TRACE - Checking brkfi17.dat * baseTimestamp=19700101000000000000: BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 161: SqlThrowExceptionStatement::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#ROC9999D
TRACE - : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 161: SqlThrowExceptionStatement::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#BIP4623E
Exception condition detected by input node 'com.inetpsa.rocade.support.file.FileManager.RotateCheck'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/TimeoutControl/ImbTimeoutNotificationNode.cpp: 1581: ImbTimeoutNotificationNode::propagateToCatch: ComIbmTimeoutNotificationNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_10
#BIP2230E
Error detected whilst processing a message in node 'com.inetpsa.rocade.support.file.FileManager.Check'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbComputeNode.cpp: 489: ImbComputeNode::evaluate: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#BIP2488E
 (com.inetpsa.rocade.support.file.TimerManager_Check.Main, 66.4) Error detected whilst executing the SQL statement 'DECLARE nextTimestamp TIMESTAMP computeNextTimestamp(CURRENT_TIMESTAMP, timerRef);'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp: 643: SqlStatementGroup::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#BIP2934E
Error detected whilst executing the function or procedure 'computeNextTimestamp'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlRoutine.cpp: 628: SqlRoutine::invoke: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#BIP2488E
 (com.inetpsa.rocade.support.file.computeNextTimestamp, 10.2) Error detected whilst executing the SQL statement 'RETURN getNextEvent(timer.TimerMinute || ' ' || timer.TimerHour || ' ' || timer.TimerDay || ' ' || timer.TimerMonth || ' ' || timer.TimerDayOfWeek, baseTimestamp);'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp: 643: SqlStatementGroup::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#BIP2934E
Error detected whilst executing the function or procedure 'getNextEvent'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlRoutine.cpp: 811: SqlRoutine::invoke: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11
#BIP2919E
(com.inetpsa.rocade.support.timing.getNextEvent, 1.1) : An error occuirred when invoking the Java method 'com.inetpsa.rocade.support.timing.CronFunctions.getNextEvent'. The parameters passed to the method were 'NULL,  '''. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlExternalJava.cpp: 1422: SqlExternalJavaMethod::executeJavaMethod: :
#BIP2917E
A call to the Java method 'com.inetpsa.rocade.support.timing.CronFunctions.getNextEvent' with parameters '' failed because a Java Exception occurred. The Java Exception information is 'java.lang.NullPointerException'. : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlExternalJava.cpp: 1261: SqlExternalJavaMethod::executeJavaMethod: :



It looks like "timer" has lost all fields (timer.TimerDay for example) between the line:
CALL logInfo('ROCADE',9999,'Checking ' || timer.FileName || ' ' || timer.TimerDay || ' baseTimestamp=' || CAST(baseTimestamp AS CHARACTER),'','','');

and the very next line:

DECLARE nextTimestamp TIMESTAMP computeNextTimestamp(CURRENT_TIMESTAMP, timerRef);

it is confirmed by this "void" trace (nothing after "TRACE -" in the ESQL computeNextTimestamp function) :
Code:
TRACE - : BROKER01.3fc701b8-3701-0000-0080-ca2b86e47352: /build/S700_P/src/DataFlowEngine/ImbRdl/ImbRdlThrowExceptionStatements.cpp: 161: SqlThrowExceptionStatement::execute: ComIbmComputeNode: com/inetpsa/rocade/support/file/FileManager#FCMComposite_1_11


and in the java code:
Code:
[Tue Jun 04 15:19:12 MEST 2013] Entering getNextEvent: '*/20 * * * *' TS:Tue Jun 04 15:20:00 MEST 2013
[Tue Jun 04 15:19:12 MEST 2013] SetTime OK: Tue Jun 04 15:40:00 MEST 2013
[Tue Jun 04 15:19:12 MEST 2013] Entering getNextEvent: 'null' TS:Tue Jun 04 15:19:11 MEST 2013
[Tue Jun 04 15:19:12 MEST 2013] null


Sorry for the big code pasting, but I wanted to show you all the code.
Some other informations:
- The activeTimers shared row is used in only one other ESQL piece of code, surrended by BEGIN ATOMIC, and no modification is done on it (just a select)
- the localtimers is loaded at startup (the linecount <> lastLinecount.... is triggered), but never after that. It means that we are always looping on the same set of (79 here) timer element
- The other machine, still running with 7.0.0.1, with the same set, has no problem (like we had no problem for the last 2 years)

Sorry for this big post, but we are totally lost, and we don't understand what's happening since we installed that fix

Any help / comment welcome !

Thank you in advance
Back to top
View user's profile Send private message
McueMart
PostPosted: Wed Jun 05, 2013 4:27 am    Post subject: Reply with quote

Chevalier

Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere

Can you share the code of the computeNextTimestamp function please?
Back to top
View user's profile Send private message
raphg
PostPosted: Wed Jun 05, 2013 4:32 am    Post subject: Reply with quote

Newbie

Joined: 04 Apr 2013
Posts: 7

McueMart wrote:
Can you share the code of the computeNextTimestamp function please?


For sure. Quite simple (I added the pattern variable for logging purpose)

Quote:
CREATE FUNCTION computeNextTimestamp(IN baseTimestamp TIMESTAMP, IN timer REFERENCE) RETURNS TIMESTAMP
BEGIN
DECLARE pattern CHARACTER timer.TimerMinute || ' ' ||
timer.TimerHour || ' ' ||
timer.TimerDay || ' ' ||
timer.TimerMonth || ' ' ||
timer.TimerDayOfWeek;

CALL logInfo('ROCADE',9999,'computeNextTimestamp baseTimestamp=' || CAST(baseTimestamp AS CHARACTER) || ' Pattern: ' || pattern,'','','');
RETURN getNextEvent(timer.TimerMinute || ' ' ||
timer.TimerHour || ' ' ||
timer.TimerDay || ' ' ||
timer.TimerMonth || ' ' ||
timer.TimerDayOfWeek,
baseTimestamp);
END;


and here is the declaration of the java method:
Code:
BROKER SCHEMA com.inetpsa.rocade.support.timing

CREATE FUNCTION getNextEvent(IN pattern CHARACTER, IN baseTimestamp TIMESTAMP) RETURNS TIMESTAMP
LANGUAGE JAVA EXTERNAL NAME "com.inetpsa.rocade.support.timing.CronFunctions.getNextEvent";
Back to top
View user's profile Send private message
McueMart
PostPosted: Wed Jun 05, 2013 8:25 am    Post subject: Reply with quote

Chevalier

Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere

So it looks like one of more of these fields are evaluating to NULL:

timer.TimerMinute
timer.TimerHour
timer.TimerDay
timer.TimerMonth
timer.TimerDayOfWeek

(remembering that concatenating a NULL with a string in ESQL results in a NULL...)

What I don't quite understand is why the call to:
Code:

CALL logInfo('ROCADE',9999,'computeNextTimestamp baseTimestamp=' || CAST(baseTimestamp AS CHARACTER) || ' Pattern: ' || pattern,'','','');


Isn't displaying in the log you posted (for the case when pattern = NULL).
As that call should be equivalent to:

CALL logInfo('ROCADE',9999,NULL,'','','');

Maybe you have something in your logInfo function which prevents the output of NULL strings?

Have you 100% checked that the value of the fields in the database for the fields:

Database.BRK01.BRKCFG.BRKQTOUTFILE.ROTATION_MINUTE
Database.BRK01.BRKCFG.BRKQTOUTFILE.ROTATION_HOUR
etc...

are all valid?
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Jun 05, 2013 8:31 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

I'm always leery of CASTing timestamps to characters without specifying a pattern.
Back to top
View user's profile Send private message
raphg
PostPosted: Thu Jun 06, 2013 11:50 pm    Post subject: Reply with quote

Newbie

Joined: 04 Apr 2013
Posts: 7

McueMart wrote:
So it looks like one of more of these fields are evaluating to NULL:

timer.TimerMinute
timer.TimerHour
timer.TimerDay
timer.TimerMonth
timer.TimerDayOfWeek

(remembering that concatenating a NULL with a string in ESQL results in a NULL...)

What I don't quite understand is why the call to:
Code:

CALL logInfo('ROCADE',9999,'computeNextTimestamp baseTimestamp=' || CAST(baseTimestamp AS CHARACTER) || ' Pattern: ' || pattern,'','','');


Isn't displaying in the log you posted (for the case when pattern = NULL).
As that call should be equivalent to:

CALL logInfo('ROCADE',9999,NULL,'','','');

Maybe you have something in your logInfo function which prevents the output of NULL strings?

Have you 100% checked that the value of the fields in the database for the fields:

Database.BRK01.BRKCFG.BRKQTOUTFILE.ROTATION_MINUTE
Database.BRK01.BRKCFG.BRKQTOUTFILE.ROTATION_HOUR
etc...

are all valid?


Hello,

Thank you for your answer.
I totally agree with "one of thoses fields are evaluating to NULL". I've traced that, no doubt about. The problem is: it should not be changed ! In fact, it's part of dynamic confirguration, loaded at startup, and it does not change after that.
Then every 30s, we're running that piece of code to detect which operations have to be done (regarding a cron expression). So we can say this code is executed 120 times an hour, without any problem, except after several hours (it could be 6, 8, 10 hours...) this code crashes, and our traces showed that one of those field went NULL. Even stranger : it's not always the same field !

Concernings logs, our method does nothing else but call LOG EVENT SEVERITY ...., and message code 9999 looks like "TRACE - {0}". I've removed this call to logInfo, replacing it with LOG EVENT SEVERITY, to avoid misleading investigations.

After several other modifications / traces added, it looks like memory corruption.

Please note that the second computer, still in 7.0.0.1, running the same code / configuration datas for nearly two years, has no problem at all.

Thank you for any help
Back to top
View user's profile Send private message
McueMart
PostPosted: Fri Jun 07, 2013 12:58 am    Post subject: Reply with quote

Chevalier

Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere

Try to create a small flow which re-creates your problem and raise as a PMR. I worked somewhere recently where someone also saw a memory corruption issue with shared rows (they raised a PMR which I thought had been fixed...).
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Jun 07, 2013 5:38 am    Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

McueMart wrote:
Try to create a small flow which re-creates your problem and raise as a PMR. I worked somewhere recently where someone also saw a memory corruption issue with shared rows (they raised a PMR which I thought had been fixed...).


If it has been fixed, it likely wasn't fixed in 7.0.0.1....
Back to top
View user's profile Send private message
raphg
PostPosted: Fri Jun 07, 2013 5:48 am    Post subject: Reply with quote

Newbie

Joined: 04 Apr 2013
Posts: 7

Unfortunately, we had no problem with 7.0.0.1, and our problems started when upgrading to 7.0.0.5
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Jun 07, 2013 6:02 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

You might want to look at your Java code. I see you are using Calendar. However AFAIK this class is not thread safe. You can alleviate this by using the ThreadLocal variant for it...

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Strange problem after 7.0.0.5 fix deployment
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.