| Author |
Message
|
| crashdog |
 Posted: Thu Mar 15, 2018 4:33 am Post subject: mq error log rotation |
 |
|
Voyager
Joined: 02 Apr 2017
Posts: 77
|
Hello,
I'm writing a process in c++ that "tails" the mq error log files AMQERR01.LOG. Basically that's straight forward and works.
The only thing that I'm a little unsure about is the rotation of the logfiles. From what I've seen on Unix systems, the file content gets piped from AMQERRO3.LOG to /dev/null. From AMQERR02.LOG to AMQERRO03.LOG and from AMQERR01.LOG to AMQERR02.LOG. The AMQERR01.LOG gets emptied but keeps it's inode number.
Now my main concern is that I would miss an entry before it's rotated. I don't want to parse the content in this process, it should only forward the messages to a queue.
So my question, how can I prevent the los/duplicate of an error log entry when the error log file gets rotated ?
ps: from what I've seen it actually works, but that could be just a coincidence.
Kind Regards,
Gerhard
_________________
You win again gravity ! |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Thu Mar 15, 2018 5:05 am Post subject: Re: mq error log rotation |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
| crashdog wrote: |
| ps: from what I've seen it actually works, but that could be just a coincidence. |
What is it?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| crashdog |
| Posted: Thu Mar 15, 2018 6:08 am Post subject: |
|
|
Voyager
Joined: 02 Apr 2017
Posts: 77
|
"it" is my process that reads the error log files continously and manages to correctly read the error file from start after it's been rotated. There are no missing or duplicate error log entries. But my fear is that it's a coincidence and that there might be some cases when my process might lose an entry or read one twice.
That would surely not happen if there's anything "hard" to rely on when the error logs rotate. Like for example the execution of the queue manager being postponed a few milliseconds during the rotation. Some kind of signal or... I don't know what's reliable to check for when the error log rotates.
Cheers,
Gerhard
_________________
You win again gravity ! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 15, 2018 7:43 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
So, your concern is that in the short time between 0001.LOG being renamed to 0002.LOG, and BEFORE a new 001.LOG is created, that some errors might NOT be recoded? Is that it?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| crashdog |
| Posted: Thu Mar 15, 2018 8:07 am Post subject: |
|
|
Voyager
Joined: 02 Apr 2017
Posts: 77
|
Basically yes. I know it sounds pedantic. But looking at timing issues in computers, sometimes milliseconds matter... when the event that the file has new information is promoted to my process, the error log might allready have rotated before the read completed and missed some important errors / information.
| Quote: |
| So, your concern is that in the short time between 0001.LOG being renamed to 0002.LOG, and BEFORE a new 001.LOG is created, that some errors might NOT be recoded? Is that it? |
actually I don't think the files are renamed... if they where, they would have different inode numbers.
_________________
You win again gravity ! |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Mar 15, 2018 9:19 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
| crashdog wrote: |
Basically yes. I know it sounds pedantic. But looking at timing issues in computers, sometimes milliseconds matter... when the event that the file has new information is promoted to my process, the error log might allready have rotated before the read completed and missed some important errors / information.
| Quote: |
| So, your concern is that in the short time between 0001.LOG being renamed to 0002.LOG, and BEFORE a new 001.LOG is created, that some errors might NOT be recoded? Is that it? |
actually I don't think the files are renamed... if they where, they would have different inode numbers. |
I seem to recall that rename does not create a new inode (but does alter date/time stamp), however I will yield to a UNIX internals specialist.
I can't imagine that MQ internals would use the copy command to physically copy the entire error log files to accomplish this rotation.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| crashdog |
| Posted: Thu Mar 15, 2018 9:47 am Post subject: |
|
|
Voyager
Joined: 02 Apr 2017
Posts: 77
|
when you rename file 001.log to 002.log the 002.log will new have 001.log's inode number and 001.log needs a new inode number.
| Code: |
4500794 -rw-r--r-- 1 xxx mqm 0 Mar 15 18:43 001.log
4500802 -rw-r--r-- 1 xxx mqm 0 Mar 15 18:43 002.log
mv 002.log 003.log
mv 001.log 0002.log
4500794 -rw-r--r-- 1 xxx mqm 0 Mar 15 18:43 0002.log
4500802 -rw-r--r-- 1 xxx mqm 0 Mar 15 18:43 003.log
|
But in case of MQ I've seen that the inode numbers stay the same. Therefore my guess is that the file content is piped to the 002 and 003 logs.
_________________
You win again gravity ! |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Mar 15, 2018 3:32 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2527
Location: Melbourne, Australia
|
| crashdog wrote: |
| Therefore my guess is that the file content is piped to the 002 and 003 logs. |
File piping / copying does not sound very efficient, particularly if the error logs are large. I have always assumed that MQ Deletes 03, Renames 02 to 03, Renames 01 to 02, Writes to new 01.
_________________
Glenn |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Mar 16, 2018 8:28 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
It's really the inode that uniquely identifies a file in Unix. It has been a while since I last checked, but I do remember the behavior that the inodes do not change on the AMQERR0x.LOG queue manager error logs as the data is moved between the three files. This probably means that MQ is just programatically moving the data between these three files. So when it is time to move the data from AMQERR01.LOG to AMQERR02.LOG, MQ probably just programmatically clears the records from AMQERR02.LOG and then writes the data from AMQERR01.LOG into AMQERR02.LOG.
If your requirement is to capture all the messages that could get written to the queue manager error logs, you probably need a more sophisticated program that is sweeping through the different AMQERR0x.LOG files and checking file update timestamps to make sure it is not missing any data due to an error log rotation.
_________________
Working with MQ since 2010.
Last edited by tczielke on Fri Mar 16, 2018 9:21 am; edited 1 time in total |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Mar 16, 2018 9:08 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
| gbaddeley wrote: |
| crashdog wrote: |
| Therefore my guess is that the file content is piped to the 002 and 003 logs. |
File piping / copying does not sound very efficient, particularly if the error logs are large. I have always assumed that MQ Deletes 03, Renames 02 to 03, Renames 01 to 02, Writes to new 01. |
My recollection is similar to that of gbaddeley.
I suspect that someone from IBM will be along to offer a more definitive explanation. I suspect, as well, that this is one of those 'how does x work' vs. 'exactly how does x work' instances.
Exactly how IBM internals accomplish any given task is proprietary, and may/will change over time and technology.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Mar 16, 2018 10:23 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5859
Location: UK
|
They're probably all en-route to Vegas..... 
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 16, 2018 10:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| They're probably all en-route to Vegas..... |
And any updates while they're there must be viewed with suspicion (unless accompanied by a negative breathalyzer reading!)
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| markt |
| Posted: Fri Mar 16, 2018 3:23 pm Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
|
| Back to top |
|
|
| rekarm01 |
| Posted: Fri Mar 16, 2018 8:28 pm Post subject: Re: mq error log rotation |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
When all else fails, check the Knowledge Center.
| crashdog wrote: |
| The only thing that I'm a little unsure about is the rotation of the logfiles. From what I've seen on Unix systems, the file content gets piped from AMQERRO3.LOG to /dev/null ... |
No piping. No /dev/null. But the behavior does depend on the version of MQ.
Prior to MQ v9.0.4, when AMQERR01.LOG gets full, the qmgr copies AMQERR02.LOG to AMQERR03.LOG, then copies AMQERR01.LOG to AMQERR02.LOG, then truncates AMQERR01.LOG. It is possible for one process to truncate a file, before another process can finish reading it.
As of MQ v9.0.4, when AMQERR01.LOG gets full, the qmgr removes AMQERR03.LOG, then renames AMQERR02.LOG to AMQERR03.LOG, then renames AMQERR01.LOG to AMQERR02.LOG, then creates a new AMQERR01.LOG. A process with an already open file handle can continue reading from an otherwise renamed (or even removed) file, but it would have to open a new file handle to read from a new file.
| bruce2359 wrote: |
| I seem to recall that rename does not create a new inode (but does alter date/time stamp), however I will yield to a UNIX internals specialist. |
The filename is a property of the directory entry, not the file. Changing the name in the directory entry alters the directory's timestamp, but has no effect whatsoever on the file that the directory entry points to. |
|
| Back to top |
|
|
| crashdog |
| Posted: Sat Mar 17, 2018 2:00 am Post subject: |
|
|
Voyager
Joined: 02 Apr 2017
Posts: 77
|
The knowledge center was the first place that I checked, before even starting this thread. https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.tro.doc/q039570_.htm . But the KC does not describe anything on how the error log files are rotated.
By the way CD is out of scope in the organization that I currently work. It was decided to go with LTS.
At one point we will start using elastic for error log file statistics. Elastic uses filebeat to forward log information. I've read this artice on it : https://www.ibm.com/developerworks/community/blogs/messaging/entry/Storing_and_searching_MQ_logs_in_Elasticsearch?lang=en .
There would also have been logstash written i Java. But the idea at the moment is to use the c++ standard library (which is always installed on our systems) with statically linked QT functionallity for file handling. We don't want to install addtional frameworks such as java or go runtime.
I will try to setup an experimental queue manager in my homelab tomorrow and try to look further into the rotating part.
_________________
You win again gravity ! |
|
| Back to top |
|
|
|
|
