ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexIBM MQ Installation/Configuration SupportUnique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL V7

Post new topicReply to topic Goto page 1, 2  Next
Unique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL V7 View previous topic :: View next topic
Author Message
praveenhegde
PostPosted: Fri Oct 13, 2017 6:22 am Post subject: Unique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL V7 Reply with quote

Novice

Joined: 14 Aug 2012
Posts: 18

Hi All,

We have a very unique issue related to AMQERROR03.log issue under /data/mqm/qmgrs/QmgrName/errors.

Background:

IBM MQ9 is installed on RHEL V7

Version:
dspmqver
Name: IBM MQ
Version: 9.0.2.0
Level: p902-L170310
BuildType: IKAP - (Production)
Platform: IBM MQ for Linux (x86-64 platform)
Mode: 64-bit
O/S: Linux 3.10.0-514.2.2.el7.x86_64
InstName: Installation1
InstDesc:
Primary: Yes
InstPath: /opt/mqm
DataPath: /var/mqm
MaxCmdLevel: 902
LicenseType: Production

Server Details:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release: 7.3
Codename: Maipo


Issue: As we have created new Qmgr on IBM MQ 9 , We could see that 3 error log file were created. AMQERROR01/2/3.

But as the MQ transactions started , we observed that AMQ logs were properly filled AMQ1.LOG ----->AMQ2.LOG when AMQ1 reached 2MB but as soon as AMQ2.LOG becomes 2MB ideally it should push to AMQ3.LOG either by creating the new file in case it does not exist or push the old logs if it exists.

But we are seeing that AMQ2 OLD logs are being disappeared after new log details
comes from AMQ1 log to AMQ2. Nothing is being written to AMQ3.LOG after AMQ2.log reaches 2 MB

Irrespective of servers or Qmgr , i.e the issue persists across all our DEV / QA and now we are going to Production.

What Action did we take so far for the testing?
1. Verified whether we had enough space under /data/mqm/Qmgrs/QMGRNAME/errors

2. Verified whether mqm user and group have proper access to write the logs, everything looks good.

3. Just to ensure whether Qmgr is generating AMQERROR03.LOG or not. We have deleted AMQERROR03.log and stopped and started the Qmgr as well. We have found that AMQ3.log being generated.

4.We have manaully pushed logs to AMQ3.log to see for any right issues etc, but we can do the same. PMR is not raised yet. We have not defined anything in qm.ini or any variables about QMerrorStana values.

Do you people have any idea, or faced similar kind of issues ?

Any quicker suggestions or findings appriciated



_________________
Thanks & Regards,
Praveen Hegde
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Sun Oct 15, 2017 2:45 pm Post subject: Reply with quote

Padawan

Joined: 25 Mar 2003
Posts: 1732
Location: Melbourne, Australia

Are you sure you have the correct file names?
They are AMQERR01.LOG, AMQERR02.LOG, AMQERR03.LOG
No other names can be used.
Can you provide an 'ls -l' of the errors directory?
_________________
Glenn
Back to top
View user's profile Send private message
zpat
PostPosted: Mon Oct 16, 2017 2:09 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5558
Location: UK

I like to increase the error log file size as the default is far too low and allow repeating errors to wrap the logs too quickly.

But in general terms, why are you getting errors anyway?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Mon Oct 16, 2017 3:34 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17448

amqerr logs are error logs. Not sure why you would care if one of them got deleted - other than that you lost the data in them.

Easiest way, and possible safest way to fill them is to keep making telnet connections to the listener port... or otherwise send junk data to it in new connections.

The listener will throw error messages - more than one, I think? - every time it gets non-mq traffic on the port.

Likely the "probelm" you are seeing is that MQ keeps a persistent file handle, once it's open, to each of the amqerr files. when you delete the file, that doesn't close the handle.

Raise a PMR if you feel strongly about it. s

Or don't actually recreate the deleted file. That might cause MQ to notice it's file handle is bad and then create a new file.

Thirdly, you might have missed some special bits - things that would have shown up as "s" in 'ls -l' output... Which might be why Glenn wants you to provide it...
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
rammer
PostPosted: Mon Oct 16, 2017 4:33 am Post subject: Reply with quote

Partisan

Joined: 02 May 2002
Posts: 343
Location: England majority USA the rest..

I can vouch for this issue. Just spun up a Reh Hat 7 and MQ 9 installation and tested the AMERR* scenarios out with rolling data to 2 & 3 and going above 2Mb etc and it does look strange!...
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Oct 16, 2017 4:36 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19380
Location: LI,NY

And last I'm not sure you understand how the file rollover works for the errorlogs....

MQ only writes to a single log: AMQERR01.LOG. Always. It is always the most current log and the latest entries are at the bottom of the log...
Once the log is full, AMQERR02.LOG gets copied/renamed to AMQERR03.LOG; AMQERR01.LOG get copied/renamed to AMQERR02.LOG and the queue manager continues to log into a new AMQERR01.LOG.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
rammer
PostPosted: Mon Oct 16, 2017 5:08 am Post subject: Reply with quote

Partisan

Joined: 02 May 2002
Posts: 343
Location: England majority USA the rest..

Yes understand that, but it doesnt explain why the defaults are being used of 2Mb yet the file increments upto 3Mb + and then only seem to rollover / copy / rename what ever you wish to call it after a long period of time or the only way I could trigger it was to stop / start the queue manager....
Back to top
View user's profile Send private message
praveenhegde
PostPosted: Mon Oct 23, 2017 10:23 pm Post subject: Correct Names Reply with quote

Novice

Joined: 14 Aug 2012
Posts: 18

gbaddeley wrote:
Are you sure you have the correct file names?
They are AMQERR01.LOG, AMQERR02.LOG, AMQERR03.LOG
No other names can be used.
Can you provide an 'ls -l' of the errors directory?


Though while writing I may have done some spelling mistake.. below is the correct File Names under Qmgr error logs
Code:
==================
 ls -ltr
total 3340
-rw-rw---- 1 mqm mqm 2097530 Oct  9 14:26 AMQERR02.LOG
-rw-rw---- 1 mqm mqm      55 Oct 10 11:49 AMQERR03.LOG.2017-10-10-11:50:16.gz
-rw-r--r-- 1 mqm mqm      58 Oct 11 12:27 AMQERR03.LOG.2017-10-11-12:28:28.gz
-rw-rw---- 1 mqm mqm       0 Oct 12 11:55 AMQERR03.LOG
-rw-rw---- 1 mqm mqm 1304337 Oct 24 08:01 AMQERR01.LOG

_________________
Thanks & Regards,
Praveen Hegde
Back to top
View user's profile Send private message
praveenhegde
PostPosted: Mon Oct 23, 2017 10:28 pm Post subject: Query regarding logs getting generated Reply with quote

Novice

Joined: 14 Aug 2012
Posts: 18

zpat wrote:
I like to increase the error log file size as the default is far too low and allow repeating errors to wrap the logs too quickly.

But in general terms, why are you getting errors anyway?


In general we are not getting too much errors.. but you must be aware that , these error logs will also capture channel start and stop etc as well.. if we have too much frequent transactions, logs would get generated. In this case by default is 2MB which is sufficient i believe and regularly we are having backup of AMQEROR03 log , hope now it makes bit more sense about the issue..
_________________
Thanks & Regards,
Praveen Hegde
Back to top
View user's profile Send private message
praveenhegde
PostPosted: Mon Oct 23, 2017 10:38 pm Post subject: Reply with quote

Novice

Joined: 14 Aug 2012
Posts: 18

fjb_saper wrote:
And last I'm not sure you understand how the file rollover works for the errorlogs....

MQ only writes to a single log: AMQERR01.LOG. Always. It is always the most current log and the latest entries are at the bottom of the log...
Once the log is full, AMQERR02.LOG gets copied/renamed to AMQERR03.LOG; AMQERR01.LOG get copied/renamed to AMQERR02.LOG and the queue manager continues to log into a new AMQERR01.LOG.

Have fun


I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange ! one of out architect tried RHEL V7 and IBM MQ 9 on VM.. he is stating each time we need to stop and start Qmgr to ensure AMQ1 gets copied to AMQ2 and again stop and start Qmgr to AMQ2 -->AMQ3 ...We were totally confused! something really wrong with IBM MQ 9 and RHEL V7 combination?
_________________
Thanks & Regards,
Praveen Hegde
Back to top
View user's profile Send private message
praveenhegde
PostPosted: Tue Oct 24, 2017 4:56 am Post subject: Re: Unique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL Reply with quote

Novice

Joined: 14 Aug 2012
Posts: 18

praveenhegde wrote:
Hi All,

We have a very unique issue related to AMQERROR03.log issue under /data/mqm/qmgrs/QmgrName/errors.

Background:

IBM MQ9 is installed on RHEL V7

Version:
dspmqver
Name: IBM MQ
Version: 9.0.2.0
Level: p902-L170310
BuildType: IKAP - (Production)
Platform: IBM MQ for Linux (x86-64 platform)
Mode: 64-bit
O/S: Linux 3.10.0-514.2.2.el7.x86_64
InstName: Installation1
InstDesc:
Primary: Yes
InstPath: /opt/mqm
DataPath: /var/mqm
MaxCmdLevel: 902
LicenseType: Production

Server Details:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release: 7.3
Codename: Maipo


Issue: As we have created new Qmgr on IBM MQ 9 , We could see that 3 error log file were created. AMQERROR01/2/3.

But as the MQ transactions started , we observed that AMQ logs were properly filled AMQ1.LOG ----->AMQ2.LOG when AMQ1 reached 2MB but as soon as AMQ2.LOG becomes 2MB ideally it should push to AMQ3.LOG either by creating the new file in case it does not exist or push the old logs if it exists.

But we are seeing that AMQ2 OLD logs are being disappeared after new log details
comes from AMQ1 log to AMQ2. Nothing is being written to AMQ3.LOG after AMQ2.log reaches 2 MB

Irrespective of servers or Qmgr , i.e the issue persists across all our DEV / QA and now we are going to Production.

What Action did we take so far for the testing?
1. Verified whether we had enough space under /data/mqm/Qmgrs/QMGRNAME/errors

2. Verified whether mqm user and group have proper access to write the logs, everything looks good.

3. Just to ensure whether Qmgr is generating AMQERROR03.LOG or not. We have deleted AMQERROR03.log and stopped and started the Qmgr as well. We have found that AMQ3.log being generated.

4.We have manaully pushed logs to AMQ3.log to see for any right issues etc, but we can do the same. PMR is not raised yet. We have not defined anything in qm.ini or any variables about QMerrorStana values.

Do you people have any idea, or faced similar kind of issues ?

Any quicker suggestions or findings appriciated




Let me clarify one more thing..! IBM MQ 7.5 on RHEL V7 is perfectly working fine. Issue is only with IBM MQ 9.0.2 on RHEL V7
_________________
Thanks & Regards,
Praveen Hegde
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Oct 24, 2017 4:58 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19380
Location: LI,NY

praveenhegde wrote:

I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange ! one of out architect tried RHEL V7 and IBM MQ 9 on VM.. he is stating each time we need to stop and start Qmgr to ensure AMQ1 gets copied to AMQ2 and again stop and start Qmgr to AMQ2 -->AMQ3 ...We were totally confused! something really wrong with IBM MQ 9 and RHEL V7 combination?

Your understanding is wrong. AMQERR02.LOG can be full and it will not get copied / renamed to AMQERR03.LOG until AMQERR01.LOG is full.
Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
gbaddeley
PostPosted: Tue Oct 24, 2017 2:38 pm Post subject: Reply with quote

Padawan

Joined: 25 Mar 2003
Posts: 1732
Location: Melbourne, Australia

Quote:
I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange !

The error logs aren't copied at rollover, that would be too ineffiecent. Size management is only done on log 1. When it reaches the threshold, log 3 is deleted, log 2 is renamed to log 3, log 1 is renamed to log 2, and a new log 1 is created and used.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Oct 24, 2017 4:26 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 7844
Location: US: west coast, almost. Otherwise, enroute.

The KC and IBM training doc continue to imprecisely use 'copy' and 'copied' to describe the mq error log file rotation process.

For example, https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.tro.doc/q039560_.htm says:

Quote:
As error messages are generated, they are placed in AMQERR01. When AMQERR01 gets bigger than 2 MB (2 097 152 bytes) it is copied to AMQERR02. Before the copy, AMQERR02 is copied to AMQERR03.LOG. The previous contents, if any, of AMQERR03 are discarded.

_________________
I didn't know that Schrdinger had a cat.
Back to top
View user's profile Send private message
praveenhegde
PostPosted: Wed Nov 08, 2017 10:41 pm Post subject: Real Issue identified with IBM MQ 9.0.2/3!IBM Provided Ifix Reply with quote

Novice

Joined: 14 Aug 2012
Posts: 18

gbaddeley wrote:
Quote:
I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange !

The error logs aren't copied at rollover, that would be too ineffiecent. Size management is only done on log 1. When it reaches the threshold, log 3 is deleted, log 2 is renamed to log 3, log 1 is renamed to log 2, and a new log 1 is created and used.


I agree.. It was a typo! I should have mentioned AMQ1! instead of AMQ2

I now understand clearly AMQ1--->AMQ2-->AMQ3 : once AMQ1 is 2MB it will be renamed to AMQ2, and earlier AMQ2 will get renamed to AMQ3 .


Lets come to real issue!.

IBM MQ 9.0.2 and IBM MQ 9.0.3 CD has the AMQ Error log issue. below is APAR link.

Ref: http://www-01.ibm.com/support/docview.wss?uid=swg1IT22283


Issue would be fixed in IBM MQ 9.0.5(not yet released!).

They have provided the Ifix for IBM MQ 9.0.2 and IBM MQ 9.0.3 currently after raising the PMR.

Thanks to all who have given suggestions and gyaan on how IBM MQ Logs works!

Praveen Hegde
_________________
Thanks & Regards,
Praveen Hegde
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexIBM MQ Installation/Configuration SupportUnique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL V7
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.