| Author |
Message
|
| praveenhegde |
 Posted: Fri Oct 13, 2017 6:22 am Post subject: Unique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL V7 |
 |
|
Novice
Joined: 14 Aug 2012
Posts: 20
|
Hi All,
We have a very unique issue related to AMQERROR03.log issue under /data/mqm/qmgrs/QmgrName/errors.
Background:
IBM MQ9 is installed on RHEL V7
Version:
dspmqver
Name: IBM MQ
Version: 9.0.2.0
Level: p902-L170310
BuildType: IKAP - (Production)
Platform: IBM MQ for Linux (x86-64 platform)
Mode: 64-bit
O/S: Linux 3.10.0-514.2.2.el7.x86_64
InstName: Installation1
InstDesc:
Primary: Yes
InstPath: /opt/mqm
DataPath: /var/mqm
MaxCmdLevel: 902
LicenseType: Production
Server Details:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release: 7.3
Codename: Maipo
Issue: As we have created new Qmgr on IBM MQ 9 , We could see that 3 error log file were created. AMQERROR01/2/3.
But as the MQ transactions started , we observed that AMQ logs were properly filled AMQ1.LOG ----->AMQ2.LOG when AMQ1 reached 2MB but as soon as AMQ2.LOG becomes 2MB ideally it should push to AMQ3.LOG either by creating the new file in case it does not exist or push the old logs if it exists.
But we are seeing that AMQ2 OLD logs are being disappeared after new log details
comes from AMQ1 log to AMQ2. Nothing is being written to AMQ3.LOG after AMQ2.log reaches 2 MB
Irrespective of servers or Qmgr , i.e the issue persists across all our DEV / QA and now we are going to Production.
What Action did we take so far for the testing?
1. Verified whether we had enough space under /data/mqm/Qmgrs/QMGRNAME/errors
2. Verified whether mqm user and group have proper access to write the logs, everything looks good.
3. Just to ensure whether Qmgr is generating AMQERROR03.LOG or not. We have deleted AMQERROR03.log and stopped and started the Qmgr as well. We have found that AMQ3.log being generated.
4.We have manaully pushed logs to AMQ3.log to see for any right issues etc, but we can do the same. PMR is not raised yet. We have not defined anything in qm.ini or any variables about QMerrorStana values.
Do you people have any idea, or faced similar kind of issues ?
Any quicker suggestions or findings appriciated
 
_________________
Thanks & Regards,
Praveen Hegde |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Sun Oct 15, 2017 2:45 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Are you sure you have the correct file names?
They are AMQERR01.LOG, AMQERR02.LOG, AMQERR03.LOG
No other names can be used.
Can you provide an 'ls -l' of the errors directory?
_________________
Glenn |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Oct 16, 2017 2:09 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5867
Location: UK
|
I like to increase the error log file size as the default is far too low and allow repeating errors to wrap the logs too quickly.
But in general terms, why are you getting errors anyway?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Oct 16, 2017 3:34 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
amqerr logs are error logs. Not sure why you would care if one of them got deleted - other than that you lost the data in them.
Easiest way, and possible safest way to fill them is to keep making telnet connections to the listener port... or otherwise send junk data to it in new connections.
The listener will throw error messages - more than one, I think? - every time it gets non-mq traffic on the port.
Likely the "probelm" you are seeing is that MQ keeps a persistent file handle, once it's open, to each of the amqerr files. when you delete the file, that doesn't close the handle.
Raise a PMR if you feel strongly about it. s
Or don't actually recreate the deleted file. That might cause MQ to notice it's file handle is bad and then create a new file.
Thirdly, you might have missed some special bits - things that would have shown up as "s" in 'ls -l' output... Which might be why Glenn wants you to provide it...
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| rammer |
| Posted: Mon Oct 16, 2017 4:33 am Post subject: |
|
|
Partisan
Joined: 02 May 2002
Posts: 359
Location: England
|
| I can vouch for this issue. Just spun up a Reh Hat 7 and MQ 9 installation and tested the AMERR* scenarios out with rolling data to 2 & 3 and going above 2Mb etc and it does look strange!... |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Oct 16, 2017 4:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
And last I'm not sure you understand how the file rollover works for the errorlogs....
MQ only writes to a single log: AMQERR01.LOG. Always. It is always the most current log and the latest entries are at the bottom of the log...
Once the log is full, AMQERR02.LOG gets copied/renamed to AMQERR03.LOG; AMQERR01.LOG get copied/renamed to AMQERR02.LOG and the queue manager continues to log into a new AMQERR01.LOG.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rammer |
| Posted: Mon Oct 16, 2017 5:08 am Post subject: |
|
|
Partisan
Joined: 02 May 2002
Posts: 359
Location: England
|
| Yes understand that, but it doesnt explain why the defaults are being used of 2Mb yet the file increments upto 3Mb + and then only seem to rollover / copy / rename what ever you wish to call it after a long period of time or the only way I could trigger it was to stop / start the queue manager.... |
|
| Back to top |
|
|
| praveenhegde |
| Posted: Mon Oct 23, 2017 10:23 pm Post subject: Correct Names |
|
|
Novice
Joined: 14 Aug 2012
Posts: 20
|
| gbaddeley wrote: |
Are you sure you have the correct file names?
They are AMQERR01.LOG, AMQERR02.LOG, AMQERR03.LOG
No other names can be used.
Can you provide an 'ls -l' of the errors directory? |
Though while writing I may have done some spelling mistake.. below is the correct File Names under Qmgr error logs
| Code: |
==================
ls -ltr
total 3340
-rw-rw---- 1 mqm mqm 2097530 Oct 9 14:26 AMQERR02.LOG
-rw-rw---- 1 mqm mqm 55 Oct 10 11:49 AMQERR03.LOG.2017-10-10-11:50:16.gz
-rw-r--r-- 1 mqm mqm 58 Oct 11 12:27 AMQERR03.LOG.2017-10-11-12:28:28.gz
-rw-rw---- 1 mqm mqm 0 Oct 12 11:55 AMQERR03.LOG
-rw-rw---- 1 mqm mqm 1304337 Oct 24 08:01 AMQERR01.LOG |
_________________
Thanks & Regards,
Praveen Hegde |
|
| Back to top |
|
|
| praveenhegde |
| Posted: Mon Oct 23, 2017 10:28 pm Post subject: Query regarding logs getting generated |
|
|
Novice
Joined: 14 Aug 2012
Posts: 20
|
| zpat wrote: |
I like to increase the error log file size as the default is far too low and allow repeating errors to wrap the logs too quickly.
But in general terms, why are you getting errors anyway? |
In general we are not getting too much errors.. but you must be aware that , these error logs will also capture channel start and stop etc as well.. if we have too much frequent transactions, logs would get generated. In this case by default is 2MB which is sufficient i believe and regularly we are having backup of AMQEROR03 log , hope now it makes bit more sense about the issue.. 
_________________
Thanks & Regards,
Praveen Hegde |
|
| Back to top |
|
|
| praveenhegde |
| Posted: Mon Oct 23, 2017 10:38 pm Post subject: |
|
|
Novice
Joined: 14 Aug 2012
Posts: 20
|
| fjb_saper wrote: |
And last I'm not sure you understand how the file rollover works for the errorlogs....
MQ only writes to a single log: AMQERR01.LOG. Always. It is always the most current log and the latest entries are at the bottom of the log...
Once the log is full, AMQERR02.LOG gets copied/renamed to AMQERR03.LOG; AMQERR01.LOG get copied/renamed to AMQERR02.LOG and the queue manager continues to log into a new AMQERR01.LOG.
Have fun |
I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange ! one of out architect tried RHEL V7 and IBM MQ 9 on VM.. he is stating each time we need to stop and start Qmgr to ensure AMQ1 gets copied to AMQ2 and again stop and start Qmgr to AMQ2 -->AMQ3 ...We were totally confused! something really wrong with IBM MQ 9 and RHEL V7 combination?
_________________
Thanks & Regards,
Praveen Hegde |
|
| Back to top |
|
|
| praveenhegde |
| Posted: Tue Oct 24, 2017 4:56 am Post subject: Re: Unique Issue with IBMMQ9 AMQERR03.LOG installed on RHEL |
|
|
Novice
Joined: 14 Aug 2012
Posts: 20
|
| praveenhegde wrote: |
Hi All,
We have a very unique issue related to AMQERROR03.log issue under /data/mqm/qmgrs/QmgrName/errors.
Background:
IBM MQ9 is installed on RHEL V7
Version:
dspmqver
Name: IBM MQ
Version: 9.0.2.0
Level: p902-L170310
BuildType: IKAP - (Production)
Platform: IBM MQ for Linux (x86-64 platform)
Mode: 64-bit
O/S: Linux 3.10.0-514.2.2.el7.x86_64
InstName: Installation1
InstDesc:
Primary: Yes
InstPath: /opt/mqm
DataPath: /var/mqm
MaxCmdLevel: 902
LicenseType: Production
Server Details:
lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 7.3 (Maipo)
Release: 7.3
Codename: Maipo
Issue: As we have created new Qmgr on IBM MQ 9 , We could see that 3 error log file were created. AMQERROR01/2/3.
But as the MQ transactions started , we observed that AMQ logs were properly filled AMQ1.LOG ----->AMQ2.LOG when AMQ1 reached 2MB but as soon as AMQ2.LOG becomes 2MB ideally it should push to AMQ3.LOG either by creating the new file in case it does not exist or push the old logs if it exists.
But we are seeing that AMQ2 OLD logs are being disappeared after new log details
comes from AMQ1 log to AMQ2. Nothing is being written to AMQ3.LOG after AMQ2.log reaches 2 MB
Irrespective of servers or Qmgr , i.e the issue persists across all our DEV / QA and now we are going to Production.
What Action did we take so far for the testing?
1. Verified whether we had enough space under /data/mqm/Qmgrs/QMGRNAME/errors
2. Verified whether mqm user and group have proper access to write the logs, everything looks good.
3. Just to ensure whether Qmgr is generating AMQERROR03.LOG or not. We have deleted AMQERROR03.log and stopped and started the Qmgr as well. We have found that AMQ3.log being generated.
4.We have manaully pushed logs to AMQ3.log to see for any right issues etc, but we can do the same. PMR is not raised yet. We have not defined anything in qm.ini or any variables about QMerrorStana values.
Do you people have any idea, or faced similar kind of issues ?
Any quicker suggestions or findings appriciated
|
Let me clarify one more thing..! IBM MQ 7.5 on RHEL V7 is perfectly working fine. Issue is only with IBM MQ 9.0.2 on RHEL V7
_________________
Thanks & Regards,
Praveen Hegde |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 24, 2017 4:58 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20767
Location: LI,NY
|
| praveenhegde wrote: |
I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange ! one of out architect tried RHEL V7 and IBM MQ 9 on VM.. he is stating each time we need to stop and start Qmgr to ensure AMQ1 gets copied to AMQ2 and again stop and start Qmgr to AMQ2 -->AMQ3 ...We were totally confused! something really wrong with IBM MQ 9 and RHEL V7 combination? |
Your understanding is wrong. AMQERR02.LOG can be full and it will not get copied / renamed to AMQERR03.LOG until AMQERR01.LOG is full.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Oct 24, 2017 2:38 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Quote: |
| I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange ! |
The error logs aren't copied at rollover, that would be too ineffiecent. Size management is only done on log 1. When it reaches the threshold, log 3 is deleted, log 2 is renamed to log 3, log 1 is renamed to log 2, and a new log 1 is created and used.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Oct 24, 2017 4:26 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9486
Location: US: west coast, almost. Otherwise, enroute.
|
The KC and IBM training doc continue to imprecisely use 'copy' and 'copied' to describe the mq error log file rotation process.
For example, https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.tro.doc/q039560_.htm says:
| Quote: |
| As error messages are generated, they are placed in AMQERR01. When AMQERR01 gets bigger than 2 MB (2 097 152 bytes) it is copied to AMQERR02. Before the copy, AMQERR02 is copied to AMQERR03.LOG. The previous contents, if any, of AMQERR03 are discarded. |
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| praveenhegde |
| Posted: Wed Nov 08, 2017 10:41 pm Post subject: Real Issue identified with IBM MQ 9.0.2/3!IBM Provided Ifix |
|
|
Novice
Joined: 14 Aug 2012
Posts: 20
|
| gbaddeley wrote: |
| Quote: |
| I do understand how logs works in MQ, but the strange thing here is once AMQ2 is full, by default it must get copied to AMQ3 once it reaches 2MB or more.. thing to be noted AMQ1 got copied to AMQ2 without any issue after AMQ1 reached 2MB..! really strange ! |
The error logs aren't copied at rollover, that would be too ineffiecent. Size management is only done on log 1. When it reaches the threshold, log 3 is deleted, log 2 is renamed to log 3, log 1 is renamed to log 2, and a new log 1 is created and used. |
I agree.. It was a typo! I should have mentioned AMQ1! instead of AMQ2
I now understand clearly AMQ1--->AMQ2-->AMQ3 : once AMQ1 is 2MB it will be renamed to AMQ2, and earlier AMQ2 will get renamed to AMQ3 .
Lets come to real issue!.
IBM MQ 9.0.2 and IBM MQ 9.0.3 CD has the AMQ Error log issue. below is APAR link.
Ref: http://www-01.ibm.com/support/docview.wss?uid=swg1IT22283
Issue would be fixed in IBM MQ 9.0.5(not yet released!).
They have provided the Ifix for IBM MQ 9.0.2 and IBM MQ 9.0.3 currently after raising the PMR.
Thanks to all who have given suggestions and gyaan on how IBM MQ Logs works!
Praveen Hegde
_________________
Thanks & Regards,
Praveen Hegde |
|
| Back to top |
|
|
|
|
