| Author |
Message
|
| vikki1107 |
 Posted: Tue Mar 03, 2015 9:57 am Post subject: AMQ6125: An internal WebSphere MQ error has occurred. |
 |
|
Newbie
Joined: 25 Nov 2014
Posts: 7
|
We have received this alert into our system and below is the gist of the FDC file. I tried to google the probe id but there is no much info in google. So reaching out all the experts here. Please help me what this error is about.
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Sun July 27 2014 16:32:14 CET |
| UTC Time :- 1406471534.469930 |
| UTC Time Offset :- 120 (CET) |
| Host Name :- clsppal05 |
| Operating System :- AIX 6.1 |
| PIDS :- 5724H7221 |
| LVLS :- 7.0.1.6 |
| Product Long Name :- WebSphere MQ for AIX |
| Vendor :- IBM |
| Probe Id :- XC076012 |
| Application Name :- MQM |
| Component :- xcsGetMessage |
| SCCS Info :- lib/cs/amqxeimx.c, 1.83.1.2 |
| Line Number :- 1332 |
| Build Date :- Jul 25 2011 |
| CMVC level :- p701-106-110725 |
| Build Type :- IKAP - (Production) |
| Effective UserID :- 501 (mqm) |
| Real UserID :- 501 (mqm) |
| Program Name :- runmqsc |
| Addressing mode :- 64-bit |
| Process :- 39321706 |
| Thread(n) :- 1 |
| UserApp :- FALSE |
| Last HQC :- 0.0.0-0 |
| Last HSHMEMB :- 0.0.0-0 |
| Major Errorcode :- xecC_E_NOT_INIT |
| Minor Errorcode :- OK |
| Probe Type :- INCORROUT |
| Probe Severity :- 2 |
| Probe Description :- AMQ6125: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| Comment1 :- Cat CCSID 819, user CCSID 819, message ID '10006209'X |
| Comment2 :- NLSPATH is |
| /usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat, LC_MESSAGES is <NULL>, |
| LANG is en_US |
| Comment3 :- catopen attempts: "", "", "amq.cat" (errno=0) |
| |
+-----------------------------------------------------------------------------+ |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Mar 03, 2015 10:22 am Post subject: Re: AMQ6125: An internal WebSphere MQ error has occurred. |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Never mind the probe id, this is probative:
| vikki1107 wrote: |
Comment1 :- Cat CCSID 819, user CCSID 819, message ID '10006209'X |
| Comment2 :- NLSPATH is |
| /usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat, LC_MESSAGES is <NULL>, |
| LANG is en_US |
| Comment3 :- catopen attempts: "", "", "amq.cat" (errno=0) |
| |
+-----------------------------------------------------------------------------+ |
It's trying to generate a message and amq.cat can't be opened.
If this is a new system, check the installation.
If this is a previously working system, check what's changed in the
installation.
Particually why that environment variable is NULL.
You'll also find a description of the process (albeit talking about more exotic code pages than 819) by googling the probe id. Be aware that the fixes described are for applications; that's why they have a major code E_CONV_NOT_SUP and you have E_NOT_INIT
_________________
Honesty is the best policy.
Insanity is the best defence.
Last edited by Vitor on Tue Mar 03, 2015 10:25 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 03, 2015 10:25 am Post subject: Re: AMQ6125: An internal WebSphere MQ error has occurred. |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
Never mind the probe id, this is probative:
| vikki1107 wrote: |
Comment1 :- Cat CCSID 819, user CCSID 819, message ID '10006209'X |
| Comment2 :- NLSPATH is |
| /usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat, LC_MESSAGES is <NULL>, |
| LANG is en_US |
| Comment3 :- catopen attempts: "", "", "amq.cat" (errno=0) |
| |
+-----------------------------------------------------------------------------+ |
It's trying to generate a message and amq.cat can't be opened. |
And it's
| Quote: |
| | Program Name :- runmqsc | |
that's failing to find the amq.cat file. |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Mar 03, 2015 11:53 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
| The subset of the FDC that is provided is too small to identify the issue, however I don't believe it's because amq.cat cannot be opened. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Mar 03, 2015 3:03 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Andyh wrote: |
| The subset of the FDC that is provided is too small to identify the issue, however I don't believe it's because amq.cat cannot be opened. |
In other words, please post another 50 lines from the FDC.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| vikki1107 |
| Posted: Tue Jul 14, 2015 11:21 pm Post subject: |
|
|
Newbie
Joined: 25 Nov 2014
Posts: 7
|
| bruce2359 wrote: |
| Andyh wrote: |
| The subset of the FDC that is provided is too small to identify the issue, however I don't believe it's because amq.cat cannot be opened. |
In other words, please post another 50 lines from the FDC. |
Apologies I had been away.
Here is the following lines from FDC file.
MQM Function Stack
xcsInitialize
xxxInitialize
xehHandleAsySignal
xcsDisplayMessageForQmgr
xcsDisplayMessage
xcsGetMessage
xcsFFST
MQM Trace History
{ xppInitialiseDestructorRegistrations
} xppInitialiseDestructorRegistrations rc=OK
{ xcsGetEnvironmentInteger
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
} xcsGetEnvironmentInteger rc=xecE_E_ENV_VAR_NOT_FOUND
{ xcsGetEnvironmentInteger
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
} xcsGetEnvironmentInteger rc=xecE_E_ENV_VAR_NOT_FOUND
{ xcsGetEnvironmentInteger
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
} xcsGetEnvironmentInteger rc=xecE_E_ENV_VAR_NOT_FOUND
{ xcsGetEnvironmentInteger
-{ xcsGetEnvironmentString
-} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
} xcsGetEnvironmentInteger rc=xecE_E_ENV_VAR_NOT_FOUND
{ xcsGetEnvironmentString
} xcsGetEnvironmentString rc=xecE_E_ENV_VAR_NOT_FOUND
{ xcsCheckDirectory
} xcsCheckDirectory rc=OK
{ xstVerifySysVIdInFile
-{ xcsGetMQMGroupId
--{ xcsGetpwnam
--} xcsGetpwnam rc=OK
-} xcsGetMQMGroupId rc=OK
-{ xcsQueryBootTime
--{ xcsLocaltime
--} xcsLocaltime rc=OK
-} xcsQueryBootTime rc=OK
Data: 0x00000003
} xstVerifySysVIdInFile rc=OK
Data: 0x00000000
{ InitProcessInitialisation
-{ xcsIsEnvironment
-} xcsIsEnvironment rc=OK
-{ xcsLoadFunction
--{ xcsQueryValueForSubpool
--} xcsQueryValueForSubpool rc=OK
--{ xcsGetMemFn
--} xcsGetMemFn rc=OK
Also I found this from the env variable further down
Component Dumps (Thread 00000001)
-------------------------------------
Environment Variables:
_=/usr/bin/runmqsc
LANG=en_US
WSM_WS_CMD="startsrc -s http4websm"
LOGIN=mqm
CLCMD_PASSTHRU=1
PATH=/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java14/jre/bin:/usr/java14/bin
LC__FASTMSG=true
CGI_DIRECTORY=/var/docsearch/cgi-bin
HISTPATH=/usr/local/log/history/current
HISTFILE=/usr/local/log/history/current/mqm/.sh_history.mqm.27-07-14.16:32.pts2 |
|
| Back to top |
|
|
| Andyh |
| Posted: Tue Jul 14, 2015 11:53 pm Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Sorry, "another 50 lines" (not my suggestion) isn't enough in this case.
Has a PMR been opened for this issue ? In which case what is the PMR number? and was the FDC attached to the PMR ?
Thanks
Andy. |
|
| Back to top |
|
|
| vikki1107 |
| Posted: Wed Jul 15, 2015 1:09 am Post subject: |
|
|
Newbie
Joined: 25 Nov 2014
Posts: 7
|
| Andyh wrote: |
Sorry, "another 50 lines" (not my suggestion) isn't enough in this case.
Has a PMR been opened for this issue ? In which case what is the PMR number? and was the FDC attached to the PMR ?
Thanks
Andy. |
Yes I have raised one now. Its 03592,032,866 and yeah I have attached the FDC files in there |
|
| Back to top |
|
|
| Andyh |
| Posted: Wed Jul 15, 2015 1:49 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
The FDC appears to have been raised because runmqsc has been sent an asynchronous signal. At the time runmqsc received the signal it was initializing data conversion services, and then it fails to properly generate the FDC because without data conversion facilities it can't include the appropriate message text in the FDC.
If this was a one off FDC then I'd put it down to being likely to someone ctrl-C'ing out of runmqsc at an unexpected time, however that seems unlikely given that the FDC in the PMR was raised today. Assuming it's a repeatable failure then it would suggest there's some issue in data conversion initialization, leading to those services not initializing in a timely manner, and then the end user trying to break out of runmqsc.
Rather than Ctrl-C'ing straight out of runmqsc I'd suggest that you run procstack against the runmqsc process before interrupting it, or collect an early (strmqtrc -e) trace of the runmqsc process in the hope that would show why data conversion isn't initializing properly.
Have you made any recent changes to ccsid.tbl ? Is runmqsc the only program that's failing in this manner ? (This level of initialization is pretty common across all MQ processes). |
|
| Back to top |
|
|
| tczielke |
| Posted: Wed Jul 15, 2015 5:15 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
If your runmqsc command is hanging, another debugging tip to do is issue a truss on the pid. So if your runmqsc pid is 1234, I believe on AIX it would be:
truss -f -p 1234 2>/tmp/truss.out
This will start sending any system calls that 1234 makes (or children of 1234) to the /tmp/truss.out file. Also, if 1234 is stuck inside a system call, that is usually reported immediately. For example, maybe your runmqsc process is trying to reference a file across a network file system and the system call is hung for some reason. A system call is part of the operating system. So if your process is hung in a system call, it is a Unix issue, not an MQ issue. Don't let a Unix admin tell you otherwise. 
I have seen runmqsc commands hang inside system calls on Solaris 10, when runmqsc was issuing a file lock system call. In our case, I think there was a bug in the file locking client on the Solaris non-global zone, that was causing the system call to hang.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
|
|
