MQSeries.net :: View topic - Analysing FDCs on Unix

dgolding · Posted: Tue Jul 09, 2002 5:15 am Post subject:

Many thanks to Justin Fries for his permission to republish this (from Vienna MQ List April 2000):

Hi, all.

If you haven't seen an FDC with a probeid like XC130001 or XC130003,
you soon will. These FDCs are created on UNIX systems when an application
does something foolish like trying to access memory outside its address
space (a segmentation violation), or trying to access non word-aligned
memory (a bus violation). UNIX sends signals to misbehaved processes which
try to do these things--SIGSEGV and SIGBUS in the two examples I gave. If
the application is a queue manager process or a connected user application,
the MQSeries signal handler will write out one of these XC13000x FDCs to
help us debug the problem.

If an XC13000x FDC is created by a core queue manager
program--amqcrsta, amqzxma0, runmqsc, or any other program written entirely
by IBM--then the problem is definitely ours. In that case, we'll need to
examine the FDC and perhaps gather other information to diagnose the
problem. If an XC13000x FDC is created by a user application which is
merely connected to the queue manager, there is a possibility that the
failure is in the customer's code and not ours. For example, a user thread
may cause a segmentation violation by doing something foolish (access NULL
pointers, for example), and this will cause UNIX to send signal 11
(SIGSEGV) to the process. Because MQSeries has its own handler installed
for SIGSEGV, we print out an FDC for the problem. In this case, we're
really reporting someone else's error, but it almost looks as if we are the
culprits.

The MQSeries function which handles these synchronous signals is
called xehExceptionHandler, and it can create six different FDCs with
probes XC130001 through XC130006. Here is your guide to interpreting these
FDCs. To make it a bit easier to find your answer, blue suggests a problem
with our code is possible while red suggests a problem with user code:

|----------->
| XC130001 |
|----------->
>-----------------------------------------------------------------------|
| When any thread does an MQCONN, a control block called pCtl is |
| allocated for it to hold information about the state of that thread. |
| This FDC is created when that pCtl control block does not exist. If a|
| thread has no pCtl control block, it is most likely not connected to |
| MQSeries, so the problem is likely to be in user code. If the FDC has|
| no trace in it, you can be almost certain the problem is in the |
| customer application. |
>-----------------------------------------------------------------------|
|----------->
| XC130002 |
|----------->
>-----------------------------------------------------------------------|
| There are times when MQSeries needs to do dangerous things, like |
| calling user exits. Because we can't really trust exits to be |
| perfectly written, we set up a recovery address where we can go if the|
| exit raises an exception. This FDC indicates that something--most |
| likely a user exit--raised an exception. MQSeries will jump to the |
| recovery address and continue to process, but this FDC suggests a |
| problem in the exit. |
>-----------------------------------------------------------------------|
|----------->
| XC130003 |
|----------->
>-----------------------------------------------------------------------|
| This is the most common probe our exception handler generates. It |
| indicates that a thread connected to MQSeries ran into unexpected |
| trouble. However, it is possible that the thread which created the |
| FDC might have been running customer code at the time. If the |
| function stack shows only xcsFFST, and if the last two entries in the |
| trace show a return from an MQI call followed by an entry into xcsFFST|
| , then the problem is again likely to be in the customer's code. If |
| the process is one of the core queue manager processes, or if the |
| function stack shows other MQSeries functions, then this FDC suggests |
| a problem within MQSeries. |
>-----------------------------------------------------------------------|
|----------->
| XC130004 |
|----------->
>-----------------------------------------------------------------------|
| As I said before, MQSeries does need to do dangerous things from time |
| to time. For example, MQPUT needs to make sure that the pointer to |
| the message data passed in by the customer application is valid. The |
| only way to do this is to try to access the pointer, even though that |
| may case a segmentation violation. MQSeries handles this by setting |
| up a recovery address to which it can jump when a segmentation |
| violation occurs, then accessing the pointer. If the pointer is in |
| fact bad, then MQSeries cuts this FDC and recovers automatically. |
| This FDC usually means that the customer application passed in a bad |
| pointer, although there may be some cases where MQSeries itself is at |
| fault. |
>-----------------------------------------------------------------------|
|----------->
| XC130005 |
|----------->
>-----------------------------------------------------------------------|
| I have never seen this probe and never expect to. It means that this |
| thread is expecting a possible exception, but unlike the XC130002 or |
| XC130004 FDCs, there is no recovery area set up. MQSeries is very |
| careful about setting up recovery areas when needed, so this FDC |
| indicates a definite problem in our code. |
>-----------------------------------------------------------------------|
|----------->
| XC130006 |
|----------->
>-----------------------------------------------------------------------|
| This FDC also indicates that a thread connected to MQSeries ran into |
| unexpected trouble. However, the thread is marked as being unprepared|
| to handle exceptions, which means that it is probably not in MQSeries |
| code. As with XC130003, if the function stack shows only xcsFFST, the|
| problem is likely to be with customer code. If the function stack |
| includes other functions, this could be a more serious problem within |
| MQSeries. |
>-----------------------------------------------------------------------|

Here's an example of a problem in customer code. This FDC was created
because a thread with no pCtl (a thread not currently connected to
MQSeries) raised an exception:

+------------------------------------------------------------------
-----------+
|

|
| MQSeries First Failure Symptom Report
|
| =====================================
|
|

|
| Date/Time :- Tuesday February 02 18:05:16 EST 1999
|
| Host Name :- calvin
|
| PIDS :- 5765B73
|
| LVLS :- 500
|
| Product Long Name :- MQSeries for AIX
|
| Vendor :- IBM
|
| Probe Id :- XC130001
|
| Application Name :- MQM
|
| Component :- xehExceptionHandler
|
| Build Date :- Nov 23 1998
|
| UserID :- 00000010 (UNKNOWN)
|
| Process :- 00100836
|
| Major Errorcode :- xecSTOP
|
| Minor Errorcode :- OK
|
| Probe Type :- HALT6109
|
| Probe Severity :- 1
|
| Probe Description :- AMQ6109: An internal MQSeries error has
occurred. |
|

|
+------------------------------------------------------------------
-----------+

Here's an FDC which generally indicates a problem in our code. But if
you look at the function stack and read my description earlier, you can in
fact see that the problem is most likely with the customer's code. Even
better, the Arith1 field in this FDC prints out the particular signal which
was raised for this exception. In this case (and this is the most common
case), the signal was number 11, or SIGSEGV. This means that the problem
was a segmentation violation (the thread trying to access memory which is
outside its address space):

+------------------------------------------------------------------
-----------+
|

|
| MQSeries First Failure Symptom Report
|
| =====================================
|
|

|
| Date/Time :- Tuesday February 02 16:58:44 EST 1999
|
| Host Name :- calvin
|
| PIDS :- 5765B73
|
| LVLS :- 500
|
| Product Long Name :- MQSeries for AIX
|
| Vendor :- IBM
|
| Probe Id :- XC130003
|
| Application Name :- MQM
|
| Component :- xehExceptionHandler
|
| Build Date :- Nov 23 1998
|
| UserID :- 00000010 (hobbes)
|
| Process :- 00109802
|
| Thread :- 00000001
|
| QueueManager :- SUSIE
|
| Major Errorcode :- xecSTOP
|
| Minor Errorcode :- OK
|
| Probe Type :- HALT6109
|
| Probe Severity :- 1
|
| Probe Description :- AMQ6109: An internal MQSeries error has
occurred. |
| Arith1 :- 11 b
|
|

|
+------------------------------------------------------------------
-----------+

MQM Function Stack
xcsFFST

MQM Trace History
[Lots of stuff removed here]

<-- zcpDeletePacket rc=OK

<-- ziiMQPUT rc=OK

<-- zstMQPUT rc=OK

<-- MQPUT rc=OK

--> xcsFFST

Please realise that these exception handler FDCs are and ever will be
common. They are created whenever an exception is raised in any process
connected to MQSeries, so there is no fix for XC130001, and CSD04 does not
fix XC130003. It is possible for an APAR or a CSD to correct a specific
problem which can cause an XC130003, but to say that these FDCs are
completely fixed by some patch or efix is incorrect.

In order to solve these problems, I suggest exporting
AMQ_ABORT_ON_EXCEPTION=TRUE. If this variable is set in the environment in
which the application is running, the MQSeries exception handler will abort
the process after cutting the FDC. This will produce a core file which we
can investigate to determine conclusively who is at fault and what needs to
be done to solve the problem. If the failing program is an MQSeries
program we can examine the core file, but if it is a customer program they
will need to use dbx, gdb, or their favourite debugger to dump the stacks
for each thread in the process. We can examine cores from customer
applications only when we have their compiled application as well as all
libraries on which it depends. If their application is linked with
MQSeries, Tuxedo, Oracle, CORBA, and half a dozen other things, forget it.
If it's a straight MQSeries application we may be able to work on it
provided that we have the binary program, the core file, and a system at
the same OS and maintenance level. If not, things get very tricky very
quickly.

I hope this helps.

Cheers,

Justin T. Fries
MQSeries Support
Raleigh, North Carolina
(919) 254-1422 TL 444-1422
Email: justinf@us.ibm.com