Author |
Message
|
Shalini |
Posted: Tue Apr 19, 2005 9:15 am Post subject: SIGSEGV FDC External Application |
|
|
Master
Joined: 30 Apr 2002 Posts: 224 Location: India
|
Hi All,
AIX MQ5.3 CSD throwing following FDC
Looking @ the FDC I belive the FDC is generated due to UID ("LINK" under mqm group) and due to the exe "kinc.exe".
But really do not know why the exe is failing (due to this we are loosing the messages Its a trigger process).
Is there issue with kinc.exe ???
Can any body suggest looking @ FDC whats the issue with kinc or with MQ (I think its not due to MQ).
+-----------------------------------------------------------------------------+
| |
| WebSphere MQ First Failure Symptom Report |
| ========================================= |
| |
| Date/Time :- Tuesday April 05 19:15:44 SAUST 2005 |
| Host Name :- linkprd (AIX 5.1) |
| PIDS :- 5724B4101 |
| LVLS :- 530.9 CSD09 |
| Product Long Name :- WebSphere MQ for AIX |
| Vendor :- IBM |
| Probe Id :- XC130003 |
| Application Name :- MQM |
| Component :- xehExceptionHandler |
| Build Date :- Dec 13 2004 |
| CMVC level :- p530-09-L041213 |
| Build Type :- IKAP - (Production) |
| UserID :- 00000102 (link) |
| Program Name :- kinc.exe |
| Process :- 00053542 |
| QueueManager :- LINK |
| Major Errorcode :- STOP |
| Minor Errorcode :- OK |
| Probe Type :- HALT6109 |
| Probe Severity :- 1 |
| Probe Description :- AMQ6109: An internal WebSphere MQ error has occurred. |
| FDCSequenceNumber :- 0 |
| Arith1 :- 11 b |
| Comment1 :- SIGSEGV |
| |
| |
+-----------------------------------------------------------------------------+
MQM Function Stack
xcsGetpwnam
xcsFFST
MQM Trace History
} zstVerifyPCD rc=OK
{ zstCheckODAddressability
{ xcsCheckPointer
} xcsCheckPointer rc=OK
} zstCheckODAddressability rc=OK
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ ziiMQOPEN
{ ziiCreateIPCCMessage
{ zcpCreateMessage
} zcpCreateMessage rc=OK
} ziiCreateIPCCMessage rc=OK
{ ziiSendReceiveAgent
{ zcpSendOnPipe
{ xcsResetEventSem
} xcsResetEventSem rc=OK
{ xcsPostEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsPostEventSem rc=OK
} zcpSendOnPipe rc=OK
{ zcpReceiveOnPipe
{ xcsWaitEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsWaitEventSem rc=OK
} zcpReceiveOnPipe rc=OK
} ziiSendReceiveAgent rc=OK
{ zcpDeleteMessage
} zcpDeleteMessage rc=OK
} ziiMQOPEN rc=OK
} zstMQOPEN rc=OK
} MQOPEN rc=OK
{ MQGET
{ zstMQGET
{ zstVerifyPCD
} zstVerifyPCD rc=OK
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ ziiMQGET
{ ziiCreateIPCCMessage
{ zcpCreateMessage
} zcpCreateMessage rc=OK
} ziiCreateIPCCMessage rc=OK
{ ziiSendReceiveAgent
{ zcpSendOnPipe
{ xcsResetEventSem
} xcsResetEventSem rc=OK
{ xcsPostEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsPostEventSem rc=OK
} zcpSendOnPipe rc=OK
{ zcpReceiveOnPipe
{ xcsWaitEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsWaitEventSem rc=OK
} zcpReceiveOnPipe rc=OK
} ziiSendReceiveAgent rc=arcE_NO_MSG_AVAILABLE
{ zcpDeleteMessage
} zcpDeleteMessage rc=OK
} ziiMQGET rc=OK
} zstMQGET rc=arcE_NO_MSG_AVAILABLE
} MQGET rc=arcE_NO_MSG_AVAILABLE
{ MQCLOSE
{ zstMQCLOSE
{ zstVerifyPCD
} zstVerifyPCD rc=OK
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ ziiMQCLOSE
{ ziiCreateIPCCMessage
{ zcpCreateMessage
} zcpCreateMessage rc=OK
} ziiCreateIPCCMessage rc=OK
{ ziiSendReceiveAgent
{ zcpSendOnPipe
{ xcsResetEventSem
} xcsResetEventSem rc=OK
{ xcsPostEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsPostEventSem rc=OK
} zcpSendOnPipe rc=OK
{ zcpReceiveOnPipe
{ xcsWaitEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsWaitEventSem rc=OK
} zcpReceiveOnPipe rc=OK
} ziiSendReceiveAgent rc=OK
{ zcpDeleteMessage
} zcpDeleteMessage rc=OK
} ziiMQCLOSE rc=OK
} zstMQCLOSE rc=OK
} MQCLOSE rc=OK
{ MQDISC
{ zstMQDISC
{ xcsCheckPointer
} xcsCheckPointer rc=OK
{ zstVerifyPCD
} zstVerifyPCD rc=OK
{ ziiMQDISC
{ ziiCreateIPCCMessage
{ zcpCreateMessage
} zcpCreateMessage rc=OK
} ziiCreateIPCCMessage rc=OK
{ ziiSendReceiveAgent
{ zcpSendOnPipe
{ xcsResetEventSem
} xcsResetEventSem rc=OK
{ xcsPostEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsPostEventSem rc=OK
} zcpSendOnPipe rc=OK
{ zcpReceiveOnPipe
{ xcsWaitEventSem
-{ xlsLockEvent
-} xlsLockEvent rc=OK
-{ xlsUnlockEvent
-} xlsUnlockEvent rc=OK
} xcsWaitEventSem rc=OK
} zcpReceiveOnPipe rc=OK
} ziiSendReceiveAgent rc=OK
{ zcpDeleteMessage
} zcpDeleteMessage rc=OK
} ziiMQDISC rc=OK
{ ziiClearUpAgent
{ ziiStopAllPlugServices
{ xcsDisconnectSharedMemSet
{ xcsFreeMem
} xcsFreeMem rc=OK
{ xcsFreeMem
} xcsFreeMem rc=OK
{ xstDisconnectExtent
} xstDisconnectExtent rc=OK
} xcsDisconnectSharedMemSet rc=OK
} ziiStopAllPlugServices rc=OK
{ xlsSetSuspendEvent
} xlsSetSuspendEvent rc=OK
{ zcpDetachPipe
{ xcsUnregisterDestructor
} xcsUnregisterDestructor rc=OK
} zcpDetachPipe rc=OK
{ ziiFinishedWithAgent
} ziiFinishedWithAgent rc=OK
{ xcsUnregisterDestructor
} xcsUnregisterDestructor rc=OK
{ xcsTerminate
{ xcsDisconnectSharedSubpool
{ xlsThreadTermination
-{ xcsFreeQuickCell
--{ xllSpinLockRequest
--} xllSpinLockRequest rc=OK
--{ xllSpinLockRelease
--} xllSpinLockRelease rc=OK
-} xcsFreeQuickCell rc=OK
} xlsThreadTermination rc=OK
{ xcsDettachSharedSubpool
-{ xcsGetSetConnectCount
--{ xstGetExtentConnectCount
--} xstGetExtentConnectCount rc=OK
-} xcsGetSetConnectCount rc=OK
-{ xcsFreeQuickCell
--{ xllSpinLockRequest
--} xllSpinLockRequest rc=OK
--{ xllSpinLockRelease
--} xllSpinLockRelease rc=OK
-} xcsFreeQuickCell rc=OK
-{ xcsDisconnectSharedMemSet
--{ xcsFreeMem
--} xcsFreeMem rc=OK
--{ xcsFreeMem
--} xcsFreeMem rc=OK
-} xcsDisconnectSharedMemSet rc=OK
-{ xcsDisconnectSharedMemSet
--{ xcsFreeMem
--} xcsFreeMem rc=OK
--{ xcsFreeMem
--} xcsFreeMem rc=OK
-} xcsDisconnectSharedMemSet rc=OK
-{ xstDisconnectExtent
-} xstDisconnectExtent rc=OK
-{ xstDisconnectExtent
-} xstDisconnectExtent rc=OK
-{ xcsFreeMem
-} xcsFreeMem rc=OK
-{ xcsFreeMem
-} xcsFreeMem rc=OK
} xcsDettachSharedSubpool rc=OK
} xcsDisconnectSharedSubpool rc=OK
{ TermPrivateServices
{ xxxTerminate
-{ xcsFreeMem
-} xcsFreeMem rc=OK
} xxxTerminate rc=OK
} TermPrivateServices rc=OK
} xcsTerminate rc=OK
} ziiClearUpAgent rc=OK
{ zstDeletePCD
{ xcsFreeMem
} xcsFreeMem rc=OK
} zstDeletePCD rc=OK
} zstMQDISC rc=OK
} MQDISC rc=OK
{ MQCONN
{ zstMQCONN
{ xcsInitialize
} xcsInitialize rc=OK
{ zstMQConnect
{ zutBlankPad
} zutBlankPad rc=OK
{ zutIsItBlank
} zutIsItBlank rc=OK
{ zutCheckQMName
{ zutCheckValidName
} zutCheckValidName rc=OK
} zutCheckQMName rc=OK
{ zutCvtMQName2Str
} zutCvtMQName2Str rc=OK
{ zstGetPCDbyTID
} zstGetPCDbyTID rc=OK
{ zstInitCS
{ xcsInitialize
{ InitPrivateServices
} InitPrivateServices rc=OK
{ xcsGetpwnam
-{ xcsFFST
 |
|
Back to top |
|
 |
Shalini |
Posted: Thu Apr 21, 2005 1:22 am Post subject: |
|
|
Master
Joined: 30 Apr 2002 Posts: 224 Location: India
|
Hi,
Can any one please comment/suggest ...
 |
|
Back to top |
|
 |
clindsey |
Posted: Thu Apr 21, 2005 5:28 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002 Posts: 586 Location: Dallas, Tx
|
Shalini,
This might help you with where to look. Take a look at the code for kinc.
It appears that is disconnect from the queue manager and then connects again. The FDC is generated during the MQCONN call on the reconnect. See if you can spot any problems in this area. First of all, I would question why you are doing a MQDISC and MQCONN in a triggered program.
Hope this helps,
Charlie |
|
Back to top |
|
 |
clindsey |
Posted: Thu Apr 21, 2005 5:50 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002 Posts: 586 Location: Dallas, Tx
|
http://www-1.ibm.com/support/docview.wss?uid=swg1IY52676
A similar problem has been reported on Linux. You should open a pmr and report that you are having this problem on AIX. It looks like the problem is related to gathering information for user Link as you suspected.
It is possible that resources for this user, like file descriptors, is limited.
Charlie |
|
Back to top |
|
 |
Shalini |
Posted: Fri Apr 22, 2005 3:12 am Post subject: |
|
|
Master
Joined: 30 Apr 2002 Posts: 224 Location: India
|
Hi Charlie,
1)
Quote: |
First of all, I would question why you are doing a MQDISC and MQCONN in a triggered program. |
.
We are doing disconnect and connect again in a trigger type First Which takes the depth of the queue in one shot.
If I donot disconnect The client will develop the open orphan connection which our QMgr will fail with "Max channel Connectivity problem".
TCPKeepAlive will not help us as it will release the connection after 2 hrs (approx) but by that time QMGr may fail due to orphan connection
2)
Quote: |
It is possible that resources for this user, like file descriptors, is limited. |
As U suggested abt the file descriptors
"A per-process unique, non-negative integer used to identify an open file for the purpose of file access. The value of a file descriptor is from zero to {OPEN_MAX}. A process can have no more than {OPEN_MAX} file descriptors open simultaneously. File descriptors may also be used to implement message catalog descriptors and directory streams; "
How can I find the (or any command to know) the file descriptor limit
or
Is the file descriptor limit to be increased in the system ?????
 |
|
Back to top |
|
 |
clindsey |
Posted: Fri Apr 22, 2005 5:06 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002 Posts: 586 Location: Dallas, Tx
|
Definitely, you should do the MQDISC before your application exits. In the stack trace of the FDC, there is a MQDISC followed by an MQCONN. Since your app is a client app, this could be for connection recovery logic, I suppose. At any rate, the second connect should not cause an FDC.
To get the file descriptors, log in as user link and run 'ulimit -a' or 'ulimit -n' to see just file descriptors. Change it with 'ulimit -n unlimited'. You would have to set this in the .profile to make it permanent.
Are you using runmqtrmc as the trigger monitor, or one of your own? Can you run the code standalone (not triggered) without problems?
File desc is only a possible problem that could happen in getting user info. Try it to see if it helps. I think you may need to get help from IBM support on this one. I appended the reference to the APAR because it was reported on linux. The apar fix may be only for linux and one is needed for aix. IBM support could answer that for you.
Last edited by clindsey on Fri Apr 22, 2005 12:04 pm; edited 1 time in total |
|
Back to top |
|
 |
PeterPotkay |
Posted: Fri Apr 22, 2005 11:11 am Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Shalini wrote: |
We are doing disconnect and connect again in a trigger type First Which takes the depth of the queue in one shot.
|
Why?
You get triggered, you connect, read the queue until it is empty, maybe do one more MQGET with wait to see if any new messages arrive in the next few seconds and disconnect.
You should need to inquire on the depth of the queue.
Regardless, even if you do this extra work for nothing, you should not be gettings FDCs like Charlie mentioned. _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
JT |
Posted: Fri Apr 22, 2005 11:16 am Post subject: |
|
|
Padawan
Joined: 27 Mar 2003 Posts: 1564 Location: Hartford, CT.
|
PeterPotkay thought he wrote: |
You should NOT need to inquire on the depth of the queue. |
I believe this was what Peter intended.  |
|
Back to top |
|
 |
PeterPotkay |
Posted: Fri Apr 22, 2005 11:49 am Post subject: |
|
|
 Poobah
Joined: 15 May 2001 Posts: 7722
|
Yes we have no contact admin today.
Thanks JT!
You should NOT have to inquire..... _________________ Peter Potkay
Keep Calm and MQ On |
|
Back to top |
|
 |
jefflowrey |
Posted: Fri Apr 22, 2005 12:55 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002 Posts: 19981
|
PeterPotkay wrote: |
You should NOT inquire..... |
Fixed...
 _________________ I am *not* the model of the modern major general. |
|
Back to top |
|
 |
|