MQSeries.net :: View topic - Lost Connections & Interpretation of Traces (STRMQTRC)

LastSaskPirate

Hi,

I need some help in troubleshooting an intermittent problem that I can't reproduce

, and I'm hoping that one of you MQ Gurus has some insight for me.

The application I support randomly locks up on users, with no apparent errors nor common factors (userids, time of day, etc.). The app is a "heavy client" build in Delphi 7, using MQ v2 (no choice in the matter here!) to talk to a mainframe IMS app for it's data.

I've implemented MQ tracing (STRMQTRC) on a couple of workstations, and have seen some interesting things. Basically, when it does fail, it's on a cciTcpReceive, but the only way I know this is by comparing traces from a successful transaction to an unsuccessfull one. Here are a couple of examples:

Good:

354:904 -----> (1636) cciTcpReceive
354:904 ------> (1636) ccxAllocMem

Bad:

804:657 <------ (1636) ccxAllocMem (rc=OK)
809:188 ------> (1636) rrxError
809:188 <------ (1636) rrxError (rc=OK)
809:188 ------> (1636) ccxFreeMem
809:188 <------ (1636) ccxFreeMem (rc=OK)
809:188 <----- (1636) cciTcpReceive (rc=rrcE_CONNECTION_CLOSED)
809:188 <---- (1636) ccxReceive (rc=rrcE_CONNECTION_CLOSED)
809:188 ----> (1636) rriCommsError
809:188 <---- (1636) rriCommsError (rc=OK)

Note that the error reflects a rrcE_CONNECTION_CLOSED error, after which, there is some error processing, then endless "xcsSleep", requiring the users to forcibly terminate the application. No FDC files are generated in the client side "Errors" folder.

I checked the server "Errors" directory...for every "lockup" and resulting error as shown above, there is an FDC file for the channel process. The FDCs point to a generic termination error ("rrcE_CHANNEL_TERMINATED"), but when I compare the timestamp when a request is first sent to the mainframe (which starts the channel process) to the time noted in the server FDCs, the time is almost exactly 5 minutes. The process setting on the MQ mgr is set to 300 seconds (5 min.), so it appears that the channel is timing out. However, the client then tries to locate the process, but fails, since it's not there.

I still can't tell what exactly is happening here...it is possible that the client PUT / GET is losing it's thread...other PUTs / GETs (sometimes many) are usually running when this happens. It's also possible that the connection is getting "lost" in the network or delayed due to resource constraints, but I can't tell yet.

Does anyone out there have any insight / experience with this and can offer some suggestions, or at least, a good place to start / look?

Is there anyone that can help me interpret the trace message flow? If I knew exactly how the <PID.Thread> can relate to realtime or to server PIDs, that might help.

Thanks!
LaSkPi
_________________
"If the women don't find you handsome,
they should at least find you handy!"
- Red Green