MQSeries.net :: View topic

jrsetters · Posted: Wed Sep 19, 2012 1:45 pm Post subject:

We have an external system that that has a TCPIP receiver that crashes occasionally. Usually the technicians simply restart it and it will reconnect with WebSphere and continue receiving messages.

Every couple of weeks when they go down and restart, WebSphere will not connect. We do not get anything in our Traces, however the flow actually continues to run attempting to send messages until it times out. At this point we will try to start and stop the message flow in MQ Explorer which usually works.

Sometimes even that does not work and we get partial information in our log file that captures the following from the message tree:

LocalEnvironment.HL7.ErrorCondition
TCPIP error. The HL7 message was not sent.(UNHANDLED EXCEPTION IN PLUGIN METHOD.): sendData

InputExceptionList.RecoverableException.RecoverableException.Text
Node throwing exception

When this happens we cannot reconnect even stopping and starting the message flow, we have to redeploy it from the BAR.
I have been able to successfully reproduce the error that happens in test when the receiving side goes down, and here is the exception list at that time:

LocalEnvironment
HL7
RetryCount 3
Error true
Retry YES
ErrorCondition TCPIP error. The HL7 message was not sent.
FlowMilestoneReached SENDHL7
RetryLimit 5
Attempt Attempt 4 OF 5
Environment
Monitoring
EventCorrelation
localTransactionId 414d5120514d494e46545354303120204fcbbbc8283e9406
ExceptionList
RecoverableException
File /build/S700_P/src/DataFlowEngine/ImbDataFlowNode.cpp
Line 1073
Function ImbDataFlowNode::createExceptionList
Type ComIbmTCPIPClientOutputNode
Name com/trh/esb/msgflows/MFL_DOLBEY_OUT_ORDERS_PRODUCE#FCMComposite_1_2.hl7out/HL7Output#FCMComposite_1_2
Label com.trh.esb.msgflows.MFL_DOLBEY_OUT_ORDERS_PRODUCE.ms102.corp.trihealth.net:6152.Send HL7
Catalog BIPmsgs
Severity 3
Number 2230
Text Node throwing exception
RecoverableException
File MbErrorHandler.java
Line 104
Function createConnection
Type
Name
Label
Catalog BIPmsgs
Severity 0
Number 3586
Text connection refused
Insert
Type 5
Text ms102.corp.trihealth.net
Insert
Type 5
Text 5152
Insert
Type 5
Text Too many open files

Then once they restart we get the below information (I captured the first 3 retries). I tried this a few times in test and once I restarted the message flow it worked again.

I could not reproduce the event listed above (TCPIP error. The HL7 message was not sent.(UNHANDLED EXCEPTION IN PLUGIN METHOD.): sendData). So I am not sure what the exact conditions are that will cause us to have to redeploy the flow.

All of these errors are reported by the GenericTCPIPOut node.

FIRST RETRY
LocalEnvironment
HL7
RetryCount 0
Error true
Retry YES
ErrorCondition TCPIP error. The HL7 message was not sent.
FlowMilestoneReached SENDHL7
RetryLimit 5
Attempt Attempt 1 OF 5
Environment
Monitoring
EventCorrelation
localTransactionId 414d5120514d494e46545354303120204fcbbbc8283e95d3
ExceptionList
RecoverableException
File /build/S700_P/src/DataFlowEngine/ImbDataFlowNode.cpp
Line 1073
Function ImbDataFlowNode::createExceptionList
Type ComIbmTCPIPClientOutputNode
Name com/trh/esb/msgflows/MFL_DOLBEY_OUT_ORDERS_PRODUCE#FCMComposite_1_2.hl7out/HL7Output#FCMComposite_1_2
Label com.trh.esb.msgflows.MFL_DOLBEY_OUT_ORDERS_PRODUCE.ms102.corp.trihealth.net:6152.Send HL7
Catalog BIPmsgs
Severity 3
Number 2230
Text Node throwing exception
RecoverableException
File MbErrorHandler.java
Line 104
Function createConnection
Type
Name
Label
Catalog BIPmsgs
Severity 0
Number 3586
Text connection refused
Insert
Type 5
Text ms102.corp.trihealth.net
Insert
Type 5
Text 5152
Insert
Type 5
Text A connection with a remote socket was reset by that socket.

RETRY 2
LocalEnvironment
HL7
RetryCount 1
Error true
Retry YES
ErrorCondition TCPIP error. The HL7 message was not sent.
FlowMilestoneReached SENDHL7
RetryLimit 5
Attempt Attempt 2 OF 5
Environment
Monitoring
EventCorrelation
localTransactionId 414d5120514d494e46545354303120204fcbbbc8283e95d3
ExceptionList
RecoverableException
File /build/S700_P/src/DataFlowEngine/ImbDataFlowNode.cpp
Line 1073
Function ImbDataFlowNode::createExceptionList
Type ComIbmTCPIPClientOutputNode
Name com/trh/esb/msgflows/MFL_DOLBEY_OUT_ORDERS_PRODUCE#FCMComposite_1_2.hl7out/HL7Output#FCMComposite_1_2
Label com.trh.esb.msgflows.MFL_DOLBEY_OUT_ORDERS_PRODUCE.ms102.corp.trihealth.net:6152.Send HL7
Catalog BIPmsgs
Severity 3
Number 2230
Text Node throwing exception
RecoverableException
File MbErrorHandler.java
Line 137
Function Connection
Type
Name
Label
Catalog BIPmsgs
Severity 0
Number 2110
Text exception in Connection
RecoverableException
File MbErrorHandler.java
Line 286
Function throwableToMbException
Type
Name
Label
Catalog BIPmsgs
Severity 0
Number 3450
Text exception in Connection
Insert
Type 5
Text A system call received a parameter that is not valid.\n

RETRY 3
LocalEnvironment
HL7
RetryCount 3
Error true
Retry YES
ErrorCondition TCPIP error. The HL7 message was not sent.
FlowMilestoneReached SENDHL7
RetryLimit 5
Attempt Attempt 4 OF 5
Environment
Monitoring
EventCorrelation
ExceptionList
RecoverableException
File /build/S700_P/src/DataFlowEngine/ImbDataFlowNode.cpp
Line 1073
Function ImbDataFlowNode::createExceptionList
Type ComIbmTCPIPClientOutputNode
Name com/trh/esb/msgflows/MFL_DOLBEY_OUT_ORDERS_PRODUCE#FCMComposite_1_2.hl7out/HL7Output#FCMComposite_1_2
Label com.trh.esb.msgflows.MFL_DOLBEY_OUT_ORDERS_PRODUCE.ms102.corp.trihealth.net:6152.Send HL7
Catalog BIPmsgs
Severity 3
Number 2230
Text Node throwing exception
RecoverableException
File MbErrorHandler.java
Line 104
Function createConnection
Type
Name
Label
Catalog BIPmsgs
Severity 0
Number 3586
Text connection refused
Insert
Type 5
Text ms102.corp.trihealth.net
Insert
Type 5
Text 5152
Insert
Type 5
Text A remote host refused an attempted connect operation.