| Author |
Message
|
| Sam99 |
 Posted: Fri Feb 06, 2015 9:14 am Post subject: SOAP Reply Node Error under load test (urgent!!!) |
 |
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
Hello Friends,
Flow Details:
SOAPInput -> Compute -> SOAP Request -> Compute - >SOAP Reply
Time Out :
SOAP Input : 15 secs
SOAP Request : 5 secs
Threads : 5 instances on two servers each
Flow usually takes 1-2 secs to respond. We are doing a peak load test and getting below error after few minutes of the test. Test is supposed to run for 1.5 hrs with a peak load of 60 transactions/minute i.e. 1 transaction/sec.
One more observation, We log all the request and response messages in the DB and I could see that all the response message are getting logged successfully however failing while sending the reply back.
Error Details :
The SOAP Reply node 'wsg.Request.SOAP Reply' encountered an error while processing a reply message. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOAPReplyNode.cpp:
396: ImbSOAPReplyNode::evaluate: ComIbmSOAPReplyNode: wsg/Request#FCMComposite_1_11
Feb 6 12:04:40 Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP3701E: A Java exception was thrown whilst calling the Java
JNI method 'Axis2Invoker_sendReplyDataSoap'. The Java exception was 'org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is allowed to
be sent back during timeout processing.'. The Java stack trace was 'Frame : 0 org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is
allowed to be sent back during timeout processing.| @: com.ibm.broker.axis2.Axis2Invoker.sendReplyInternalNotThroughPipeline(Axis2Invoker.java:1482)| @: com.ibm.broker.
axis2.Axis2Invoker.sendReplyDataSoap(Axis2Invoker.java:1127)|'. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOA
PReplyHelper.cpp: 693: ImbSOAPReplyHelper::makeSOAPReply: :
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP2628E: Exception condition detected on input node 'wsg.Requ
est.SOAP Input'. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/DataFlowEngine/TemplateNodes/ImbCallableTemplateNode.cpp: 1763: ImbCallable
TemplateNode::processMessage: ComIbmSOAPInputNode: wsg/Request#FCMComposite_1_1
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP2230E: Error detected whilst processing a message in node '
wsg.Request.ExceptionHandler'. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/DataFlowEngine/ImbComputeNode.cpp: 497: ImbComputeNode::evalu
ate: ComIbmComputeNode: wsg/Request#FCMComposite_1_9.wsg/ExceptionHandling#FCMComposite_1_1
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP2488E: (wsg.ExceptionHandling_ExceptionHandler.Main, 113.3
) Error detected whilst executing the SQL statement 'PROPAGATE TO TERMINAL 'out2' FINALIZE DEFAULT DELETE DEFAULT;'. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd:
/build/slot1/S800_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp: 666: SqlStatementGroup::execute: :
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP3752E: The SOAP Reply node 'wsg.Request.SOAP Reply' encount
ered an error while processing a reply message. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOAPReplyNode.cpp:
396: ImbSOAPReplyNode::evaluate: ComIbmSOAPReplyNode: wsg/Request#FCMComposite_1_11
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP3701E: A Java exception was thrown whilst calling the Java
JNI method 'Axis2Invoker_sendReplyDataSoap'. The Java exception was 'org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is allowed to
be sent back during timeout processing.'. The Java stack trace was 'Frame : 0 org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is
allowed to be sent back during timeout processing.| @: com.ibm.broker.axis2.Axis2Invoker.sendReplyInternalNotThroughPipeline(Axis2Invoker.java:1482)| @: com.ibm.broker.
axis2.Axis2Invoker.sendReplyDataSoap(Axis2Invoker.java:1127)|'. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOA
PReplyHelper.cpp: 693: ImbSOAPReplyHelper::makeSOAPReply: :
Please help
Last edited by Sam99 on Fri Feb 06, 2015 9:55 am; edited 1 time in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Fri Feb 06, 2015 9:40 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Well it looks like you are hitting an extreme.
Are you monitoring the flow? What is the elapsed/wait time per node?
Now @ 60 tps with 15 s / transaction you will need 900 instances !!!
my guess is that the time spent waiting for a free thread to process counts as well (time spent on the listener stack) and that you are quickly hitting a case where the message is expired before it even is taken off the stack...
Have you considered making your SOAP request an asynchronous one?
15 seconds total with only 5 seconds on the SOAP Request? What are you doing during these additional 10 seconds? Where is your bottleneck??
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Sam99 |
| Posted: Fri Feb 06, 2015 9:43 am Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
| I am soo sorry, 60 transactions/minute. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Feb 06, 2015 9:44 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Regarding your logging.
Are you doing it synchronously or asynchronously?
IMHO, you get a better throughput if you do it asynchtonously.
By this I mean write the data (non transactionally) to a set of queues and have a number of flows read them and do the DB work.
Flow 1 logs all the requests
Flow 2 logs all the replies
Flow 3 Logs all the errors.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Sam99 |
| Posted: Fri Feb 06, 2015 9:51 am Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
Thanks smdavies99 and fjb_saper for your reply.
Initially the timeout was SOAPInput : 4 secs and SOAPRequest : 2 secs. I just increased it to see if it has any impact on the test.
For me performance is not an issue, but I am not sure why I am getting
below errors even after increasing the time out.
Changing the complete design at this point of time is not feasible for me and I am quite sure that this problem is not because of DB transactions however I will still try to speak to my designer about this.
An attempt was made to use a SOAP message ReplyIdentifier after it had timed out. The SOAP message ReplyIdentifier was 534f41500000000c000000000000000700000000000069bc. : BRKREPST_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOAPHandleManager.cpp: 1403: ImbInFlightMessageHandle::throwHandleException: :
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP3752E: The SOAP Reply node 'wsg.Request.SOAP Reply' encount
ered an error while processing a reply message. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOAPReplyNode.cpp:
396: ImbSOAPReplyNode::evaluate: ComIbmSOAPReplyNode: wsg/Request#FCMComposite_1_11
Server1 user:err|error WebSphere Broker v8002[19988612]: (BRKR_Q1.EG1)[10282]BIP3701E: A Java exception was thrown whilst calling the Java
JNI method 'Axis2Invoker_sendReplyDataSoap'. The Java exception was 'org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is allowed to
be sent back during timeout processing.'. The Java stack trace was 'Frame : 0 org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is
allowed to be sent back during timeout processing.| @: com.ibm.broker.axis2.Axis2Invoker.sendReplyInternalNotThroughPipeline(Axis2Invoker.java:1482)| @: com.ibm.broker.
axis2.Axis2Invoker.sendReplyDataSoap(Axis2Invoker.java:1127)|'. : BRKR_Q1.0dbcce79-4a01-0000-0080-c56c99cb9bfd: /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOA
PReplyHelper.cpp: 693: ImbSOAPReplyHelper::makeSOAPReply: : |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 06, 2015 9:56 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
| Sam99 wrote: |
| I am soo sorry, 60 transactions/minute. |
ok so with 1 tps (60 per minute) @ 15 seconds you need 15 + instances.
15 + because it is too tight 16 would be nice...
With 10 instances across 2 servers you're still quite short of your goal and will hit the "backup" problem quite quickly when ramping up...
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Feb 06, 2015 10:07 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY
|
Are you by any chance trying to send a valid reply with http return code 500 (from the time out on the request node ) ?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Sam99 |
| Posted: Fri Feb 06, 2015 10:15 am Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
I also tried with 20 instances and the same issue occurred. 15 secs is the worst case scenario and normally request takes around 2 -3 secs.
All the request are getting processed successfully however somehow SOAPReply node is behaving weirdly if the time out happens.
We ran three test;
Test 1 : Everything was good for first 30 mins and after that continuously same error
Test 2 : Same configuration, next day, same issue starts after 10 mins but with java error in addition.
Test 3 : Increased the time out and threads, not much of an impact same issue after few minutes.
[/img] |
|
| Back to top |
|
|
| Sam99 |
| Posted: Fri Feb 06, 2015 10:18 am Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
| fjb_saper wrote: |
| Are you by any chance trying to send a valid reply with http return code 500 (from the time out on the request node ) ? |
Sorry didn't catch this, can you please elaborate a little.
I am not setting HTTP reply code specifically. For all the errors we create a valid SOAP Response but for this issue we are getting below SOAP Fault.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<soapenv:Fault>
<faultcode>soapenv:Server</faultcode>
<faultstring>BIP3113E: Exception detected in message flow wsg.Request.SOAP Input (broker BRKREPST_Q1)</faultstring>
<detail>
<Text>BIP2230E: Error detected whilst processing a message in node 'wsg.Request.ExceptionHandler'.
The message broker detected an error whilst processing a message in node 'wsg.Request.ExceptionHandler'. An exception has been thrown to cut short the processing of the message.
See the following messages for details of the error. : /build/slot1/S800_P/src/DataFlowEngine/ImbComputeNode.cpp: 497: ImbComputeNode::evaluate: ComIbmComputeNode: wsg/Request#FCMComposite_1_9.wsg/ExceptionHandling#FCMComposite_1_1
BIP2488E: ('wsg.ExceptionHandling_ExceptionHandler.Main', '113.3') Error detected whilst executing the SQL statement ''PROPAGATE TO TERMINAL 'out2' FINALIZE DEFAULT DELETE DEFAULT;''.
The message broker detected an error whilst executing the given statement. An exception has been thrown to cut short the SQL program.
See the following messages for details of the error. : /build/slot1/S800_P/src/DataFlowEngine/ImbRdl/ImbRdlStatementGroup.cpp: 666: SqlStatementGroup::execute: :
BIP3752E: The SOAP Reply node 'wsg.Request.SOAP Reply' encountered an error while processing a reply message.
An error occurred during reply message processing.
See previous error messages to determine the cause of the error. : /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOAPReplyNode.cpp: 396: ImbSOAPReplyNode::evaluate: ComIbmSOAPReplyNode: wsg/Request#FCMComposite_1_11
BIP3701E: A Java exception was thrown whilst calling the Java JNI method ''Axis2Invoker_sendReplyDataSoap''. The Java exception was ''org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is allowed to be sent back during timeout processing.''. The Java stack trace was ''Frame : 0 org.apache.axis2.AxisFault: Tried to send Non-Fault message when only a Fault message is allowed to be sent back during timeout processing.| @: com.ibm.broker.axis2.Axis2Invoker.sendReplyInternalNotThroughPipeline(Axis2Invoker.java:1482)| @: com.ibm.broker.axis2.Axis2Invoker.sendReplyDataSoap(Axis2Invoker.java:1127)|''.
Correct the error, and if necessary redeploy the flow. : /build/slot1/S800_P/src/WebServices/WSLibrary/ImbSOAPReplyHelper.cpp: 693: ImbSOAPReplyHelper::makeSOAPReply: :</Text>
</detail>
</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope> |
|
| Back to top |
|
|
| Sam99 |
| Posted: Fri Feb 06, 2015 10:22 am Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
| Also, we did a average load test 10 transcation/minute for 1.5 hrs and there was no issue found. |
|
| Back to top |
|
|
| sumit |
| Posted: Fri Feb 06, 2015 11:25 am Post subject: |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
Switch on auditing on your flow and see which node is taking how much time. Compute node usually takes less time as compared to SoapRequest node. You might want to give some more time to your soapRequest node.
To me, it looks like the avg wait time for a message is much more than 15 secs, when they are down in sequence. See if by increasing the soapRequest timeout, you are getting a better throughput.
_________________
Regards
Sumit |
|
| Back to top |
|
|
| mgk |
| Posted: Fri Feb 06, 2015 1:52 pm Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1638
|
The error:
| Quote: |
| "Tried to send Non-Fault message when only a Fault message is allowed to be sent back during timeout processing" |
Only occurs if there is a timeout whilst processing the message on the main flow, and so the timeout terminal on the SOAPInput node is fired and down that terminal you try to send back a message that is not a SOAPFault message from a SOAPReply node. Change the timeout flow to send back a valid SOAPFault or do not wire the timeout terminal.
Kind regards,
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Last edited by mgk on Fri Feb 06, 2015 1:59 pm; edited 1 time in total |
|
| Back to top |
|
|
| Sam99 |
| Posted: Fri Feb 06, 2015 1:57 pm Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
Thanks mgk..
I am trying it now, will keep this thread updated with the outcome. |
|
| Back to top |
|
|
| Sam99 |
| Posted: Wed Feb 11, 2015 8:09 am Post subject: |
|
|
Apprentice
Joined: 19 Jul 2012
Posts: 43
|
I have implemented the suggestions given.
Increased the thread to 20 each and also unwired the HTTP timeout terminal but still I can see few timeouts happening however they have reduced. One more thing is that they make a peak and goes back to normal after few mins and again starts after few minutes.
Also I am getting below error now. We call a webservice (HTTPs) as part of this flow.
A timeout error occurred whilst performing an SSL socket operation-read-4985-java.net.SocketTimeoutException: Read timed out |
|
| Back to top |
|
|
|
|
