| Author |
Message
|
| gabrielj |
 Posted: Sun Nov 16, 2014 3:49 am Post subject: SOAPInput takes long time to accept the request |
 |
|
Novice
Joined: 16 Nov 2014
Posts: 23
Location: Muscut,Perth,Sydney,Bangalore,Hydrabad,Coimbatore
|
Hi Experts,
I have a situation. We have developed one application(Webserive consumer) and deployed into WAS 8.5 and another application(Webservice Provider) is deployed in WMB 7.0.0.7. This is the existing setup of our environment.
Some times we got read timedout error from consumer application. 1 of 1000 messages are failed. We checked the logs files(both application logs, WAS logs, and WMB syslog). we find one thing from the log. WAS application call the SOAP request and the request reach the middleware after 80 seconds. these values are we arrived based on the logs we have. Both WAS and WMB servers are resides in same network.
My question is Why SOAPinput node takes so much time to receives the message? |
|
| Back to top |
|
 |
| shashivarungupta |
| Posted: Sun Nov 16, 2014 7:19 pm Post subject: |
|
|
Grand Master
Joined: 24 Feb 2009
Posts: 1343
Location: Floating in space on a round rock.
|
Did you configure WMB User trace to find out more details ?
Monitoring events can be applied as well on each of the critical nodes (WMB) to find out the transaction logs and related timestamp value while message traverse.
_________________
*Life will beat you down, you need to decide to fight back or leave it. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Nov 16, 2014 9:36 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
if truly the WAS call takes 80 seconds to get to WMB, you NEED to involve the network people. It could be something as easy as a misconfigured DNS lookup, bad ip resolution sequence, bad routing, etc ...
Now 80 ms (0.080 s) would be a normal time frame.
Check as well what ping and tracert(traceroute) do and what time they advertise as they make their way through the network.
You did not specify the size of the message, nor the distance it has to travel.
You can have machines on both sides of the country in the same network, however traffic will be considerably slower than if they were side by side on the same RAC.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gabrielj |
| Posted: Mon Nov 17, 2014 9:41 pm Post subject: |
|
|
Novice
Joined: 16 Nov 2014
Posts: 23
Location: Muscut,Perth,Sydney,Bangalore,Hydrabad,Coimbatore
|
Thanks fjb_saper and shashivarungupta for your valuable suggestion.
Just elaborate this issue little bit.
1. these time delay not coming in SIT or UAT Environment. It comes only production Environment. We can't ennable trace in production environment.
2. Both WAS and WMB boxes are sitting in same room. So there is no delay in DNS lookup. Just have a look the traceroute and ping output attached bellow.
Tracert takes less than 1 ms and one hop
tracert 10.0.6.2
1 <1ms <1ms <1ms [dnsname][10.0.6.2]
Ping take less than 1 ms
3. Message sizes is less than 1KB
I am thinking about embedded http listener. Listener only pickups SOAP messages from SYSTEM.BROKER.WS.INPUT queue. I suspect listener takes time to pickup the message. But I can't prove that the problem because of Listener.
Here the example of log. Here you can see the difference of timing. FYI Both boxes are SAME timezone and same time settings.
WAS runs on WINDOWS machine. WMB runs on HP UX machine.
WAS Log
--------------------------------------------------
[2014-10-10 16:30:24.455] [INFO] - CALL SOAP - TRXID #MOB201410101234
MW LOG
[2014-10-10 16:31:06.125] [INFO] - [FLOW NAME] [REC SOAP REQUEST ] - TRXID #MOB201410101234
[2014-10-10 16:31:06.985] [INFO] - [FLOW NAME] [SENT SOAP RESPONSE ] - TRXID #MOB201410101234
Please provide any other suggestion to look up this issue. Why is the delay for few of the messages?. Where should I look up ? HTTP Listener level or Network Level.
Thanks again for you suggestion |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Nov 17, 2014 9:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
This proves nothing as you did not specify if both boxes are time sync'd or not (port 13 for time service)
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gabrielj |
| Posted: Mon Nov 17, 2014 10:11 pm Post subject: |
|
|
Novice
Joined: 16 Nov 2014
Posts: 23
Location: Muscut,Perth,Sydney,Bangalore,Hydrabad,Coimbatore
|
hi fjb_saper,
As I said earlier both boxes sync with same time value.. there is no different between two boxes timezone..
IF it is happen for all message then no problem for me. it happens in very few messages only |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 18, 2014 6:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| gabrielj wrote: |
| As I said earlier both boxes sync with same time value.. there is no different between two boxes timezone.. |
You said there was no difference in time zone. You've not said that the two boxes are time synced. If they're in the same time zone (or even the same room) and the clock has been set manually there could be any level of error based on the fatness of the operator's fingers. Especially if the daylight saving settings are not in use and the base (GMT) time is being reset manually twice a year.
| gabrielj wrote: |
| IF it is happen for all message then no problem for me. it happens in very few messages only |
This sounds a lot like a transitory network problem.
| gabrielj wrote: |
| We can't ennable trace in production environment |
Of course you can - it's the same commanda as in SIT & UAT. It will require more change control & more management (you certainly can't leave it running for a few days) but you either get evidence to diagnose this problem or you don't.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 18, 2014 6:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Did you run out of instances of the flow to process the number of simultaneous requests you get coming in? |
|
| Back to top |
|
|
| gabrielj |
| Posted: Tue Nov 18, 2014 9:20 pm Post subject: |
|
|
Novice
Joined: 16 Nov 2014
Posts: 23
Location: Muscut,Perth,Sydney,Bangalore,Hydrabad,Coimbatore
|
Thanks mqjeff and Vitor for your thoughts.
@mqjeff: The particular service hardly receives 300 request per day. Default SOAP HTTP listener can accept 200 request simultaneously. Please correct me if I am wrong
The stderr file has recently reported the following exceptions:
| Code: |
Exception in thread "Thread-12" 2014-11-03 22:06:58.652 20
java.io.IOException: Stream closed due to HTTPConnector timeout
2014-11-03 22:06:58.653 20 at
com.ibm.broker.inlinehttp.tomcatthreadpool.InlineHTTPOutputStream.write(
InlineHTTPOutputStream.java:237)
2014-11-03 22:06:58.661 20 at
com.ibm.broker.inlinehttp.tomcatthreadpool.WMBOutboundHTTPTransport.writ
eOutputMessageBody(WMBOutboundHTTPTransport.java:304)
2014-11-03 22:06:58.661 20 at
com.ibm.broker.axis2.Axis2Invoker.sendReplyInternalNotThroughPipeline(Ax
is2Invoker.java:1424)
2014-11-03 22:06:58.661 20 at
com.ibm.broker.axis2.Axis2Invoker.sendReplyDataNonSoap(Axis2Invoker.java
:1179) |
This may indicate that the WMB message flow is not able to process
inbound requests quick enough to keep up with the increased load,
and that this is causing the requests to time out on the HTTPConnector,
resulting in a FAULT message back to the originating client.
The embedded tomcat server in the EG will have maxThreads set to 200
by default, which should be enough if the max number of concurrent
clients is set to 50
So planning to increasing the
number of additional instances for both the request and response message
flows. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Nov 18, 2014 11:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Use support pack IS03 to analyze your flow.
What is the peak throughput requested from your flow
What is the average throughput achieved.
Where is the bottleneck on your flow (hint most likely not the broker, but I expect that to be a service being called from the flow, or poorly written flow code... IS03 will help you tell).
What can be done to make the bottle neck go away.
How many instances of the flow do I need (peak demand(/sec) * avg resp time (avg throughput in sec) = avg nbr of threads needed.)
Example
peak = 300 tps avg processing time = 0.333 s (333 ms) => 100 instances needed.
Have fun
_________________
MQ & Broker admin
Last edited by fjb_saper on Tue Nov 18, 2014 11:10 pm; edited 1 time in total |
|
| Back to top |
|
|
| gabrielj |
| Posted: Tue Nov 18, 2014 11:06 pm Post subject: |
|
|
Novice
Joined: 16 Nov 2014
Posts: 23
Location: Muscut,Perth,Sydney,Bangalore,Hydrabad,Coimbatore
|
| @fjb_saper: will try out what you suggested and post my response |
|
| Back to top |
|
|
|
|
