| Author |
Message
|
| sravan |
 Posted: Fri Mar 23, 2012 10:49 am Post subject: socket closed exception for http request. |
 |
|
Centurion
Joined: 02 Apr 2010
Posts: 104
Location: Charlotte
|
We are making a http request to java based webservice provider and ocassionally recieve below exception in production.
An error occured whilst performing a TCP/IP socket operation. The socket was closed before all expected data was received, as the end of the HTTP headers have not been found.
The request MQ message is put into a backout queue if failure happens,along with logging the exception in failure queue.
The same request will be succesful if it is resend from backout queue.
Let me know your thoughts on this error if retry logic will help from failure terminal or anyother fix.We do have a PMR but so far unable to recreate it in test environment to provide the trace.
_________________
skr_wmb |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Fri Mar 23, 2012 11:12 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Changing the retry and error handling of your flow is not going to stop the java application or the intervening firewall from closing the socket before the whole data is sent.
You could investigate the timeout on the HTTPRequest node you're using.... |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Mar 23, 2012 12:49 pm Post subject: Re: socket closed exception for http request. |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| sravan wrote: |
| The request MQ message is put into a backout queue if failure happens,along with logging the exception in failure queue. |
SUPERB error handling, sir, if I do say so myself. !! 
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Sat Mar 24, 2012 7:44 pm Post subject: Re: socket closed exception for http request. |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Increase the timeout can improve the situation (as much so that the problem disappears)... Though TCP/IP is not really reliable... MQ-Server to Server (and a channel) would result in a reliable connetion: Either you get the whole message or none. With TCP/IP you might have to "implement that yourself": Retry until all data is there / until there wasnt any timeout.
Probably you might want to test the payload for at least well-formedness (to verify you received all)... "unexpected end of http-header" sounds as if your payload is empty ?
Waiting longer might not help, since the request is probably "lost".
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Mar 25, 2012 12:08 pm Post subject: Re: socket closed exception for http request. |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| sravan wrote: |
| An error occured whilst performing a TCP/IP socket operation. The socket was closed before all expected data was received, as the end of the HTTP headers have not been found. |
What was the actual Exception? For example, something like "BIP3161: A socket was closed before all expected data was received. Total data received: <insert_1>"? What data, if any, was received? Were there other Exceptions that could offer additional information?
It seems as though the remote server is closing the connection. In that case, increasing the HTTPRequest node's request timeout won't help; examine the remote server to determine the root cause.
| sravan wrote: |
| Let me know your thoughts on this error if retry logic will help from failure terminal or any other fix. |
If duplicate requests can cause unintended side effects, then it's not a good idea to blindly retry them. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Sun Mar 25, 2012 11:20 pm Post subject: Re: socket closed exception for http request. |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| rekarm01 wrote: |
| examine the remote server to determine the root cause. |
good point !
| rekarm01 wrote: |
| If duplicate requests can cause unintended side effects, then it's not a good idea to blindly retry them. |
yes, consider this 
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| sravan |
| Posted: Mon Mar 26, 2012 7:18 am Post subject: |
|
|
Centurion
Joined: 02 Apr 2010
Posts: 104
Location: Charlotte
|
Below is the exception,There is no data from response -
</XmlResponse>
===========================
<Catalog>BIPv610</Catalog><Severity>1</Severity><Number>3162</Number
This is an Update service so there is impact as update already happened in the sending system and the response data has to be sent to the requester which is not happening due to socket exception.
But we do resend the message from backot queue when failure happens.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ErrorMsg><Identification><ErrorUUID>72fe0ed2-15e7-11e1-a9f8-000000000000</ErrorUUID><ErrorTimeStamp>TIMESTAMP '2011-11-23 Broker_Thrown_Error</CBEmsg></Identification><Environment><Variables><BrokerData><LastError><Label></Label><Number>3161</Number><Text>An error occured whilst performing a TCP/IP socket operation. The socket was closed before all expected data was received, as the end of the HTTP headers have not been found. Total data received:
_________________
skr_wmb |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Tue Mar 27, 2012 12:09 am Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
you could use code-tags...
TCP/IP is unreliable... there are ACKs, but... you can actually loose data.
Either you have a physical reliable connection: E.g. you are on the same network segment
Or you have a software reliable connection: In the Broker-World this is MQ-Server to MQ-Server.
So if retry is not possible (e.g. you are doing an update) then you can either move the two boxes close to each other (physical reliable) or you put in software.
You can also search with Wireshark or a network (switches&routers) administrator where your (TCP/IP) packets might get lost.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Wed Mar 28, 2012 1:55 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| sravan wrote: |
| Code: |
| <Catalog>BIPv610</Catalog><Severity>1</Severity><Number>3162</Number |
| Code: |
| ...<LastError><Label></Label><Number>3161</Number><Text>An error occured whilst performing a TCP/IP socket operation. The socket was closed before all expected data was received, as the end of the HTTP headers have not been found. Total data received: |
|
Both of these Exceptions are incomplete; they appear to have been truncated. There should be additional information about the HTTP Request header and HTTP Reply header contents, (among other things).
| mqsiuser wrote: |
| TCP/IP is unreliable... there are ACKs, but... you can actually [lose] data. |
More precisely, TCP does offer reliable transport, which means, in this case, that the broker can reliably detect that the remote server closed the connection. Whether the remote application received the HTTP request, accepted it, or discarded it, is another matter; it's not possible for the broker to determine what happened remotely, without an HTTP response. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Thu Mar 29, 2012 5:29 pm Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| rekarm01 wrote: |
| There should be additional information about the HTTP Request header and HTTP Reply header contents, (among other things). |
The truth is on the wire: If you want to know exactly use (ethereal or) wireshark... these tools require a good understanding of computer networks, protocols and TCP/IP.
Note that a typical integration-architecture is to have a broker (with an MQ-Server) somewhere and somewhere else ("remote") an ERP-System (or a DB), where you want to send and/or receive data from/to. To have a (from a practical point of view) reliable connection you'd put an MQ-Server next to the (remote) box. Now you will (actually must or 'have to') get your data from/to the remote system to/into the (remote) MQ-Server: Either you tell them (the team responsible for the ERP-System/DB) to do it (you provide a SVRCONN-Channel & some queues on the remote MQ-Server for them) or you do it (and use e.g. WBI-adapters or ... there are actually new adapters).
Then the 2 MQ-Servers can be connected through a channel-pair (sender/receiver) or an MQ-cluster.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sat Mar 31, 2012 2:32 pm Post subject: Re: socket closed exception for http request. |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mqsiuser wrote: |
| The truth is on the wire: If you want to know exactly use (ethereal or) wireshark... these tools require a good understanding of computer networks, protocols and TCP/IP. |
A packet sniffer can be useful, but it may require elevated privileges, or access to an SSL private key that the web service provider may not be willing to share, to decrypt network traffic. It can generate a lot of data and impact network performance, so it's better suited for errors that are easily reproducible in a non-production environment. Packet analysis is also time-consuming; it's worthwhile to exhaust the available application and system logs first, (including those on the web service host), before deciding that packet analysis is necessary to gather additional data. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Sat Mar 31, 2012 11:02 pm Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
Good point: Network sniffer in production... that would be very un-productive (far too much data passing by).
| Code: |
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ErrorMsg>
<Identification>
<ErrorUUID>72fe0ed2-15e7-11e1-a9f8-000000000000</ErrorUUID>
<ErrorTimeStamp>TIMESTAMP '2011-11-23 [...?...]
[...?...]Broker_Thrown_Error</CBEmsg>
</Identification>
<Environment>
<Variables>
<BrokerData>
<LastError>
<Label></Label>
<Number>3161</Number>
<Text>An error occured whilst performing a TCP/IP socket operation. The socket was closed before all expected data was received, as the end of the HTTP headers have not been found. Total data received: [...] |
At least you are missing some closing tags. Probably "Total data received" was the empty string ('') ?!... I guess it is your (custom) error format (retrieved from the logging-DB/text-file-logs from production).
| sravan wrote: |
| But we do resend the message from backout queue when failure happens. |
You can do that... you can resend your update (even if it already succeeded). You might update based on a unique ID and the reply might likely be "updated <uniqueId> successfully (commited)". This is cheaper than buying MQ-Server and http is also easier / more commonly used... there are more people with http-knowledge, than people who actually can set up and troubleshoot MQ-channels successfully. You will need to have an (part-time) MQ-Admin dedicated for that.
Though http is "unreliable"... might the OP consider using SOAP over JMS instead of SOAP over http ?
@OP: What is the distance between your caller and the receiver: Same computer-center? How many network-hops (network components) are there in between... depending on that either you are trying to go through a too big distance or you have a defect (cisco-)switch (might get replaced ?!).
Also: Retrying while your network hangs doesn't succeed either: But well... there are timeouts (which delay e.g. by 5 minutes) and retry-counts, e.g. 12.
I think you have to extend your (existing) retry... to retry until "commit really successfully confirmed", so also retry when receiving this (socket-timeout-)exception ?! ... I think the original developer just missed (this situation) when he implemented the retry-logic.
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Apr 01, 2012 3:25 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mqsiuser wrote: |
| Though http is "unreliable"... might the OP consider using SOAP over JMS instead of SOAP over http ? |
HTTP is "unreliable" in the sense that either side may close a persistent connection at any time, and the other side must handle any recovery. The OP has not indicated that the webservice is using SOAP. Either way, switching protocols is not a one-sided change. Better to figure out what's going wrong first, before scrapping the design and starting over.
| mqsiuser wrote: |
| @OP: What is the distance between your caller and the receiver: Same computer-center? How many network-hops (network components) are there in between... depending on that either you are trying to go through a too big distance or you have a defect (cisco-)switch (might get replaced ?!). |
The error indicates an unexpected close, not a timeout. This suggests an issue with the application-layer, not the network-layer. The remote side (or intermediate proxy) deliberately closed the connection for some reason. If the cause is external to the broker, then the broker is not the best place to look for an explanation. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Sun Apr 01, 2012 9:47 pm Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
You can (often/sometimes) correct/adjust errors (and (intentionally) missing features) of lower network layers in upper/higher layers. Note that Broker/ESQL is probably quite "high" in the network stack.
I guess that the original developer has left the company. The OP would have turned to him (first). @OP: There needs to be a new person (or a team) responsible for the interface (development). Also... now a cool thing about ESQL: The code can always be retrieved from the (deployed) bar-file (in case your sources got lost). Just unzip (win- or 7-zip)... though there are always processes for everything in big companies, you should recheck with all stakeholders!
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Apr 02, 2012 3:45 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| mqsiuser wrote: |
| Also... now a cool thing about ESQL: The code can always be retrieved from the (deployed) bar-file (in case your sources got lost). Just unzip (win- or 7-zip)... though there are always processes for everything in big companies, you should recheck with all stakeholders! |
DO NOT SUGGEST THIS.
DO NOT RELY ON THIS.
DO NOT ATTEMPT THIS.
DO NOT USE THIS AS JUSTIFICATION FOR ONLY KEEPING BAR FILES.
ALWAYS SAVE YOUR SOURCE CODE UNDER PROPER VERSION CONTROL.
ALWAYS SAVE YOUR BAR FILES UNDER PROPER VERSION CONTROL. |
|
| Back to top |
|
|
|
|
