ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportHTTP Reply node problem

Post new topicReply to topic
HTTP Reply node problem View previous topic :: View next topic
Author Message
wmqstankela
PostPosted: Thu Nov 22, 2018 11:27 am Post subject: HTTP Reply node problem Reply with quote

Voyager

Joined: 29 Feb 2016
Posts: 82

Hi all,

I have rest web service that has timeout of 90s. This service invoke 3rd party service and responce back to client. Sometimes my service does not send response to client in that 90s and then send timeout message, but in log I saw just before http reply that it takes 30s total. It looks like sometimes http reply node can't send back response message to client.

Is someone has any idea how to solve this kind of problem?

Thanks in advance
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Nov 26, 2018 5:50 am Post subject: Re: HTTP Reply node problem Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25365
Location: Ohio, USA

wmqstankela wrote:
It looks like sometimes http reply node can't send back response message to client.


So what's stopping it? Does the HTTP Reply node get an ack from the 3rd party service, an HTTP 200 or some other issue?

wmqstankela wrote:
Is someone has any idea how to solve this kind of problem?


Figure out what's stopping the reply.

You talk about "log" and if you don't mean "user trace" that's your next step. If the HTTP Reply node replies after 30 seconds, what's happening for the next 60?

I see 3 likely problems:

- your code is hosing up the session id and sometimes replies to the wrong client (especially if you have multiple instances and store session ids in any kind of shared memory)
- something like a firewall is terminating the connection, the reply is bouncing off that and the HTTP Reply node is not coded to respond to the HTTP failure code
- something like a load balancer is accepting the reply and then either eating it or replying to the wrong client.

This does not preclude your problem actually being a 4th, less likely scenario.


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
wmqstankela
PostPosted: Tue Nov 27, 2018 6:03 am Post subject: Reply with quote

Voyager

Joined: 29 Feb 2016
Posts: 82

Thanks Vitor for your answer!

HTTP Request get proper response with http 200 code. Log that I am able to see is from DB. I log message in queue and another application store message from queue to db. I log message few time during flow. Last log is just before http reply node, and in that log I see good response from 3rd party service.

I have haproxy load balancer in front of an IIB, and its timeout is bigger than timeout on http input node of my flow.

The whole situation is a little bit strange, because same request to IIB one time respond with http 504 timeout, and the next time respond with code 200. It looks like some network issue, maybe firewall like you said.

Where in message tree is stored session id? I thought that http reply node reads info, about where to send reply from LocalEnvironment.Destination.HTTP.RequestIdentifier.

I would be very thankful if you have another idea how to troubleshoot this problem.
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Nov 27, 2018 7:50 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25365
Location: Ohio, USA

wmqstankela wrote:
The whole situation is a little bit strange, because same request to IIB one time respond with http 504 timeout, and the next time respond with code 200. It looks like some network issue, maybe firewall like you said.


504 isn't a timeout, it's a flat out error - "Gateway error" according to Mr. Google. Ask your network people what's wrong.

wmqstankela wrote:
Where in message tree is stored session id? I thought that http reply node reads info, about where to send reply from LocalEnvironment.Destination.HTTP.RequestIdentifier.


It is.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
wmqstankela
PostPosted: Thu Nov 29, 2018 6:18 am Post subject: Reply with quote

Voyager

Joined: 29 Feb 2016
Posts: 82

This is default IIB response when service does not respond in time
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<title>504 Gateway Timeout</title>
</head>
<body>
<h1>504 Gateway Timeout</h1>
<p>
Integration node TESTNODE did not provide a response within the specified time interval (5 seconds). The message reached the main flow, but timed out during processing.
</p>
<hr>
<i>
IBM Integration Bus v10.0.0.5
</i>
</body>
</html>

So 504 gateway timeout is generate on IIB
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Nov 29, 2018 7:41 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25365
Location: Ohio, USA

wmqstankela wrote:
This is default IIB response when service does not respond in time


I'm confused. How does this tie up with:

wmqstankela wrote:
because same request to IIB one time respond with http 504 timeout


That's not a request to IIB, that's a request from IIB.

Likewise:

wmqstankela wrote:
It looks like sometimes http reply node can't send back response message to client.


You need to be very, very clear on exactly what your topology is here. Is it your rest service that's in TESTNODE?

wmqstankela wrote:
So 504 gateway timeout is generate on IIB


And that HTML is from an IIB log or the calling client? How have you determined that IIB is generating that message and not that it's propagating the 504 message the HTTPRequest node got when it called the 3rd party service?
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
wmqstankela
PostPosted: Thu Nov 29, 2018 8:03 am Post subject: Reply with quote

Voyager

Joined: 29 Feb 2016
Posts: 82

Yes, my service is on TESTNODE and this is response from my service to the client. This message was generated on IIB.

I simplified my scenario, just for troubleshooting. Whitout haproxy, just calling my service directly.

Everything is pointing on network issue. I need to consult with network department.

Thanks a lot Vitor for your answers!
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Nov 29, 2018 8:07 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25365
Location: Ohio, USA

wmqstankela wrote:
Yes, my service is on TESTNODE and this is response from my service to the client. This message was generated on IIB.


No - passed back by IIB not generated by IIB.

wmqstankela wrote:
Everything is pointing on network issue. I need to consult with network department.


Like I said, 504 is a hard error from a network component.

wmqstankela wrote:
Thanks a lot Vitor for your answers!


Welcome I'm sure
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mgk
PostPosted: Fri Nov 30, 2018 1:26 am Post subject: Reply with quote

Padawan

Joined: 31 Jul 2003
Posts: 1590
Location: IBM Hursley, UK

Quote:
No - passed back by IIB not generated by IIB.


Actually I can confirm that this 504 message IS generated by IIB, in certain timeout cases, such as the main flow not sending a reply before the timeout period expires (which seems like this case) or the "timeout" terminal of the HTTPInput node being connected, but a reply not being sent by the flow during the timeout processing which is essentially "a timeout occurring during timeout processing". The message is a fallback to make sure that the client at least gets a reply in either case and is not left hanging around. I would start looking at any timeout processing your flow is doing... Also, is it possible your load balancer is getting confused in some way?
_________________
MGK
IBM Global Blockchain Enablement
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Back to top
View user's profile Send private message
wmqstankela
PostPosted: Fri Nov 30, 2018 2:24 am Post subject: Reply with quote

Voyager

Joined: 29 Feb 2016
Posts: 82

Thanks mgk. Yes, this message is generated when flow does not respond in time (maximum client wait time in HTTP Input properties).

If it is problem with load balancer then this 504 message also will not propagate to client. My opinion is that some network issue does not allow http reply node to send back message to load balancer and when message flow timeout is reached http input node send this 504 error.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Nov 30, 2018 5:06 am Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 25365
Location: Ohio, USA

mgk wrote:
Quote:
No - passed back by IIB not generated by IIB.


Actually I can confirm that this 504 message IS generated by IIB, in certain timeout cases, such as the main flow not sending a reply before the timeout period expires (which seems like this case) or the "timeout" terminal of the HTTPInput node being connected, but a reply not being sent by the flow during the timeout processing which is essentially "a timeout occurring during timeout processing". The message is a fallback to make sure that the client at least gets a reply in either case and is not left hanging around. I would start looking at any timeout processing your flow is doing... Also, is it possible your load balancer is getting confused in some way?


I'm enlightened
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportHTTP Reply node problem
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.