| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| WMB Webservice issue invoked using DP |
« View previous topic :: View next topic » |
| Author |
Message
|
| mqxplorer |
 Posted: Mon Sep 08, 2014 6:04 pm Post subject: WMB Webservice issue invoked using DP |
 |
|
Master
Joined: 22 Jun 2009
Posts: 206
|
Hi All,
I am not sure if this is the right forum to post this. I have been struggling with a problem for few days.
I have WMB, DP and MQ in my flow.
Application A puts messaes into a queue QA1 with ReplyToQ and ReplyToQmgr in MQMD. DP MQFSH of MPGW gets messages from the queue QA1 and the MPGW service saves certain information like MsgId, ReplyToQ, and ReplyToQmgr and calls a WMB Web service. When response come back from the WMB Webservice, we reattach the ReplytoQ, ReplyToQmgr and copy message id to corerelid and send it back. Everything works fine whne we application sends 10K messages to queue QA1, however, when the count goes from 10K to 50K and above, we see errors as below:
| Code: |
16:37:09 mpgw error 234652944 error 0x01130011 mpgw (MPGW_XX_Service_1.0): Failed to process response headers
16:37:09 mpgw error 234652944 0x80e0012b mpgw (MPGW_XX_Service_1.0): Backside header ('N/A') failed to parse due to: Failed to process response headers, URL: http://<load balancergroup URL>
16:37:09 mpgw error 234652944 0x80e00626 mpgw (MPGW_XX_Service_1.0): Error occurred when connecting to URL 'http://<load balancergroup URL>'
16:37:09 mpgw error 234652944 0x80e0062a mpgw (MPGW_XX_Service_1.0): The header is empty when connecting to URL '<load balancergroup URL>'. |
DP MPGW load balance group is configured with WMB Webservices URLSs (broker1 and broker2). and the WMB web service flow is running with 5 additiaonal instances on both broekrs.
Based on the follwoing link, it says the error is happening because of the backend.
http://www-01.ibm.com/support/docview.wss?uid=swg21292808
| Code: |
In almost every event, this error is caused by a connection failure.
The error is correct, the service did not receive any response headers or data. |
| Code: |
The classic causes of this are either of these two:
• A slow response that causes the connection to timeout before receiving the response
• A failed connection attempt due to a closed persistent connection or some other connection issue that prevents the WebSphere® DataPower device® from connecting to the backend. For example, a reused connection would be attempted and found to already be closed out by something outside of DataPower®.
|
Based on what I read so far (too many documents), I feel like it is the problem with my WMB Webservcie flow handling the traffic sent from DP. I was confident that my WMB web service would send a timeout message through timeout terminal to DP MPGW. However, based on some info in other DP related forums, it says that the DP is getting an empty reposne which I though would not happen from WMB.
May be dumb question but I do not know the answer at all...
What does WMB web service flow do " If the flow is hit by DP service more than the capacity it has (not enough additioanl threads etc), what happens to the requests that exceed the capacity of the flow? It is not an MQ inetraction, so no queuing concept involed. How does a web servcie handle the traffic if its capacity is less than what traffic it is receiving?
Please help me with this. I ahve been struggling for the past few days with this issue.
Thanks
mqxplorer |
|
| Back to top |
|
 |
| mqxplorer |
| Posted: Tue Sep 09, 2014 2:01 am Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
| Can anybody help me with this please ? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Sep 09, 2014 4:43 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqxplorer wrote: |
| Can anybody help me with this please ? |
Apparently not, or they'd have posted.
If you need help within an SLA, or don't get help here, raise a PMR. IBM has whole buildings filled with people who can help you.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Sep 09, 2014 5:02 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It sounds like the WMB flow is taking too long. It should be presenting a timeout message back to DP, but apparently it's coded not to, or something is preventing that from happening.
You can review the timeouts across the whole chain - the value at DP, the value for general requests coming to the WMB http server, the maximum number of simultaneous requests the wmb http server can handle, the network layer to see if it breaks connections after x seconds, and finally the message flow to see if it does something to prevent a timeout response.
Oh, and the number of instances of the wmb flow. |
|
| Back to top |
|
|
| mqxplorer |
| Posted: Wed Sep 10, 2014 5:18 am Post subject: |
|
|
Master
Joined: 22 Jun 2009
Posts: 206
|
| mqjeff wrote: |
It sounds like the WMB flow is taking too long. It should be presenting a timeout message back to DP, but apparently it's coded not to, or something is preventing that from happening.
You can review the timeouts across the whole chain - the value at DP, the value for general requests coming to the WMB http server, the maximum number of simultaneous requests the wmb http server can handle, the network layer to see if it breaks connections after x seconds, and finally the message flow to see if it does something to prevent a timeout response.
Oh, and the number of instances of the wmb flow. |
Thank you mqjeff for your inputs.
I do not have my SOAPInput node's HTTPTimeout terminal connected to SOAPReply node. However. based on my experience with SAOP nodes, even if the HTTPTimeOut terminal is not connected to SOAPReply node, a timeout message is automatically sent to the client application calling WMB WebService. Based on this I can say that there is nothing that is preventing the timeout message going out from my mwmb web service flow to consumer application (in my case it is DP).
I have the flow running on two brokers with 5 additional instances running on both brokers. We are getting only 50 K messages from DP and based on the complexity of my WMB Flow, this load for flows running on both brokers with 5 additiaonl instances each.
I am involing network guys today to see what is going on between DP and WMB servers.
Is there anuthing else that can cause this issue.
May be dumb question but I am asking again..
When a web service (can be wmb or any other kind) receives a burst of requests that it cannot handle (in terms of not enough additioanl instances), what happens to the messages that came in? If it is a queue based transaction, I see the messages queueing up but what happens in case of web service.
Thanks
mqxplorer |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Sun Sep 14, 2014 2:07 pm Post subject: Re: WMB Webservice issue invoked using DP |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mqxplorer wrote: |
| I feel like it is the problem with my WMB Webservice flow handling the traffic sent from DP. |
Maybe, but that's not the only explanation. Out of 50K transactions, about how many are failing? For transactions that fail, what is the actual elapsed time between sending the request and getting any empty response? Is it possible to increase the logging level to debug, or to get a packet capture? That would provide the exact cause of the connection failures.
For the DataPower MQ Front Side Handler, what is the number of concurrent MQ connections set to? For the WMB Execution Group HTTP listener, what is maxThreads set to? The rate at which the DataPower MPGW can send requests is limited by the number of concurrent MQ connections in the MQ FSH. As long as this is less than or equal to the number of message flow instances, and the total number of incoming connection requests to the EG is less than the EG HTTP Listener maxThreads, and the load balancer group distributes connections fairly evenly between brokers, the message flows should be able to receive requests as fast as DataPower can send them. Unless there is some other bottleneck in the message flow itself, the volume alone is not likely to cause WMB timeout faults.
For the DataPower MPGW, what is the Back Side Timeout set to? For the WMB message flow SOAPInput node, what is the Maximum client wait time set to? The MPGW timeout should be greater than the WMB timeout plus any expected network delay, otherwise the MPGW would error out with an empty response and close the connection, before it could receive any WMB timeout fault.
For the MPGW, are Persistent Connections enabled, and if so, what is the Back Persistent Connection Timeout set to? For the WMB Execution Group HTTP listener, what are maxKeepAliveRequests, and connectionTimeout set to? There is a potential race condition with persistent connections; if the WMB closes the connection, but the MPGW manages to send a request before it receives the close event, then the MPGW would error out with an empty response, and the WMB would never receive the request, nor send a timeout fault. To prevent the WMB from intentionally closing a persistent connection, either disable persistent connections, (in the MPGW, or by setting maxKeepAliveRequests to 1, or both), or enable persistent connections, but set maxKeepAliveRequests to 0 or -1 to allow unlimited requests, and set the timeouts so that the MPGW Back Persistent Connection Timeout is less than the connectionTimeout.
Ultimately though, HTTP is not MQ, and does not offer assured delivery. Sometimes network connections break, beyond the control of either client or server, and requests, responses, or faults don't get delivered. The HTTP standard requires that clients and servers "MUST be able to recover from asynchronous close events", and advises that clients SHOULD retransmit aborted requests at least once, if it's safe to do so. DataPower does not automatically retry aborted requests; the MPGW policy would have to provide the additional logic for that. One way to do that is to get MQ messages under syncpoint, and use the MQ backout/retry when a connection failure occurs. If all else fails, then the sending application needs to figure out how to handle failed transactions. |
|
| Back to top |
|
|
| Armageddon123 |
| Posted: Mon Sep 15, 2014 4:10 am Post subject: |
|
|
Acolyte
Joined: 11 Feb 2014
Posts: 61
|
Hi rekarm01
nice to see your posts again.
We are a small team of WMB developers based in India. and when I joined the team as a fresher and had troubles understanding wmb encoding, ccsid, etc,etc,
i was asked to go to Mqseries forum search link, search for author="rekarm01", and read all his replies/posts to understand about the wmb encoding.
Your posts have been very very helpful for a fresher like me.I just wanted to thankyou for your services. |
|
| Back to top |
|
|
| rekarm01 |
| Posted: Tue Sep 23, 2014 9:48 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 1415
|
| mqxplorer wrote: |
| Can anybody help me with this please ? |
So, ... did that help?
| Armageddon123 wrote: |
Hi rekarm01
nice to see your posts again ... |
Thanks for the feedback. Although now I feel the need to double-check my old posts for mistakes ... |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
