| Author |
Message
|
| satya2481 |
 Posted: Tue Apr 13, 2010 5:36 am Post subject: Issue with WebService Retry Logic - HTTPRequestNode |
 |
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hello All,
We have a requirement of Retrying the WebService request for certain number of times when the WebServer is down. I am facing a strange issue in the logic which is developed. If any one have an idea on what could be the possible reason for failure please let me know.
I am using WMB V6.1 Toolkit and Broker Runtime in Windows platform.
The logic is as below
1. MQInput
2. Compute (Construct WebService Mesage)
3. HTTPRequest
4a. If WebService Reply is SUCCESS then stop the execution
4b. If WebService response is FAIL then capture the error reply and stop execution
4c. If WebServer is down (when message is sent to Failure node, considering it as a common issue). Resend the request after 10 Minutes. That is go back to step 2
In the step 2 I am storing entire Input message in Environment tree and restoring it back in step 4c so that the same message is considered as a new message when it reaches to Compute node
First time when message is constructed and sent to HTTP Request its working fine. When the execution goes to step 4c and the same message is reconstructed once again and sent to HTTP request then its failing.
I get below errors in the Server system event logs
---------------------------------------------------------------------------------------------
| Code: |
Broker process terminating abnormally (stack full): diagnostic information for IBM 'Severe Abend Error: A Stack overflow has occured, further logging may cause a double fault and will be attempted with caution. Abend Files and/or dumps may not be produced.
'.
A broker process is terminating abnormally because its stack is full. For execution group processes, this may be caused by message flows containing loops or more than 500 nodes in series. Otherwise the following information will be required when contacting IBM: 'Severe Abend Error: A Stack overflow has occured, further logging may cause a double fault and will be attempted with caution. Abend Files and/or dumps may not be produced.
'.
Check the definitions of all message flows deployed on the named execution group for loops and excessive path length. If this does not resolve the problem, contact your IBM support center. |
---------------------------------------------------------------------------------------------
| Code: |
Broker process terminating abnormally: The following diagnostic information will be required when contacting IBM: '
Severe Abend Error detected.
For full details see Abend File: default.3964.2576.Abend
A summary of the Error follows:
An Unhandled Exception detected in process 3964, on thread 0xA10.
Type: EXCEPTION_STACK_OVERFLOW (C00000FD).
Address: 001B:2F8EC7C7.
Exception Parameters: 2
Param#0: 0x00000000
Param#1: 0x082e2000
'.
A broker process is terminating abnormally. |
One more point I want to add is I have created a Dummy webService in the broker to simulate the actual webservice. I am stopping the message flow execution in this simulator flow to test condition 4c. Like timeout test case.
I have tried to compare all trees before sending the message to HTTP request node, in the first attempt and in the second attempt, both are same except some time and date values.
Thanks in advance
Satya |
|
| Back to top |
|
 |
| elvis_gn |
| Posted: Tue Apr 13, 2010 7:43 am Post subject: Re: Issue with WebService Retry Logic - HTTPRequestNode |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi satya2481,
| satya2481 wrote: |
1. MQInput
2. Compute (Construct WebService Mesage)
3. HTTPRequest
4a. If WebService Reply is SUCCESS then stop the execution
4b. If WebService response is FAIL then capture the error reply and stop execution
4c. If WebServer is down (when message is sent to Failure node, considering it as a common issue). Resend the request after 10 Minutes. That is go back to step 2
In the step 2 I am storing entire Input message in Environment tree and restoring it back in step 4c so that the same message is considered as a new message when it reaches to Compute node |
How is the compute node resending the request ? You have some waiting logic ?
The problem has to do with your flow accumulating data for one and also holding it for such long periods...basically this is not good architecture.
If you want 10 min wait, then you should rather put the MQ message into a Db or another temporary queue and schedule or expire it to auto trigger the next attempt.
Regards. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Tue Apr 13, 2010 8:30 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi,
Once the message arrives at HTTP Request node Failure terminal I am adding a logic to introduce Delay of about 10 minutes (using MQGet node). Then the original message will be resent to the compute node so that the request will be resent to HTTP Request node.
I need a quick fix for this as it needs to be moved to Production asap 
Regards
Satya |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 13, 2010 9:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| satya2481 wrote: |
| I am adding a logic to introduce Delay of about 10 minutes (using MQGet node). |
How? A get with wait will end as soon as a message arrives, not when the wait expires.
| satya2481 wrote: |
| I need a quick fix for this as it needs to be moved to Production asap |
Then you need to raise a PMR. We're just a bunch of volunteers, IBM have a manned support desk and it's got access to all sorts of clever people who respond within SLAs.
It looks to me like you've got some kind of recurvise step that's filling the stack connected to your retry processing. I must say at first glance your design is a bit odd. Why not disconnect the flows & use Timer nodes?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Tue Apr 13, 2010 10:16 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi satya2481,
| satya2481 wrote: |
Once the message arrives at HTTP Request node Failure terminal I am adding a logic to introduce Delay of about 10 minutes (using MQGet node). Then the original message will be resent to the compute node so that the request will be resent to HTTP Request node.
I need a quick fix for this as it needs to be moved to Production asap |
As Vitor mentioned, the MQGet node will not help you pick the message ONLY AFTER 10 min, it will only wait for max 10 min WHEN the message you need is not on that queue.
Or else if you are getting putting into queue until you elapse 10 min, then that would be a crazy logic.
All said and done, this solution will not work...you need to either dump the message onto a temp queue and set expiry on that for 10 min, after which the expiry will create a report which you need to use as trigger. Else use the timer nodes as Vitor suggests.
Regards. |
|
| Back to top |
|
|
| mgk |
| Posted: Tue Apr 13, 2010 11:56 pm Post subject: |
|
|
Padawan
Joined: 31 Jul 2003
Posts: 1642
|
From the above it looks like you have created a "loop" in your flow and this is causing the abend as your "wait" is not actually waiting as pointed out by others. Try not to loop as it leads to problems, instead use the ESQL "PROPAGATE" statement to drive the part of the flow that must repeat... However, I do think that splitting this into multiple flows would be best as otherwise your flow will be paused for 10 mins for each message that must be retried and this will stop redeploys, stop messages and any other configuration messages getting to the flow during this period...
_________________
MGK
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions. |
|
| Back to top |
|
|
| napier |
| Posted: Wed Apr 14, 2010 8:29 am Post subject: |
|
|
Apprentice
Joined: 09 Oct 2007
Posts: 48
Location: USA
|
| You are in V6.1 why don't you try Timer nodes for your retry request delay? |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Apr 14, 2010 8:39 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| napier wrote: |
| You are in V6.1 why don't you try Timer nodes for your retry request delay? |
It's like there's an echo in here.... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Apr 14, 2010 10:22 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You should be aware that the Timer nodes are not designed to handle a full stream of input messages, but rather to handle a small number of recurring events.
If you, for example, decide to delay every message that comes in for a fixed period of time, you will find that you suffer a significant performance impact and see a lot of CPU usage.
If you need to delay a reasonably large number of messages ( more than perhaps 100 at a time), you will strongly want to use some other mechanism than the Timer nodes to do so. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Wed Apr 14, 2010 11:20 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi All,
Thank you very much for your replies. Few points I need to mention regarding the requirements.
- The requirement is like this, If there are 5 messages in the queue and webservice is down. Then flow should take first message and try for an hour to send that message to Webservice. Till that time other messages should be there in the queue. After an hour if the webservice is not up then send the first message to error queue and then pick next message. To achieve this I have the loop in the same flow and not divided the transaction. If I create 2 flows then all the messages picked up by the flow1 will be sent to flow2
- I have tried using the Timer nodes but its breaking the transaction. I mean when the timer node introducing the delay for the first message then flow picks next message from the queue. So its not satisfying my requirement
Coming back to original issue, I have managed to resolve the issue and also fullfill the requirement with below logic.
1. MQInput
2. Compute (Construct WebService Mesage)
3. HTTPRequest
4a. If WebService Reply is SUCCESS then stop the execution
4b. If WebService response is FAIL then capture the error reply and stop execution
4c. If WebServer is down (when message is sent to Failure node, considering it as a common issue). Add MQRFH2 header and save the retries made in the user folder. Change the message Priority (Existing Priority + 1). Put the message back to Input queue
I have tested this flow and looks working fine, even when there are more than one message in the Input queue. It takes the one which is having high priotity that is nothing but the mesage which is in Retry mode.
The only thing to be considered is all mesages coming to the queue should be of same Priority. And queue should be defined of Sequence of type Priority and not 'FIFO'.
Any other ideas or suggessions ? Please post.
Note : One more thing I have noticed is that the original logic of looping in the same flow in V6.0 broker which was working fine. I need to cross check once again, any how.
Regards
Satya
_________________
IBM Certified Solution Developer WebSphere Message Broker V6.0
IBM Certified System Administrator WebSphere MQ V6.0 |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Apr 15, 2010 1:53 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
The order/priority of the messages read from the queue is really outside the control of Broker.
IMHO, It is down to two things.
1) The way the queue is defined
2) How the sending app puts the message.
If the OOTB Default for priority/ordering is used on both then there is no problem. However, if some smart alec programmer decided to send his particular message with a higher than normal priority then there is really nowt you can do about it except kick back at the programmer.
This is what Priority Based Message Queueing systems is all about, as I learnt back in 1975 as a student studying Queueing Theory.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Svp |
| Posted: Thu Apr 15, 2010 11:59 am Post subject: |
|
|
Apprentice
Joined: 18 Feb 2010
Posts: 40
|
Hi Satya,
Check whether this works.
MainFlow:
MQi/p(Q1)--->HTTPRequestFailureTerminal --->MQo/p(RetryQueue)OutTerminal-->JavaComputeNode(Stop the CurrentFlow using CMP)to make MainFlow not to pick any other message from Q1.
When the RetryQueue gets the message it starts the RetryFlow
RetryFlow:
MQi/p(RetryQueue)-->Compute(Set MQMD priority=1 & delay for x minutes)--->timeoutControl TimeoutNotification--->MqO/p(Q1)-->Java
Compute(Start the MainFlow using CMP).
This results same Message is retried until Webserver comes up. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Apr 15, 2010 12:21 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Svp wrote: |
Hi Satya,
Check whether this works.
MainFlow:
MQi/p(Q1)--->HTTPRequestFailureTerminal --->MQo/p(RetryQueue)OutTerminal-->JavaComputeNode(Stop the CurrentFlow using CMP)to make MainFlow not to pick any other message from Q1.
When the RetryQueue gets the message it starts the RetryFlow
RetryFlow:
MQi/p(RetryQueue)-->Compute(Set MQMD priority=1 & delay for x minutes)--->timeoutControl TimeoutNotification--->MqO/p(Q1)-->Java
Compute(Start the MainFlow using CMP).
This results same Message is retried until Webserver comes up. |
The flow here is that 0 is the lowest priority and 9 the highest. So every message with a priority of 2 or higher would be processed in front of you... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Svp |
| Posted: Thu Apr 15, 2010 12:31 pm Post subject: |
|
|
Apprentice
Joined: 18 Feb 2010
Posts: 40
|
If the application puts the message to the Queue(Q1) is not going to set the priority then all the messages in the Q1 will have MQMD.Priority = 0.
In the retry flow I am going to set MQMD.Priority =1;
When this message goes back to MainFlow it will sit on the top of all other
messages. As the Priority is 0 for all other messages and they are not picked as the MainFlow is stopped. |
|
| Back to top |
|
|
| satya2481 |
| Posted: Thu Apr 15, 2010 10:23 pm Post subject: |
|
|
Disciple
Joined: 26 Apr 2007
Posts: 170
Location: Bengaluru
|
Hi SVP,
Yes, we can do as you said. But we are not using CMP API in any of the interfaces. To use it again I need to check with my Architects and access needs to be given to the CMP Api to be used etc etc.... its a long process just to start with. So I am not going to that solution.
Regarding priority of the messages I agree with you all. If sending application changes priority then my logic and design will fail. But that will taken care when the situation arrives. As of now application not setting any priority its always 0. I am adding +1 inside the code each time its retries. So macimum retries are only 5 so it will not go beyond 9.
Regards
Satya
_________________
IBM Certified Solution Developer WebSphere Message Broker V6.0
IBM Certified System Administrator WebSphere MQ V6.0 |
|
| Back to top |
|
|
|
|
