| Author |
Message
|
| mqbrks |
 Posted: Wed Feb 22, 2012 7:32 am Post subject: Does broker respond to the changes of failover/takeover |
 |
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
Hi All,
We have a application, where the broker uses(calls) Erlang(java procedure) to interface with a peer system. The peer system has the primary and secondary nodes. Whenever the primary goes down, the secondary takes over and becomes active in few seconds.
The broker sends data to this peer system. The logic in erlang code is to repeatedly see which node is responding and send data to it.
When the primary went down, the broker was trying to repeatedly poll the primary and secondary and could not connect to secondary or primary when the primary went down. All the messages were piled up in queue as broker was not able to commit the transaction.
Once the EG was recycled, the broker responded to the peer system and sent the message successfully.
Why did the broker not respond to secondary when the primary went down?
Do we need to bounce the broker whenever the failover/takeover happens on the peer system. |
|
| Back to top |
|
 |
| lancelotlinc |
| Posted: Wed Feb 22, 2012 7:36 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Your question is why did erlang not behave properly. As I understand your story, erlang does not equal broker. Broker has nothing to do with it, other than a host for erlang.
Debug your code and find out the answer. By the way, you should host third-party code in a Jar not in a JCN. The JCN should call the Jar. The Jar should have test cases written, hopefully in Junit, that proves or disproves the implementation is successful.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 22, 2012 7:42 am Post subject: Re: Does broker respond to the changes of failover/takeover |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqbrks wrote: |
| Why did the broker not respond to secondary when the primary went down? |
Either the secondary node of this peer system seamlessly takes over the role of the primary (a failover) or it's a just an alternative destination for the traffic. If it's the former then the broker shouldn't know or care that the primary has failed. If it's the latter then either you need a specific method to handle the fact the connection has been lost. Either code in the Java or bouncing the broker.
This is a design issue not a software one.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Wed Feb 22, 2012 8:02 am Post subject: Re: Does broker respond to the changes of failover/takeover |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| Vitor wrote: |
Either the secondary node of this peer system seamlessly takes over the role of the primary (a failover) or it's a just an alternative destination for the traffic. If it's the former then the broker shouldn't know or care that the primary has failed. If it's the latter then either you need a specific method to handle the fact the connection has been lost. Either code in the Java or bouncing the broker.
This is a design issue not a software one. |
Thank You Vitor for the response. Secondary node of the peer system takes over the role of primary. That is how it is designed. So the only solution is routing the traffic to secondary(java func to be accordingly implemented) or bouncing the broker? |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Feb 22, 2012 8:05 am Post subject: Re: Does broker respond to the changes of failover/takeover |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Feb 22, 2012 8:11 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You just need to revisit your Java code. presumably the Erlang java APIs have functions for this that have not somehow been implemented.
As lancelotlinc suggests, it can be useful to run tests against this kind of thing from the java command line, rather than having to use a message flow. So you might run some standard erlang failover tests using standard java code run on the broker machine from the command line and make sure you understand the behavior. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 22, 2012 8:15 am Post subject: Re: Does broker respond to the changes of failover/takeover |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqbrks wrote: |
| Secondary node of the peer system takes over the role of primary. |
If it takes over the role of the primary (i.e. it becomes the primary) then nothing (Java, WMB, whatever) should notice the change. If it takes over the workload of the primary then something in your design needs to handle that change.
| mqbrks wrote: |
| That is how it is designed. |
Then there's a missing piece of your design.
| mqbrks wrote: |
| So the only solution is routing the traffic to secondary(java func to be accordingly implemented) or bouncing the broker? |
Or fixing the design so that the secondary node fully assumes the role of the primary.
IMHO bouncing the broker is going to take too long for the average failover scenario. Aside from the impact on everything else running on the broker that's perfectly fine but has to take an outage just because this peer system's gone over. Fix the Java. Or fix this peer system.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Feb 22, 2012 9:27 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
A couple of erlang-specific things you can check:
1. Is the security cookie on node A the same as the security cookie on node B?
2. Use ip address in your erlang code, not host systems (ie. avoid DNS).
3. Can you ping both systems ?
One design question:
Why do you not just use broker to communicate directly with your destination nodes? Erlang provides no added value over what broker does natively.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Feb 22, 2012 12:15 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| mqbrks |
| Posted: Wed Feb 22, 2012 12:42 pm Post subject: Re: Does broker respond to the changes of failover/takeover |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
thank you vitor and lancelotlinc.
We tried to replicate the issue.( The logic to connect to primary and secondary is already implemented in java(erlang code). )
Today it worked like charm. We found no issues polling to primary when secondary is down. connecting to secondary when primary is down.
Not sure why the messages were sitting in the queue yesterday and the logic was not working fine(to connect to primary/secondary).
can it be the issue of jms queues.
And about lancelotlinc question.
Why Erlang?
Even I was thinking to implement TCP IP nodes to connect to peer system.
This design and code was implemented long back. So my team is still using the same. I would definitely try to use this feature. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Feb 22, 2012 12:49 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
Ok, good luck. I know you can do it. 
TCPIPClient nodes are very easy to use and you should have little problems using them to talk to any Telnet server, including erlang servers.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| mqbrks |
| Posted: Wed Feb 22, 2012 12:54 pm Post subject: Re: Does broker respond to the changes of failover/takeover |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| mqbrks wrote: |
Not sure why the messages were sitting in the queue yesterday and the logic was not working fine(to connect to primary/secondary).
. |
Or it could be that the primary and secondary both have not come up. |
|
| Back to top |
|
|
| mqbrks |
| Posted: Wed Feb 22, 2012 1:01 pm Post subject: |
|
|
Voyager
Joined: 17 Jan 2012
Posts: 75
|
| lancelotlinc wrote: |
Ok, good luck. I know you can do it.
TCPIPClient nodes are very easy to use and you should have little problems using them to talk to any Telnet server, including erlang servers. |
Thank you for the support  |
|
| Back to top |
|
|
| zpat |
| Posted: Tue May 22, 2012 11:38 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Set the DNS TTL value. The default is unlimited which means once Java (e.g. a broker node) has resolved the DNS name to an IP address it will never update that IP address even when the DNS changes (until the JVM ends).
This breaks some fail-over designs which update the DNS to reflect a change to the active server.
You can set the TTL value in the Java security policy or using a line like this in your environment (broker .profile)
| Code: |
### stop unlimited Java DNS caching - reduce to 300 seconds ###
export MQSIJVERBOSE=-Dsun.net.inetaddr.ttl=300 |
|
|
| Back to top |
|
|
|
|
