| Author |
Message
|
| vickas |
 Posted: Tue Nov 28, 2017 4:42 am Post subject: consequences which lead to broker shutdown |
 |
|
Centurion
Joined: 18 Aug 2013
Posts: 126
|
Hi All ,
can we get to know or someone can list out in what all situation broker will shutdown Automatically ?
Especially in situations where the broker nodes are managed by Veritas cluster.. curios on knowing what happens when broker node cannot access coordination server point .
Thanks . |
|
| Back to top |
|
 |
| Vitor |
| Posted: Tue Nov 28, 2017 6:22 am Post subject: Re: consequences which lead to broker shutdown |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vickas wrote: |
| can we get to know or someone can list out in what all situation broker will shutdown Automatically ? |
It won't. Not by itself.
| vickas wrote: |
| Especially in situations where the broker nodes are managed by Veritas cluster.. curios on knowing what happens when broker node cannot access coordination server point . |
The broker may be shutdown automatically by a Veritas script, but that's not the same as the broker automatically shutting down. If the broker is being managed by Veritas then this is a Veritas question not a broker one; it's all about how you've configured the node.
It's a common practice (which may or may not be in use at your site) to shutdown working components (like broker) as part of a stack failover. A common example would be the broker and queue manager being shutdown (in that order) as a result of a failure in an application that's using them (like a non-zero application exit or a database problem). Another common example is a fail over due to a memory shortage on the server.
It's entirely reliant on how Veritas is set up; both what it does in the event of an issue and (equally importantly) what it considers an issue. It may well, in the example I quote above, react to a non-zero application exit by simply restating the application on the same server. It may equally try that n times and then decide to move the stack to the passive node.
Speak to your Veritas administrator.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Nov 28, 2017 6:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
For the record, I endorse none of the Veritas strategies I outline above, nor do I consider them complete instructions for using Veritas to support MQ, IIB or any application. They are for illustration only and the exact Veritas configuration needed for any given site should be determined in conjunction with the Veritas documentation and the specific requirements of the site.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| vickas |
| Posted: Sun Dec 03, 2017 10:47 pm Post subject: |
|
|
Centurion
Joined: 18 Aug 2013
Posts: 126
|
Hi Vitor ,
thanks for your response.
we find some veritas logs (Resource coordpoint (Owner: Unspecified, Group: CPSFENCE) is FAULTED ) as below and soon after that ,broker instructs all the Integration servers to stop and atlast shutdowns itself .
why does this happen ? can you throw some light , pls . 
19:55:22 AgentFramework[9770]: VCS CRITICAL V-16-10061-639 CoordPoint:coordpoint:monitor:Administrator intervention is required because either the local node cannot access the coordination point server [xx.xxx.xxx.xxx]:443 or it is not registered on the coordination point server. If split brain happens in this condition, all nodes in the cluster may panic. Ensure that the local node can access the coordination points and the keys are registered. Refer to the cpsadm (1m) man page for more information.
19:55:23 VCS ERROR V-16-1-54031 Resource coordpoint (Owner: Unspecified, Group: CPSFENCE) is FAULTED on sys
19:55:24 (Msg 1/1) BIP2152I: Configuration message received from broker
19:55:24 BIP2153I: About to 'Stop' an execution group
19:55:56 BIP2002W: The IBM Integration Bus service has stopped |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Dec 04, 2017 5:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| vickas wrote: |
| 19:55:23 VCS ERROR V-16-1-54031 Resource coordpoint (Owner: Unspecified, Group: CPSFENCE) is FAULTED on sys |
Either something is wrong with your cluster setup (owner: unspecified) or your network or something (firewall) prevented access to a critical point of your infrastructure... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Dec 04, 2017 6:09 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vickas wrote: |
we find some veritas logs (Resource coordpoint (Owner: Unspecified, Group: CPSFENCE) is FAULTED ) as below and soon after that ,broker instructs all the Integration servers to stop and atlast shutdowns itself .
why does this happen ? |
Vertias lost contact with it's coordination server and failed over. This entailed shutting down IIB.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| vickas |
| Posted: Sat Dec 09, 2017 11:05 pm Post subject: |
|
|
Centurion
Joined: 18 Aug 2013
Posts: 126
|
| fjb_saper wrote: |
| vickas wrote: |
| 19:55:23 VCS ERROR V-16-1-54031 Resource coordpoint (Owner: Unspecified, Group: CPSFENCE) is FAULTED on sys |
Either something is wrong with your cluster setup (owner: unspecified) or your network or something (firewall) prevented access to a critical point of your infrastructure... |
Yo saper , thanks for the response.
Yes we observed some issues with the network , even the network team have acknowledged this but all we want to do is avoid the broker switchover when such things happen .
how can we do it ? is it possible to avoid the switch over ?
we have disabled another node to avoid switch over but then the broker keeps on restarting on the same server soon after the occurence of coordination point Fencing issue .
I jus want to understand what is making the broker to restart ? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Dec 11, 2017 6:00 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| vickas wrote: |
| fjb_saper wrote: |
| vickas wrote: |
| 19:55:23 VCS ERROR V-16-1-54031 Resource coordpoint (Owner: Unspecified, Group: CPSFENCE) is FAULTED on sys |
Either something is wrong with your cluster setup (owner: unspecified) or your network or something (firewall) prevented access to a critical point of your infrastructure... |
Yo saper , thanks for the response.
Yes we observed some issues with the network , even the network team have acknowledged this but all we want to do is avoid the broker switchover when such things happen .
how can we do it ? is it possible to avoid the switch over ?
we have disabled another node to avoid switch over but then the broker keeps on restarting on the same server soon after the occurence of coordination point Fencing issue .
I jus want to understand what is making the broker to restart ? |
The broker /application restart is just following the Veritas logic.
You need to fix the root problem which apparently is your network.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Dec 11, 2017 6:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| vickas wrote: |
| how can we do it? |
Fix the network.
| vickas wrote: |
| is it possible to avoid the switch over ? |
Ask the Veritas admin, but I suspect not.
| vickas wrote: |
| jus want to understand what is making the broker to restart ? |
Veritas, issuing an mqsistop command and then an mqsistart command.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
