| Author |
Message
|
| mqdev |
 Posted: Mon Jul 01, 2019 12:57 pm Post subject: procedure to replace a drbd-node in a working drbd-cluster? |
 |
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Hello,
We recently had a situation wherein one of the drbd nodes in a cluster had to be replaced with a new server - and had to practically reinstall RDQM from scratch on all 3nodes. The prob statement is as follows:
Lets say a/b/c are implementing a 3-node drbd cluster on which RDQM is installed and configured. There are a bunch of Queue Managers (in our setup, we had 4 QMs). All was well in the MQ world at this point. However, when migrating an MQ application, to the RDQM, we ran a script on one of the nodes - say c, which (by mistake), changed the permissions on the root FS - thereby breaking that node. We had to replace node c by a new node d. So the intent was to have a,b,d as the drbd-cluster.
We tried editing rdqm.ini and reconfigure drbd cluster ( rdqmadm -U for delete the drbd cluster followed by rdqmadm -c with the a,b,d in rdqm.ini). This just did not work. Long story short, we had to uninstall and reinstall RDQM on a,b,d (as if a new RDQM being installed on a/b/d) and recreate the QMs afresh on a/b/d.
That brings up a question - what is the IBM recommended procedure to replace one or more drbd-nodes in a working (i.e. has one or more healthy RDQM Queue Managers running) drbd-cluster?
Thanks
-mqdev |
|
| Back to top |
|
 |
| exerk |
| Posted: Mon Jul 01, 2019 11:06 pm Post subject: Re: procedure to replace a drbd-node in a working drbd-clust |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6191
|
| mqdev wrote: |
| ...That brings up a question - what is the IBM recommended procedure to replace one or more drbd-nodes in a working (i.e. has one or more healthy RDQM Queue Managers running) drbd-cluster?... |
Basically, THIS.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
 |
|
| Back to top |
|
|
| mqdev |
| Posted: Mon Jul 15, 2019 5:25 am Post subject: THIS does not work... |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Thanks exerk for the response.
We are not able to make the cluster "believe" that one of the node is replaced (A/B/C is now A/B/D). The RDQM commands keep looking for C and the QM is broke. The only way to get this to work was to dismantle the entire drbd cluster with A/B/C and build a new one with A/B/D - lot of work for a simple end-result and may not be practical all the time (imagine if these were Production....)
cheers
-mqdev |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 15, 2019 5:37 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6191
|
Did you read the content of the link I posted? it clearly states that A/B/D will NOT work, and the simplest way would to have rebuilt C, or a clone of C - as stated in the KC.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
|
|
| Back to top |
|
|
| mqdev |
| Posted: Mon Jul 15, 2019 7:11 am Post subject: |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
exerk: See the second bullet under "Procedure" ( If the replacement node has a different configuration to the failed node:) - this is what applicable to A/B/D and that is not working!
In my case, I do not have an option to do A/B/C (by restoring C from an image/backup etc). Specifically, the IP of D is different from C (even if I build a new box from clone of C, I cannot get back the same IP for totally un-related reasons). |
|
| Back to top |
|
|
| exerk |
| Posted: Mon Jul 15, 2019 7:35 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6191
|
| mqdev wrote: |
| exerk: See the second bullet under "Procedure" ( If the replacement node has a different configuration to the failed node:) - this is what applicable to A/B/D and that is not working! |
If you are absolutely sure that you followed the procedure, and that procedure did not work, raise a PMR.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
|
|
| Back to top |
|
|
|
|
