| Author |
Message
|
| migz0901 |
 Posted: Mon Oct 12, 2015 1:28 am Post subject: DR DRILL exercise cluster issue |
 |
|
Apprentice
Joined: 01 Nov 2012
Posts: 28
|
Hi Guys,
.
We had our DR (Disaster recovery exercise) DRILL last September and we hit a problem in wmq cluster between Mainframe and Aix server and No firewall in place. BCV split in Mainframe happened on Friday morning from PROD to DR environment. We bring up MQ qmgr and chinit and run jobs to alter channel conname,etc to point to DR ip address on Friday afternoon. Then, open system noticed that our DR MQ poisoned their prod qmgr on actual DR day which is Sunday.
Is there any manual for best practice in cluster for this kind of activity (DR DRILL) to avoid the same issue in our next DR drill exercise.
1. AIX server - Full rep
2. Mainframe - partial rep
AIX Dr server only brought up on Sunday.
Thanks in advance!
Migz0901 |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Mon Oct 12, 2015 3:39 am Post subject: Re: DR DRILL exercise cluster issue |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| migz0901 wrote: |
Hi Guys,
.
We had our DR (Disaster recovery exercise) DRILL last September and we hit a problem in wmq cluster between Mainframe and Aix server and No firewall in place. BCV split in Mainframe happened on Friday morning from PROD to DR environment. We bring up MQ qmgr and chinit and run jobs to alter channel conname,etc to point to DR ip address on Friday afternoon. Then, open system noticed that our DR MQ poisoned their prod qmgr on actual DR day which is Sunday.
Is there any manual for best practice in cluster for this kind of activity (DR DRILL) to avoid the same issue in our next DR drill exercise.
1. AIX server - Full rep
2. Mainframe - partial rep
AIX Dr server only brought up on Sunday.
Thanks in advance!
Migz0901 |
Without a lot more details, I'm inclined to say working as designed...
Define "our DR MQ poisoned their prod qmgr"... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| migz0901 |
| Posted: Mon Oct 12, 2015 6:08 pm Post subject: |
|
|
Apprentice
Joined: 01 Nov 2012
Posts: 28
|
our DR qmgr connects to the aix prod MQ server during exercise.
How to avoid this in future?
Thanks. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Oct 13, 2015 4:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| migz0901 wrote: |
our DR qmgr connects to the aix prod MQ server during exercise.
How to avoid this in future?
Thanks. |
Change the config of your DR testing scenarios and environment.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| migz0901 |
| Posted: Fri Oct 16, 2015 1:05 am Post subject: |
|
|
Apprentice
Joined: 01 Nov 2012
Posts: 28
|
| We run jobs to alter the DR MQ config but somehow our DR qmgr connects to prod AIX server in a cluster setup. In WMQ cluster perspective is there a best practice in bringing up DR MQ cluster qmgr after prod bcv split. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Oct 16, 2015 4:30 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You need to be aware of your configuration. If you can't afford to have the DR qmgr to "poison" Prod, then you can't have the DR qmgr be a member of the Prod cluster...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rammer |
| Posted: Fri Oct 16, 2015 4:31 am Post subject: |
|
|
Partisan
Joined: 02 May 2002
Posts: 359
Location: England
|
To confirm you changed the conname on the defined Cluster Sender on the Mainframe that points to your Full Repos on AIX?
And also changed the IP on the Cluster Receiver on the Mainframe? |
|
| Back to top |
|
|
| migz0901 |
| Posted: Sat Oct 17, 2015 8:48 am Post subject: |
|
|
Apprentice
Joined: 01 Nov 2012
Posts: 28
|
Hi fjb_saper,
Definitely our DR MQ cluster is separate from our MQ prod cluster.. but somehow after our DR MQ process executed which we are doing for years, our DR cluster connects to AIX prod cluster. |
|
| Back to top |
|
|
| migz0901 |
| Posted: Sat Oct 17, 2015 8:50 am Post subject: |
|
|
Apprentice
Joined: 01 Nov 2012
Posts: 28
|
Hello rammer,
Yes we run jobs to alter the channel conname in DR machine to point to DR IP addresses of AIX servers, this is part of our process.
thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Oct 17, 2015 7:03 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| migz0901 wrote: |
Hi fjb_saper,
Definitely our DR MQ cluster is separate from our MQ prod cluster.. but somehow after our DR MQ process executed which we are doing for years, our DR cluster connects to AIX prod cluster. |
Before running any active MQ messages through run a display mqtrace a few times to see which paths are being taken through the cluster.
Also don't change just before sending messages. Depending on the size of the cluster you may have to change a week before. Remember the cluster will need some time to propagate the information... Make sure you stop and restart any channels you may have changed. And follow the cluster change procedure from the manual to the letter.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| migz0901 |
| Posted: Sun Oct 18, 2015 8:04 am Post subject: |
|
|
Apprentice
Joined: 01 Nov 2012
Posts: 28
|
Hi Poobah,
any link to cluster change procedure?
TIA |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sun Oct 18, 2015 8:36 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| migz0901 wrote: |
Hi Poobah,
any link to cluster change procedure?
TIA |
Go to google. In the search text box, type: mq clusters manual
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
