ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » MQ Active-Passive DR using VM SRM

Post new topic  Reply to topic Goto page 1, 2  Next
 MQ Active-Passive DR using VM SRM « View previous topic :: View next topic » 
Author Message
Old Lag
PostPosted: Mon Dec 12, 2016 2:51 am    Post subject: MQ Active-Passive DR using VM SRM Reply with quote

Novice

Joined: 02 Jul 2014
Posts: 13

Dear all,
Has anyone any experience of the above please ?
The customer has a 'standard' MQ clustered system providing access to IIB nodes and wishes to setup a Passive DR site.
I have told the customer is that since MQ is really a "transient pipe" in the entire business transaction real-estate, it should come up at the DR site as a clean system with the same config (albeit running on different IP addresses) and using 'standard' dmpmqcfg, copying ini files, binaries, etc. have everything ready at the DR site should the dreadful day ever arrive.
So at the DR site, we would - essentially - recreate the config, have a nice new EMPTY clustered system and be ready and open for new business.
But the infrastructure env. is Linux under VMWare, and the customer is keen on exploiting VM and SRM.
Has anyone any advice/experience in this area ?
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Dec 12, 2016 5:48 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Have you looked at what VM - SRM does https://www.google.com/search?q=VM+SRM&oq=VM+SRM&ie=UTF-8
That should be driving you...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Mon Dec 12, 2016 6:03 am    Post subject: Re: MQ Active-Passive DR using VM SRM Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Old Lag wrote:
Has anyone any advice/experience in this area ?


Yes to both.

My advice, if you want a clean copy of IIB & MQ on a passive DR site, is to keep your head down and let the VM people handle it at their level. It's going to be much easier to use SRM to do this than any kind of application level replication of config or files.

I agree with your comment about "transient pipe" and you should continue to emphasize this with everyone who doesn't run away fast enough. My personal problem with this set up is that we moved to this from a SAN replication DR model, and all the customers were used to a situation where the queue managers came up in the DR site with all the messages still on the queues (because we replicated the disc they were on).

In the new world we come up much faster, but we (quite literally) come up empty and this has caused some applications some problems.

This is a great example of the difference between assured and guaranteed delivery. In this VM model, persistent messages are still assured to be delivered, but only when we fail back to the original data center.....
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Old Lag
PostPosted: Mon Dec 12, 2016 7:09 am    Post subject: MQ Active-Passive DR using VM SRM Reply with quote

Novice

Joined: 02 Jul 2014
Posts: 13

Thank you both for your replies.
I have tried to read about SRM but so far I haven't really got down far enough.
Vitor says - "In the new world we come up much faster, but we (quite literally) come up empty and this has caused some applications some problems.".
I don't know how this can be achieved without having some knowledge of the MQ Config info which is captured by dmpmqcfg (let's leave IIB to one side for now).
I currently have a bunch of Qmgrs in an MQ Cluster (so I have partial and full repos info scattered around the various sys..cluster.xmit.qs) and config. data in sys..auth.data.q.
If I want to capture the config.data, then I either replicate the s.a.d.q data (along with ALL the other queue data which I don't want - especially possibly inconsistent repos info) or I issue the dmpmqcfg cmd anyway in which case why don't I simply stick to the mq-proprietary mechanism.
Any guidance here much appreciated.
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Dec 12, 2016 7:12 am    Post subject: Re: MQ Active-Passive DR using VM SRM Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Old Lag wrote:
I don't know how this can be achieved without having some knowledge of the MQ Config info which is captured by dmpmqcfg (let's leave IIB to one side for now).


Because the VMs have knowledge of that configuration, and can replicate it for you.

Your client wants a passive site that's a clean copy of the configuration as at set up time. So get SRM to make it for you.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Dec 12, 2016 7:33 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

@Vitor: Does your SRM solution not integrate the replication of the Data and log files?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Mon Dec 12, 2016 7:52 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

fjb_saper wrote:
@Vitor: Does your SRM solution not integrate the replication of the Data and log files?


No.

At least not the contents. We have a replicated copy of the queue manager / broker set up in our DR site which has all of the data, log, audit, etc files in all the same paths as they do in the "live" site. They don't contain the same data as they do in the other site.

Note that our DR is a "passive" site in the sense that it's the one we're not using. We fail forward to the passive site about once a quarter at which point it becomes the active one and the other one becomes passive.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Old Lag
PostPosted: Mon Dec 12, 2016 9:18 am    Post subject: MQ Active-Passive DR using VM SRM Reply with quote

Novice

Joined: 02 Jul 2014
Posts: 13

Vitor and fjb_saper.
I think I am starting to see how this will work and I thank you both for your indulging me in what is essentially a VMWare and SRM tutorial. I will be talking to a VM SME shortly to make certain I really understand the technology but from Vitor's latest post, it sounds like he has done what I would like - namely stand-up a DR "clone" of the Primary config but NOT try replicating the data (for all the reasons we understand).
Currently hard-coded IP addresses are used in the Qmgr, Channel and Clint Connections but since this is MQ V7.5, these could be changed to use Hostname and DNS switching should take care of the IP address change.

So if we started off like at T0 (using Hostname) with identical configs at the Primary and DR site, then I can see how it should work. If we apply MQ maintenance at the Primary at T1, then clearly we need to capture that at the DR site. Likewise if we change the config (new queues, ACLs, etc.) at the primary at T2, then we must reflect that also at the DR site and both of the above should be part of operational procedures.

I think I can see how this will work (but maybe that simply means I don't fully understand it !)
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Dec 12, 2016 9:38 am    Post subject: Re: MQ Active-Passive DR using VM SRM Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Old Lag wrote:
Currently hard-coded IP addresses are used in the Qmgr, Channel and Clint Connections but since this is MQ V7.5, these could be changed to use Hostname and DNS switching should take care of the IP address change.


You shouldn't have been using hard coded addresses prior to v7.5


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Old Lag
PostPosted: Mon Dec 12, 2016 11:47 am    Post subject: MQ Active-Passive DR using VM SRM Reply with quote

Novice

Joined: 02 Jul 2014
Posts: 13

"You shouldn't have been using hard coded addresses prior to v7.5".
True, but there is a lot of history here.
One final point and that is the failback situation.

You pointed out that if we just came back up in the Primary, we would indeed come back (or at least try to assuming we hadn't lost some or all of the disk data) at the point of failure, whereas what we really want to do is once again come up with a clean system.

With failover, I can ensure that I have the same disk paths but pointing at essentially clean files.
On failback, do I have to take some separate action to clean the file data - i..e. delete and redefine the Queue/Log stores, etc. - prior to firing up the VMs ?
Is something that SRM can orchestrate ?
I really don't want to inherit the stuff that was there and then have to clean up the messages, issue RESET CLUSTER, etc.

Thanks again for all your help.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Mon Dec 12, 2016 5:48 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

Ideally, apps can deal with missing messages and duplicate messages.
Ideally.

Replicate it all. When you come up in the alternate data center, the apps should know there is a potential for missing or duplicate messages due to the nature of asynchronous replication. But since the apps are properly written (ha!) to deal with a missing or duplicate message during normal operations, they can deal with a few more missing or duplicates in a DR.

Let the database people deal with no loss or no duplication of data. They are good at that.

My feeling is the primary goal for an IIB or MQ Admin when it comes to DR is to be ready ASAP post DR for the next transaction. You do not care about the data (app messages). The data is the app's and the DBA's worry). MQ is not a database.

Don't over-complicate your MQ/IIB DR design chasing an impossible dream (zero message loss or duplication) at the cost of a monstrous Frankenstein solution that doesn't achieve the dream anyway, and makes it take hours or days to get functional for the next transaction after a DR.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Old Lag
PostPosted: Tue Dec 13, 2016 1:39 am    Post subject: MQ Active-Passive DR using VM SRM Reply with quote

Novice

Joined: 02 Jul 2014
Posts: 13

Peter,
In this scenario, the DR site will come up open for NEW business and there will be NO attempt to recover either application or System messages.
Why ? The customer does not want the overhead of Sync - or Async with consistency groups - replication and I have NO problem with that.
(How they are going to work out where they are in terms of System of Record from their DBs without replication is another question but one which I am staying out of for now).
Anyway, given a Passive DR design of - essentially - come up as a clean system open for new business but with no residue of the data at the point of failure at the primary, in my last post, I was simply wondering about the failback story (assuming there is anything left of the Primary to fail back to).
Here, the design could be smarter since - presumably - the DR has been survived, business has continued at the DR site, the data mess has been sorted out and the primary is now back in a state where processing can switch back to it.
In this case, a more orderly switchover could be planned but unless we have Sync/Async group disk replication, we still have the issue of data consistency and I suggest that coming up as a clean system would be the easiest solution. I could imagine a scheduled outage to enable this.
Any advice re. failback from anyone ?
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Dec 13, 2016 5:07 am    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

If you are replicating the virtual server from the primary site to the DR site, you are going to get app and system messages in the queues whether you like it or not.

Your DR plan may call for clearing the app queues on the DR copy of the server when it comes up before releasing the environment to the apps.

Not sure you want to mess with messages in any of the SYSTEM.* queues. Generally you need those to be as is, so the replicated copy of the MQ server functions as the primary copy. So will be crucial to be left alone (like SYSTEM.AUTH.DATA.QUEUE), others are harmless if they have messages (like Event queues).

Regarding failback, interesting how everyone in the room dances around that topic and generally wants to avoid talking about it. At the highest level, its simply an orderly "DR" of your DR, and all the things you plan for and do to go from Primary to DR occur when you go from DR to Primary. This assumes that if you relied on replication to have DR look like Primary, after a real disaster, you very quickly establish your tertiary environment, or rebuild your primary, and get all the replication going ASAP from DR to the restored primary or new tertiary site (the DR for the DR if you will).
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Dec 13, 2016 5:47 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

PeterPotkay wrote:
Regarding failback, interesting how everyone in the room dances around that topic and generally wants to avoid talking about it. At the highest level, its simply an orderly "DR" of your DR, and all the things you plan for and do to go from Primary to DR occur when you go from DR to Primary. This assumes that if you relied on replication to have DR look like Primary, after a real disaster, you very quickly establish your tertiary environment, or rebuild your primary, and get all the replication going ASAP from DR to the restored primary or new tertiary site (the DR for the DR if you will).




We move from one site to another about once a quarter, just to see what doesn't work.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Old Lag
PostPosted: Tue Dec 13, 2016 6:47 am    Post subject: MQ Active-Passive DR using VM SRM Reply with quote

Novice

Joined: 02 Jul 2014
Posts: 13

Many thanks again Peter.
You say -
"Your DR plan may call for clearing the app queues on the DR copy of the server when it comes up before releasing the environment to the apps."
and
"Not sure you want to mess with messages in any of the SYSTEM.* queues."

No disagreement with the 2nd statement which brings me neatly to MQ Clustering.

Unless you are using Sync Disk (or Async with Consistency groups) - neither of which my customer is planning to use by the way - how on earth are you going to avoid the sort of mess you could end up with inconsistent Cluster state across all the SYSTEM.CLUSTER.REPOSITORY.QUEUEs ?

And I can't think that you'd want DR operational procedures that called on you to issue RESET CLUSTER cmds all over the place as everyone is panickig trying to get the systems back up.

Once again, coming up with a clean system open for NEW business sounds the way to go.

P.S. I am SO grateful to you and Vitor for your help here.

Are you
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » MQ Active-Passive DR using VM SRM
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.