| Author |
Message
|
| JanB_193 |
 Posted: Wed Oct 22, 2014 4:00 am Post subject: WMQ and IIB DR concept/Ideas |
 |
|
Novice
Joined: 22 Aug 2014
Posts: 15
|
Hello all
Our shop has a running Installation of WMQ and IIB which I currently try to set up in a more controlled manner. Since our SLAs are fairly low - an outage of up to 24h is tolerated - and virtually no MQ know-how is available in our operating departement. I'm forced to take small steps and live with some constraints. Currently there is no dedicated backup or DR strategy around for the MQ environment. After all so far everthing has run smooth for 2 years, so why.... 
So I have a setup with three productive QMs WMQ V8 each with a MB (IIB V9) together one one virtual machine (Windows). All QMs with circular logging, no MQ clustering, no HW clustering.
I'm now considering at least to introduce regular backups of the QMs and Brokers as well as regular snap-shots of the VMWare Image.
As far as I understand a proper backup of a QM requires that it is stopped where as a Broker can be backed up while running (w/o config changes going on). Right?
Do I have to expect problems if I take snapshots of the VM while ist up an running?
And generally what options do I have in my scenario, apart from above mentioned? Backup QMs are only possible with linear logs and clustering is considered too expensive. Any other suggestions/Ideas?
Thanks in advance
Jan |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Wed Oct 22, 2014 4:27 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you looked into the "Multi-Instance" option for MQ and IIB? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Oct 22, 2014 4:35 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Typically a site used for DR is going to be farther away than what's going to be feasible to use for the second instance of the Multi Instance QM.
For High Availability, OK. But not for real DR unless your DR Data Center is close enough to allow synchronous updates of data.
Jan,
Check out the Disaster Recover presentation here:
http://www.mqtechconference.com/sessions.html
Use dmpmqcfg to take object definition backups of your queue managers while they run. Store them and your IIB backups NOT on the same servers as your QMs and Brokers. This is so you can manually rebuild the QM and Broker if all else fails. Replicate the storage that holds these backups off site. Include the info in the qm.ini file in these backups. Ideally you build your QM via a script - have backups of that script.
Talk to your VMware team about getting that VMWare session replicating asynchronously to your DR data center for DR.
Talk to them about snapshot backups at the storage layer for taking backups of the VMware server while it and MQ run unaware of the backup.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Wed Oct 22, 2014 5:01 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
| The consequence of using Circular Logging is that you are OK with loosing all the msgs on a queue (even if they are persistent), if the queue object becomes damaged. A queue object can get damaged when SAN gets jerked out from under the Qmgr or somebody/process modifies the file somehow. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Oct 22, 2014 5:37 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If the SAN gets jerked out from the QM, you are still vulnerable with Linear Logging, presumably because there is no better place to put your logs than on the SAN, same as your queue files. Yeah, you may win the battle and get those SAN guys to segregate your q storage from your log storage, but if you are still on the same frame being accessed by the same fiber, I would bet any problem severe enough to fry your q files is going to blitz those logs anyway.
Yes, there can surely be a scenario where the q object is damaged AND the linear log is fine AND the queue had messages in it AND the messages were important AND the application had no way of recreating the message (wait, who started using MQ is the system of record?).
I wonder, if MQ was just a twinkle in Father MQ's eye in Q4 2014, and they were designing it from the ground up aware of all the advancements with storage technology, would they even offer a Circular / Linear option?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| JanB_193 |
| Posted: Wed Oct 22, 2014 10:59 pm Post subject: |
|
|
Novice
Joined: 22 Aug 2014
Posts: 15
|
Thanks for the answers.
As clustering and HA is considered to expensive options like Multiinstance QMS will not be feasible. 
So I guess I'll have to start with regular backups and prepare scripts to rebuild the environment in case of a disaster.
After such a case I presume, that the other options will become more accepted... 
Regards Jan |
|
| Back to top |
|
|
| exerk |
| Posted: Wed Oct 22, 2014 11:27 pm Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| JanB_193 wrote: |
...So I guess I'll have to start with regular backups and prepare scripts to rebuild the environment in case of a disaster.
After such a case I presume, that the other options will become more accepted... |
That's the spirit!  Just make sure you have a huge 'I told you so' grin on your face when it happens. Frustrating though it is, remember that whilst your employers pay you for your skills and advice they're not obligated to use them.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Oct 23, 2014 4:47 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| exerk wrote: |
| JanB_193 wrote: |
...So I guess I'll have to start with regular backups and prepare scripts to rebuild the environment in case of a disaster.
After such a case I presume, that the other options will become more accepted... |
That's the spirit! Just make sure you have a huge 'I told you so' grin on your face when it happens. Frustrating though it is, remember that whilst your employers pay you for your skills and advice they're not obligated to use them. |
Also ensure that you have a document (indeed several copies of a document) in which you lay out clustering and HA as options, with supporting documentation on when it was ruled too expensive and by whom. One of the well known effects of a disaster is post-traumatic management amnesia, where none of the management can remember being told the cheap DR they'd agreed to was insufficient or indeed remember any other option being discussed with them.
Not only can such documentation save your job and your career, it's really good fun to produce it in the post-mortum meeting and watch them squirm. If you can bear to wait until they've done the speech about "unforeseen failings in the recovery strategy" and "need to undertake an urgent investigation into other options", it will make your day and ruin theirs.
I speak from experience  
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Thu Oct 23, 2014 5:02 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
Not only can such documentation save your job and your career, it's really good fun to produce it in the post-mortum meeting and watch them squirm. If you can bear to wait until they've done the speech about "unforeseen failings in the recovery strategy" and "need to undertake an urgent investigation into other options", it will make your day and ruin theirs.
I speak from experience |
I know of a system located on the West Coast of the USA where the Management spent $500M on the building but were too mean to make a system that is essential to the smooth operation of the whole place resilient. The whole place could have to revert to Paper for everything if this one system fails. All for less than $500K total costs.
Needless to say six months after go live, a PSU Failed. 19hours later it was back running. But, and here is the rub, the beancounters decided that this was ok, as long as the outage was under 24hours and less than once a year and despite the outage costing more than $1M.
I left their employ the next month for obvious reasons.
What Vitor is saying is 100% correct. Protect your own back.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| JanB_193 |
| Posted: Thu Oct 23, 2014 6:23 am Post subject: |
|
|
Novice
Joined: 22 Aug 2014
Posts: 15
|
Thanks a lot for all the good advice - having worked in a very large and well known IT company I know about the importance of "CYA" documents... 
So those will be in place ASAP.
In the meantime I've digged through a lot of material reg. backup and recovery of MQ/IIB. And I understand that it normally makes more sense to backup only the definitions needed te recreate the environment in a fairly quick way if need be and forget about using a snapshot of the QM with messages in the queues
I plan this approach:
1. create scripts to create all MQ artefacts like QM, Queues, channels etc. from scratch
2. have the right stuff backed up to then update the created environment with all configuration changes
3. (possibly, but presumably rather not) recreate the queue content from log backups
Now my questions:
1. As most of the references in the web refer to Unix installations I wonder if I got all I need for Windows? I planned to stop the QM every night (somebody mentioned 3:30 am which sounds good to me) and backup the following directories for object definitions:
C:\...\IBM\WMQ\Qmgrs\@SYSTEM
C:\...\IBM\WMQ\Qmgs\<QMNAME>
and this for the logs:
...\IBM\WMQ\log\<QMNAME>
Is that enough or am I missing vital parts, which I will need for a rebuild?
2. As mentioned before we use circular logging. So rcdmqimg doesn't work for me. Does a backup of the logs make any sense at all then, or do I just live with the fact that any messages might be lost? This last one might sound stupid, but I'm somewhat confused about this.
Regards Jan |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Thu Oct 23, 2014 9:16 am Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
| Check your windows Qmgr to see if it has a qm.ini file. Older ones will not and in that case, you need to get/set the Qmgr stanzas with amqmdain commands. |
|
| Back to top |
|
|
|
|
