| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Some Backup questions |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Thu Aug 09, 2012 6:53 pm Post subject: Some Backup questions |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
MQ 7 on Linux is my use case.
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqzag.doc/fa14700_.htm
This makes it fairly simple - stop the QM, back up the files, start the QM.
Assuming you do this, do you find the restoring from these backups practical? Isn't the possability of reintroducung a whole bunch of messages back into the queues that were already processed a big issue?
In order to minimize down time for the QM
1. Can /opt/mqm be successfully backed up while the QM runs? I was taught no, that backups touching MQ files while MQ runs will at best give you a useless backup and at worst will cause MQ to start farting FDCs. But can't find explicit doc one way or the other when it comes to backing up the *binaries* when the Queue Manager is running using those binaries. The link I posted to above only refers to "working" files, which I take to mean /var/mqm stuff and not /opt/mqm stuff.
2. If the QM is under control of an HA solution like Veritas Clustering, and the QM's queues and logs have been moved out of /var/mqm over to another file system using SAN, can what remains in /var/mqm be successfully backed up while the QM is running?
3. Or have you come to conclusion that MQ does not allow "hot" backups, you must incur the Queue Manager outage it takes to create a full and consistent backup, and the apps just have to deal with a brief outage nightly or weekly. And if you ever have to restore, apps have to deal with duplicate or missing messages, but its better than dealing with no Queue Manager at all if you had to rebuild it from scratch?
_________________
Peter Potkay
Keep Calm and MQ On
Last edited by PeterPotkay on Fri Aug 10, 2012 6:08 am; edited 1 time in total |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Aug 09, 2012 11:21 pm Post subject: Re: Some Backup questions |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
This is another reason why you may want to use linear logs.
Remember the problem with damaged objects? Pray that they were empty when they got damaged if you have circular logs...
| PeterPotkay wrote: |
In order to minimize down time for the QM
1. Can /opt/mqm be successfully backed up while the QM runs? I was taught no, that backups touching MQ files while MQ runs will at best give you a useless backup and at worst will cause MY to start farting FDCs. But can't find explicit doc one way or the other when it comes to backing up the *binaries* when the Queue Manager is running using those binaries. The link I posted to above only refers to "working" files, which I take to mean /var/mqm stuff and not /opt/mqm stuff. |
Like you I would not worry about the binaries (/opt/mqm).
How ever you are hitting a raw point here with "working" files.
If you issue the "advance log" command I would expect that you could "backup" (take a file backup) of all logs but the ones that the qmgr is writing to and holds as next ones... This should be much more clear in a linear log environment than in a circular one. In any case the log pointer file is always in use and can only be safely backed up if the qmgr is down...
| PeterPotkay wrote: |
| 2. If the QM is under control of an HA solution like Veritas Clustering, and the QM's queues and logs have been moved out of /var/mqm over to another file system using SAN, can what remains in /var/mqm be successfully backed up while the QM is running? |
And would not what remains in /var/mqm be nothing... Remember the SAN is being used on the other box... I would imagine you'd call it N A S as it is attached to multiple boxes simultaneously?
| PeterPotkay wrote: |
| 3. Or have you come to conclusion that MQ does not allow "hot" backups, you must incur the Queue Manager outage it takes to create a full and consistent backup, and the apps just have to deal with a brief outage nightly or weekly. And if you ever have to restore, apps have to deal with duplicate or missing messages, but its better than dealing with no Queue Manager at all if you had to rebuild it from scratch? |
Not quite. Consider linear logging and running rcdmqimg... This apparently creates a consistent backup of sorts... You don't have to shut down the qmgr, but if you want to achieve maximum consistency you better run it while the qmgr is at a max idle state i.e. in the timeframe of least traffic.
If just before running it you force an advance log... And no nothing is perfect... you may still not be able to restart a qmgr with this method because the log pointer file is constantly in use and does then not show a consistent log state... But at least it should allow you to ship off the logs to a DR queue mgr and update the DR qmgr (up to the latest log just before you issued advance log)
Conclusion: If you want a backup that allows you a restart of the qmgr, you need a cold backup and shutdown the qmgr for the duration of it.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 10, 2012 6:26 am Post subject: Re: Some Backup questions |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
This is another reason why you may want to use linear logs.
Remember the problem with damaged objects? Pray that they were empty when they got damaged if you have circular logs...
|
If the disk holding your logs is fried, circular versus linear don't matter much. In either case you need the QM stopped to get a good backup.
| fjb_saper wrote: |
| PeterPotkay wrote: |
In order to minimize down time for the QM
1. Can /opt/mqm be successfully backed up while the QM runs? I was taught no, that backups touching MQ files while MQ runs will at best give you a useless backup and at worst will cause MY to start farting FDCs. But can't find explicit doc one way or the other when it comes to backing up the *binaries* when the Queue Manager is running using those binaries. The link I posted to above only refers to "working" files, which I take to mean /var/mqm stuff and not /opt/mqm stuff. |
Like you I would not worry about the binaries (/opt/mqm).
|
I just found an old PMR of mine, where on Windows L2 support said binaries and working data should be excluded from backups if the QM is running. And from anti virus software. The PMR was for a Windows MQ problem.
| fjb_saper wrote: |
| PeterPotkay wrote: |
| 2. If the QM is under control of an HA solution like Veritas Clustering, and the QM's queues and logs have been moved out of /var/mqm over to another file system using SAN, can what remains in /var/mqm be successfully backed up while the QM is running? |
And would not what remains in /var/mqm be nothing... Remember the SAN is being used on the other box... I would imagine you'd call it N A S as it is attached to multiple boxes simultaneously?
|
Its SAN, attached to both servers, but the cluster software insures only one server can 'own' it at a time. But there is plenty of stuff left over in /var/mqm after your move the QM data off to another filesystem. I suspect there is nothing QM specific left behind, outside of some references to the QM in the mqs.ini file, but otherwise I think /var/mqm in this case is 'generic' between 2 systems who have their QM data moved off to a SAN location. Permissions being another difference 2 systems.
Question being can you backup that left over data in /var/mqm while the QM runs? Or even though there might not be anything QM specific in there, the QM is still relying on it, accessing it, and if a backup grabs some of those files during run time you got issues?
| fjb_saper wrote: |
| PeterPotkay wrote: |
| 3. Or have you come to conclusion that MQ does not allow "hot" backups, you must incur the Queue Manager outage it takes to create a full and consistent backup, and the apps just have to deal with a brief outage nightly or weekly. And if you ever have to restore, apps have to deal with duplicate or missing messages, but its better than dealing with no Queue Manager at all if you had to rebuild it from scratch? |
Not quite. Consider linear logging and running rcdmqimg... This apparently creates a consistent backup of sorts... You don't have to shut down the qmgr, but if you want to achieve maximum consistency you better run it while the qmgr is at a max idle state i.e. in the timeframe of least traffic. |
A 'max idle state' is a term used in the manuals. Max Idle for me may be 10 messages a minute. Its the max I can be in terms of idle.
My conclusion, and happy to be corrected please: I think it has to be completely down. MQ has no concept of warm or hot backups like Oracle. If you want the reassurance of having a good point in time backup, you must be willing to pay the price of an outage to take those backups. And that includes /opt/mqm, the QM data wherever it lives (SAN if under VCS or HACMP), and whatever remains in /var/mqm. And then you must deal with restoring BACK to a point in time and potentially reintroducing already processed messages.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 10, 2012 6:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Here's the real question.
What is the purpose of the backup?
Either you want the ability to recover a set of messages at a point in time, or you don't.
If you *don't* want the ability to recover a set of messages at a point in time, then you can get away with a much simpler backup procedure. To whit, if the queue manager dies, delete it and rebuilt it from saveqmgr/dmpmqcfg scripts.
This is *slightly* more complicated if the qmgr is part of a cluster - because of the qmid - but there are various ways to solve that. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Aug 10, 2012 6:57 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
Some of the newer, more expensive disk devices support concurrent copy - a backup while the device (what you are backing up) is in use by other applications.
As the backup progresses, concurrent copy hardware/software detects that a previously backed up portion of the disk has been updated. When this occurs, the newly updated portion of disk is again written to the backup copy.
The backup copy is at the point in time when the backup completes. The completed backup can be much larger than the source disk.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 10, 2012 10:57 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
mqjeff,
Forget about messages for a second. Consider the mount point where /opt/mqm and /var/mqm is "corrupted and unavailable". (Translation - I bet someone deleted the filesystem that these were linked to)
So there are good saveqmgr/dmpmqcfg scripts, and assume the messages are either OK on the SAN, or you don't care about them.
If you had point in time backups of /var/mqm and /opt/mqm I'm guessing you could just restore and move on. But without backups...
You need to reinstall, but will the reinstall balk because the O/S still sees MQ installed?
Will the uninstall balk because there is nothing in /opt/mqm and /var/mqm?
Assuming you can get a good install going, what to do with the QM if all the QM data is on the SAN - does the QM need to be rebuilt or will it magically sync up with the new install?
Could you copy /var/mqm (what's on local disk, not that QM data on that SAN) and /opt/mqm from another system and assuming permissions are all the same GID and UID #, that would work? Don't forget all QM data is on the SAN.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 10, 2012 12:50 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| Could you copy /var/mqm (what's on local disk, not that QM data on that SAN) and /opt/mqm from another system and assuming permissions are all the same GID and UID #, that would work? Don't forget all QM data is on the SAN. |
You should be able to copy at least /opt/mqm, although this is probably not technically supported.
Most of /var/mqm is going to be okay, but at least until 7.1 or later, the lock files and shared memory files for the qmgr are going to be in there - some review of the HA documentation may assist...
So, yes, I would be tempted to delete /opt/mqm and /var/mqm, copy over with ones from a fresh install on another machine on the same platform, and then attempt to use dltmqm and crtmqm with saveqmgr/dmpmqcfg recovery of object. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Aug 10, 2012 1:54 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
But if all the QM data is safe and sound on the shared SAN filesystem, perhaps the rebuilding of the QM will not be needed. I wonder other than mqs.ini are there any other references to the QM left behing in /var/mqm that could have been lost that need to be recreated.
MQ 7.0.1.*
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 10, 2012 2:54 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
| MQ 7.0.1.* |
Okay.
So did you create the qmgr as a multi-instance queue manager? Regardless of whether you ever set up another instance?
Again, the docs on configuring HA... 
If you've done that, then *all* of the information about the queue manager, other than that it exists, will be on the SAN drive.
Also a review of the docs on manually uninstalling may apply as well. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 11, 2012 9:39 am Post subject: Re: Some Backup questions |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
| PeterPotkay wrote: |
- stop the QM, back up the files, start the QM.
Assuming you do this, do you find the restoring from these backups practical? Isn't the possability of reintroducung a whole bunch of messages back into the queues that were already processed a big issue? |
No. A gracefully quiesced qmgr has all messages not yet consumed resting comfortably their queues, and there are no UofW's in-flight.
No. If the qmgr is less-than-gracefully ended, when the qmgr is restarted, logs will be replayed. Messages in UofW's successfully committed out of queues (gets) will be purged from queues. Messages in UofW's committed into queues (puts) will be restored to queues.
No loss, no duplication.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Aug 11, 2012 3:44 pm Post subject: Re: Some Backup questions |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| bruce2359 wrote: |
| PeterPotkay wrote: |
- stop the QM, back up the files, start the QM.
Assuming you do this, do you find the restoring from these backups practical? Isn't the possability of reintroducung a whole bunch of messages back into the queues that were already processed a big issue? |
No. A gracefully quiesced qmgr has all messages not yet consumed resting comfortably their queues, and there are no UofW's in-flight.
No. If the qmgr is less-than-gracefully ended, when the qmgr is restarted, logs will be replayed. Messages in UofW's successfully committed out of queues (gets) will be purged from queues. Messages in UofW's committed into queues (puts) will be restored to queues.
No loss, no duplication. |
There are some messages in some queues. Call these the "A" messages.
You gracefully stop the QM at 01:00:00
You back up at 01:01:00.
You restart the QM.
In the next 12 hours, the "A" messages get processed by the app.
During the same 12 hours, new messages arrive and sit in other queues. Call these the "B" messages.
At 13:00, the file systems get corrupted.
You now restore from the 01:01:00 backup and start the QM.
"WHERE ARE ALL MY "B" MESSAGES?!?!?!?" <--- Lost messages.
"OMG - WHY ARE ALL THESE "A" MESSAGES FLOWING THRU OUR APPS AGAIN"?!?!? <---- Duplicate messages.
I know the sending apps should be able to resend the B messages if properly written.
I know the apps should be able to deal with duplicates of the A messages if properly written.
The question is, how valuable is a point in time back up of a QM if:
1. You need to take an outage to accomplish it (no hot backups).
2. There is no ability to to roll forward to a specific point in time even with linear logs.
3. You risk duplicate messages and missing messages.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sat Aug 11, 2012 4:11 pm Post subject: Re: Some Backup questions |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| PeterPotkay wrote: |
"WHERE ARE ALL MY "B" MESSAGES?!?!?!?" <--- Lost messages.
"OMG - WHY ARE ALL THESE "A" MESSAGES FLOWING THRU OUR APPS AGAIN"?!?!? <---- Duplicate messages. |
Again.
Either you *want* the ability to recover a set of messages at a point in time, or you don't.
Everything else is arguing about *which* point in time.
| PeterPotkay wrote: |
I know the sending apps should be able to resend the B messages if properly written.
I know the apps should be able to deal with duplicates of the A messages if properly written.
The question is, how valuable is a point in time back up of a QM if:
1. You need to take an outage to accomplish it (no hot backups).
2. There is no ability to to roll forward to a specific point in time even with linear logs.
3. You risk duplicate messages and missing messages. |
Again, the argument here is that it was the wrong point in time.
The simple fact is that you should take a point-in-time backup at the right point in time, which I hereby fully define for you.
The only correct point in time to back up messages on a queue manager is when the set of application-produced messages on the queue manager is the empty set.
I.e. only ever back up the queue manager when it's known that there are absolutely no non-system generated messages on it. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 11, 2012 4:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9470
Location: US: west coast, almost. Otherwise, enroute.
|
Whether you are backing up a complete or partial filesystem, or backing up with applications or utilities, a point-in-time backup is just that - historical. Once the backup is complete, subsequent message activity become at-risk for corruption.
Mirrored disk, RAID, and offsite replication, reduce the likelihood of data loss at a single point of failure (SPOF) due to filesystem corruption.
If your design is minimalist - no mirrored disk, no RAID, no replication, no concurrent copy, the risk goes up.
DR planning must contemplate catastrophic outages (loss of data center), single server failure, o/s software failure, disk failure, WMQ software failure, application-specific failure, network failure, ... to loss of a single message.
I recommend to my clients multiple concurrent solutions to minimize loss. Some see the wisdom; some believe they are immune (since they've never experienced a loss before); some don't want to spend the $. It's a choice.
But you know all of this stuff.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Aug 12, 2012 6:21 am Post subject: Re: Some Backup questions |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| PeterPotkay wrote: |
2. There is no ability to to roll forward to a specific point in time even with linear logs. |
Beg to differ. The difference is that you can't control the point in time as it is going to be the point of failure. No duplicates or missing messages expected either...
As a really advanced topic I believe there is the possibility to do a point of synchronization restore (syncpoint being between logs and log pointer entry in log files) but you'd have to check it. (Not for the faint of heart) ...
In either way the point of time is not controlled by you, it is controlled by the qmgr. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sun Aug 12, 2012 3:41 pm Post subject: Re: Some Backup questions |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| fjb_saper wrote: |
| PeterPotkay wrote: |
2. There is no ability to to roll forward to a specific point in time even with linear logs. |
Beg to differ. The difference is that you can't control the point in time as it is going to be the point of failure. No duplicates or missing messages expected either...
|
IF you have all the logs since the last backup....but in our hypothetical situation you took a presumambly good backup at Time A, you had a failure of disk at Time B, all you can do is restore to Time A, a point in time at which time the apps might find messages again already processed and won't find messages added to queues between Time A and B.
What I was really getting at by making the point about not being able to roll forward to a particular point in time is assume you still have all your logs post successful backup - MQ does not allow you to specify any specific point in time when it comes time to relay the logs. They replay all they way to the end.
jeff hit it home for me....If you can take your backups at quiet times, when you can best afford the QM outage and the queues are most likely to be empty, that is your best chance of having a useful backup with the least amount of issues related to reintroducing messages already processed. Any new messages since the backup would be gone, but I think everyone would realize that nothing else would be possible. Again, this is for a catostropic failure where you lost the disks holding both the queues and (linear) logs.
Using Backups QMs (MQ's "log shipping") would help minimize message loss, since a copy of your last complete log would be at an alternate site and protected from whatever harm came by your primary.
As bruce pointed out, lots of technologies to help minimize the loss of those disks or the data due to corruption. Unfortunately none that will protect against the rocket scientest that deletes your dir and all the data in it. In that case all you got is your last backup.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
