ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » MQ Backup strategies

Post new topic  Reply to topic
 MQ Backup strategies « View previous topic :: View next topic » 
Author Message
blorro
PostPosted: Wed Mar 28, 2018 4:21 am    Post subject: MQ Backup strategies Reply with quote

Acolyte

Joined: 09 Jan 2014
Posts: 57
Location: Sweden

Hello.
What kind of MQ Backups are you running "out there" in the real world ?
Are you running 3rd party products for backing up Production Queue managers or have you crafted own solutions ?
If "hom crafted", what kind of structure are you using ?
Daily backups ?
How long do you store Queue manager backups for ?
_________________
"Anything is possible, all the time."
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Mar 28, 2018 5:27 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

We have the queue manager software install automated (along with all the other software installs), so we can quickly rebuild a failed queue manager, and the definition scripts are held in Git. We've also taught our deployment tool (which does our Java and so forth) to run those scripts for us so that post-rebuild the queue structure can be put back the way it was.

Application developers know better than to keep messages on queues for extended periods of time. We have databases for that.

We have backups of the queue managers because we backup the underlying file systems. Because we backup all the underlying file systems. Even the DASD. I swear the operational risk people would backup the restrooms if a product became available. In the event of an proper outage, the Linux people would probably use that to put the server back as it was yesterday, including any queue manager that happened to be hosted there. If we just lost the queue manager but the underlying server was ok, we'd remove and rebuild the queue manager as described above.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Mar 28, 2018 8:19 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Multiple strategies are required to address the various possibilities of loss, from loss of object(s) in a specific qmgr, loss of a specific qmgr, loss of a disk volume, loss of an o/s, loss of an LPAR, loss of hardware platform, loss of a computer facility.

Which are you asking about?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
blorro
PostPosted: Wed Mar 28, 2018 8:24 am    Post subject: Reply with quote

Acolyte

Joined: 09 Jan 2014
Posts: 57
Location: Sweden

Well,mainly MQ objects , queue manager ,queues,channel defs, authority. And then having a sound policy of maintaining the backups,making sure they run ok and so forth.
_________________
"Anything is possible, all the time."
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Wed Mar 28, 2018 6:22 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

Vitor wrote:
...We have backups of the queue managers because we backup the underlying file systems. Because we backup all the underlying file systems. Even the DASD. I swear the operational risk people would backup the restrooms if a product became available. In the event of an proper outage, the Linux people would probably use that to put the server back as it was yesterday, including any queue manager that happened to be hosted there. If we just lost the queue manager but the underlying server was ok, we'd remove and rebuild the queue manager as described above.
Backing up files systems while qmgrs are running is not reliable for DR. In a live DR, if the backed up recovery logs were restored, the qmgr may not start due to corrupted logs, and the qmgr will need to be recreated, or a cold-start set of logs copied in.

Also in a live DR, all application queues should be handed over empty. Resolution of transaction streams is the responsibility of apps, not MQ. Random app messages sitting on queues after a DR restore can cause a lot of headaches.
_________________
Glenn
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Mar 28, 2018 6:41 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

blorro wrote:
Well,mainly MQ objects , queue manager ,queues,channel defs, authority. And then having a sound policy of maintaining the backups,making sure they run ok and so forth.


Write a script to execute dmpmqcfg in client mode from a central server to reach out to each QM and grab a backup of all the definitions. Search for dmpmqcfg on this site and read all you can.

Use the central corporate job scheduling software at your company to make sure your backup job runs when its supposed to, and to successful completion. Or use cron / Windows Scheduler as a 2nd choice.

Maintain a rolling x days worth of these backups. What is x? I dunno, start with 32 (one month is always available)

Backup up the backups - wherever they live for their 32 days should itself be backed up at the file system level.

Feel free to duplicate ALL of the above on ANOTHER server. Parallel backups.

qm.ini and mqs.ini and other config files on the MQ server will need to be backed up some other way, but change so infrequently that any corporate backup team can and should just treat them as regular files for regular incremental and full backups. Since they change so infrequently, the gold copies you keep in source control are probably good enough. Unless every QM has a different qm.ini file beyond the name of the QM itself ( I hope not).

Do not let the backup team backup up your transaction logs and queue files. OK, maybe in 2018 its better, but back in the ol' days backups and virus scanners would give a queue manager fits when their software got all touchy-feely with MQ files that the QM was trying to use.

Backups are nothing more than untested restore files.
So actually use your backups every so often for a test restore.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Mar 28, 2018 6:59 pm    Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

gbaddeley wrote:

Also in a live DR, all application queues should be handed over empty. Resolution of transaction streams is the responsibility of apps, not MQ. Random app messages sitting on queues after a DR restore can cause a lot of headaches.

I agree.
I'll also say that applications should be written to handle missing and duplicate messages when dealing with an asynchronous messaging technology. And if they can deal with 1 or more missing or duplicate messages in Production, they can do the same in DR. Ehhhhh, I am not going to blow away messages from application queues on a replicated queue manager that comes up in DR mode at the DR site.
I completely understand the viewpoint of someone who says they will.

In either case, the bigger priority is to be ready for the NEXT transaction to flow thru MQ after a DR. My priority is to have the plumbing functional to get business going again. Not an expensive, complicated and inevitably fruitless attempt to try to have every app message that was somewhere in MQ be available in DR, which due to the annoying laws of physics governing the speed of light between data centers across the country says its impossible no matter what you do if you are replicating asynchronously.

I guess I could see myself saying it doesn't matter if the app queues are handed over empty or not in a DR. The app should be able to handle either scenario by reconciling against its database. The database is the system of record, not the queue. Since it (probably) doesn't matter, and its impossible to get 100% right, don't bother over complicating DR trying to do the impossible. Focus on doing DR simply so you know the simple solution understood by all works 100% of the time. And let the DBAs worry about the data.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Vitor
PostPosted: Thu Mar 29, 2018 5:29 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

gbaddeley wrote:
Backing up files systems while qmgrs are running is not reliable for DR.


I never said we backed them up while the queue manager was running.

gbaddeley wrote:
In a live DR, if the backed up recovery logs were restored, the qmgr may not start due to corrupted logs, and the qmgr will need to be recreated, or a cold-start set of logs copied in.


In a live DR we'd probably rebuild the servers as I describe.

gbaddeley wrote:
Also in a live DR, all application queues should be handed over empty. Resolution of transaction streams is the responsibility of apps, not MQ. Random app messages sitting on queues after a DR restore can cause a lot of headaches.


In a live DR, there's an expectation that the applications (and indeed the business) will need to spend some time picking shrapnel out of the walls. My comment below about application developers knowing better than to keep messages on queues is grounded in my stated position that any such messages will not survive most DR scenarios, which obviously has implications for in-flight business transactions and hence (again as I previously stated) status should be held in databases.

Aforesaid databases are replicated in near-real time (sub second SLA) as part of the site DR strategy.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mvic
PostPosted: Thu Mar 29, 2018 2:31 pm    Post subject: Reply with quote

Jedi

Joined: 09 Mar 2004
Posts: 2080

Remember also that if you restore a clustered queue manager from backup, you will have to run REFRESH CLUSTER on it to give it up-to-date internal sequence numbers so that it participates in the cluster properly. If you don't, then the queue manager will proceed with "old" sequence numbers and any updates it sends to the full repositories will be ignored, leading to other problems of missing state information.
See:
(1) https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.pla.doc/q004830_.htm
(2) https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.pla.doc/q004820_.htm
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Apr 02, 2018 4:15 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2492
Location: Melbourne, Australia

PeterPotkay wrote:

I'll also say that applications should be written to handle missing and duplicate messages when dealing with an asynchronous messaging technology. And if they can deal with 1 or more missing or duplicate messages in Production, they can do the same in DR. Ehhhhh, I am not going to blow away messages from application queues on a replicated queue manager that comes up in DR mode at the DR site.
I completely understand the viewpoint of someone who says they will.

Agree. A live DR can be chaotic. If servers are automatically moved to a DR site and brought up, the MQ admins may have no control over when the qmgrs are started, and when they can do appropriate healthchecks and queue clean ups, before the apps start using MQ. The apps should be tolerant of the infrastructure state (servers, DBs, MQ and other integration points). However, if MQ is clean, its one less thing for them to be concerned about.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » General IBM MQ Support » MQ Backup strategies
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.