| Author |
Message
|
| kash3338 |
 Posted: Sat Jan 30, 2016 2:06 am Post subject: Handling downtime of IIB interfaces |
 |
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
Hi,
We have a HA Broker environment in our Production. Its a pretty new Production server and only about 20 Message Flows have been deployed so far. All these flows are consumed by a Mobile App (Android based) and open in public space (Play Store).
Now, we have many more Message Flows being developed, which are not related to the Mobile Application that is already live, and planned to be moved to Production within the next 6 months.
The problem that I am foreseeing here is, every deployment has either a new DSN creation, SSL Certificate Configuration or some other activity which requires a Broker restart. Whenever we do such deployment, it may require a Broker restart, which will have a downtime of the Mobile application though they are not related to each other.
How do we overcome this problem of downtime's and what is the standard approach followed across Middle-ware solutions? |
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Sat Jan 30, 2016 4:24 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Schedule the restarts for a time when activity is at a minimum, say 02:00 on a monday morning.
Is this Active-Active or Active-Passive? IF the latter, simply fail it over to the other side.
The Downtime for a properly configured system should be just a few minutes.
If it is Active-Active then having a load balancer fronting it will/could solve your problems.
You switch off connects in the LB (Again at a time of minimum traffic) to one broker and restart it. If everything is hunky-dory then get traffic going by enabling it in the Load Balancer. Rince and repeat with the other system.
You will have to consider any MQ Connected clients as well. You can setup clients to use the first active QMGR so if you stop the data flows in the LB, then stopping the APP Connected Listener/channels could do it for you.
Each site has different connectivity so what might work for one won't always work for another. You will have to develop procedures specific to your site for this operation.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Feb 01, 2016 8:55 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
Thanks smdavies99.
We have a Active-Passive setup. Though the downtime is very small, say few minutes, the problem is, the App users needs to be notified with a SMS on every downtime. Since there is a series of deployments planned in a period of six months, we may be sending too many SMS's on downtime. This will surly have a bad impact on the users, which is a major concern. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 01, 2016 8:58 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kash3338 wrote: |
| We have a Active-Passive setup. Though the downtime is very small, say few minutes, the problem is, the App users needs to be notified with a SMS on every downtime. Since there is a series of deployments planned in a period of six months, we may be sending too many SMS's on downtime. This will surly have a bad impact on the users, which is a major concern. |
Then you need 2 Active/Passive nodes with a load balancer in front of them. Route all traffic to one pair, update the other, verify it, switch all the traffic to that and update the other. Increases uptime and eliminates the risk that a few minutes downtime while you restart turns into an hour or so while you diagnose the problems (and possibly roll back).
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 01, 2016 9:04 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Vitor wrote: |
| kash3338 wrote: |
| We have a Active-Passive setup. Though the downtime is very small, say few minutes, the problem is, the App users needs to be notified with a SMS on every downtime. Since there is a series of deployments planned in a period of six months, we may be sending too many SMS's on downtime. This will surly have a bad impact on the users, which is a major concern. |
Then you need 2 Active/Passive nodes with a load balancer in front of them. Route all traffic to one pair, update the other, verify it, switch all the traffic to that and update the other. Increases uptime and eliminates the risk that a few minutes downtime while you restart turns into an hour or so while you diagnose the problems (and possibly roll back). |
 
If your site won't stump up the money then you need to agree with your users to have a regular weekly outage slot. Then you need ot only notify the users for exceptional outages.
It can be done. At my last place of employment we had a regular outage period of 5-7pm on a Friday. Well, it is Beer O'Clock isn't it?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Feb 01, 2016 9:08 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| Vitor wrote: |
| Then you need 2 Active/Passive nodes with a load balancer in front of them. Route all traffic to one pair, update the other, verify it, switch all the traffic to that and update the other. Increases uptime and eliminates the risk that a few minutes downtime while you restart turns into an hour or so while you diagnose the problems (and possibly roll back). |
This is possibly the last option we have considered for now as there are not too many services running currently. I agree this is a permanent and a proper solution, but just want to know is there is a way out with current setup.
Why is the Broker server / EG level restart ever required for certain configurations? Cant there be a way around for refreshing the configurations in such a critical backbone product? With a product like IIB v9 this was expected to be handled differently, but.... |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Feb 01, 2016 9:15 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| kash3338 wrote: |
| Why is the Broker server / EG level restart ever required for certain configurations? |
For the same reason you sometimes need to restart WAS / Apache / Tomcat / other critical backbone products.
| kash3338 wrote: |
| Cant there be a way around for refreshing the configurations in such a critical backbone product? |
Probably. Raise an RFE and post the link here.
| kash3338 wrote: |
| With a product like IIB v9 this was expected to be handled differently, but.... |
I'd turn the question round, because I'd expect your code to be handled differently. IIB is not WAS, where each application is in it's own isolated container but is a singular entity running a variety of applications. Why does each application need a new DSN or need an individual & unique SSL? Why are these broker level shared resources not being set up once and shared by all applications? Apart from that's how they've been designed because the designers are used to WAS?
Also SSL is a poor example. You can set up SSL at an EG level, and an EG reload is no time at all unless you've hideously overloaded it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Mon Feb 01, 2016 9:18 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| kash3338 wrote: |
Why is the Broker server / EG level restart ever required for certain configurations? Cant there be a way around for refreshing the configurations in such a critical backbone product? With a product like IIB v9 this was expected to be handled differently, but....
|
Once your production environment is fully setup then you don't make many changes.
Why not plan the changes (such as mqsichangeproperties etc) so you roll up a bumber of them and even do them in advance of the deployment of the ssociated mesage flows?
Planning and Forethought are essential in this world
I made a change to a flow on one of our sites today. Apart from Product Updates/Upgrades none of the flows apart from this one had been changed in more than two years. This is 24/7 operation.
There are planned outages of all system on this site. Every active/passive environment is failed over every month at 02:00 on the first Monday of the month.
Planning and forethought.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Mon Feb 01, 2016 9:28 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| smdavies99 wrote: |
Once your production environment is fully setup then you don't make many changes.
Why not plan the changes (such as mqsichangeproperties etc) so you roll up a bumber of them and even do them in advance of the deployment of the ssociated mesage flows?
Planning and Forethought are essential in this world
I made a change to a flow on one of our sites today. Apart from Product Updates/Upgrades none of the flows apart from this one had been changed in more than two years. This is 24/7 operation.
There are planned outages of all system on this site. Every active/passive environment is failed over every month at 02:00 on the first Monday of the month.
Planning and forethought. |
|
|
| Back to top |
|
|
| kash3338 |
| Posted: Tue Feb 09, 2016 1:05 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
| Vitor wrote: |
| Probably. Raise an RFE and post the link here. |
Raised a RFE for this request here |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Feb 09, 2016 2:24 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Link does not work.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Feb 09, 2016 5:24 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| zpat wrote: |
| Link does not work. |
I think you have to be logged in to developerworks first. Although, I do sometimes get an 500 error back from thr website.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| kash3338 |
| Posted: Wed Feb 10, 2016 12:36 am Post subject: |
|
|
Shaman
Joined: 08 Feb 2009
Posts: 709
Location: Chennai, India
|
|
| Back to top |
|
|
|
|
