| Author |
Message
|
| mqluddite |
 Posted: Wed Mar 26, 2008 4:47 pm Post subject: MQHA failover: messaged inaccessable at failover |
 |
|
Newbie
Joined: 26 Mar 2008
Posts: 5
|
Here's the set up: An application running on SERVER C is "put"ing messages onto the shared cluster queue QUEUE.LQ. There are two instances of QUEUE.LQ, one belonging to cluster queue manager QM.A hosted by SERVER A and one belonging to cluster queue manager QM.B hosted by SERVER B. The WMQ cluster takes care of the load balancing - great, I understand that.
I have another application which is processing the messages that are being placed onto the cluster queue QUEUE.LQ. This application is deployed to a WAS ND cluster which consists of a WAS on SERVER A and a WAS on SERVER B. Currently WAS on SERVER A reads messages from the instance of QUEUE.LQ that belongs to the cluster queue manager QM.A - IE the WAS and cluster queue manager that are hosted on the same machine (SERVER A). Ditto on SERVER B. This work well enough during normal operation.
I wanted to make MQ highly available so I implented MQHA, which work as expected. So when SERVER A blows up SERVER B simply hosts QM.A and the application on SERVER C continues to put messages to both instances of QUEUE.LQ. However the WAS application is not processing messages from the queue belonging to QM.A!
Please could somebody give me an idea how I may access these messages?
Initially I thought that configuring my WAS queue destination with a null queue manager would do the trick from the WAS side - but no, that's not a goer.
And then I thought that the application could issue a mqopen or mqget with an option, which didn't dictate which QM the target queue belonged to (I'm no programmer so I've not given up hope on this yet).
I beginning to think that the only solution open to me would be a method that prevents further messages being sent to the failed cluster queue (belonging to QM.A) and then moves the messages to the cluster queue belonging to QM.B. This seem a bit Heath-Robinson?
There has to be a better way?
SERVER A and SERVER B are running AIX5200-06
I'm running WAS ND 6.0.2.17 and WMQ 6.0 (plus some fixpak)
This is my first posting. Reading through it, it seems pretty convoluted. Hopefully it's clear enough.
Thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Mar 27, 2008 1:21 am Post subject: Re: MQHA failover: messaged inaccessable at failover |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqluddite wrote: |
I wanted to make MQ highly available so I implented MQHA, which work as expected. So when SERVER A blows up SERVER B simply hosts QM.A and the application on SERVER C continues to put messages to both instances of QUEUE.LQ. However the WAS application is not processing messages from the queue belonging to QM.A!
Please could somebody give me an idea how I may access these messages?
|
Ok, I'm not clear on what you mean by "MQHA". Link, description, etc please? Possibly a new term for something I've not heard. Mr Google doesn't seem to have heard of it either.
Moving on, an MQ cluster as you've described provides load balancing as you've seen. What it doesn't provide (hence my question above) is high availability. Each instance of the cluster queue (QUEUE.LQ in your example) is hosted by a given queue manager, and neither it nor the messages it hosts can be accessed when the queue manager is not running.
If QM.A crashes, the cluster will detect this fact and stop putting messages to it. How quickly it detects this fact is dependant on a lot of factors and it's well possible that 1-n messages will be addressed to the downed queue manager. If you search the forum for "stuck messages" you'll see some discussion round this.
Typically a queue manager will be made highly available using something HACMP, so if QM.A crashes then another instance of QM.A starts, which by the magic of HACMP can access the messages and life goes on as normal. I presume (but only presume) from your description this MQHA thing is doing something similar by starting a QM.A on the other server but I don't see how it's going to access the messages or correctly imitate QM.A to the cluster.
In short - I don't know what this thing is so can't really advise, but a WMQ cluster used for HA will exhibit exactly the problem you're describing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqluddite |
| Posted: Thu Mar 27, 2008 1:48 am Post subject: |
|
|
Newbie
Joined: 26 Mar 2008
Posts: 5
|
Thanks for your reply. When I use the term MQHA I am refering to the method that is used by HACMP to make a queue manager highly available. So ,in my example, HACMP detects that SERVER A has fallen over and intervenes by varying on the volume containg the physical queues belonging to QM.A, aliasing the IP addresses that is used to communicate to QM.A onto SERVER B and then start the queue manager QM.A. So now as far as the MQ cluster is concerned all is well with the cluster queue managers - QM.A is still available at the IP address and port that it has always listened on; it just so happens to be hosted by a different machine.
My problem is that the WAS application cannot access the messages on the instance of QUEUE.LQ, which belongs to QM.A. (Further more messages will accumulate on this [physical] queue because the cluster is still work load balancing). I hope that is clear?
I think that I have discovered that my problem is a symptom of a (probably a sensible) "limitation" of MQ. I beleive that if my WAS application were issuing a MQPUT to QUEUE.LQ the MQ cluster would round-robin "put" messages to the queue instances. However the application is actually (under the covers) invoking a MQGET from QUEUE.LQ. It is now my understanding that an MQGET can only take messages from a queue that belongs to the cluster queue manager that the calling application is connected to.
I need a new approach? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Mar 27, 2008 1:59 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You need to put the WAS instance in the failover group with the queue manager.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Mar 27, 2008 2:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqluddite wrote: |
It is now my understanding that an MQGET can only take messages from a queue that belongs to the cluster queue manager that the calling application is connected to.
|
Quite true, but if HACMP has failed over the queue manager to a new location, it should have failed over the WAS as well. Or you should at least have a process to reconnect the WAS (not a big WAS person so forgive any terminology blunders) as from an application standpoint QM.A has crashed and restarted.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqluddite |
| Posted: Thu Mar 27, 2008 2:09 pm Post subject: |
|
|
Newbie
Joined: 26 Mar 2008
Posts: 5
|
The WAS is a constituent member of A WAS ND cluster, which is designed to continue functioning despite the loss of members. Each WAS is a pretty heavy weight JVM so failing the WAS as well as the QM would result in a significant drain on resources of the machine they're newly hosted by! (In fact, I don't believe you could fail a WAS to another machine?).
I'm currently thinking that I could re-architect my WAS ND topology by doubling the the number of WAS JVMs on each machine (upto two) and reducing their heap sizes (maybe this would make efficient use of AIX SMT?). One WAS on SERVER B could talk to QM.B (on the same host) and the other WAS could talk directly to QM.A (on SERVER A). Should SERVER A fail HACMP will move QM.A onto SERVER B. Hey presto - I'm back in business!
I'll look into this further and keep you posted. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 28, 2008 1:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqluddite wrote: |
| (In fact, I don't believe you could fail a WAS to another machine?). |
Yes you can.
| mqluddite wrote: |
| One WAS on SERVER B could talk to QM.B (on the same host) and the other WAS could talk directly to QM.A (on SERVER A). Should SERVER A fail HACMP will move QM.A onto SERVER B. Hey presto - I'm back in business! |
And if SERVER B fails, HACMP moves QM.B onto SERVER A, where there's nothing and you're out of business! 
If, as I think you meant, you have 2 WAS instances on each machine, one for QM.A & one for QM.B, how do you plan to control which WAS instance is using which queue manager?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Mar 28, 2008 4:54 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You can consider using client bindings instead of server bindings for your MQ connections.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Mar 28, 2008 5:06 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jefflowrey wrote: |
| You can consider using client bindings instead of server bindings for your MQ connections. |
Unless you have XA transactions.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Mar 28, 2008 6:10 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| Vitor wrote: |
| jefflowrey wrote: |
| You can consider using client bindings instead of server bindings for your MQ connections. |
Unless you have XA transactions. |
That is one thing to put into the consideration...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mqluddite |
| Posted: Mon Mar 31, 2008 1:25 am Post subject: |
|
|
Newbie
Joined: 26 Mar 2008
Posts: 5
|
| SERVER A JVM1 talks QM.A and SERVER A JVM2 talks to QM.B. It doesn't matter where QM.B is hosted as long as it is accessable (IP and port) to JVM2. The set up on SERVER B will be similar! |
|
| Back to top |
|
|
| JLRowe |
| Posted: Mon Mar 31, 2008 2:42 am Post subject: |
|
|
Yatiri
Joined: 25 May 2002
Posts: 664
Location: South East London
|
| Surely you don't need 2 WAS instances, you can the deploy enterprise application that accesses the queue twice, once against each queue. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Apr 02, 2008 7:19 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Vitor wrote: |
| jefflowrey wrote: |
| You can consider using client bindings instead of server bindings for your MQ connections. |
Unless you have XA transactions. |
ETC would allow client connections even with an XA requirement. An option in this here scenario?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| mqluddite |
| Posted: Wed Apr 02, 2008 11:43 pm Post subject: |
|
|
Newbie
Joined: 26 Mar 2008
Posts: 5
|
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Apr 03, 2008 12:11 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Extended Transactional Client. The standard client doesn't allow the application to participate in XA transactions, the ETC does allow applications connected through it to participate in an XA transaction.
Be aware it's a licensed piece of software that is the same sort of price as a server license (speak to your IBM rep for authoratative prices) rather than the standard client which is free.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
