| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| QPASA and Hardware clusters |
« View previous topic :: View next topic » |
| Author |
Message
|
| PeterPotkay |
 Posted: Tue Sep 23, 2008 6:44 am Post subject: QPASA and Hardware clusters |
 |
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
We have QPASA running successfully in a Microsoft hardware cluster. The benefit it that our MQ alerts work regardless of which node the QM is on. Additionally the end users don't need to guess what node the QM is running on when accessing the Management Console or the Config Manager. They just see the VIP as a node in the QPASA tree and so get to the QM each time.
Have any of you done the same for QPASA on a UNIX hardware cluster? The vendor does provide a doc for how to do it on Windows MSCS, but nothing for UNIX, although they say it should be similar.
The QPASA config files (eaa.xml and eaapi.ini) are 2 things that have to live on the share drive so you have only 1 copy of each. I would prefer to have the QPASA agents themselves be installed on the physical hard drives on both nodes (i.e. /opt/QPASA, like /opt/mqm is installed on both), but it seems to me that won't work, and I need to have the QPASA agents live on the share drive that holds the config files. What I'm specifically curious about is it "safe" or generally an allowable best practice to have executables (the QPASA agents in this case) run on the shared drive that floats between the 2 nodes? If yes I would just make a QPASA folder in /MQHA/queuemanagername/DATA (the dir specified in MC91 and IC91 for MQ and WMB stuff) and put the agents and config files there.
Anyone doing this successfully out there that can share their experience?
Linux x86-64
RHEL 5.2 Cluster
QPASA 4.0.0.180
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Sep 23, 2008 9:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
We don't run HA but we place the agents into /var/mqm/qpasa_version.
As I assume your /var/mqm is shared and this should not be a problem.
However this would only cover an active/passive kind of scenario. If you are running active/active I don't know how you are going to correctly monitor.
Get in touch with MQSoftware support. They are usually quite responsive.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Sep 24, 2008 5:26 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
I posted on the list serve and got a reply from someone doing this with about 10 QMs running on a 4 node VCS cluster. On each Queue Manager's data drive that is managed by the clustering software, they have a set of QPASA agents running. So if and when 2 or more QMs are runing on the same node, there are 2 or more sets of QPASA Agents (qpea, qpmon and qpcfg) running on the server, apparently with no problem, since each as its own storage and its own VIP. Additionally they don't use the cluster to stop and start the QPASA Agents, the use the QM's Service object. If the QM comes up, so does QPASA. If the QM goes down, so does QPASA, but they rely on the qpea and qpmon down alerts built in the QPASA central services to let them know of that situation, after x minutes. X being long enough to allow for a typical failover. Interesting.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hallmark |
| Posted: Thu Jan 22, 2009 3:05 am Post subject: |
|
|
Voyager
Joined: 10 Mar 2005
Posts: 76
|
Hi Peter,
Did you ever get your qpasa working as though independent on your resilient environment?
We are using hacmp and as such we have 2 environments (one on each node of the hardware) both serviced by san disks that are shared between each node. This means that the qpasa agent is installed in a directory that can be failed from one physical box to another. However when both nodes are on the same physical hardware I have been unable to start the agent for the second node? I suspect I may only need the one agent on a box but just wanted to check what your set up ended up being
Thanks
Rob |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jan 22, 2009 5:09 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
We do have QPASA working as a clustered resource in MSCS on the Windows servers.
On our Linux servers running with Veritas Clustering, we do have QPASA go back and forth between the 2 nodes along with the QM, VIP and disks, but its not VCS that's doing it - its the QM. We took advanatage of MQ 6.0's Services. We made a dir for QPASA on the shared drive used by the QM, put all the agents there, and told the QM to bring QPASA up and down as the QM comes up and down, just like it would a Trigger Monitor Service.
"Clustering" QPASA works great for both UNIX and Windows*. The QM shows up under the Virtual Name in the QPASA "trees", so no matter what node its running on, all of us only have to go to the VIP name. All our alerts are based off the VIP, so whichever node the QM is running on, all the alerts are valid.
*There was no way to have the QM control QPASA on Windows like we did on Linux if you are dealing with Hardware clusters, i.e. MSCS. Issues with Windows Services and the way QPASA installs on Windows versus UNIX. You have to introduce the 3 agents into MSCS as clustered resources. No biggie. It works and MQSoftware provides doc on how to do it.
I have not done 2 QMs on a single cluster with clustered QPASA yet.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| hallmark |
| Posted: Thu Jan 22, 2009 5:57 am Post subject: |
|
|
Voyager
Joined: 10 Mar 2005
Posts: 76
|
Thanks for the response,
After some more reading and some assistance from the boffins at MQSoftware I have successfully set up the scenario I outlined. The reason why the second agent wouldn't start on the same hardware was because of a port conflict. The addition of an eaapi.ini file to the agent directory with the service address for the node in question resolves this conflict. So both agent installs have their own eaapi.ini file each detailing their own service ip address. Works a treat
Rob |
|
| Back to top |
|
|
| SAFraser |
| Posted: Wed Jan 28, 2009 3:02 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
But... umm... If the QPasa agent is a service object of the queue manager, and the queue manager shuts it down when it goes down, how does QPasa tell you that the queue manager is down? The only alert possible is "agent down", right?
What am I missing? |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 28, 2009 3:30 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Yes, we monitor for QPASA qpea and qpmon down and treat those as seriously as a QM down alert. You should do this even if you are not using the QM to bring QPASA up and down.
And for the cases where the QM is controlling QPASA, I suppose its possible the QM could die in such a way that it did not have a chance to end the QPASA Service gracefully, leavin the QPASA service running, hence the reason for keeping the QM down monitor as well.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| SAFraser |
| Posted: Wed Jan 28, 2009 4:44 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
Perhaps I did not explain my point very well.
I consider an agent unreachable as a serious event. If the agent is unreachable, it cannot tell me the status of the queue manager.
An "agent down" alert from QPasa can signal network issues and server issues that are unrelated to the queue manager, symptoms of situations that bear further investigation. It tells me more than just the status of qpea and qpmon.
I think QPasa should be the last thing down and the first thing up. I don't see how making a monitoring tool dependent upon the product it's monitoring makes any sense.
Just my opinion, of course! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jan 28, 2009 4:49 pm Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I'd expect that, if the QPasa agent was being told, politely, to stop by the Queue Manager, that it should be able to inform the QPasa monitoring system that "Hey, the qmgr is stopping, and it asked me to stop too. So I'm gonna go away now, but *THE QMGR IS STOPPING*"
Perhaps I expect too much? |
|
| Back to top |
|
|
| SAFraser |
| Posted: Wed Jan 28, 2009 5:01 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
QPasa will tell you anything you want, if you write the alert correctly. And that is exactly what should be done, otherwise the QP console will show its last known status of the queue manager, which will be "up" - at the point the queue manager stops the agent, the agent has recorded the queue manager as "up". Now there are also ways to deal with that, custom views and so forth.
But why would I want to fuss with all that customization? Why would I want the bodyguard to go off duty before the important personage is safely tucked into bed? I wouldn't. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 28, 2009 7:33 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
If you depend on Process A to tell you how Process B is doing, you cannot assume Process B is fine if Process A is down. Its irrelevant how A or B got started, right?
Replace Process A with qpmon/qpea and Process B with Queue Manager.
As soon as the QPASA agent is down on a server, you can no longer assume the health of any QMs on that server. You are flying blind and its a page-able event, same as if the QM was reported down. It doesn't matter that the QM starts and stops QPASA.
Whether QPASA Agent Down or Queue Manager Down is more more serious than the other is probably not worth discussing once you've determined you have to get paged for both. I concede that a QM down alert is more serious, since a QPASA down alert still leaves some hope that the QM is still up and running, and the business is not impacted. This is more likely if QPASA starts and stops indepedently of the QM. But if QPASA starts and stops independently, and you get a QPASA down alert, how can you not assume the box is about to fall over?
I see no problem with having the QM start and stop QPASA if you page for QPASA down, and think its the best way inside a VCS cluster.
_________________
Peter Potkay
Keep Calm and MQ On
Last edited by PeterPotkay on Thu Jan 29, 2009 3:38 am; edited 1 time in total |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 28, 2009 7:37 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| SAFraser wrote: |
| Why would I want the bodyguard to go off duty before the important personage is safely tucked into bed? I wouldn't. |
If you find the bodyguard knocked out, can you assume the important person is safe just because you didn't hear any cries for help from the important person? Nope you got a serious situation, until you confirm the important person, coincidentally called Mr.CueEmm, safe and sound.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| SAFraser |
| Posted: Wed Jan 28, 2009 8:30 pm Post subject: |
|
|
Shaman
Joined: 22 Oct 2003
Posts: 742
Location: Austin, Texas, USA
|
Well, golly, what part of this wasn't clear? And what in the world implied it's not a pageable event?
| SAFraser wrote: |
I consider an agent unreachable as a serious event. |
Just because qpea and qpmon are up does not mean the agent is reachable and the QP database is being updated. That is one reason why I prefer an agent alert.
What do you do about reporting in the Mgmt Console when QP goes down first? It will display the most recent state information stored in the DB, which will be inaccurate.
If the bodyguard gets knocked out, of course I will check on the important personage right away. But a bodyguard getting knocked out is beyond my control; sending him home early before his work is done is another matter. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Thu Jan 29, 2009 4:21 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| SAFraser wrote: |
Well, golly, what part of this wasn't clear? And what in the world implied it's not a pageable event?
| SAFraser wrote: |
I consider an agent unreachable as a serious event. |
|
Sorry, missed that. I was already reading Jeff's reply and your reply to him.
| SAFraser wrote: |
What do you do about reporting in the Mgmt Console when QP goes down first? It will display the most recent state information stored in the DB, which will be inaccurate.
|
I don't report on the QM level, so I haven't thought of this. Being an H.A. QM I would expect it to come up on one of the other alternate nodes relativily quickly. I have sent a requirement into QPASA that it not cache and display last know state like this.
| SAFraser wrote: |
I think QPasa should be the last thing down and the first thing up. I don't see how making a monitoring tool dependent upon the product it's monitoring makes any sense.
|
It does sound funny, I agree. We struggled with it when we first contemplated as well. But I don't feel we have an exposure to have a QM down and not knowing about it, even if the QM starts QPASA:
1. The QM goes down, but not completely. The QPASA Agents stay running. We get paged for QM Down - no problem.
2. The QM goes down, and takes the QPASA Agents with it. We get paged for QPASA down. We log on and the first thing we check is the QM. Still OK.
3. The QM stays up, the QPASA Agents die. We get paged. We log on, see that the QM is still up (phew!), then work on the QPASA Agent problem.
We do the same whether the QM started QPASA or a start/stop script.
In a hardware cluster, the QPASA agents need to move with the QM, VIP, disks, Broker. Consider a 4 node hardware cluster where the QM is only ever running on one node at a time. You can't install QPASA on all 4 nodes and build alerts on all 4 nodes, because 3 of them at any given time would be reporting QM down.
So you can make the QPASA agents clustered resources, controlled directly by the cluster, or let the QM control them like we did. Either of these 2 methods lets the agents move with the QM, and allows all the alerts you build on Node A work equally well on Node B, C and D. And you can tell QPASA to report the virtual name in its trees, so you never have to guess which Node the QM is on - click on Node ABCD, and you connect to where ever the cluster group is.
So, why did we pick the QM starting it versus the VCS cluster? If we had the cluster do it, we could create three QPASA resources and have the QM be dependent on them. That would insure that they come up before the QM does. BUT, what if the QM could come up but QPASA could not? Or what if the QPASA agent dies but the QM was fine? No way was I going to make the QM dependent on QPASA. So we were going to make the QPASA agents come up in the VCS cluster independent of what the QM was going to do, possibly starting before or after the QM depending on timing. It was at that point we said, why complicate the cluster set up with 3 extra resources (qpea, qpmon, qpcfg), its not really buying us anything over letting the QM start and stop QPASA directly.
Here is my QPASA Service as defined at the QM, if anyone wants to implement something like it:
| Code: |
DEFINE SERVICE ('QPASA') +
* ALTDATE (2008-11-19) +
* ALTTIME (21.52.23) +
DESCR('Starts & stops all 3 QPasa services whenever QMGR starts & stops') +
STARTCMD('/MQHA/+QMNAME+/data/qpasa_scripts_logs/qpasa_cluster_start.sh') +
STARTARG('+QMNAME+') +
STOPCMD('/MQHA/+QMNAME+/data/qpasa_scripts_logs/qpasa_cluster_stop.sh') +
STOPARG('+QMNAME+') +
STDOUT(' ') +
STDERR('/MQHA/+QMNAME+/data/qpasa_scripts_logs/qp.err') +
CONTROL(QMGR) +
SERVTYPE(COMMAND) +
REPLACE
|
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
