| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Recreating a Cluster |
« View previous topic :: View next topic » |
| Author |
Message
|
| TonyD |
 Posted: Thu Jan 17, 2002 10:25 am Post subject: |
 |
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
We have a quite simple but critical (to the business) cluster consisting of 2 * NT queue managers and 2* HP queue managers. The NT queue managers are identical (MQSI brokers with workload balancing) and there are probably in excess of 100 cluster queues defined.
I want to specify a procedure that will allow us to recreate our cluster very quickly and with full integrity if there are problems.
Currently we have saved definitions of our four queue managers. When we have had problems with the cluster (which we have had!) in testing we have deleted the queue managers, recreated them, recreated the cluster and added the queue managers to the cluster. This works fine....the repositories exchange the information about the cluster queues and channels and the non-repository queue managers automatically rebuild their partial repositories over time.
However under this approach we lose any messages..not good for a Production environment
I therefore want to achieve the same result without recreating the queue managers, i.e. 'rebuild' the repositories. I have experimented with removing all queue managers, including repository qms, from the cluster, recreating the cluster (in the Explorer) and then adding the queue managers back into it. However this appears to deliver inconsistent results, particularly with respect to the second repository.
I have the Cluster Utilities Supportpac and wonder if they can be used to achieve what I want.
Ideally I want to remove all entries from the SYSTEM.CLUSTER.REPOSITORY.QUEUES on all queue managers and then rebuild them from scratch. However I do not want to lose the cluster properties of the cluster queues and channels.
Any suggestions or advice would be gratefully received.
In connection with the above topic I would really like to know a bit more about how MQ clusters work under the covers. I have looked at the messages on the Repository queues and compared them with DUMP REPOS and amqrfdm outputs to get a bit of an idea but would feel more comfortable if I understood a bit more about what causes information to be exchanged, how often it happens, exactly what REFRESH does etc. Is there any undocumented doc that has this sort of info? |
|
| Back to top |
|
 |
| bduncan |
| Posted: Thu Jan 17, 2002 2:44 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Well I have a lot of personal experience in this area, though I can't point to much documentation outside of the IBM manuals. Basically, rebuilding the cluster without blowing everything away is tough, and as you pointed out, the results are inconsistent. As far as the REFRESH command works, it goes something like this. When you are issuing it on a queue manager that isn't a full repository, it will remove all cluster object defintions from it's own partial repository, and ask the full repository that you manually created channels to for updated information. Keep in mind that it will only inquire about cluster objects that were already in its partial repository. On the other hand, if you issue the REFRESH command on a full repository machine, it appears to force it to negotiate with the other full repository(s) in an attempt to synchronize their repositories. For instance, I have QM1 and QM2 which are both full repositories, and for whatever reason, QM2 doesn't see a clustered queue called 'A' on QM3. So I issue a REFRESH on QM2, and it appears to negotiate with QM1, sees that QM1 has a cluster object definition it doesn't, and then get that definition from QM1. However, the converse sometimes happens where you have a full-repository with corrupt information, and by issuing a REFRESH (in an attempt to get it to drop the corrupt objects) actually infects the other repository with the bad information. This happens very often in the case of "ghost" queues. You have a clustered queue which you remove, and only one of the full repositories removes it. The other full repository still sees it, and in an attempt to get it to flush this ghost definition, you issue a REFRESH, and only end up in propogating the ghost definition to the other repository....
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| TonyD |
| Posted: Fri Jan 18, 2002 7:22 am Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
Thanks for comments Brandon.
I have done some experimentation to recreate repositories by running the 'Remove' and 'Clean' cluster utilities against each of my cluster QMs, clearing the messages off all repository queues (had to stop amqrrfma to do this), recreating the cluster, adding the queue managers back into the cluster, and then finally running the 'Restore' utility to make the cluster queues available.
Seemed to work OK for the repository queue managers but cluster queues on non-repository QMS do not get propagated to the repositories....on my first attempt anyway. Will proceed with further experimentation. |
|
| Back to top |
|
|
| bower5932 |
| Posted: Fri Jan 18, 2002 12:34 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
Tony,
I attended a session once that went into some internal details on clusters. I don't remember much other than:
- You can't really look at the messages on the queues and figure anything out. MQSeries keeps an internal cache and writes messages on the queues to rebuild the cache. The messages are in some kind of internal format.
- You can definitely get into problems with deleting queue managers and recreating them. The new queue manager will not be recognized as a replacement for the old one because of some internal name (QMID from runmqsc - dis qmgr?).
- Just to be safe, deletes will hang around since a repository can be getting notifications from multiple places. MQSeries needs to make sure that something is really deleted or not.
After listening in, the conclusion that I came to was to be careful (which I think you already know).
The other place that you might be able to get detailed information is at the MQSeries Conference in April in Dallas. A lot of developers actually make presentations, and you should be able to either find a session or somebody who can give you some specifics.
Ron |
|
| Back to top |
|
|
| bduncan |
| Posted: Fri Jan 18, 2002 2:15 pm Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
Also, keep in mind that any clustered queues which you delete but aren't actually removed from the cluster - you type DIS QCLUSTER(*) on a full-repository and still see the deleted queue, it's not actually a big problem. If there are other instances of this queue in the cluster, then the workload balancing will avoid the deleted instance because it will appear to be unavailable. If it's the only instance, and applications are still trying to PUT or GET from it, they will get errors. Then after 90 days, the definition will automatically get flushed. Apparently 90 days of inactivity for a cluster object removes it from the repositories.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
| TonyD |
| Posted: Sun Jan 20, 2002 9:56 am Post subject: |
|
|
Knight
Joined: 15 May 2001
Posts: 540
Location: New Zealand
|
Thanks for your comments guys...yes, I am very conscious that it would be very easy to make things worse. This was why I wanted to actually destroy the repositories themselves, both full and partial.
I managed to get what appears to be a reasonably safe procedure working, after a bit of experimentation. The Cluster utility CLEAN lets one remove all knowledge of the cluster from a queue manager and also creates a Restore file which puts the Cluster attribute back on the queues. I therefore ran this utility against my four queue managers. The cluster satisfactorily disappeared. The bit I am uncomfortable with was the need then to manually stop AMQRRMFA so I could clear the cluster queues including the repository...one would have to ensure that NOTHING else was going on in the queue managers at this time.
I then stopped and started the QMs to get AMQRRFMA started again and each Repository queue was initialised with s single 12K msg. It was then necssary to delete the cluster sender channels between the repostories in order to explicitly define them....if this is not done, the second repository never gets populated. I then ran the Restores (ex CLEAN) to restore the cluster attributes to my queues and appeared to have a sound cluster again.
The purpose of this exercise is for emergency use only and then only under very controlled circumstances....I will also test it more. However if you have an unusable cluster I guess you can't make things much worse???!! |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
