| Author |
Message
|
| csmith28 |
 Posted: Fri Apr 02, 2004 4:25 pm Post subject: Odd problem with CLUSSDR channels after upgrade |
 |
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
For reasons that are not fully understood, after upgrading to MQ5.3 CSD05 from 5.2.0.7 on a AIX 5.1.0.0 server the cluster sender channels from MQWF Full Cluster repository MQManager to the two remote Cluster Participants would not start normally. After attempting to start they would go into a RETRYING status and the logs said the receiving channels were not properly defined. However we were able to confirm that the receiving channels were there and defined properly. We had the same problem with the CLUSSDR channels on the other two particpant but after starting them manually once the problem did not reoccur even after having bounce MQ and WF but the proble remained on the Full Repository server.
On the Full Repository MQManager attempts to resolve and back out the messages on the sender channels to the other participants failed.
Also we noticed that when we stopped these channels on the Full Repository Manager and then stopped and restarted that MQManager the status of the channel would come back as STOPPED which is abnormal. That channel should not have had any status after the MQManager had been stopped and restarted.
It was as if these channels were shadow or ghost channels. We could see the status of the channels but we could not display the channel attributes. We could stop the channels with mode(force) but mode(quiesce) would not work (as in the channel would stop but instead of going into a "Satus not found" state they would show STOPPED). We could reset the channel and ping it but we couldn't resolve them or delete them.
Eventually we re-defined these two CLUSSDR channels on the Full Repository MQManager to the two other Cluster Particpants and bounced everthing again. This appears to have resolved the problems we were having.
Note we did not alter or redefine the CLUSRCVR channels.
Has anyone run into this problem before? Can anyone explain why this happened?
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
 |
| PeterPotkay |
| Posted: Fri Apr 02, 2004 9:56 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Quote: |
Eventually we re-defined these two CLUSSDR channels on the Full Repository MQManager to the two other Cluster Particpants and bounced everthing again
|
You are not supposed to manually define CLUSSNDR channels on any QM in the cluster, except for one manual CLUSSNDR channel to one Full Repository. The only CLUSSNDR channel you would ever define on a Full Repository is one to the other Full Repository.
All other CLUSSNDR channels will be automatically defined by MQ. If you have manualy defined ones in the way, I don't know what the effect would be, but it is definitly wrong to have them. This may be causing your problem.
Moderator, please move this post to the Cluster Forum.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| csmith28 |
| Posted: Sat Apr 03, 2004 10:08 am Post subject: |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
[quote="PeterPotkay"]
| Quote: |
Eventually we re-defined these two CLUSSDR channels on the Full Repository MQManager to the two other Cluster Particpants and bounced everthing again
You are not supposed to manually define CLUSSNDR channels on any QM in the cluster, except for one manual CLUSSNDR channel to one Full Repository. The only CLUSSNDR channel you would ever define on a Full Repository is one to the other Full Repository. |
Yes I know.
| Quote: |
All other CLUSSNDR channels will be automatically defined by MQ. If you have manualy defined ones in the way, I don't know what the effect would be, but it is definitly wrong to have them. This may be causing your problem. |
Yes I know that the CLUSSDR channels are supposed to be defined based on the definition of the CLUSRCVR channel attributes.
But in this instance manually defining the CLUSSDR channels actually restored service like I said.
| Quote: |
| Moderator, please move this post to the Cluster Forum. |
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Sat Apr 03, 2004 11:26 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Quote: |
Eventually we re-defined these two CLUSSDR channels on the Full Repository MQManager to the two other Cluster Particpants and bounced everthing again.
|
When you say REdefine, to me that means they must have been originally defined, which is not recommended, which may have caused your problems.
If your Full Repositories had bad cluster channels to a QM, issuing the RESET command on one of the full repositories specifying the problem QM would cause the cluster to wipe all info about that QM away, including those bad channels. Then go to the problem QM, insure its CLUSRCVR is correct, insure that it has a CLUSSNDR properly defined to one of the Full Repositories, and then issue the REFRESH command from that QM (with the REPOS(YES) option), which publishes all its info out to the Full Repositories, at which time the Full Repositories would then have the proper info to dynamically create the CLUSSNDRs back to this QM.
If you defined CLUSSNDRs manually on your Full Repositories back to this QM, they weren't used by the cluster. The only time a manually defined CLUSSNDR channel is used is the very first time a QM enters a cluster. Thats how it gets its info to one of the Full Repositories. From that point forward, that manually defined channel will never be used, even if the QM needs to talk to that Full Repository. It will always use an Auto CLUSSNDR channel. The only Exception to this rule is if you issue REFRESH with the REPOS(YES) option, which is telling that QM (among other things) to erase any auto CLUSSNDR channels it has, and to use the manual channel you defined for just that one time of the initial introduction.
| Quote: |
Eventually we re-defined these two CLUSSDR channels on the Full Repository MQManager to the two other Cluster Particpants and bounced everthing again.
|
I wonder if it wasn't the "bounced everything" that fixed the problem?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| csmith28 |
| Posted: Mon Apr 05, 2004 11:40 pm Post subject: Point taken..... |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
OK look, I am sure that is is basic MQ knowledge that you are not "supposed to manually define CLUSSNDR channels on any QM in the cluster".
When I said re-defined, perhaps I should have been more clear.
I expected that MQ would define the CLUSSDR channels based on CLUSRCVR definitions. Perhaps foolishly I assumed that you would expect that I am not a dumb@ss.
But like I stated after the upgrade from 5.2.0.7 to 5.3.05 for some unknown reason MQ would define the CLUSSDR channel but after they were defined the would be started then go into a RETRYING status and the AMQERR0X.LOG's would start complaining that the remote CLUSRCVR channels were not defined or improperly named.
It's the chicken or the egg, Catch 22.
I am traveling right now and will revisit this when I return home, get some rest and can approach this with a clear head.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| JasonE |
| Posted: Tue Apr 06, 2004 4:25 am Post subject: |
|
|
Grand Master
Joined: 03 Nov 2003
Posts: 1220
Location: Hursley
|
Manually defining clussdr channels other than one to the main full repos can occasianally work around a problem where an auto sender has been lost from the cluster cache for one reason or another. It is definitely not something which should be required, and possably masks other problems as well.
If you had serious problems like you describe following an update like that you should probably involve IBM service - they can do more accurate diagnosis based off collected documentation (such as cluster cache dumps) rather then speculation.
What Peter said was accurate and was trying to be helpful - Given the varying level of cluster knowledge it always pays to start from the basics and work upwards.
I would also suggest you put any msgids in your problem report (here or to ibm) as it helps on searchability. One possibility is the problem documented in
| Quote: |
| http://www-1.ibm.com/support/search.wss?apar=include&q=IC37728 |
|
|
| Back to top |
|
|
| csmith28 |
| Posted: Tue Apr 06, 2004 7:22 am Post subject: @Peter |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
I did open a ticket with IBM just after I saved off all the .FDC files and erro logs for that day and just before I posted this thread. However I haven't had the opportunity to really give them a good look. I have been traveling.
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
| warrenJ |
| Posted: Tue Apr 06, 2004 10:56 pm Post subject: |
|
|
Apprentice
Joined: 11 Jan 2004
Posts: 29
Location: AUSTRALIA
|
|
| Back to top |
|
|
| csmith28 |
| Posted: Wed Apr 07, 2004 9:48 am Post subject: Thanks Warren |
|
|
Grand Master
Joined: 15 Jul 2003
Posts: 1196
Location: Arizona
|
I had just finished mining all the data IBM wanted. This looks exactly like the beast I encountered.
It's good to be home. Note to all, if you go to San Francisco do not make reservations at the Hotel Majestic on 1500 Sutter Street. 
_________________
Yes, I am an agent of Satan but my duties are largely ceremonial. |
|
| Back to top |
|
|
|
|
