ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexIBM MQ SecurityLDAP Authentication failure after MQ Server restart

Post new topicReply to topic
LDAP Authentication failure after MQ Server restart View previous topic :: View next topic
Author Message
saurabh25281
PostPosted: Mon Jun 24, 2019 11:38 pm Post subject: LDAP Authentication failure after MQ Server restart Reply with quote

Voyager

Joined: 05 Nov 2006
Posts: 98
Location: Bangalore

Hi All,

We faced an issue in our environment which blocked our access to MQ Explorer connecting to remote Queue Manager.

We have a Cluster setup which was mistakenly configured with usage as Normal for the System Transmit queue. This caused the below error causing our Queue Manager to go unresponsive.

Code:
----- amqrrmfa.c : 2205 -------------------------------------------------------
AMQ9509E: Program cannot open queue manager object.

EXPLANATION:
The attempt to open either the queue or queue manager object '' on queue
manager '' failed with reason code 2085.
ACTION:
Ensure that the queue is available and retry the operation.

----- amqrfxra.c : 2834 -------------------------------------------------------
AMQ9511E: Messages cannot be put to a queue.

EXPLANATION:
The attempt to put messages to queue 'SYSTEM.CLUSTER.TRANSMIT.QUEUE' on queue
manager 'D01A03' failed with reason code 2092.
ACTION:
Ensure that the required queue is available and operational.

----- amqrrmfa.c : 20064 ------------------------------------------------------                 
AMQ9878I: Cluster repository command DB_AVAILABLE for TC02 encountered a
problem

EXPLANATION:
An internal cluster repository command failed to complete successfully,
reporting error reason 536909073. Earlier messages in the log will contain
details of the problem. The command originated from queue manager
'D01A12_2019-06-17_12.59.45', relating to object 'TC02'. Failure to
successfully process a command may leave your cluster in an inconsistent state.
ACTION:
Investigate the problems reported. Use the information provided when contacting
MQ support.

----- amqrrmfa.c : 3667 -------------------------------------------------------
AMQ9448E: Repository manager failed.  Retry in 10 minutes, queue manager will
terminate in 7060 minutes

EXPLANATION:
Repository manager encountered a severe problem. See the earlier messages in
the queue manager or system error logs for details. The repository manager will
retry the command in 10 minutes. If the problem is not rectified in 7060
minutes the queue manager will terminate. Until this problem is rectified no
further cluster management activity will occur, this will affect the
availability of cluster resources accessed or hosted by this queue manager.
ACTION:
If possible, rectify the identified problem, otherwise contact your IBM support
center. To postpone the queue manager from terminating due to this problem set
the SYSTEM.CLUSTER.COMMAND.QUEUE queue to be GET(DISABLED). Once the problem
has been rectified, set the queue to be GET(ENABLED) and wait for the
repository manager to retry the command or restart the queue manager.

----- amqrccca.c : 439 --------------------------------------------------------
AMQ9509E: Program cannot open queue manager object.

EXPLANATION:
The attempt to open either the queue or queue manager object
'SYSTEM.CLUSTER.TRANSMIT.TC02.D01A12' on queue manager 'D01A03' failed with
reason code 2085.
ACTION:
Ensure that the queue is available and retry the operation.

----- amqrmssa.c : 978 --------------------------------------------------------
AMQ9999E: Channel 'TC02.D01A12' to host 'hostname(1414)' ended
abnormally.

EXPLANATION:
The channel program running under process ID 23563 for channel 'TC02.D01A12'
ended abnormally. The host name is 'hostname(1414)'; in some cases
the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
--------------------------------------------------------------------------------


We restarted the MQ server and since then, we lost connectivity to remote Queue Managers using MQ Explorer which validates our remote connections using LDAP. We are getting the below LDAP errors.

Code:
----- amqzfula.c : 2882 -------------------------------------------------------
AMQ9557E: Queue Manager User ID initialization failed for 'MyMCAUser'.

EXPLANATION:
The call to initialize the User ID 'MyMCAUser' failed with CompCode 2 and Reason
2035. If an MQCSP block was used, the User ID in the MQCSP block was ''. If a
userID flow was used, the User ID in the UID header was 'MyUser' and
any CHLAUTH rules applied prior to user adoption were evaluated
case-sensitively against this value.
ACTION:
Correct the error and try again.

----- cmqxrsrv.c : 2311 -------------------------------------------------------
AMQ5530E: Error from LDAP authentication and authorization service

EXPLANATION:
The LDAP authentication and authorization service has failed. The
'ldap_simple_bind' call returned error 49 : 'Invalid credentials'.  The context
string is 'UserDN@LDAPServer:389
'. Additional code is 0.
ACTION:
Correct the LDAP configuration. Look at the LDAP server logs for additional
error information.

----- amqzfula.c : 2882 -------------------------------------------------------


Our access was working fine until we restarted our MQ Servers, so its puzzling to me, if there is any link between the Cluster errors in the logs and the authentication failure. Can someone help in understanding what might be causing the authentication failure and if there is any link between the two issues?
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
exerk
PostPosted: Mon Jun 24, 2019 11:47 pm Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6072

Have you resolved the MQRC 2085 issue with queue SYSTEM.CLUSTER.TRANSMIT.TC02.D01A12 ?

Have you tracked what changes were made (if any) to your queue manager or its objects, before this happened?

Any FDC files cut and if so, what is their content?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
saurabh25281
PostPosted: Tue Jun 25, 2019 12:49 am Post subject: Reply with quote

Voyager

Joined: 05 Nov 2006
Posts: 98
Location: Bangalore

Quote:
Have you resolved the MQRC 2085 issue with queue SYSTEM.CLUSTER.TRANSMIT.TC02.D01A12 ?

This issue happened on our testing environment and we had to bring our systems up urgently and hence had to re-create the Qmgr which resolved all the issue. We kept a copy of the .LOG files though.

Quote:
Have you tracked what changes were made (if any) to your queue manager or its objects, before this happened?

We tried to create a Cluster, which failed. Although the Cluster sender channels were successfully created.

Quote:
Any FDC files cut and if so, what is their content?

No FDC files were generated.
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
hughson
PostPosted: Tue Jun 25, 2019 3:17 pm Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1182
Location: Bay of Plenty, New Zealand

saurabh25281 wrote:
Our access was working fine until we restarted our MQ Servers, so its puzzling to me, if there is any link between the Cluster errors in the logs and the authentication failure. Can someone help in understanding what might be causing the authentication failure and if there is any link between the two issues?


saurabh25281 wrote:
This issue happened on our testing environment and we had to bring our systems up urgently and hence had to re-create the Qmgr which resolved all the issue.


You have mentioned restarting the MQ Server and recreating the Queue Manager. One of these things is not like the other.

If you recreated the queue manager, presumably you have not recreated the objects that control configuration to your LDAP server correctly.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
saurabh25281
PostPosted: Tue Jun 25, 2019 10:57 pm Post subject: Reply with quote

Voyager

Joined: 05 Nov 2006
Posts: 98
Location: Bangalore

Quote:
You have mentioned restarting the MQ Server and recreating the Queue Manager. One of these things is not like the other.

Yes these 2 actions are different.
1. Restarting the Qmgr was done to get over the unresponsiveness of the Qmgr. This did not solve the problem, rather added another issue, i.e. blocking access for MQ Explorer users.
2. Re-creating the Qmgr was done to make the setup usable when the 1st option failed. We were able to re-create & resolve incorrect Xmitq usage errors & access to MQ Explorer was restored.

Quote:
If you recreated the queue manager, presumably you have not recreated the objects that control configuration to your LDAP server correctly.

We deleted the Qmgr and Recreated it. While re-creating the Qmgr, we used a standard definition which is a tested configuration of our solution. Re-creation of Qmgr resolved all our issues.
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
hughson
PostPosted: Wed Jun 26, 2019 1:38 am Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1182
Location: Bay of Plenty, New Zealand

I have read and re-read your question a number of times (because clearly I didn't understand it the first time!). It seems you are saying that you had the cluster XmitQ error, and while that error was present in the error logs you still had remote access to the queue manager via MQ Explorer. This would seem to suggest that the answer to your question:-

saurabh25281 wrote:
Our access was working fine until we restarted our MQ Servers, so its puzzling to me, if there is any link between the Cluster errors in the logs and the authentication failure.


... is no. Since you did not have authentication failures when you did have the cluster errors prior to the restart.

You then restarted the queue manager - which did not solve the cluster error and led you to ultimately recreate the queue manager which not only solved the cluster error, it also solved the authentication issue.

saurabh25281 wrote:
While re-creating the Qmgr, we used a standard definition which is a tested configuration of our solution. Re-creation of Qmgr resolved all our issues.


So, your question is simply, why did restarting your queue manager cause the LDAP setup to be affected such that you received the following error message:-

saurabh25281 wrote:
Code:
AMQ5530E: Error from LDAP authentication and authorization service

EXPLANATION:
The LDAP authentication and authorization service has failed. The
'ldap_simple_bind' call returned error 49 : 'Invalid credentials'.  The context
string is 'UserDN@LDAPServer:389
'. Additional code is 0.
ACTION:
Correct the LDAP configuration. Look at the LDAP server logs for additional
error information.

----- amqzfula.c : 2882 -------------------------------------------------------


Can you tell us whether the context string shown in the above error message is correct, as per the recreated configuration that now works? If, so that would seem to suggest that the password was wrong. You have told us that all you have left of this queue manager are the error log files, so we cannot look into anything else from this now deleted queue manager, so is this the only error message that mentions LDAP? And, as is suggested in the error above, what do the LDAP server logs have to say about the issue?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
saurabh25281
PostPosted: Wed Jun 26, 2019 3:03 am Post subject: Reply with quote

Voyager

Joined: 05 Nov 2006
Posts: 98
Location: Bangalore

Quote:
So, your question is simply, why did restarting your queue manager cause the LDAP setup to be affected such that you received the error message

Yes, that was the simple question, but I felt it was necessary to give a background of what steps we did to get this error.

Quote:
Can you tell us whether the context string shown in the above error message is correct, as per the recreated configuration that now works? If, so that would seem to suggest that the password was wrong.

Yes the context string works well after re-creation of Qmgr. In the absence of any other evidence, it seems that the password was wrong, however, no changes were made to the configurations or the password.

Is it possible for the password that is configured to go corrupt?

Quote:
You have told us that all you have left of this queue manager are the error log files, so we cannot look into anything else from this now deleted queue manager, so is this the only error message that mentions LDAP?

As mentioned in my initial post, there are 2 error codes relating to the LDAP authentication AMQ5530E & AMQ9557E.

Quote:
And, as is suggested in the error above, what do the LDAP server logs have to say about the issue?

We are trying to get the LDAP logs and would keep you posted.
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
fjb_saper
PostPosted: Wed Jun 26, 2019 5:32 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20069
Location: LI,NY

I would also expect changes to have been made to the authentication without the corresponding refresh security type (connauth). These would then quick in upon qmgr restart... that might also explain why MQE stopped connecting...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
hughson
PostPosted: Wed Jun 26, 2019 4:23 pm Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1182
Location: Bay of Plenty, New Zealand

fjb_saper wrote:
I would also expect changes to have been made to the authentication without the corresponding refresh security type (connauth). These would then quick in upon qmgr restart... that might also explain why MQE stopped connecting...

Very good suggestion!

Is this possible saurabh25281?

Could changes have been made to LDAP config that only kicked in upon queue manager restart due to omitted REFRESH SECURITY TYPE(CONNAUTH) command?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
saurabh25281
PostPosted: Thu Jun 27, 2019 1:06 am Post subject: Reply with quote

Voyager

Joined: 05 Nov 2006
Posts: 98
Location: Bangalore

Quote:
I would also expect changes to have been made to the authentication without the corresponding refresh security type (connauth). These would then quick in upon qmgr restart... that might also explain why MQE stopped connecting

We normally take a backup of our Qmgrs (using dmpmqcfg) before re-creating one. Hence, I was able to retrieve some information from the backups and compared the creation time of the Qmgr and the alteration time of the Connauth object which negates the above assumption.

Code:
AUTHINFO
*  ALTDATE(2019-05-24) +
*  ALTTIME(19.25.23) +

Qmgr Creation Time
*  CRDATE(2019-05-24) +
*  CRTIME(19.22.08) +


The issue occured almost after a month on 2019-06-21 and hence there was no change in between. The time delay of approx 3 mins, between Qmgr creation and Connauth creation is as per our scripts that perform some other configurations after the creation of Qmgr. So this is understandable.

One question still remains, is it possible that the configured password is somehow corrupted?

Regards
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
fjb_saper
PostPosted: Thu Jun 27, 2019 4:29 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20069
Location: LI,NY

saurabh25281 wrote:

We normally take a backup of our Qmgrs (using dmpmqcfg) before re-creating one. Hence, I was able to retrieve some information from the backups and compared the creation time of the Qmgr and the alteration time of the Connauth object which negates the above assumption.

Code:
AUTHINFO
*  ALTDATE(2019-05-24) +
*  ALTTIME(19.25.23) +

Qmgr Creation Time
*  CRDATE(2019-05-24) +
*  CRTIME(19.22.08) +


The issue occured almost after a month on 2019-06-21 and hence there was no change in between. The time delay of approx 3 mins, between Qmgr creation and Connauth creation is as per our scripts that perform some other configurations after the creation of Qmgr. So this is understandable.

One question still remains, is it possible that the configured password is somehow corrupted?

Regards

Number one: You did not specify the frequency of your backups. So it is entirely possible that something got changed that was not backed up yet.
Number two: the alteration time is past the creation time. However there is no guarantee that after the alteration time a refresh security type(connauth) was run. Again this would impact the behavior you reported and show up at the next qmgr restart.

And remember to go live you need a change in authinfo, or / and a change in the qmgr attributes (connauth) and finally a refresh of security type(connauth).

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
saurabh25281
PostPosted: Thu Jun 27, 2019 11:03 am Post subject: Reply with quote

Voyager

Joined: 05 Nov 2006
Posts: 98
Location: Bangalore

Quote:
Number one: You did not specify the frequency of your backups. So it is entirely possible that something got changed that was not backed up yet.

As i mentioned earlier, that this was a non-prod environment and hence there were no regular backups. However, the procedure is to take a backup before we delete a Qmgr. The info that I am quoting is from the backup that was taken just before the Qmgr was removed. So the above assumption is invalid.

Quote:
Number two: the alteration time is past the creation time. However there is no guarantee that after the alteration time a refresh security type(connauth) was run. Again this would impact the behavior you reported and show up at the next qmgr restart.

You have a valid point here. I was unable to find any Qmgr restarts from the logs after the Connauth alteration time.
Somehow the modification was done on the Connauth Object, but the Qmgr restart or connauth refresh was not performed after that.

Thanks for the pointers fjb & Morag.

Regards
Saurabh
Back to top
View user's profile Send private message Send e-mail Yahoo Messenger
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexIBM MQ SecurityLDAP Authentication failure after MQ Server restart
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.