ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexIBM MQ Installation/Configuration SupportQueue manager quiescing

Post new topicReply to topic
Queue manager quiescing View previous topic :: View next topic
Author Message
kalam475
PostPosted: Fri Mar 10, 2017 10:08 am Post subject: Queue manager quiescing Reply with quote

Acolyte

Joined: 16 Jan 2015
Posts: 56

Hi team,

We have a situation where my client application running in Jboss server is connecting to my queue managers participating in cluster through CCDT file. My cluster has three qmgrs. two of them hosting the cluster queues(QM1 and QM2) and one qmgr doesn't have any queues it is only for cluster (a full repository). All three queue managers are multi-instance queue manager.

so we wanted to do some failover testing with some load.
System1
QM1(active and full repository)
QM2(standby and half repository)
QM3(standby and half repository)

system 2
QM1(standby and full repository)
QM2(active and half repository)
QM3(active and full repository)

we started the puting the load and i stoped the qmgr QM1 with endmqm -i -s QM1. It is gracefully switchover to system2 and becomes active. meanwhile my application in jboss using CCDT will connect to QM2 and put the message in one of the cluster queue in it.once QM1 is also active in system 2 my application works fine and accepting the load and distributes the load b/w two cluster queues.

The same test we wanted to do it when system 1 shuts down abruptly rather than ending queue manager.While we shutdown the system queue manage is going into quisece state for 2 minute and then switchover to standby during which heavy load comes to JBOSS connection factory. the connection factory assign all its connections to quisece qmgr(because it is the first qmgr in CCDT file) and was not able to close the connections and soon my jboss hangs.

Below is the error:

Code:

INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar: JMSCC3036:An exception has been delivered to the connection's exception listener: '
11:55:04,803 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                        Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar: An error has occurred with the WebSphere MQ JMS connection.
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar: Use the linked exception to determine the cause of this error.
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                          Class : class com.ibm.msg.client.jms.DetailedJMSException
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                          Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:595)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:215)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:786)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.api.RemoteHconn.callEventHandler(RemoteHconn.java:2918)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.api.RemoteHconn.driveEventsEH(RemoteHconn.java:629)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.processHconn(RemoteDispatchThread.java:659)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:221)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
11:55:04,804 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:341)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1227)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:      Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                          Class : class com.ibm.mq.MQException
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                          Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:203)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:786)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.api.RemoteHconn.callEventHandler(RemoteHconn.java:2918)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.api.RemoteHconn.driveEventsEH(RemoteHconn.java:629)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.processHconn(RemoteDispatchThread.java:659)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:221)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:319)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:99)
11:55:04,805 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:341)
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:312)
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar:                                : com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1227)
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession[connectionId=414D51434541492E43514D3120202020BF39C258278A2B24]]) wmq.jmsra.rar: '.
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (default task-44) wmq.jmsra.rar: JMSCC3036:An exception has been delivered to the connection's exception listener: '
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (default task-44) wmq.jmsra.rar:                        Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (default task-44) wmq.jmsra.rar: An error has occurred with the WebSphere MQ JMS connection.
11:55:04,806 INFO  [org.jboss.as.connector.deployers.RaXmlDeployer] (default task-44) wmq.jmsra.rar: Use the linked exception to determine the cause of this error.
11:55:04,806 ERROR [com.eb.util.GetConnection] (default task-33)


So I was thinking to run script that shutsdown to queue managers before system going down and test it. But want to know where the bottle neck is. Is it because the QMGR is in quisece state for too long i think so because i even tried to kill the QMGR process and was switching within 30 seconds where as if i shutdown the system it is taking 2-3 minutes to switchover and is in quisece state.

Thanks in advance.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sat Mar 11, 2017 1:34 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 18948
Location: LI,NY

Change your connection factory properties to include FAIL_IF_QUIESCING.
This way when JBOSS tries the first queue manager (quiescing) in the CCDT it will move to the next one...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
kalam475
PostPosted: Sat Mar 11, 2017 3:39 am Post subject: Reply with quote

Acolyte

Joined: 16 Jan 2015
Posts: 56

If you don't mind me asking how can we set FAIL_IF_QUIESCING in JBOSS 7 standalone-full.xml. I tried to find it out but not able to find

This is my adapter settings:


Code:

<subsystem xmlns="urn:jboss:domain:resource-adapters:1.1">
            <resource-adapters>
                <resource-adapter id="wmq.jmsra.rar">
                    <archive>
                        wmq.jmsra.rar
                    </archive>
                    <connection-definitions>
                        <connection-definition class-name="com.ibm.mq.connector.outbound.ManagedConnectionFactoryImpl" jndi-name="MQDEVJNDI" pool-name="MQ.CONNECTIONFACTORY.NAME">
                            <config-property name="ccdtURL">
                                file://opt/MQ/AMQCLCHL.TAB
                            </config-property>
                            <config-property name="transportType">
                                CLIENT
                            </config-property>
                            <config-property name="queueManager">
                                *EAIGRP
                            </config-property>
                            <security>
                                <security-domain-and-application>JmsXARealm</security-domain-and-application>
                            </security>
                            <recovery>
                                <recover-credential>
                                    <user-name>mqadmin</user-name>
                                    <password>Mqadmin@123</password>
                                </recover-credential>
                            </recovery>
                        </connection-definition>
                    </connection-definitions>
                    <admin-objects>
                        <admin-object class-name="com.ibm.mq.connector.outbound.MQQueueProxy" jndi-name="jndi/RechareReqQ" pool-name="MQ.QUEUE.NAME">
                            <config-property name="baseQueueName">
                                RECHARGE.REQUEST
                            </config-property>
                            <config-property name="baseQueueManagerName">
                                EAI.QM
                            </config-property>
                        </admin-object>
                    </admin-objects>
                </resource-adapter>
            </resource-adapters>
        </subsystem>


Thanks ...
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexIBM MQ Installation/Configuration SupportQueue manager quiescing
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.