ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportApplication connecting Listeners are dropping itself

Post new topicReply to topic Goto page 1, 2  Next
Application connecting Listeners are dropping itself View previous topic :: View next topic
Author Message
vicks_mq
PostPosted: Wed Apr 17, 2019 1:28 pm Post subject: Application connecting Listeners are dropping itself Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

Hi, We have an application which connect to QMGR A on LINUX MQ 8 using SVRCONN, and when the application starts it shows 16 listeners on the Q and the 16 instances of SVRCONN channel.
Those 16 listeners always stays whether we are getting any messages or not.

Now we have migrated our MQ to MQ appliance V9 and our SVRCONN channel parameters remains the same except few listed differences, We didn't migrate our application, only point the application to this new QMGR on MQ Appliance.
When application starts it shows 16 listeners but then it drops down to 2. and it does't increase itself again even if large number of messages for consumption are coming in Queue.
"We want to keep the listeners count high and don't want these listeners to drop".

I did apple to apple comparing of SVRCONN channel in MQ8 Linux and MQ9 appliance and below are the details

Old MQ qmgr on LINUX version - 8.0.0.10
SVRCONN channel parameters
DISCINT(0), HBINT(300) KAINT(AUTO), MAXINST(999999999), MAXINSTC(999999999) SHARECNV(10)


New MQ appliance V9
DISCINT(0), HBINT(300) KAINT(AUTO), MAXINST(100), MAXINSTC(100) SHARECNV(1)
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Wed Apr 17, 2019 3:23 pm Post subject: Reply with quote

Padawan

Joined: 25 Mar 2003
Posts: 1995
Location: Melbourne, Australia

Quote:
When application starts it shows 16 listeners but then it drops down to 2

Does the number of running SVRCONN instances also start off as 16 and then drop down to 2?

By listeners, I assume you mean "input processes" on the local queue.
_________________
Glenn
Back to top
View user's profile Send private message
vicks_mq
PostPosted: Wed Apr 17, 2019 7:53 pm Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

gbaddeley wrote:
Quote:
When application starts it shows 16 listeners but then it drops down to 2

Does the number of running SVRCONN instances also start off as 16 and then drop down to 2?

By listeners, I assume you mean "input processes" on the local queue.


Yes to both the questions.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Apr 18, 2019 2:11 am Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1212
Location: Bay of Plenty, New Zealand

Any MQ return code given to these 14 instances of the application that have decided to stop?

Any error messages in the queue manager error log?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
vicks_mq
PostPosted: Thu Apr 18, 2019 6:20 am Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

hughson wrote:
Any MQ return code given to these 14 instances of the application that have decided to stop?

Any error messages in the queue manager error log?

Cheers,
Morag


Hi Morag, I am checking the logs. By the way, there is one more parameter difference in the above configuration that is SHARECVN, i read about it, I believe that should not cause the issue.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Apr 18, 2019 2:57 pm Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1212
Location: Bay of Plenty, New Zealand

vicks_mq wrote:
By the way, there is one more parameter difference in the above configuration that is SHARECVN, i read about it, I believe that should not cause the issue.

Indeed so. The change in SHARECNV would simply change whether the 16 connections each have their own running channel instance or whether they share. It would not change whether they were running at all.

The output from DISPLAY CONN would be the same, even if the output from DISPLAY CHSTATUS was different.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
vicks_mq
PostPosted: Thu Apr 18, 2019 4:32 pm Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

hughson wrote:
vicks_mq wrote:
By the way, there is one more parameter difference in the above configuration that is SHARECVN, i read about it, I believe that should not cause the issue.

Indeed so. The change in SHARECNV would simply change whether the 16 connections each have their own running channel instance or whether they share. It would not change whether they were running at all.

The output from DISPLAY CONN would be the same, even if the output from DISPLAY CHSTATUS was different.

Cheers,
Morag

Don't know the reason, but I took a chance and changed the value of SHARECNV back to 10, and it resolved the issue. Although nowhere in my understanding of SHARECNV points to that increasing its value will resolve the issue, but it worked.
"You use theSHARECNVparameter to specify the maximum number of conversations to be shared over a particular TCP/IP client channel instance."
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Apr 18, 2019 4:56 pm Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1212
Location: Bay of Plenty, New Zealand

That makes no sense. Especially since, from your initial description, the applications were NOT sharing channel instances:-

vicks_mq wrote:
when the application starts it shows 16 listeners on the Q and the 16 instances of SVRCONN channel


I would still be curious what return code was given to the application to cause 14 of them to stop, and any error messages in the error log.

It feels like there is a problem lurking there that is not understood that the change to SHARECNV simply masks. The only thing that I can think of is that there were already 98 instances of that SVRCONN running, and so only room for 2 more with SHARECNV(1) but 16 applications were all able to connect using the 2 spare slots when they shared connections.

With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
vicks_mq
PostPosted: Fri Apr 19, 2019 1:03 am Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

hughson wrote:
That makes no sense. Especially since, from your initial description, the applications were NOT sharing channel instances:-

vicks_mq wrote:
when the application starts it shows 16 listeners on the Q and the 16 instances of SVRCONN channel


I would still be curious what return code was given to the application to cause 14 of them to stop, and any error messages in the error log.

It feels like there is a problem lurking there that is not understood that the change to SHARECNV simply masks. The only thing that I can think of is that there were already 98 instances of that SVRCONN running, and so only room for 2 more with SHARECNV(1) but 16 applications were all able to connect using the 2 spare slots when they shared connections.

With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag

No, I see less than 16.
Also below are application error logs and QMGR error logs, QMGR error logs doesn't tell much except that there is an error.


QMGR LOGS
----- amqrccca.c : 1131 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(11.122.133.144)
CommentInsert3(select() [TIMEOUT] 65 seconds)

AMQ9271E: Channel 'BOC.TO.QMGR1' timed out.

EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'BOC.TO.QMGR1'. The address of the remote end of the connection was
'11.122.133.144'.
ACTION:
The return code from the select() [TIMEOUT] 65 seconds call was 0 (X'0').
Record these values and tell the systems administrator.
----- amqccita.c : 4779 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(828781)
CommentInsert3(11.122.133.144)

AMQ9999E: Channel 'BOC.TO.QMGR1' to host '11.122.133.144' ended abnormally.

EXPLANATION:
The channel program running under process ID 828781 for channel
'BOC.TO.QMGR1' ended abnormally. The host name is '11.122.133.144'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.

APPLICATION ERROR LOGS
14:16:16,384 WARN [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] (JMSCCThreadPoolWorker-20) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@66c9d12b[state=NORMAL managed connection=com.ibm.mq.connector.outbound.ManagedConnectionImpl@4bc392c5 connection handles=1 lastUse=1555523115705 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@14d8ea5f pool internal context=SemaphoreArrayListManagedConnectionPool@2d030daa[pool=java:/jms/BOCQCF]]: com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
An error has occurred with the WebSphere MQ JMS connection.
Use the linked exception to determine the cause of this error.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:585) [com.ibm.msg.client.wmq.common.jar:]
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:221) [com.ibm.msg.client.wmq.common.jar:]
at com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:785) [com.ibm.msg.client.wmq.jar:]
at com.ibm.mq.jmqi.remote.api.RemoteHconn.callEventHandler(RemoteHconn.java:2621) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.mq.jmqi.remote.api.RemoteHconn.driveEventsEH(RemoteHconn.java:601) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.processHconn(RemoteDispatchThread.java:668) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:244) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:214) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:105) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:229) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:303) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1241) [com.ibm.msg.client.commonservices.j2se.jar:]
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:209) [com.ibm.msg.client.wmq.common.jar:]

]]) Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
14:16:16,435 INFO [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] (JMSCCThreadPoolWorker-22) IJ000302: Unregistered handle that was not registered: com.ibm.mq.connector.outbound.ConnectionWrapper@12fec5ae for managed connection: com.ibm.mq.connector.outbound.ManagedConnectionImpl@2224a1dd
14:16:16,435 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (ApplicationMdpListener-4) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@25c5cffb' - trying to recover. Cause: MQJCA1020: The session is closed.
14:16:16,437 INFO [stdout] (ApplicationMdpListener-4) MQJCA4016:Unregistered connection handle being closed: 'com.ibm.mq.connector.outbound.ConnectionWrapper@12fec5ae'.
14:16:16,505 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (ApplicationMdpListener-4) Successfully refreshed JMS Connection
14:16:17,423 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener- Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@26fb8fdb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,432 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener- Successfully refreshed JMS Connection
14:16:17,448 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-11) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@26fb8fdb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,469 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-11) Successfully refreshed JMS Connection
14:16:17,480 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (ApplicationMdpListener-10) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@25c5cffb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,481 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-17) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@26fb8fdb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,482 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-17) Successfully refreshed JMS Connection
Back to top
View user's profile Send private message
hughson
PostPosted: Fri Apr 19, 2019 2:59 am Post subject: Reply with quote

Grand Master

Joined: 09 May 2013
Posts: 1212
Location: Bay of Plenty, New Zealand

vicks_mq wrote:
hughson wrote:
With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag

No, I see less than 16.

I assume you see 16 queue listeners though, and if you run DISPLAY CHSTATUS and add up the number of CURSHCNV on each, you get a total of 16?

vicks_mq wrote:

APPLICATION ERROR LOGS
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').

QMGR LOGS
----- amqrccca.c : 1131 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(11.122.133.144)
CommentInsert3(select() [TIMEOUT] 65 seconds)

AMQ9271E: Channel 'BOC.TO.QMGR1' timed out.

EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'BOC.TO.QMGR1'. The address of the remote end of the connection was
'11.122.133.144'.
ACTION:
The return code from the select() [TIMEOUT] 65 seconds call was 0 (X'0').
Record these values and tell the systems administrator.


As 65 second timeout suggests that the channel is still in early channel negotiation, since it is not making use of the actual heartbeat interval value of 300 for its timeout.

vicks_mq wrote:
Now we have migrated our MQ to MQ appliance V9 and our SVRCONN channel parameters remains the same except few listed differences, We didn't migrate our application, only point the application to this new QMGR on MQ Appliance.


Something else that will be different is the firewall. Have you checked whether anything is being blocked there? The timeout error strongly suggests that. However, if changing SHARECNV(1 -> 10) fixed your issue, I guess it can't be that. I am not sure if it did fix it or not from your comments above.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
vicks_mq
PostPosted: Fri Apr 19, 2019 4:47 am Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

hughson wrote:
vicks_mq wrote:
hughson wrote:
With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag

No, I see less than 16.

I assume you see 16 queue listeners though, and if you run DISPLAY CHSTATUS and add up the number of CURSHCNV on each, you get a total of 16?

vicks_mq wrote:

APPLICATION ERROR LOGS
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').

QMGR LOGS
----- amqrccca.c : 1131 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(11.122.133.144)
CommentInsert3(select() [TIMEOUT] 65 seconds)

AMQ9271E: Channel 'BOC.TO.QMGR1' timed out.

EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'BOC.TO.QMGR1'. The address of the remote end of the connection was
'11.122.133.144'.
ACTION:
The return code from the select() [TIMEOUT] 65 seconds call was 0 (X'0').
Record these values and tell the systems administrator.


As 65 second timeout suggests that the channel is still in early channel negotiation, since it is not making use of the actual heartbeat interval value of 300 for its timeout.

vicks_mq wrote:
Now we have migrated our MQ to MQ appliance V9 and our SVRCONN channel parameters remains the same except few listed differences, We didn't migrate our application, only point the application to this new QMGR on MQ Appliance.


Something else that will be different is the firewall. Have you checked whether anything is being blocked there? The timeout error strongly suggests that.
Quote:
However, if changing SHARECNV(1 -> 10) fixed your issue, I guess it can't be that. I am not sure if it did fix it or not from your comments above.
Hi Morag, the issue is fixed after changing SHARECNV to 10, all these logs are before I changed the value of SHARECNV, now no more errors are coming.

Cheers,
Morag
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Apr 19, 2019 6:20 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20089
Location: LI,NY

If upping the sharecnv fixed the problem you might not have had enough channels defined (max channels stanza) in the qm.ini ...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
vicks_mq
PostPosted: Sat Apr 20, 2019 6:20 am Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

fjb_saper wrote:
If upping the sharecnv fixed the problem you might not have had enough channels defined (max channels stanza) in the qm.ini ...

Yup that was the 1st thought crossed in my mind, but we have around 500 channels and the count never reached that high for the channel in question.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Mon Apr 22, 2019 4:54 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20089
Location: LI,NY

Well in that case were you channel restrictions set high enough (maxinst, maxinstc ) ?
_________________
MQ & Broker admin


Last edited by fjb_saper on Mon Apr 22, 2019 11:35 am; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail
vicks_mq
PostPosted: Mon Apr 22, 2019 7:25 am Post subject: Reply with quote

Disciple

Joined: 03 Oct 2017
Posts: 162

fjb_saper wrote:
Well in that case were you channel restrictions set high enough (maxinst, maxinstc ) ?


Hi fjb_saper, I think you are right, we need to increase the value of this - because they are set to 100, which is probably not enough.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexGeneral IBM MQ SupportApplication connecting Listeners are dropping itself
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.