| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Application connecting Listeners are dropping itself |
« View previous topic :: View next topic » |
| Author |
Message
|
| vicks_mq |
 Posted: Wed Apr 17, 2019 1:28 pm Post subject: Application connecting Listeners are dropping itself |
 |
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
Hi, We have an application which connect to QMGR A on LINUX MQ 8 using SVRCONN, and when the application starts it shows 16 listeners on the Q and the 16 instances of SVRCONN channel.
Those 16 listeners always stays whether we are getting any messages or not.
Now we have migrated our MQ to MQ appliance V9 and our SVRCONN channel parameters remains the same except few listed differences, We didn't migrate our application, only point the application to this new QMGR on MQ Appliance.
When application starts it shows 16 listeners but then it drops down to 2. and it does't increase itself again even if large number of messages for consumption are coming in Queue.
"We want to keep the listeners count high and don't want these listeners to drop".
I did apple to apple comparing of SVRCONN channel in MQ8 Linux and MQ9 appliance and below are the details
Old MQ qmgr on LINUX version - 8.0.0.10
SVRCONN channel parameters
DISCINT(0), HBINT(300) KAINT(AUTO), MAXINST(999999999), MAXINSTC(999999999) SHARECNV(10)
New MQ appliance V9
DISCINT(0), HBINT(300) KAINT(AUTO), MAXINST(100), MAXINSTC(100) SHARECNV(1) |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Wed Apr 17, 2019 3:23 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Quote: |
| When application starts it shows 16 listeners but then it drops down to 2 |
Does the number of running SVRCONN instances also start off as 16 and then drop down to 2?
By listeners, I assume you mean "input processes" on the local queue.
_________________
Glenn |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Wed Apr 17, 2019 7:53 pm Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| gbaddeley wrote: |
| Quote: |
| When application starts it shows 16 listeners but then it drops down to 2 |
Does the number of running SVRCONN instances also start off as 16 and then drop down to 2?
By listeners, I assume you mean "input processes" on the local queue. |
Yes to both the questions. |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Apr 18, 2019 2:11 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
Any MQ return code given to these 14 instances of the application that have decided to stop?
Any error messages in the queue manager error log?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Thu Apr 18, 2019 6:20 am Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| hughson wrote: |
Any MQ return code given to these 14 instances of the application that have decided to stop?
Any error messages in the queue manager error log?
Cheers,
Morag |
Hi Morag, I am checking the logs. By the way, there is one more parameter difference in the above configuration that is SHARECVN, i read about it, I believe that should not cause the issue. |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Apr 18, 2019 2:57 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| vicks_mq wrote: |
| By the way, there is one more parameter difference in the above configuration that is SHARECVN, i read about it, I believe that should not cause the issue. |
Indeed so. The change in SHARECNV would simply change whether the 16 connections each have their own running channel instance or whether they share. It would not change whether they were running at all.
The output from DISPLAY CONN would be the same, even if the output from DISPLAY CHSTATUS was different.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Thu Apr 18, 2019 4:32 pm Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| hughson wrote: |
| vicks_mq wrote: |
| By the way, there is one more parameter difference in the above configuration that is SHARECVN, i read about it, I believe that should not cause the issue. |
Indeed so. The change in SHARECNV would simply change whether the 16 connections each have their own running channel instance or whether they share. It would not change whether they were running at all.
The output from DISPLAY CONN would be the same, even if the output from DISPLAY CHSTATUS was different.
Cheers,
Morag |
Don't know the reason, but I took a chance and changed the value of SHARECNV back to 10, and it resolved the issue. Although nowhere in my understanding of SHARECNV points to that increasing its value will resolve the issue, but it worked.
"You use the SHARECNV parameter to specify the maximum number of conversations to be shared over a particular TCP/IP client channel instance." |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Apr 18, 2019 4:56 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
That makes no sense. Especially since, from your initial description, the applications were NOT sharing channel instances:-
| vicks_mq wrote: |
| when the application starts it shows 16 listeners on the Q and the 16 instances of SVRCONN channel |
I would still be curious what return code was given to the application to cause 14 of them to stop, and any error messages in the error log.
It feels like there is a problem lurking there that is not understood that the change to SHARECNV simply masks. The only thing that I can think of is that there were already 98 instances of that SVRCONN running, and so only room for 2 more with SHARECNV(1) but 16 applications were all able to connect using the 2 spare slots when they shared connections.
With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Fri Apr 19, 2019 1:03 am Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| hughson wrote: |
That makes no sense. Especially since, from your initial description, the applications were NOT sharing channel instances:-
| vicks_mq wrote: |
| when the application starts it shows 16 listeners on the Q and the 16 instances of SVRCONN channel |
I would still be curious what return code was given to the application to cause 14 of them to stop, and any error messages in the error log.
It feels like there is a problem lurking there that is not understood that the change to SHARECNV simply masks. The only thing that I can think of is that there were already 98 instances of that SVRCONN running, and so only room for 2 more with SHARECNV(1) but 16 applications were all able to connect using the 2 spare slots when they shared connections.
With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag |
No, I see less than 16.
Also below are application error logs and QMGR error logs, QMGR error logs doesn't tell much except that there is an error.
QMGR LOGS
----- amqrccca.c : 1131 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(11.122.133.144)
CommentInsert3(select() [TIMEOUT] 65 seconds)
AMQ9271E: Channel 'BOC.TO.QMGR1' timed out.
EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'BOC.TO.QMGR1'. The address of the remote end of the connection was
'11.122.133.144'.
ACTION:
The return code from the select() [TIMEOUT] 65 seconds call was 0 (X'0').
Record these values and tell the systems administrator.
----- amqccita.c : 4779 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(828781)
CommentInsert3(11.122.133.144)
AMQ9999E: Channel 'BOC.TO.QMGR1' to host '11.122.133.144' ended abnormally.
EXPLANATION:
The channel program running under process ID 828781 for channel
'BOC.TO.QMGR1' ended abnormally. The host name is '11.122.133.144'; in some
cases the host name cannot be determined and so is shown as '????'.
ACTION:
Look at previous error messages for the channel program in the error logs to
determine the cause of the failure. Note that this message can be excluded
completely or suppressed by tuning the "ExcludeMessage" or "SuppressMessage"
attributes under the "QMErrorLog" stanza in qm.ini. Further information can be
found in the System Administration Guide.
APPLICATION ERROR LOGS
14:16:16,384 WARN [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] (JMSCCThreadPoolWorker-20) IJ000305: Connection error occured: org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener@66c9d12b[state=NORMAL managed connection=com.ibm.mq.connector.outbound.ManagedConnectionImpl@4bc392c5 connection handles=1 lastUse=1555523115705 trackByTx=false pool=org.jboss.jca.core.connectionmanager.pool.strategy.OnePool@14d8ea5f pool internal context=SemaphoreArrayListManagedConnectionPool@2d030daa[pool=java:/jms/BOCQCF]]: com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has occurred.
An error has occurred with the WebSphere MQ JMS connection.
Use the linked exception to determine the cause of this error.
at com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:585) [com.ibm.msg.client.wmq.common.jar:]
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:221) [com.ibm.msg.client.wmq.common.jar:]
at com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:785) [com.ibm.msg.client.wmq.jar:]
at com.ibm.mq.jmqi.remote.api.RemoteHconn.callEventHandler(RemoteHconn.java:2621) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.mq.jmqi.remote.api.RemoteHconn.driveEventsEH(RemoteHconn.java:601) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.processHconn(RemoteDispatchThread.java:668) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.mq.jmqi.remote.impl.RemoteDispatchThread.run(RemoteDispatchThread.java:244) [com.ibm.mq.jmqi.remote.jar:7.5.0.7 - p750-007-160812]
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:214) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.java:105) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:229) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.workqueue.WorkQueueManager.runWorkQueueItem(WorkQueueManager.java:303) [com.ibm.msg.client.commonservices.jar:]
at com.ibm.msg.client.commonservices.j2se.workqueue.WorkQueueManagerImplementation$ThreadPoolWorker.run(WorkQueueManagerImplementation.java:1241) [com.ibm.msg.client.commonservices.j2se.jar:]
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
at com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:209) [com.ibm.msg.client.wmq.common.jar:]
]]) Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
14:16:16,435 INFO [org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener] (JMSCCThreadPoolWorker-22) IJ000302: Unregistered handle that was not registered: com.ibm.mq.connector.outbound.ConnectionWrapper@12fec5ae for managed connection: com.ibm.mq.connector.outbound.ManagedConnectionImpl@2224a1dd
14:16:16,435 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (ApplicationMdpListener-4) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@25c5cffb' - trying to recover. Cause: MQJCA1020: The session is closed.
14:16:16,437 INFO [stdout] (ApplicationMdpListener-4) MQJCA4016:Unregistered connection handle being closed: 'com.ibm.mq.connector.outbound.ConnectionWrapper@12fec5ae'.
14:16:16,505 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (ApplicationMdpListener-4) Successfully refreshed JMS Connection
14:16:17,423 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener- Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@26fb8fdb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,432 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener- Successfully refreshed JMS Connection
14:16:17,448 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-11) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@26fb8fdb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,469 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-11) Successfully refreshed JMS Connection
14:16:17,480 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (ApplicationMdpListener-10) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@25c5cffb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,481 WARN [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-17) Setup of JMS message listener invoker failed for destination 'com.ibm.mq.connector.outbound.MQQueueProxy@26fb8fdb' - trying to recover. Cause: MQJCA1018: Only one session per connection is allowed.
14:16:17,482 INFO [org.springframework.jms.listener.DefaultMessageListenerContainer] (2ApplicationMdpListener-17) Successfully refreshed JMS Connection |
|
| Back to top |
|
|
| hughson |
| Posted: Fri Apr 19, 2019 2:59 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| vicks_mq wrote: |
| hughson wrote: |
With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag |
No, I see less than 16. |
I assume you see 16 queue listeners though, and if you run DISPLAY CHSTATUS and add up the number of CURSHCNV on each, you get a total of 16?
| vicks_mq wrote: |
APPLICATION ERROR LOGS
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
QMGR LOGS
----- amqrccca.c : 1131 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(11.122.133.144)
CommentInsert3(select() [TIMEOUT] 65 seconds)
AMQ9271E: Channel 'BOC.TO.QMGR1' timed out.
EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'BOC.TO.QMGR1'. The address of the remote end of the connection was
'11.122.133.144'.
ACTION:
The return code from the select() [TIMEOUT] 65 seconds call was 0 (X'0').
Record these values and tell the systems administrator. |
As 65 second timeout suggests that the channel is still in early channel negotiation, since it is not making use of the actual heartbeat interval value of 300 for its timeout.
| vicks_mq wrote: |
| Now we have migrated our MQ to MQ appliance V9 and our SVRCONN channel parameters remains the same except few listed differences, We didn't migrate our application, only point the application to this new QMGR on MQ Appliance. |
Something else that will be different is the firewall. Have you checked whether anything is being blocked there? The timeout error strongly suggests that. However, if changing SHARECNV(1 -> 10) fixed your issue, I guess it can't be that. I am not sure if it did fix it or not from your comments above.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Fri Apr 19, 2019 4:47 am Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| hughson wrote: |
| vicks_mq wrote: |
| hughson wrote: |
With your "solution" in place, do you still see 16 instances of the SVRCONN, or are they now sharing?
Cheers,
Morag |
No, I see less than 16. |
I assume you see 16 queue listeners though, and if you run DISPLAY CHSTATUS and add up the number of CURSHCNV on each, you get a total of 16?
| vicks_mq wrote: |
APPLICATION ERROR LOGS
Caused by: com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2009' ('MQRC_CONNECTION_BROKEN').
QMGR LOGS
----- amqrccca.c : 1131 -------------------------------------------------------
04/18/19 08:48:00 - Process(828781.127589) User(mqsystem) Program(amqrmppa)
Host(ABC001234) Installation(MQAppliance)
VRMF(9.1.0.0) QMgr(QMGR1)
Time(2019-04-18T12:48:00.966Z)
RemoteHost(11.122.133.144)
CommentInsert1(BOC.TO.QMGR1)
CommentInsert2(11.122.133.144)
CommentInsert3(select() [TIMEOUT] 65 seconds)
AMQ9271E: Channel 'BOC.TO.QMGR1' timed out.
EXPLANATION:
A timeout occurred while waiting to receive from the other end of channel
'BOC.TO.QMGR1'. The address of the remote end of the connection was
'11.122.133.144'.
ACTION:
The return code from the select() [TIMEOUT] 65 seconds call was 0 (X'0').
Record these values and tell the systems administrator. |
As 65 second timeout suggests that the channel is still in early channel negotiation, since it is not making use of the actual heartbeat interval value of 300 for its timeout.
| vicks_mq wrote: |
| Now we have migrated our MQ to MQ appliance V9 and our SVRCONN channel parameters remains the same except few listed differences, We didn't migrate our application, only point the application to this new QMGR on MQ Appliance. |
Something else that will be different is the firewall. Have you checked whether anything is being blocked there? The timeout error strongly suggests that.
| Quote: |
| However, if changing SHARECNV(1 -> 10) fixed your issue, I guess it can't be that. I am not sure if it did fix it or not from your comments above. |
Hi Morag, the issue is fixed after changing SHARECNV to 10, all these logs are before I changed the value of SHARECNV, now no more errors are coming.
Cheers,
Morag |
|
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Apr 19, 2019 6:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If upping the sharecnv fixed the problem you might not have had enough channels defined (max channels stanza) in the qm.ini ... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Sat Apr 20, 2019 6:20 am Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| fjb_saper wrote: |
| If upping the sharecnv fixed the problem you might not have had enough channels defined (max channels stanza) in the qm.ini ... |
Yup that was the 1st thought crossed in my mind, but we have around 500 channels and the count never reached that high for the channel in question. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Apr 22, 2019 4:54 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Well in that case were you channel restrictions set high enough (maxinst, maxinstc ) ?
_________________
MQ & Broker admin
Last edited by fjb_saper on Mon Apr 22, 2019 11:35 am; edited 1 time in total |
|
| Back to top |
|
|
| vicks_mq |
| Posted: Mon Apr 22, 2019 7:25 am Post subject: |
|
|
Disciple
Joined: 03 Oct 2017
Posts: 162
|
| fjb_saper wrote: |
| Well in that case were you channel restrictions set high enough (maxinst, maxinstc ) ? |
Hi fjb_saper, I think you are right, we need to increase the value of this - because they are set to 100, which is probably not enough. |
|
| Back to top |
|
|
|
|
|
  |
Goto page 1, 2 Next |
Page 1 of 2 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
