| Author |
Message
|
| dakoroni |
 Posted: Tue Sep 08, 2020 3:25 am Post subject: PROD MQ Client opens/closes connection to z/OS QM every 3sec |
 |
|
Acolyte
Joined: 10 Jan 2020
Posts: 50
|
Hello IBM MQ support forum users,
I have a Windows MQ client v9.1.4 with IP: 192.168.164.163 (Production DMZ) which connects to Queue Manager named MQSP on z/OS (Prod - DMZ) v8 with IP: 192.168.202.85 - port 1414 via
MQServer environment variable (MQSERVER=MQIDDOCS.TO.MQSP/TCP/192.168.202.85(1414).
Looking into z/OS log it looks like every 3 seconds MQ client open the connection channel successfully and then closes it within the same second..
23.00.01 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 149
149 connection 192.168.164.163
23.00.01 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.04 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 197
197 connection 192.168.164.163
23.00.04 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.07 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 200
200 connection 192.168.164.163
23.00.07 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.10 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 222
222 connection 192.168.164.163
23.00.10 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.13 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 954
954 connection 192.168.164.163
23.00.13 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.16 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 007
007 connection 192.168.164.163
23.00.16 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
The above results to massive messaging (huge log volumes) on z/Os system & MQ logs.
From the Windows MQ client side the error log is full of the following error msg:
Process(11644.25) User(SYSTEM) Program(etIdDocsInfo.exe)
Host(V0000A0118) Installation(Installation1)
VRMF(9.1.4.0)
Time(2020-08-23T00:35:25.357Z)
ArithInsert1(10061) ArithInsert2(10061)
CommentInsert1(192.168.202.85 (1414))
CommentInsert2(TCP/IP)
CommentInsert3(MQIDDOCS.TO.MQSP)
AMQ9202E: Remote host not available, retry later.
EXPLANATION:
The attempt to allocate a conversation using TCP/IP to host '192.168.202.85 (1414)' for channel MQIDDOCS.TO.MQSP was not successful. However the error may
be a transitory one and it may be possible to successfully allocate a TCP/IP conversation later.
In some cases the remote host cannot be determined and so is shown as '????'.
ACTION:
Try the connection again later. If the failure persists, record the error values and contact your systems administrator. The return code from TCP/IP is 10061 (X'274D'). The reason for the failure may be that this host cannot reach the destination host.
It may also be possible that the listening program at host '192.168.202.85 (1414)' was not running. If this is the case, perform the relevant operations to start the TCP/IP listening program, and try again.
--
FYI, the firewall rules between Windows MQ Client & host (z/OS) QM: MQSP are activated and the port (1414) access is enabled.
Can you pls advise if the above is caused by the application or it is a matter of MQ client setting (i.e. tcp stanza in mqclien.ini -> Connect_Timeout = number ??? )
Any advise will be much appreciated.
Cheers Nick. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Tue Sep 08, 2020 4:23 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Set the MAXINSTC on the zOS server connection channel to a low value, say 10 and verify the behavior of the Windows box. Could it be that the Windows box is constantly creating a new connection without closing the old one?
Or better still if the old one is not closed, why isn't Windows reusing it? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Sep 08, 2020 7:06 am Post subject: Re: PROD MQ Client opens/closes connection to z/OS QM every |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| dakoroni wrote: |
...I have a Windows MQ client v9.1.4 with IP: 192.168.164.163 (Production DMZ) which connects to Queue Manager named MQSP on z/OS (Prod - DMZ) v8 with IP: 192.168.202.85 - port 1414 via
MQServer environment variable... |
You have an unsecured channel in your DMZ? 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| dakoroni |
| Posted: Tue Sep 08, 2020 7:20 am Post subject: Re: PROD MQ Client opens/closes connection to z/OS QM every |
|
|
Acolyte
Joined: 10 Jan 2020
Posts: 50
|
| exerk wrote: |
| You have an unsecured channel in your DMZ? |
Yes, since TLS or SSL is NOT enabled.
In general MQ security in terms of channel or msg encryption is NOT enabled across all MQ customer's infra, at least for the moment. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Sep 08, 2020 3:50 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
10061 = WSAECONNREFUSED
The time from the CHIN log does not match up to the time in the Windows system AMQERR01.LOG.
Please check that the CHIN is not reaching its max channels limit.
Try running MQ trace on the Windows system. To see exactly when / why it is failing, and match up to messages in the CHIN log.
_________________
Glenn |
|
| Back to top |
|
|
| dakoroni |
| Posted: Wed Sep 09, 2020 6:26 am Post subject: PROD MQ Client opens/closes connection to z/OS QM every 3sec |
|
|
Acolyte
Joined: 10 Jan 2020
Posts: 50
|
| fjb_saper wrote: |
Set the MAXINSTC on the zOS server connection channel to a low value, say 10 and verify the behavior of the Windows box. Could it be that the Windows box is constantly creating a new connection without closing the old one?
Or better still if the old one is not closed, why isn't Windows reusing it? |
After discussing with z/OS MQ admin, he confirmed that MAXINSTC had been already set to a lower value (i.e. 10) but the problem remains. He claims that the problem is not on z/OS MQ side, since it treats as expected similar MQ client windows apps, but it might be on the application code related to connection handling.
From the MQClient side it is suggested KEEP ALIVE=YES in the mqclient.ini which it was already set.
Btw, What is your comment about setting DefRecon and/or ReconDelay parameters in mqclient.ini [channels] stanza on Windows box as listed on
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.con.doc/q016860_.htm#q016860___chstrecondelay?
Do you think that the above can make a difference?
What else can be advised from the application development perspective? |
|
| Back to top |
|
|
| dakoroni |
| Posted: Wed Sep 09, 2020 6:33 am Post subject: |
|
|
Acolyte
Joined: 10 Jan 2020
Posts: 50
|
| gbaddeley wrote: |
10061 = WSAECONNREFUSED
The time from the CHIN log does not match up to the time in the Windows system AMQERR01.LOG.
Please check that the CHIN is not reaching its max channels limit.
Try running MQ trace on the Windows system. To see exactly when / why it is failing, and match up to messages in the CHIN log. |
After discussing with z/OS MQ admin, there is no issue related to max channels limit.
He thinks that the root cause of the problem should be related to application connection handling coding.
In terms of Windows mqclient.ini file the recommended setting KeepAlive=YES has been already set.
I am wondering if DefRecon and/or ReconDelay settings in mqclient.ini [Channels] stanza can make a difference.. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Sep 09, 2020 9:17 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| dakoroni wrote: |
| After discussing with z/OS MQ admin, there is no issue related to max channels limit. |
'No issue' is not a very technical response from your z/OS MQ admin.
In addition to MAXCHL, there are other related qmgr settings, including: ACTCHL and TCPCHL. I'd ask the admin folks the actual values (1-9999) for each.
As a reference https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.ref.adm.doc/q085320_.htm
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| dakoroni |
| Posted: Thu Sep 10, 2020 1:40 am Post subject: MQ Client opens/closes connection to host QM every 3 sec |
|
|
Acolyte
Joined: 10 Jan 2020
Posts: 50
|
| bruce2359 wrote: |
| dakoroni wrote: |
| After discussing with z/OS MQ admin, there is no issue related to max channels limit. |
'No issue' is not a very technical response from your z/OS MQ admin.
In addition to MAXCHL, there are other related qmgr settings, including: ACTCHL and TCPCHL. I'd ask the admin folks the actual values (1-9999) for each.
|
I have asked z/OS MQ admin for these settings, expecting his feedback.
In the meantime can you pls comment if giving a value to DefRecon and/or ReconDelay parameters in mqclient.ini [channels] stanza, as listed on
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.con.doc/q016860_.htm#q016860___chstrecondelay?
can make a difference? |
|
| Back to top |
|
|
| dakoroni |
| Posted: Thu Sep 10, 2020 1:48 am Post subject: MQ Client opens/closes connection to host QM every 3 secs |
|
|
Acolyte
Joined: 10 Jan 2020
Posts: 50
|
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Sep 10, 2020 3:09 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Quote: |
MQ error log contains:
Try the connection again later. If the failure persists, record the error values and contact your systems administrator. The return code from TCP/IP is 10061 (X'274D'). The reason for the failure may be that this host cannot reach the destination host.
It may also be possible that the listening program at host '192.168.202.85 (1414)' was not running. |
You should not be seeing intermittent 10061 errors, assuming that the z/OS MQ listener is always running and accepting connections. Please check with your network team for switch or router errors in the route.
MQ trace will show at what point it is occurring, probably on the initial WinSock TCP connect() call.
| Quote: |
| Can you pls advise if the above is caused by the application or it is a matter of MQ client setting (i.e. tcp stanza in mqclien.ini -> Connect_Timeout = number ??? ) |
Probably not.
_________________
Glenn |
|
| Back to top |
|
|
|
|
