ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportPROD MQ Client opens/closes connection to z/OS QM every 3sec

Post new topicReply to topic
PROD MQ Client opens/closes connection to z/OS QM every 3sec View previous topic :: View next topic
Author Message
dakoroni
PostPosted: Tue Sep 08, 2020 3:25 am Post subject: PROD MQ Client opens/closes connection to z/OS QM every 3sec Reply with quote

Apprentice

Joined: 10 Jan 2020
Posts: 39

Hello IBM MQ support forum users,

I have a Windows MQ client v9.1.4 with IP: 192.168.164.163 (Production DMZ) which connects to Queue Manager named MQSP on z/OS (Prod - DMZ) v8 with IP: 192.168.202.85 - port 1414 via
MQServer environment variable (MQSERVER=MQIDDOCS.TO.MQSP/TCP/192.168.202.85(1414).

Looking into z/OS log it looks like every 3 seconds MQ client open the connection channel successfully and then closes it within the same second..
23.00.01 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 149
149 connection 192.168.164.163
23.00.01 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.04 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 197
197 connection 192.168.164.163
23.00.04 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.07 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 200
200 connection 192.168.164.163
23.00.07 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.10 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 222
222 connection 192.168.164.163
23.00.10 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.13 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 954
954 connection 192.168.164.163
23.00.13 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active
23.00.16 STC16733 +CSQX511I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP started 007
007 connection 192.168.164.163
23.00.16 STC16733 +CSQX512I MQSP CSQXRESP Channel MQIDDOCS.TO.MQSP no longer active

The above results to massive messaging (huge log volumes) on z/Os system & MQ logs.


From the Windows MQ client side the error log is full of the following error msg:

Process(11644.25) User(SYSTEM) Program(etIdDocsInfo.exe)

Host(V0000A0118) Installation(Installation1)

VRMF(9.1.4.0)

Time(2020-08-23T00:35:25.357Z)

ArithInsert1(10061) ArithInsert2(10061)

CommentInsert1(192.168.202.85 (1414))

CommentInsert2(TCP/IP)

CommentInsert3(MQIDDOCS.TO.MQSP)



AMQ9202E: Remote host not available, retry later.



EXPLANATION:

The attempt to allocate a conversation using TCP/IP to host '192.168.202.85 (1414)' for channel MQIDDOCS.TO.MQSP was not successful. However the error may

be a transitory one and it may be possible to successfully allocate a TCP/IP conversation later.



In some cases the remote host cannot be determined and so is shown as '????'.

ACTION:

Try the connection again later. If the failure persists, record the error values and contact your systems administrator. The return code from TCP/IP is 10061 (X'274D'). The reason for the failure may be that this host cannot reach the destination host.

It may also be possible that the listening program at host '192.168.202.85 (1414)' was not running. If this is the case, perform the relevant operations to start the TCP/IP listening program, and try again.

--

FYI, the firewall rules between Windows MQ Client & host (z/OS) QM: MQSP are activated and the port (1414) access is enabled.

Can you pls advise if the above is caused by the application or it is a matter of MQ client setting (i.e. tcp stanza in mqclien.ini -> Connect_Timeout = number ??? )

Any advise will be much appreciated.
Cheers Nick.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Sep 08, 2020 4:23 am Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20382
Location: LI,NY

Set the MAXINSTC on the zOS server connection channel to a low value, say 10 and verify the behavior of the Windows box. Could it be that the Windows box is constantly creating a new connection without closing the old one?
Or better still if the old one is not closed, why isn't Windows reusing it?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
exerk
PostPosted: Tue Sep 08, 2020 7:06 am Post subject: Re: PROD MQ Client opens/closes connection to z/OS QM every Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6191

dakoroni wrote:
...I have a Windows MQ client v9.1.4 with IP: 192.168.164.163 (Production DMZ) which connects to Queue Manager named MQSP on z/OS (Prod - DMZ) v8 with IP: 192.168.202.85 - port 1414 via
MQServer environment variable...

You have an unsecured channel in your DMZ?
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
dakoroni
PostPosted: Tue Sep 08, 2020 7:20 am Post subject: Re: PROD MQ Client opens/closes connection to z/OS QM every Reply with quote

Apprentice

Joined: 10 Jan 2020
Posts: 39

exerk wrote:
You have an unsecured channel in your DMZ?

Yes, since TLS or SSL is NOT enabled.

In general MQ security in terms of channel or msg encryption is NOT enabled across all MQ customer's infra, at least for the moment.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Sep 08, 2020 3:50 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2149
Location: Melbourne, Australia

10061 = WSAECONNREFUSED

The time from the CHIN log does not match up to the time in the Windows system AMQERR01.LOG.

Please check that the CHIN is not reaching its max channels limit.

Try running MQ trace on the Windows system. To see exactly when / why it is failing, and match up to messages in the CHIN log.
_________________
Glenn
Back to top
View user's profile Send private message
dakoroni
PostPosted: Wed Sep 09, 2020 6:26 am Post subject: PROD MQ Client opens/closes connection to z/OS QM every 3sec Reply with quote

Apprentice

Joined: 10 Jan 2020
Posts: 39

fjb_saper wrote:
Set the MAXINSTC on the zOS server connection channel to a low value, say 10 and verify the behavior of the Windows box. Could it be that the Windows box is constantly creating a new connection without closing the old one?
Or better still if the old one is not closed, why isn't Windows reusing it?


After discussing with z/OS MQ admin, he confirmed that MAXINSTC had been already set to a lower value (i.e. 10) but the problem remains. He claims that the problem is not on z/OS MQ side, since it treats as expected similar MQ client windows apps, but it might be on the application code related to connection handling.

From the MQClient side it is suggested KEEP ALIVE=YES in the mqclient.ini which it was already set.

Btw, What is your comment about setting DefRecon and/or ReconDelay parameters in mqclient.ini [channels] stanza on Windows box as listed on
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.con.doc/q016860_.htm#q016860___chstrecondelay?

Do you think that the above can make a difference?

What else can be advised from the application development perspective?
Back to top
View user's profile Send private message
dakoroni
PostPosted: Wed Sep 09, 2020 6:33 am Post subject: Reply with quote

Apprentice

Joined: 10 Jan 2020
Posts: 39

gbaddeley wrote:
10061 = WSAECONNREFUSED

The time from the CHIN log does not match up to the time in the Windows system AMQERR01.LOG.

Please check that the CHIN is not reaching its max channels limit.

Try running MQ trace on the Windows system. To see exactly when / why it is failing, and match up to messages in the CHIN log.


After discussing with z/OS MQ admin, there is no issue related to max channels limit.
He thinks that the root cause of the problem should be related to application connection handling coding.

In terms of Windows mqclient.ini file the recommended setting KeepAlive=YES has been already set.

I am wondering if DefRecon and/or ReconDelay settings in mqclient.ini [Channels] stanza can make a difference..
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Sep 09, 2020 9:17 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8769
Location: US: west coast, almost. Otherwise, enroute.

dakoroni wrote:
After discussing with z/OS MQ admin, there is no issue related to max channels limit.

'No issue' is not a very technical response from your z/OS MQ admin.

In addition to MAXCHL, there are other related qmgr settings, including: ACTCHL and TCPCHL. I'd ask the admin folks the actual values (1-9999) for each.

As a reference https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.ref.adm.doc/q085320_.htm
_________________
My life flows on in endless song;
How can I keep from singing?
Back to top
View user's profile Send private message
dakoroni
PostPosted: Thu Sep 10, 2020 1:40 am Post subject: MQ Client opens/closes connection to host QM every 3 sec Reply with quote

Apprentice

Joined: 10 Jan 2020
Posts: 39

bruce2359 wrote:
dakoroni wrote:
After discussing with z/OS MQ admin, there is no issue related to max channels limit.

'No issue' is not a very technical response from your z/OS MQ admin.

In addition to MAXCHL, there are other related qmgr settings, including: ACTCHL and TCPCHL. I'd ask the admin folks the actual values (1-9999) for each.


I have asked z/OS MQ admin for these settings, expecting his feedback.
In the meantime can you pls comment if giving a value to DefRecon and/or ReconDelay parameters in mqclient.ini [channels] stanza, as listed on
https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.con.doc/q016860_.htm#q016860___chstrecondelay?
can make a difference?
Back to top
View user's profile Send private message
dakoroni
PostPosted: Thu Sep 10, 2020 1:48 am Post subject: MQ Client opens/closes connection to host QM every 3 secs Reply with quote

Apprentice

Joined: 10 Jan 2020
Posts: 39

[/quote]
In addition to MAXCHL, there are other related qmgr settings, including: ACTCHL and TCPCHL. I'd ask the admin folks the actual values (1-9999) for each.

As a reference https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.ref.adm.doc/q085320_.htm[/quote]

I have received the settings from z/OS MQ admin:
ACTCHL = 200
TCPCHL = 200
MAXCHL = 200

Current active channels = 47 to 51 depending on time
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Thu Sep 10, 2020 3:09 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2149
Location: Melbourne, Australia

Quote:
MQ error log contains:
Try the connection again later. If the failure persists, record the error values and contact your systems administrator. The return code from TCP/IP is 10061 (X'274D'). The reason for the failure may be that this host cannot reach the destination host.
It may also be possible that the listening program at host '192.168.202.85 (1414)' was not running.


You should not be seeing intermittent 10061 errors, assuming that the z/OS MQ listener is always running and accepting connections. Please check with your network team for switch or router errors in the route.

MQ trace will show at what point it is occurring, probably on the initial WinSock TCP connect() call.

Quote:
Can you pls advise if the above is caused by the application or it is a matter of MQ client setting (i.e. tcp stanza in mqclien.ini -> Connect_Timeout = number ??? )


Probably not.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportPROD MQ Client opens/closes connection to z/OS QM every 3sec
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.