ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportJump in number of connections opened by a queue manager

Post new topicReply to topic
Jump in number of connections opened by a queue manager View previous topic :: View next topic
Author Message
andrewfemin
PostPosted: Mon Dec 16, 2019 9:35 pm Post subject: Jump in number of connections opened by a queue manager Reply with quote

Acolyte

Joined: 26 Aug 2017
Posts: 52

Hi,

We have 2 FRs, 80+ PRs, 2 Remote QMs and 3 applications connecting through SVRCONN channels in our Production. The Remote QMs and the applications using SVRCONN channels connect to FR1, thereby making FR1 a gateway QM.

We have been observing a strange behaviour. Every Monday, at precisely 6 PM CST, the CLUSSDR channels in all PRs go to BINDING or INITIALIZING status. Once we restart the FR1, the CLUSSDR channels go back to RUNNING status.

On raising a case with IBM, we were told that MQ was hitting an OS limit with pthread_create and we were asked to contact our platform team.

When we tried troubleshooting it further, we found that DIS CONN(*) returned 47 connections at 5:59 PM and it begins increasing drastically after 6:00 PM. At around 6:04 PM, it reaches 5500 connections and that's when the PRs start having the connection issue. It looks like the OS(or the QM) is hitting some limit and unable to create further connections.

We are also trying to find out which application(or channel) is creating this many connections. But when I display one of the 5500 connections, it does not have much information. Here's the result:

Code:
AMQ8276: Display Connection details.
   CONN(1E71F45DFDDC472B)
   EXTCONN(414D51434C534D515052442020202020)
   TYPE(CONN)
   PID(1449)                               TID(161)
   APPLDESC(IBM MQ Channel)                APPLTAG(amqrmppa)
   APPLTYPE(SYSTEM)                        ASTATE(NONE)
   CHANNEL( )                              CLIENTID( )
   CONNAME( )
   CONNOPTS(MQCNO_HANDLE_SHARE_BLOCK,MQCNO_SHARED_BINDING)
   USERID(mqm)                             UOWLOG( )
   UOWSTDA( )                              UOWSTTI( )
   UOWLOGDA( )                             UOWLOGTI( )
   URTYPE(QMGR)
   EXTURID(XA_FORMATID[] XA_GTRID[] XA_BQUAL[])
   QMURID(0.0)                             UOWSTATE(NONE)

The channel name is empty and APPLTAG is amqrmppa. This does not help much to identify the culprits. In such cases, is there a way to identify which component is creating that many connections? Also, DIS CHS(*) during the issue does not show any increase in the channels. I'm unable to proceed with the troubleshooting due to this. Kindly help me out with your suggestions.

Thanks,
Andrew
Back to top
View user's profile Send private message
LJM
PostPosted: Tue Dec 17, 2019 6:21 am Post subject: Reply with quote

Novice

Joined: 05 Jul 2018
Posts: 21

Version ?

my google shows a hit for a bug

https://developer.ibm.com/answers/questions/399683/mq-amqrmppa-processes-are-increasing/
Back to top
View user's profile Send private message
andrewfemin
PostPosted: Tue Dec 17, 2019 11:53 am Post subject: Reply with quote

Acolyte

Joined: 26 Aug 2017
Posts: 52

We are running MQ 9.0.0.4 on SUSE Linux Enterprise Server 12 SP3. So it must be a different issue?
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Dec 17, 2019 3:17 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2048
Location: Melbourne, Australia

At 6:00PM, when the connections begin, start MQ trace, and then end it after a few seconds. This will likely show the source. Make sure you have plenty of space in /var/mqm/trace.
_________________
Glenn
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Dec 17, 2019 5:32 pm Post subject: Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7588

andrewfemin wrote:
SUSE Linux Enterprise Server 12 SP3


andrewfemin wrote:
MQ was hitting an OS limit with pthread_create


When our SUSE servers went to SLES 12 SP4 from SLES 11, same problems. Instability with FDCs talking about issues with pthread_create.

An O/S parameter called DefaultTasksMax was the culprit in our case. Out of the box SLES 12 SP3 has that set at a stupidly low value (512 I think) limiting how many threads/processes your mqm account is going to be able to spawn.

System command to view this is "systemctl show --property DefaultTasksMax".

Also check settings for pid_max and threads-max, as well as nofile (-Hn) nofile (-Sn), nproc (-Hn), nproc (-Sn).


I found the overall problem resolution process very frustrating with no one being able to provide us a means to identify exactly what stupid O/S parameter was causing MQ to barf on pthread_create. Furious Googling and some dumb luck led us to some other poor sap dealing with pthread_create issues due to DefaultTasksMax being set too low by SUSE.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportJump in number of connections opened by a queue manager
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.