ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » xcsCreateThread error

Post new topic  Reply to topic Goto page 1, 2, 3, 4  Next
 xcsCreateThread error « View previous topic :: View next topic » 
Author Message
yanaK
PostPosted: Thu May 28, 2020 3:22 pm    Post subject: xcsCreateThread error Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

I am getting a `Resource temporarily unavailable` error from an IBM MQ queue manager. I tried changing my kernel parameters and I still see its there. Here is the top of my FDC:

| Date/Time :- Mon May 25 2020 23:58:03 PDT |

| UTC Time :- 1590389941.820409 |

| UTC Time Offset :- -420 (PST) |

| Host Name :- x0321 |

| Operating System :- Linux 3.10.0-1127.el7.x86_64 |

| PIDS :- 5724H7230 |

| LVLS :- 7.1.0.7 |

| Product Long Name :- WebSphere MQ for Linux (x86-64 platform) |

| Vendor :- IBM |

| Installation Path :- /opt/mqm |

| Installation Name :- Installation1 (1) |

| Probe Id :- XC035040 |

| Application Name :- MQM |

| Component :- xcsCreateThread |

| SCCS Info :- lib/cs/unix/linux_2/amqxprmx.c, 1.176.1.1 |

| Line Number :- 1974 |

| Build Date :- Nov 4 2015 |

| CMVC level :- p710-007-151104 |

| Build Type :- IKAP - (Production) |

| Effective UserID :- 3732 (mqm) |

| Real UserID :- 3732 (mqm) |

| Program Name :- runmqlsr |

| Arguments :- -r -m MQPSEPX -t TCP -p 1417 |

| Addressing mode :- 64-bit |

| LANG :- en_US.UTF-8 |

| Process :- 68029 |

| Process(Thread) :- 68029 |

| Thread :- 1 |

/xcsCreateThread

| Probe Description :- AMQ6119: An internal WebSphere MQ error has occurred |

| ('11 - Resource temporarily unavailable' from pthread_create.) |

| FDCSequenceNumber :- 6 |

| Arith1 :- 11 (0xb) |

| Comment1 :- '11 - Resource temporarily unavailable' from |

| pthread_create. |

|

I am on RHEL 7 and my kernel params are:

fs.file-max = 583180

kernel.shmall = 18446744073692774399

kernel.shmmax = 18446744073692774399

kernel.shmmni = 4096

kernel.sem = 500 256000 250 1024

net.ipv4.tcp_keepalive_time = 300

kernel.threads-max = 95346

This is intermittent and whenever this happens I cannot runmqsc (I get
AMQ9508: Program cannot connect to the queue manager.)

I am stuck on this - any help is appreciated.

Thanks
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Thu May 28, 2020 4:02 pm    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

Have you done a bit of Googling ? A quick search on the probe id showed this https://www.ibm.com/support/pages/resource-shortage-mq-queue-manager-linux-when-running-large-number-clients-additional-kernel-parameters does this help at all ?

How many clients are you trying to run on a single machine ?
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
yanaK
PostPosted: Thu May 28, 2020 4:25 pm    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

Yes I changed the params based on that article (and the existing queue managers we've) - if you see all my kernel params are above those numbers (although it will depend on clients and OS I think).
I still see the issue intermittently - so need to know what's causing it.

What do you mean by how many clients? This queue manager is a part of a cluster (of 5 queue managers) and the cluster has over 10k clients
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu May 28, 2020 4:53 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

Are you seeing the same error on all 5 qmgrs in the cluster?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
yanaK
PostPosted: Thu May 28, 2020 5:04 pm    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

No - this is a new one we added (replacing an old one)
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Thu May 28, 2020 5:19 pm    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

So, am I right in thinking that you are trying to run a lot of clients in to your queue manager ? Are you running these channels in the listener or are you using channel pool processes ? I would have expected new threads be continually created in the agent process or in the channel pool process AMQRMPPA. Why is it that RUNMQLSR is complaining about creating a thread ? I would recommend that you run with MQ with channel pool processes active.

Can you give us a feel for how many threads and processes you have running at the time of the failure.
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
yanaK
PostPosted: Thu May 28, 2020 5:28 pm    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

So, am I right in thinking that you are trying to run a lot of clients in to your queue manager ?
Theoretically yes (but I'd expect it to get distributed over all the q mgrs)

Are you running these channels in the listener or are you using channel pool processes ?
How do I know?

I would have expected new threads be continually created in the agent process or in the channel pool process AMQRMPPA. Why is it that RUNMQLSR is complaining about creating a thread ? I would recommend that you run with MQ with channel pool processes active.
Can you tell how?

Can you give us a feel for how many threads and processes you have running at the time of the failure.
What command do I use to get that value?
Also when I run this I get 0:
Code:
$ echo "dis conn(*) all" | runmqsc MQPOSEPD|grep EXTCONN|wc -l

Can you suggest what other info I can generate to help troubleshoot?

Thanks to all for the help.
Back to top
View user's profile Send private message
PaulClarke
PostPosted: Thu May 28, 2020 5:37 pm    Post subject: Reply with quote

Grand Master

Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand

I believe the commands to display thread information along with processes varies depending on the version of Linux you are using. I seem to remember that originally Linux used to show threads by default in the ps -ef command which confused people because they thought they had 10 listeners running etc. But I think you can still ask for threads using -L or -T or similar. You can also use something like top to display thread information.

Essentially what I am interested in is whether the bulk of your channels are running in RUNMQLSR or do you have lots of AMQRMPPA processes running ?
_________________
Paul Clarke
MQGem Software
www.mqgem.com
Back to top
View user's profile Send private message Visit poster's website
yanaK
PostPosted: Thu May 28, 2020 6:21 pm    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

Code:
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 75049 mqm       20   0  172556   2492   1624 R   0.7  0.0   0:00.79 top
 15501 mqm       20   0 1342584  40068  35588 S   0.3  0.7   0:46.05 amqzlaa0
100253 mqm       20   0 1303784  57096  31024 S   0.3  1.0   2:26.63 amqrmppa
107590 mqm       20   0 1342432  33640  29592 S   0.3  0.6   0:23.62 amqzlaa0
113602 asxk      20   0  186908   3812   1272 S   0.3  0.1   0:05.40 sshd
     1 root      20   0  202360   6832   3928 S   0.0  0.1  81:10.58 systemd
     2 root      20   0       0      0      0 S   0.0  0.0   0:16.17 kthreadd
     4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
     6 root      20   0       0      0      0 S   0.0  0.0   0:54.72 ksoftirqd/0
     7 root      rt   0       0      0      0 S   0.0  0.0   0:34.72 migration/0
     8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
     9 root      20   0       0      0      0 S   0.0  0.0  18:13.82 rcu_sched
    10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
    11 root      rt   0       0      0      0 S   0.0  0.0   0:05.98 watchdog/0
    12 root      rt   0       0      0      0 S   0.0  0.0   0:07.43 watchdog/1


I see this when I run top
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Thu May 28, 2020 8:28 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

If you changed you kernel parms but the queue manager was left running you may have to do the following:
  • logon to the box (sudo to mqm)
  • stop the queue manager (closes the old shell)
  • start the queue manager from the new shell (with the new kernel parms)

Enjoy
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
yanaK
PostPosted: Thu May 28, 2020 10:54 pm    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

I can give it a try - I did a sysctl -p but didn't do a endmqm and strmqm. Do you've any doc on this?
Like this says sysctl -p should load the new values.
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.ins.doc/q008550_.htm
Back to top
View user's profile Send private message
tczielke
PostPosted: Fri May 29, 2020 3:28 am    Post subject: Reply with quote

Guardian

Joined: 08 Jul 2010
Posts: 939
Location: Illinois, USA

What is your nproc setting for the mqm id? You can find that with "ulimit -u" when run under the mqm id.
_________________
Working with MQ since 2010.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri May 29, 2020 4:26 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

yanaK wrote:
I can give it a try - I did a sysctl -p but didn't do a endmqm and strmqm. Do you've any doc on this?
Like this says sysctl -p should load the new values.
https://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.ins.doc/q008550_.htm

It will load the new values into the current shell. Is that the same shell the queue manager was started in?
Beware also of other running queue managers. They might suddenly starve for resources because you started a queue manager in the new shell not submitted to the restrictions of the ones running in the old shells.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
yanaK
PostPosted: Fri May 29, 2020 8:20 am    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

[i]ulimit -u gives 47673
[/i]
There is only one queue manager in that box.

It will load the new values into the current shell. Is that the same shell the queue manager was started in?

No I started the queue manager few weeks back


Last edited by yanaK on Fri May 29, 2020 9:47 am; edited 1 time in total
Back to top
View user's profile Send private message
yanaK
PostPosted: Fri May 29, 2020 9:18 am    Post subject: Reply with quote

Acolyte

Joined: 28 May 2020
Posts: 69

Also I see in FDC:

xcsCreateThread rc=xecP_E_PROC_LIMIT

Does that correspond to nproc? Is there a way to find out how much processes were running at the time of the failures?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2, 3, 4  Next Page 1 of 4

MQSeries.net Forum Index » General Discussion » xcsCreateThread error
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.