ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Integration Server limits on RHEL 7

Post new topic  Reply to topic
 Integration Server limits on RHEL 7 « View previous topic :: View next topic » 
Author Message
paulau
PostPosted: Mon Feb 26, 2018 12:58 am    Post subject: Integration Server limits on RHEL 7 Reply with quote

Novice

Joined: 06 Feb 2017
Posts: 19

Hi,
We are doing a V6 to V10.0.0.8 upgrade and have attempted to define 50 integration servers on RHEL. When we get to approximately 46 servers IIB fails and triggers java and system dumps. There is plenty of free memory available and ulimits, semaphores etc. seem OK.
Has anyone been able to create & run large (100) numbers of execution groups on RHEL. The knowledge centre indicates that "there is no finite limit" and that there are no specific tuning requirements however thats not what we are seeing.

Any practical experience with RHEL IIB tuning would be appreciated.

Paul
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Mon Feb 26, 2018 3:31 am    Post subject: Reply with quote

Knight

Joined: 17 Jul 2017
Posts: 516
Location: UK

Hi Paul...as you mentioned I don't think there is any specific limit to the no. of integration servers that can created on the node. IBM seems to indicate that anything more than 100 is a high no...

https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.scenarios.doc/gop_02/topics/bj60024_.htm

Most likely it might be something to do with the memory/file limits on the server, the java/memory dumps - does it indicate the failure reason at all? if javacore* files are generated that should've readable info in it otherwise you could use heapanalyzer tools to check out the heapdump files.

To rule out a specific limit issue you could try installing the node on a different server (eg:- linux server with higher memory or even a Windows server) and try creating 50 EGs and see.
Back to top
View user's profile Send private message
paulau
PostPosted: Mon Feb 26, 2018 12:25 pm    Post subject: Reply with quote

Novice

Joined: 06 Feb 2017
Posts: 19

Hi abhi_thri, we can repeat the issue on multiple RHEL servers. So I have submitted PMR's to IBM from two different machines with different memory specs. On RHEL we found that we needed to increase the SEMMNI as each EG takes 2 semaphores. Once you get past the SEMMNI issue there is a specific limit that stops IIB starting more that 50 EGs on RHEL.

We are able to run 150 servers (EGs) on AIX so its a linux only problem. Multiple people including IBM and Red Hat have looked at the file limits and memory without noticing any specific issues. Even on the smallest RHEL box there is still 15GB of free memory and much more available memory.

We are seeing a JVM in the dumps with a max of 1GB that appears short on storage. Not sure what this JVM is for as its not a server JVM, nor does it appear to be the agent JVM. It does have a bunch of IIB classes loaded and if it needs tuning I would hope that the IBM PMR people suggest a fix as I can't seen any documented IIB commands that relate to the JVM in the dump.

Interestingly when we have triggered the short on memory issue in IIB from the lib user, an mqconfig command from a different user (mqm) also gets out of memory. So agree its a memory or limits issue somewhere on RHEL, I just can't find our which resource it is.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Feb 26, 2018 4:50 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2491
Location: Melbourne, Australia

Hi Paul,
We are running about 100 IS's on Redhat, 8 CPU, 32GB. The DataFlowEngine user has :

>ulimit -a
address space limit (Kibytes) (-M) unlimited
core file size (blocks) (-c) unlimited
cpu time (seconds) (-t) unlimited
data size (Kibytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-x) unlimited
locked address space (Kibytes) (-l) 64
message queue size (Kibytes) (-q) 800
nice (-e) 0
nofile (-n) 65536
nproc (-u) 131072
pipe buffer size (bytes) (-p) 4096
max memory size (Kibytes) (-m) unlimited
rtprio (-r) 0
socket buffer size (bytes) (-b) 4096
sigpend (-i) 127968
stack size (Kibytes) (-s) unlimited
swap size (Kibytes) (-w) not supported
threads (-T) not supported
process size (Kibytes) (-v) unlimited

> grep sem /etc/sysctl.conf
kernel.sem = 1000 512000 250 1024

If needed, IS's have JVMs inside them.
_________________
Glenn
Back to top
View user's profile Send private message
zpat
PostPosted: Tue Feb 27, 2018 1:06 am    Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5849
Location: UK

I would suggest running the IBM supplied mqconfig.sh script and making any changes recommended by that. I know it's for MQ - but they will be relevant as well.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Wed Feb 28, 2018 3:29 am    Post subject: Reply with quote

Knight

Joined: 17 Jul 2017
Posts: 516
Location: UK

In case it helps, we did a bit of tuning exercise for the linux broker running on RHEL 7.2 and these are the params we ended up with...

kernel.sem = 1000 1024000 500 8192

fs.file-max = 524288

vm.dirty_background_ratio = 10
vm.dirty_ratio = 30
vm.swappiness = 30
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000

kernel.shmmax = 137438953472
kernel.shmmni = 8192
kernel.shmall = 4294967296
kernel.pid_max = 655360
kernel.threads-max = 448823

The broker user settings - nofile (-Hn & -Sn) , nproc (-Hu & -Su) are all set as 1048576. The memory and no. of cpu varies across the environments and we do have a mix of VM and Physical servers.

PS: These may not be the ideal settings for you as it could vary depending on the kind of heavy lifting the broker node is designed for...
Back to top
View user's profile Send private message
paulau
PostPosted: Wed Feb 28, 2018 12:10 pm    Post subject: Reply with quote

Novice

Joined: 06 Feb 2017
Posts: 19

Thanks for the tuning responses. Will try them out over the next few days. Yesterdays changes recommended by IBM didn't work so less than half a dozen left to try.
It is a bit disappointing that IBM say no OS tuning is required for IIB yet a medium system clearly does need OS tuning.

re mqconfig: it did highlight the need to adjust SEMMNI but doesn't seem to cover some of IIB's 'other' requirements.
Back to top
View user's profile Send private message
paulau
PostPosted: Thu Mar 01, 2018 5:31 pm    Post subject: Reply with quote

Novice

Joined: 06 Feb 2017
Posts: 19

Hi Glenn & abhi_thri, I have applied most of your tuning recommendations and no material improvement.

These mqconfig results taken during the failure test highlight most the the settings that would normally be tuned for IIB are sufficient. It does highlight that an independent user gets impacted by the same OS shortage.

mqm@au2... ~ $ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 500000
max locked memory (kbytes, -l) 3000000
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 131072
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
mqm@au2... ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0

System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
semmns (sem:2) 52 of 64000 semaphores (0%) IBM>=4096 PASS
semopm (sem:3) 64 operations IBM>=32 PASS
semmni (sem:4) 20 of 1000 sets (2%) IBM>=128 PASS

System V Shared Memory
shmmax 18446744073692774399 bytes IBM>=268435456 PASS
shmmni 77 of 8192 sets (0%) IBM>=4096 PASS
shmall 88412 of 18446744073692774399 pages (0%) IBM>=2097152 PASS

System Settings
file-max 3872 of 524288 files (0%) IBM>=524288 PASS
pid_max 836 of 65536 processids (1%) IBM>=32768 PASS
threads-max 836 of 64768 threads (1%) IBM>=32768 PASS

Current User Limits (mqm)
nofile (-Hn) 128000 files IBM>=10240 PASS
nofile (-Sn) 128000 files IBM>=10240 PASS
nproc (-Hu) 221 of 131072 processes (0%) IBM>=4096 PASS
nproc (-Su) 221 of 131072 processes (0%) IBM>=4096 PASS

mqconfig: A PASS score means your system meets the minimum
IBM recommendations but busy systems may need
higher limits to run production workloads.
mqm@au2... ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0

System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
semmns (sem:2) 64000 semaphores /opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
mqm@au2.. ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0

System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
semmns (sem:2) 218 of 64000 semaphores (0%) IBM>=4096 PASS
semopm (sem:3) 64 operations IBM>=32 PASS
semmni (sem:4) 103 of 1000 sets (10%) IBM>=128 PASS

System V Shared Memory
shmmax 18446744073692774399 bytes IBM>=268435456 PASS
shmmni 77 of 8192 sets (0%) IBM>=4096 PASS
shmall 88412 of 18446744073692774399 pages (0%) IBM>=2097152 PASS

System Settings
file-max 12320 of 524288 files (2%) IBM>=524288 PASS
pid_max 1745 of 65536 processids (2%) IBM>=32768 PASS
threads-max 1726 of 64768 threads (2%) IBM>=32768 PASS

Current User Limits (mqm)
nofile (-Hn) 128000 files IBM>=10240 PASS
nofile (-Sn) 128000 files IBM>=10240 PASS
nproc (-Hu) 237 of 131072 processes (0%) IBM>=4096 PASS
nproc (-Su) 237 of 131072 processes (0%) IBM>=4096 PASS

mqconfig: A PASS score means your system meets the minimum
IBM recommendations but busy systems may need
higher limits to run production workloads.
mqm@au2... ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0

System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
semmns (sem:2) 64000 semaphores /opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
mqm@au2.. ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0

System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
semmns (sem:2) 206 of 64000 semaphores (0%) IBM>=4096 PASS
semopm (sem:3) 64 operations IBM>=32 PASS
semmni (sem:4) 97 of 1000 sets (9%) IBM>=128 PASS

System V Shared Memory
shmmax 18446744073692774399 bytes IBM>=268435456 PASS
shmmni 77 of 8192 sets (0%) IBM>=4096 PASS
shmall 88412 of 18446744073692774399 pages (0%) IBM>=2097152 PASS

System Settings
file-max 20960 of 524288 files (3%) IBM>=524288 PASS
pid_max 2445 of 65536 processids (3%) IBM>=32768 PASS
threads-max 2448 of 64768 threads (3%) IBM>=32768 PASS

Current User Limits (mqm)
nofile (-Hn) 128000 files IBM>=10240 PASS
nofile (-Sn) 128000 files IBM>=10240 PASS
nproc (-Hu) 258 of 131072 processes (0%) IBM>=4096 PASS
nproc (-Su) 258 of 131072 processes (0%) IBM>=4096 PASS

mqconfig: A PASS score means your system meets the minimum
IBM recommendations but busy systems may need
higher limits to run production workloads.
Back to top
View user's profile Send private message
abhi_thri
PostPosted: Fri Mar 09, 2018 6:44 am    Post subject: Reply with quote

Knight

Joined: 17 Jul 2017
Posts: 516
Location: UK

Hi Paul,
Did you made any progress on this yet?

It may be worth verifying whether the Redhat build is customized (eg:- overriding some of the OS level settings) by your Linux admin team. While diagnosing a performance we did notice that our linux team had disabled the kernel autotuning (can't remember the exact param) in the custom build while the vanilla one had it enabled by default.

Regards,
Abhi
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Mar 09, 2018 6:57 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20695
Location: LI,NY

Could it be that you're overallocated on memory?
What happens if you reduce some of these values. Will you then also get cannot fork no memory?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
paulau
PostPosted: Mon Mar 12, 2018 2:29 pm    Post subject: Reply with quote

Novice

Joined: 06 Feb 2017
Posts: 19

Hi abhi_thri,
Yes we eventually got an empty Integration Node with 42 Servers running on a 32GB machine. There were at least 5 mandatory changes that we made to get the product running so doubt the documentation saying "IBM® Integration Bus has no specific requirements for kernel configuration parameters." is applicable to anything larger than a small installation.
Some of the changes included:
  • changing the JVM to use memory above 4GB to keep the class loader and heap allocation separate.
    we had already set overcommit_memory to 2 so we needed to increase the default ratio
  • MQ, the Node and Servers used 570k worth of files so we had to increase fs.file-max
    each Server takes 2 semaphores
    and a few other ulimit and kernel changes.

Its a pity there isn't an iibconfig script to check if the OS values are suitable.
Paul
Back to top
View user's profile Send private message
skrv
PostPosted: Fri May 18, 2018 3:27 am    Post subject: Reply with quote

Centurion

Joined: 26 Oct 2012
Posts: 118

Hi,

We are having similar issues with just 20 integration servers on RHEL 7.4 servers where as no issues with Solaris Sparc 5.10.

Can you specifically mention what you guys did to resolve this issue?

thanks
Back to top
View user's profile Send private message
esbdeveloper
PostPosted: Mon Jun 11, 2018 9:34 am    Post subject: Reply with quote

Newbie

Joined: 01 Oct 2017
Posts: 6

paulau wrote:
Hi abhi_thri,
Yes we eventually got an empty Integration Node with 42 Servers running on a 32GB machine. There were at least 5 mandatory changes that we made to get the product running so doubt the documentation saying "IBM® Integration Bus has no specific requirements for kernel configuration parameters." is applicable to anything larger than a small installation.
Some of the changes included:
  • changing the JVM to use memory above 4GB to keep the class loader and heap allocation separate.
    we had already set overcommit_memory to 2 so we needed to increase the default ratio
  • MQ, the Node and Servers used 570k worth of files so we had to increase fs.file-max
    each Server takes 2 semaphores
    and a few other ulimit and kernel changes.

Its a pity there isn't an iibconfig script to check if the OS values are suitable.
Paul



Hi,
we are facing very similar issues (on RHEL 7 but with less integration servers), we are getting "cannot allocate memory" errors following falling integration servers and abend files, java core dumps and FDCs (the MQ 9 runs alongside with our IIB 10.0.0.6 on the same server).
we thought about changing the overcommit_memory parameter to 0 instead of 2 (when we had WMB 8, this parameter was set to 0 and no "cannot allocate memory" exceptions were thrown.

how can you set the semaphore as you have mentioned?
Thanks!!
Back to top
View user's profile Send private message
harimhkr
PostPosted: Fri Jun 22, 2018 7:48 am    Post subject: Reply with quote

Novice

Joined: 08 Jan 2013
Posts: 22

transparent_hugepage was set to never and this also changed the overcommit_memory to 0
the above has resolved the issue for us. before we made those changes it was failing at 25 ex groups but now i am able to create as long as the memory is sufficent (with 8Gb went alll the way up to 100 ex groups)

But will we have any issues having those settings. I am not sure on how the overcommit_memory and the transaction_huge pages will effect the system.
can anyone answer?
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Integration Server limits on RHEL 7
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.