| Author |
Message
|
| paulau |
 Posted: Mon Feb 26, 2018 12:58 am Post subject: Integration Server limits on RHEL 7 |
 |
|
Novice
Joined: 06 Feb 2017
Posts: 19
|
Hi,
We are doing a V6 to V10.0.0.8 upgrade and have attempted to define 50 integration servers on RHEL. When we get to approximately 46 servers IIB fails and triggers java and system dumps. There is plenty of free memory available and ulimits, semaphores etc. seem OK.
Has anyone been able to create & run large (100) numbers of execution groups on RHEL. The knowledge centre indicates that "there is no finite limit" and that there are no specific tuning requirements however thats not what we are seeing.
Any practical experience with RHEL IIB tuning would be appreciated.
Paul |
|
| Back to top |
|
 |
| abhi_thri |
| Posted: Mon Feb 26, 2018 3:31 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
Hi Paul...as you mentioned I don't think there is any specific limit to the no. of integration servers that can created on the node. IBM seems to indicate that anything more than 100 is a high no...
https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.scenarios.doc/gop_02/topics/bj60024_.htm
Most likely it might be something to do with the memory/file limits on the server, the java/memory dumps - does it indicate the failure reason at all? if javacore* files are generated that should've readable info in it otherwise you could use heapanalyzer tools to check out the heapdump files.
To rule out a specific limit issue you could try installing the node on a different server (eg:- linux server with higher memory or even a Windows server) and try creating 50 EGs and see. |
|
| Back to top |
|
|
| paulau |
| Posted: Mon Feb 26, 2018 12:25 pm Post subject: |
|
|
Novice
Joined: 06 Feb 2017
Posts: 19
|
Hi abhi_thri, we can repeat the issue on multiple RHEL servers. So I have submitted PMR's to IBM from two different machines with different memory specs. On RHEL we found that we needed to increase the SEMMNI as each EG takes 2 semaphores. Once you get past the SEMMNI issue there is a specific limit that stops IIB starting more that 50 EGs on RHEL.
We are able to run 150 servers (EGs) on AIX so its a linux only problem. Multiple people including IBM and Red Hat have looked at the file limits and memory without noticing any specific issues. Even on the smallest RHEL box there is still 15GB of free memory and much more available memory.
We are seeing a JVM in the dumps with a max of 1GB that appears short on storage. Not sure what this JVM is for as its not a server JVM, nor does it appear to be the agent JVM. It does have a bunch of IIB classes loaded and if it needs tuning I would hope that the IBM PMR people suggest a fix as I can't seen any documented IIB commands that relate to the JVM in the dump.
Interestingly when we have triggered the short on memory issue in IIB from the lib user, an mqconfig command from a different user (mqm) also gets out of memory. So agree its a memory or limits issue somewhere on RHEL, I just can't find our which resource it is. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Feb 26, 2018 4:50 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Hi Paul,
We are running about 100 IS's on Redhat, 8 CPU, 32GB. The DataFlowEngine user has :
>ulimit -a
address space limit (Kibytes) (-M) unlimited
core file size (blocks) (-c) unlimited
cpu time (seconds) (-t) unlimited
data size (Kibytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-x) unlimited
locked address space (Kibytes) (-l) 64
message queue size (Kibytes) (-q) 800
nice (-e) 0
nofile (-n) 65536
nproc (-u) 131072
pipe buffer size (bytes) (-p) 4096
max memory size (Kibytes) (-m) unlimited
rtprio (-r) 0
socket buffer size (bytes) (-b) 4096
sigpend (-i) 127968
stack size (Kibytes) (-s) unlimited
swap size (Kibytes) (-w) not supported
threads (-T) not supported
process size (Kibytes) (-v) unlimited
> grep sem /etc/sysctl.conf
kernel.sem = 1000 512000 250 1024
If needed, IS's have JVMs inside them.
_________________
Glenn |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Feb 27, 2018 1:06 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I would suggest running the IBM supplied mqconfig.sh script and making any changes recommended by that. I know it's for MQ - but they will be relevant as well.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Wed Feb 28, 2018 3:29 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
In case it helps, we did a bit of tuning exercise for the linux broker running on RHEL 7.2 and these are the params we ended up with...
kernel.sem = 1000 1024000 500 8192
fs.file-max = 524288
vm.dirty_background_ratio = 10
vm.dirty_ratio = 30
vm.swappiness = 30
kernel.sched_min_granularity_ns = 10000000
kernel.sched_wakeup_granularity_ns = 15000000
kernel.shmmax = 137438953472
kernel.shmmni = 8192
kernel.shmall = 4294967296
kernel.pid_max = 655360
kernel.threads-max = 448823
The broker user settings - nofile (-Hn & -Sn) , nproc (-Hu & -Su) are all set as 1048576. The memory and no. of cpu varies across the environments and we do have a mix of VM and Physical servers.
PS: These may not be the ideal settings for you as it could vary depending on the kind of heavy lifting the broker node is designed for... |
|
| Back to top |
|
|
| paulau |
| Posted: Wed Feb 28, 2018 12:10 pm Post subject: |
|
|
Novice
Joined: 06 Feb 2017
Posts: 19
|
Thanks for the tuning responses. Will try them out over the next few days. Yesterdays changes recommended by IBM didn't work so less than half a dozen left to try.
It is a bit disappointing that IBM say no OS tuning is required for IIB yet a medium system clearly does need OS tuning.
re mqconfig: it did highlight the need to adjust SEMMNI but doesn't seem to cover some of IIB's 'other' requirements. |
|
| Back to top |
|
|
| paulau |
| Posted: Thu Mar 01, 2018 5:31 pm Post subject: |
|
|
Novice
Joined: 06 Feb 2017
Posts: 19
|
Hi Glenn & abhi_thri, I have applied most of your tuning recommendations and no material improvement.
These mqconfig results taken during the failure test highlight most the the settings that would normally be tuned for IIB are sufficient. It does highlight that an independent user gets impacted by the same OS shortage.
mqm@au2... ~ $ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 500000
max locked memory (kbytes, -l) 3000000
max memory size (kbytes, -m) unlimited
open files (-n) 128000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 131072
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
mqm@au2... ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
semmns (sem:2) 52 of 64000 semaphores (0%) IBM>=4096 PASS
semopm (sem:3) 64 operations IBM>=32 PASS
semmni (sem:4) 20 of 1000 sets (2%) IBM>=128 PASS
System V Shared Memory
shmmax 18446744073692774399 bytes IBM>=268435456 PASS
shmmni 77 of 8192 sets (0%) IBM>=4096 PASS
shmall 88412 of 18446744073692774399 pages (0%) IBM>=2097152 PASS
System Settings
file-max 3872 of 524288 files (0%) IBM>=524288 PASS
pid_max 836 of 65536 processids (1%) IBM>=32768 PASS
threads-max 836 of 64768 threads (1%) IBM>=32768 PASS
Current User Limits (mqm)
nofile (-Hn) 128000 files IBM>=10240 PASS
nofile (-Sn) 128000 files IBM>=10240 PASS
nproc (-Hu) 221 of 131072 processes (0%) IBM>=4096 PASS
nproc (-Su) 221 of 131072 processes (0%) IBM>=4096 PASS
mqconfig: A PASS score means your system meets the minimum
IBM recommendations but busy systems may need
higher limits to run production workloads.
mqm@au2... ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
semmns (sem:2) 64000 semaphores /opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
mqm@au2.. ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
semmns (sem:2) 218 of 64000 semaphores (0%) IBM>=4096 PASS
semopm (sem:3) 64 operations IBM>=32 PASS
semmni (sem:4) 103 of 1000 sets (10%) IBM>=128 PASS
System V Shared Memory
shmmax 18446744073692774399 bytes IBM>=268435456 PASS
shmmni 77 of 8192 sets (0%) IBM>=4096 PASS
shmall 88412 of 18446744073692774399 pages (0%) IBM>=2097152 PASS
System Settings
file-max 12320 of 524288 files (2%) IBM>=524288 PASS
pid_max 1745 of 65536 processids (2%) IBM>=32768 PASS
threads-max 1726 of 64768 threads (2%) IBM>=32768 PASS
Current User Limits (mqm)
nofile (-Hn) 128000 files IBM>=10240 PASS
nofile (-Sn) 128000 files IBM>=10240 PASS
nproc (-Hu) 237 of 131072 processes (0%) IBM>=4096 PASS
nproc (-Su) 237 of 131072 processes (0%) IBM>=4096 PASS
mqconfig: A PASS score means your system meets the minimum
IBM recommendations but busy systems may need
higher limits to run production workloads.
mqm@au2... ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
/opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
semmns (sem:2) 64000 semaphores /opt/IBM/mqm_A/bin/mqconfig: fork: Cannot allocate memory
mqm@au2.. ~ $ mqconfig
mqconfig: Analyzing Red Hat Enterprise Linux Server release 7.4 (Maipo)
settings for IBM MQ V9.0
System V Semaphores
semmsl (sem:1) 500 semaphores IBM>=32 PASS
semmns (sem:2) 206 of 64000 semaphores (0%) IBM>=4096 PASS
semopm (sem:3) 64 operations IBM>=32 PASS
semmni (sem:4) 97 of 1000 sets (9%) IBM>=128 PASS
System V Shared Memory
shmmax 18446744073692774399 bytes IBM>=268435456 PASS
shmmni 77 of 8192 sets (0%) IBM>=4096 PASS
shmall 88412 of 18446744073692774399 pages (0%) IBM>=2097152 PASS
System Settings
file-max 20960 of 524288 files (3%) IBM>=524288 PASS
pid_max 2445 of 65536 processids (3%) IBM>=32768 PASS
threads-max 2448 of 64768 threads (3%) IBM>=32768 PASS
Current User Limits (mqm)
nofile (-Hn) 128000 files IBM>=10240 PASS
nofile (-Sn) 128000 files IBM>=10240 PASS
nproc (-Hu) 258 of 131072 processes (0%) IBM>=4096 PASS
nproc (-Su) 258 of 131072 processes (0%) IBM>=4096 PASS
mqconfig: A PASS score means your system meets the minimum
IBM recommendations but busy systems may need
higher limits to run production workloads. |
|
| Back to top |
|
|
| abhi_thri |
| Posted: Fri Mar 09, 2018 6:44 am Post subject: |
|
|
Knight
Joined: 17 Jul 2017
Posts: 516
Location: UK
|
Hi Paul,
Did you made any progress on this yet?
It may be worth verifying whether the Redhat build is customized (eg:- overriding some of the OS level settings) by your Linux admin team. While diagnosing a performance we did notice that our linux team had disabled the kernel autotuning (can't remember the exact param) in the custom build while the vanilla one had it enabled by default.
Regards,
Abhi |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Mar 09, 2018 6:57 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Could it be that you're overallocated on memory?
What happens if you reduce some of these values. Will you then also get cannot fork no memory? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| paulau |
| Posted: Mon Mar 12, 2018 2:29 pm Post subject: |
|
|
Novice
Joined: 06 Feb 2017
Posts: 19
|
Hi abhi_thri,
Yes we eventually got an empty Integration Node with 42 Servers running on a 32GB machine. There were at least 5 mandatory changes that we made to get the product running so doubt the documentation saying "IBM® Integration Bus has no specific requirements for kernel configuration parameters." is applicable to anything larger than a small installation.
Some of the changes included:
- changing the JVM to use memory above 4GB to keep the class loader and heap allocation separate.
we had already set overcommit_memory to 2 so we needed to increase the default ratio
- MQ, the Node and Servers used 570k worth of files so we had to increase fs.file-max
each Server takes 2 semaphores
and a few other ulimit and kernel changes.
Its a pity there isn't an iibconfig script to check if the OS values are suitable.
Paul |
|
| Back to top |
|
|
| skrv |
| Posted: Fri May 18, 2018 3:27 am Post subject: |
|
|
Centurion
Joined: 26 Oct 2012
Posts: 118
|
Hi,
We are having similar issues with just 20 integration servers on RHEL 7.4 servers where as no issues with Solaris Sparc 5.10.
Can you specifically mention what you guys did to resolve this issue?
thanks |
|
| Back to top |
|
|
| esbdeveloper |
| Posted: Mon Jun 11, 2018 9:34 am Post subject: |
|
|
Newbie
Joined: 01 Oct 2017
Posts: 6
|
| paulau wrote: |
Hi abhi_thri,
Yes we eventually got an empty Integration Node with 42 Servers running on a 32GB machine. There were at least 5 mandatory changes that we made to get the product running so doubt the documentation saying "IBM® Integration Bus has no specific requirements for kernel configuration parameters." is applicable to anything larger than a small installation.
Some of the changes included:
- changing the JVM to use memory above 4GB to keep the class loader and heap allocation separate.
we had already set overcommit_memory to 2 so we needed to increase the default ratio
- MQ, the Node and Servers used 570k worth of files so we had to increase fs.file-max
each Server takes 2 semaphores
and a few other ulimit and kernel changes.
Its a pity there isn't an iibconfig script to check if the OS values are suitable.
Paul |
Hi,
we are facing very similar issues (on RHEL 7 but with less integration servers), we are getting "cannot allocate memory" errors following falling integration servers and abend files, java core dumps and FDCs (the MQ 9 runs alongside with our IIB 10.0.0.6 on the same server).
we thought about changing the overcommit_memory parameter to 0 instead of 2 (when we had WMB 8, this parameter was set to 0 and no "cannot allocate memory" exceptions were thrown.
how can you set the semaphore as you have mentioned?
Thanks!! |
|
| Back to top |
|
|
| harimhkr |
| Posted: Fri Jun 22, 2018 7:48 am Post subject: |
|
|
Novice
Joined: 08 Jan 2013
Posts: 22
|
transparent_hugepage was set to never and this also changed the overcommit_memory to 0
the above has resolved the issue for us. before we made those changes it was failing at 25 ex groups but now i am able to create as long as the memory is sufficent (with 8Gb went alll the way up to 100 ex groups)
But will we have any issues having those settings. I am not sure on how the overcommit_memory and the transaction_huge pages will effect the system.
can anyone answer? |
|
| Back to top |
|
|
|
|
