| Author |
Message
|
| xcuseme123 |
 Posted: Mon Mar 02, 2020 7:28 am Post subject: IIB and MQ - performance issue |
 |
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
We are using HA clusters and Multi-instance queue managers in Performance enviornment. To support this design, we are using NFS File mount 4.1
We are testing two message flows(pattern-1 & 2) and common flow which does auditing which logs successful//failure in DB. All these 3 flows are deployed into individual EG’s. Below Flow design Pattern-1&2 completes in 1.5 hours in Performance environment using two brokers shares the load. Performance enviornment uses NFS file system as it uses HA clusters and Multi-instance. If we execute the same load having same data completes in 30 mins in DEV environment which local file system.
Flow design pattern 1: Input Queue MQ-->Compute node-->SOAP Request Node --> Compute node-->Publication Node
Flow design pattern 2: Input Queue MQ-->Compute node-->Publication Node
CPU and memory consumptions are normal in DEV and Performance enviornment
IIB : 10.0.0.3
MQ: 8.0.0.4
Server details
NAME="Red Hat Enterprise Linux Server"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"
NFS mount is set with options - rw,sync,no_wdelay,no_root_squash
Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Mar 02, 2020 9:05 am Post subject: Re: IIB and MQ - performance issue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| xcuseme123 wrote: |
| Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?. |
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Mon Mar 02, 2020 10:43 am Post subject: Re: IIB and MQ - performance issue |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
| Vitor wrote: |
| xcuseme123 wrote: |
| Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?. |
|
Thanks for quick response.
I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file. Any other options left to boost say atleast another 10% of performance please?. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 02, 2020 11:45 am Post subject: Re: IIB and MQ - performance issue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| xcuseme123 wrote: |
| Any other options left to boost say atleast another 10% of performance please?. |
You'll get more bang for your buck optimizing the NFS disc itself; make sure you're using any caching that's available, eliminating any network latency between the disc and the server, that sort of thing.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Tue Mar 03, 2020 11:47 am Post subject: Re: IIB and MQ - performance issue |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
| Vitor wrote: |
| xcuseme123 wrote: |
| Any other options left to boost say atleast another 10% of performance please?. |
You'll get more bang for your buck optimizing the NFS disc itself; make sure you're using any caching that's available, eliminating any network latency between the disc and the server, that sort of thing. |
Thanks  |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Mar 03, 2020 12:18 pm Post subject: Re: IIB and MQ - performance issue |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| xcuseme123 wrote: |
| I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file. |
What change is performance did you measure increasing q buffer sizes?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Wed Mar 04, 2020 7:10 am Post subject: Re: IIB and MQ - performance issue |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
| bruce2359 wrote: |
| xcuseme123 wrote: |
| I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file. |
What change is performance did you measure increasing q buffer sizes? |
I saw processing rate increased by 300 messages/min for HA cluster and Multi instance.
Also tested the following scenarios-
1. We are using HA cluster and Multi instance. Removed multi instance and made broker to run standalone for two brokers. Noticed all 70000 messages processed in 30min. Broker was installed as single instance(standalone) in NFS folder.
2. Same above load of messages takes 1hr in HA cluster and Multi instance |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Mar 04, 2020 1:51 pm Post subject: Re: IIB and MQ - performance issue |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| xcuseme123 wrote: |
| bruce2359 wrote: |
| xcuseme123 wrote: |
| I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file. |
What change is performance did you measure increasing q buffer sizes? |
I saw processing rate increased by 300 messages/min for HA cluster and Multi instance. |
More precisely, please: What was the q buffer sizes before you changed the qm.ini file? What size did you increase each of the q buffers to?
Were/are the messages all persistent? All non persistent? 50% persistent and 50% non-persistent?
Was this the ONLY change you made before you re-tested?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Mar 04, 2020 5:58 pm Post subject: Re: IIB and MQ - performance issue |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| xcuseme123 wrote: |
| Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?. |
What evidence do you have that I/O to your disk is the bottleneck?
Please don't say that is the only difference between the systems as it is almost surely not.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Wed Mar 04, 2020 11:31 pm Post subject: Re: IIB and MQ - performance issue |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
| bruce2359 wrote: |
| xcuseme123 wrote: |
| bruce2359 wrote: |
| xcuseme123 wrote: |
| I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file. |
What change is performance did you measure increasing q buffer sizes? |
I saw processing rate increased by 300 messages/min for HA cluster and Multi instance. |
More precisely, please: What was the q buffer sizes before you changed the qm.ini file? What size did you increase each of the q buffers to?
Were/are the messages all persistent? All non persistent? 50% persistent and 50% non-persistent?
Was this the ONLY change you made before you re-tested? |
qm.ini was not having any entry for q buffer sizes. I have increased it to 100MB
All messages were persistent.
Yes , I have tested with above change only. |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Thu Mar 05, 2020 1:41 am Post subject: Re: IIB and MQ - performance issue |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
| PeterPotkay wrote: |
| xcuseme123 wrote: |
| Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?. |
What evidence do you have that I/O to your disk is the bottleneck?
Please don't say that is the only difference between the systems as it is almost surely not. |
dont see any IO wait in nmon generated file. I see that PROC- BLOCKED graph is generated and reached upto 2.2 for 1.5 hours. Am unable to upload nmon graphs here. Please let me know how I can share it here? |
|
| Back to top |
|
|
| LJM |
| Posted: Fri Mar 06, 2020 2:17 am Post subject: |
|
|
Novice
Joined: 05 Jul 2018
Posts: 22
|
Design change time ?
perhaps use QMGR on same server as IIB ? - local connections.
for HA use 2 brokers, and put the QMGRS into an MQ_CLUS , wath a gateway for message deliuvery into the solution |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Fri Mar 06, 2020 5:54 am Post subject: Re: IIB and MQ - performance issue |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
| xcuseme123 wrote: |
| PeterPotkay wrote: |
| xcuseme123 wrote: |
| Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?. |
What evidence do you have that I/O to your disk is the bottleneck?
Please don't say that is the only difference between the systems as it is almost surely not. |
dont see any IO wait in nmon generated file. I see that PROC- BLOCKED graph is generated and reached upto 2.2 for 1.5 hours. Am unable to upload nmon graphs here. Please let me know how I can share it here? |
Broker-1 and Broker-2 are started crashing during end-end test full load test. Earlier we have tested isolation testing with same data with peak load.
Below java dump error:
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TICHARSET UTF-8
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
1TIDATETIME Date: 2020/03/05 at 12:34:22
1TIFILENAME Javacore filename: /var/mqsi/common/errors/javacore.20200305.123421.32517.0003.txt
1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)
NULL ------------------------------------------------------------------------
0SECTION GPINFO subcomponent dump routine
NULL ================================
2XHOSLEVEL OS Level : Linux 3.10.0-693.el7.x86_64
2XHCPUS Processors -
3XHCPUARCH Architecture : amd64
3XHNUMCPUS How Many : 1
3XHNUMASUP NUMA is either not supported or has been disabled by user
NULL
1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
NULL
NULL ------------------------------------------------------------------------
0SECTION ENVINFO subcomponent dump routine
NULL =================================
1CIJAVAVERSION JRE 1.7.0 Linux amd64-64 build (pxa6470_27sr3fp10-20150708_01(SR3 FP10) )
1CIVMVERSION VM build R27_Java727_SR3_20150630_2236_B255653
1CIJITVERSION tr.r13.java_20150623_94888.01
1CIGCVERSION GC - R27_Java727_SR3_20150630_2236_B255653_CMPRSS
1CIJITMODES JIT enabled, AOT enabled, FSD enabled, HCR enabled
1CIRUNNINGAS Running as a standalone JVM
1CIPROCESSID Process ID: 32517 (0x7F05)
1CICMDLINE
1CIJAVAHOMEDIR Java Home Dir: /opt/IBM/iib-10.0.0.3/common/jdk/jre
1CIJAVADLLDIR Java DLL Dir: /opt/IBM/iib-10.0.0.3/common/jdk/jre/bin
1CISYSCP Sys Classpath: /opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs/jclSC170/vm.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/se-service.jar;/o
pt/IBM/iib-10.0.0.3/common/jdk/jre/lib/math.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/jlm.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmorb.jar;/opt/IBM/iib-10.0.0
.3/common/jdk/jre/lib/ibmorbapi.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmcfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmpkcs.jar;/opt/IBM/iib-10.0.0.3/commo
n/jdk/jre/lib/ibmcertpathfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjgssfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjssefw.jar;/opt/IBM/iib-10.0.0.3/comm
on/jdk/jre/lib/ibmsaslfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjcefw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjgssprovider.jar;/opt/IBM/iib-10.0.0.3/co
mmon/jdk/jre/lib/ibmjsseprovider2.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmcertpathprovider.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/xmldsigfw.jar;/opt/IBM/i
ib-10.0.0.3/common/jdk/jre/lib/xml.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/charsets.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/resources.jar;/opt/IBM/iib-10.0.0.
3/common/jdk/jre/lib/rt.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/dataaccess.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmgpu.jar;
1CIUSERARGS UserArgs:
2CIUSERARG -Xoptionsfile=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs/options.default
2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harm
2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harm
ony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple
2CIUSERARG -Xjcl:jclse7b_27
2CIUSERARG -Dcom.ibm.oti.vm.bootstrap.library.path=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64
2CIUSERARG -Dsun.boot.library.path=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64
2CIUSERARG -Djava.library.path=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64:/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/classic:/opt/IBM/iib-10.0.0.3/ie02/lib:/var/mqsi/extensions/10.0.0/server/lib:/var/mqsi/extensions/10.0.0/lib:/opt/IBM/iib-10.0.0.3/server/xml4c/lib:/opt/IBM/iib-10.0.0.3/server/lib:/opt/IBM/iib-10.0.0.3/server/bin:/opt/IBM/iib-10.0.0.3/server/ODBC/drivers/lib:/opt/IBM/iib-10.0.0.3/server/xlxpc/lib:/opt/IBM/iib-10.0.0.3/server/dfdlc/lib:/var/mqsi/extensions/10.0.0/server/lil:/opt/IBM/iib-10.0.0.3/server/lil:/opt/IBM/iib-10.0.0.3/server/jplugin:/opt/mqm/java/lib:/usr/lib
2CIUSERARG -Djava.home=/opt/IBM/iib-10.0.0.3/common/jdk/jre
2CIUSERARG -Djava.ext.dirs=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ext
2CIUSERARG -Duser.dir=/var/mqsi
2CIUSERARG -Djava.runtime.version=pxa6470_27sr3fp10-20150708_01 (SR3 FP10)
2CIUSERARG -Xmx268435456
2CIUSERARG -Xss1048576
2CIUSERARG -Xmso1048576
2CIUSERARG -Djava.class.path=/opt/IBM/iib-10.0.0.3/server/classes/bipbroker.jar:/opt/IBM/iib-10.0.0.3/server/classes/cmnCon.jar:
2CIUSERARG -Dmqsi.64bit.build=1
2CIUSERARG -Xjit:exclude={com/ibm/p8/engine/core/types/PHPInteger*}
2CIUSERARG -Dcom.ibm.CORBA.UseSchemaIDR=false
2CIUSERARG -Dbroker.isHAEnabled=Yes
2CIUSERARG -Djava.endorsed.dir=/var/mqsi/common/profiles
2CIUSERARG -Djava.security.manager=com.ibm.broker.plugin.security.PluginSecurityManager
2CIUSERARG -Djava.library.path=/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/classic:/opt/IBM/iib-10.0.0.3/ie02/lib:/var/mqsi/extensions/10.0.0/server/lib:/var/mqsi/extensions/10.0.0/lib:/opt/IBM/iib-10.0.0.3/server/xml4c/lib:/opt/IBM/iib-10.0.0.3/server/lib:/opt/IBM/iib-10.0.0.3/server/bin:/opt/IBM/iib-10.0.0.3/server/ODBC/drivers/lib:/opt/IBM/iib-10.0.0.3/server/xlxpc/lib:/opt/IBM/iib-10.0.0.3/server/dfdlc/lib:/var/mqsi/extensions/10.0.0/server/lil:/opt/IBM/iib-10.0.0.3/server/lil:/opt/IBM/iib-10.0.0.3/server/jplugin:/opt/mqm/java/lib
2CIUSERARG -Xrs:sync
2CIUSERARG -Xdump:heap:events=user
2CIUSERARG -Xdump:snap:events=user
1CIUSERLIMITS User Limits (in bytes except for NOFILE and NPROC)
NULL ------------------------------------------------------------------------
NULL type soft limit hard limit
2CIUSERLIMIT RLIMIT_AS unlimited unlimited
2CIUSERLIMIT RLIMIT_CORE 0 unlimited
2CIUSERLIMIT RLIMIT_CPU unlimited unlimited
2CIUSERLIMIT RLIMIT_DATA unlimited unlimited
2CIUSERLIMIT RLIMIT_FSIZE unlimited unlimited
2CIUSERLIMIT RLIMIT_LOCKS unlimited unlimited
2CIUSERLIMIT RLIMIT_MEMLOCK 65536 65536
2CIUSERLIMIT RLIMIT_NOFILE 10240 10240
2CIUSERLIMIT RLIMIT_NPROC 4096 8192
2CIUSERLIMIT RLIMIT_RSS unlimited unlimited
2CIUSERLIMIT RLIMIT_STACK 8388608 unlimited
2CIUSERLIMIT RLIMIT_MSGQUEUE 819200 819200
2CIUSERLIMIT RLIMIT_NICE 0 0
2CIUSERLIMIT RLIMIT_RTPRIO 0 0
2CIUSERLIMIT RLIMIT_SIGPENDING 63462 63462 |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Mar 06, 2020 6:00 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| 1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received |
Your e.g. 's JVM is running out of memory. you will need to check if that is due to message size or memory leak in your JCN.
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| xcuseme123 |
| Posted: Fri Mar 06, 2020 6:46 am Post subject: |
|
|
Apprentice
Joined: 30 Jun 2013
Posts: 33
|
mqsichangeproperties <broker> -b agent -n jvmMinHeapSize -o ComIbmJVMManager -v 1073741824
mqsichangeproperties IDS3BRK1 -b agent -n jvmMaxHeapSize -o ComIbmJVMManager -v 3221225472
we have increase broker heap size to 1gb to 3 gb. Still it didnot help and got out of memory error. Server level we have 16GB RAM , it didnot use 100% RAM. We have seen IO wait for long time , may be it was struggling to write to NFS. Any suggestion please?.. |
|
| Back to top |
|
|
|
|
