ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » Clustering » IIB and MQ - performance issue

Post new topic  Reply to topic Goto page 1, 2  Next
 IIB and MQ - performance issue « View previous topic :: View next topic » 
Author Message
xcuseme123
PostPosted: Mon Mar 02, 2020 7:28 am    Post subject: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

We are using HA clusters and Multi-instance queue managers in Performance enviornment. To support this design, we are using NFS File mount 4.1

We are testing two message flows(pattern-1 & 2) and common flow which does auditing which logs successful//failure in DB. All these 3 flows are deployed into individual EG’s. Below Flow design Pattern-1&2 completes in 1.5 hours in Performance environment using two brokers shares the load. Performance enviornment uses NFS file system as it uses HA clusters and Multi-instance. If we execute the same load having same data completes in 30 mins in DEV environment which local file system.
Flow design pattern 1: Input Queue MQ-->Compute node-->SOAP Request Node --> Compute node-->Publication Node
Flow design pattern 2: Input Queue MQ-->Compute node-->Publication Node

CPU and memory consumptions are normal in DEV and Performance enviornment

IIB : 10.0.0.3
MQ: 8.0.0.4

Server details
NAME="Red Hat Enterprise Linux Server"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"

NFS mount is set with options - rw,sync,no_wdelay,no_root_squash

Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?.
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Mar 02, 2020 9:05 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

xcuseme123 wrote:
Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?.




_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
xcuseme123
PostPosted: Mon Mar 02, 2020 10:43 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

Vitor wrote:
xcuseme123 wrote:
Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?.





Thanks for quick response.

I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file. Any other options left to boost say atleast another 10% of performance please?.
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Mar 02, 2020 11:45 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

xcuseme123 wrote:
Any other options left to boost say atleast another 10% of performance please?.


You'll get more bang for your buck optimizing the NFS disc itself; make sure you're using any caching that's available, eliminating any network latency between the disc and the server, that sort of thing.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
xcuseme123
PostPosted: Tue Mar 03, 2020 11:47 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

Vitor wrote:
xcuseme123 wrote:
Any other options left to boost say atleast another 10% of performance please?.


You'll get more bang for your buck optimizing the NFS disc itself; make sure you're using any caching that's available, eliminating any network latency between the disc and the server, that sort of thing.


Thanks
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Mar 03, 2020 12:18 pm    Post subject: Re: IIB and MQ - performance issue Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

xcuseme123 wrote:
I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file.

What change is performance did you measure increasing q buffer sizes?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
xcuseme123
PostPosted: Wed Mar 04, 2020 7:10 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

bruce2359 wrote:
xcuseme123 wrote:
I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file.

What change is performance did you measure increasing q buffer sizes?


I saw processing rate increased by 300 messages/min for HA cluster and Multi instance.


Also tested the following scenarios-
1. We are using HA cluster and Multi instance. Removed multi instance and made broker to run standalone for two brokers. Noticed all 70000 messages processed in 30min. Broker was installed as single instance(standalone) in NFS folder.
2. Same above load of messages takes 1hr in HA cluster and Multi instance
Back to top
View user's profile Send private message
bruce2359
PostPosted: Wed Mar 04, 2020 1:51 pm    Post subject: Re: IIB and MQ - performance issue Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9394
Location: US: west coast, almost. Otherwise, enroute.

xcuseme123 wrote:
bruce2359 wrote:
xcuseme123 wrote:
I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file.

What change is performance did you measure increasing q buffer sizes?


I saw processing rate increased by 300 messages/min for HA cluster and Multi instance.

More precisely, please: What was the q buffer sizes before you changed the qm.ini file? What size did you increase each of the q buffers to?

Were/are the messages all persistent? All non persistent? 50% persistent and 50% non-persistent?

Was this the ONLY change you made before you re-tested?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Wed Mar 04, 2020 5:58 pm    Post subject: Re: IIB and MQ - performance issue Reply with quote

Poobah

Joined: 15 May 2001
Posts: 7717

xcuseme123 wrote:
Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?.


What evidence do you have that I/O to your disk is the bottleneck?
Please don't say that is the only difference between the systems as it is almost surely not.
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
xcuseme123
PostPosted: Wed Mar 04, 2020 11:31 pm    Post subject: Re: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

bruce2359 wrote:
xcuseme123 wrote:
bruce2359 wrote:
xcuseme123 wrote:
I have tried adding DefaultQBufferSize and DefaultPQBufferSize in qm.ini file.

What change is performance did you measure increasing q buffer sizes?


I saw processing rate increased by 300 messages/min for HA cluster and Multi instance.

More precisely, please: What was the q buffer sizes before you changed the qm.ini file? What size did you increase each of the q buffers to?

Were/are the messages all persistent? All non persistent? 50% persistent and 50% non-persistent?

Was this the ONLY change you made before you re-tested?


qm.ini was not having any entry for q buffer sizes. I have increased it to 100MB

All messages were persistent.

Yes , I have tested with above change only.
Back to top
View user's profile Send private message
xcuseme123
PostPosted: Thu Mar 05, 2020 1:41 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

PeterPotkay wrote:
xcuseme123 wrote:
Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?.


What evidence do you have that I/O to your disk is the bottleneck?
Please don't say that is the only difference between the systems as it is almost surely not.


dont see any IO wait in nmon generated file. I see that PROC- BLOCKED graph is generated and reached upto 2.2 for 1.5 hours. Am unable to upload nmon graphs here. Please let me know how I can share it here?
Back to top
View user's profile Send private message
LJM
PostPosted: Fri Mar 06, 2020 2:17 am    Post subject: Reply with quote

Novice

Joined: 05 Jul 2018
Posts: 22

Design change time ?

perhaps use QMGR on same server as IIB ? - local connections.

for HA use 2 brokers, and put the QMGRS into an MQ_CLUS , wath a gateway for message deliuvery into the solution
Back to top
View user's profile Send private message
xcuseme123
PostPosted: Fri Mar 06, 2020 5:54 am    Post subject: Re: IIB and MQ - performance issue Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

xcuseme123 wrote:
PeterPotkay wrote:
xcuseme123 wrote:
Can NFS file mount system affect performance for HA clusters and Multi-instance queue managers/brokers ?.


What evidence do you have that I/O to your disk is the bottleneck?
Please don't say that is the only difference between the systems as it is almost surely not.


dont see any IO wait in nmon generated file. I see that PROC- BLOCKED graph is generated and reached upto 2.2 for 1.5 hours. Am unable to upload nmon graphs here. Please let me know how I can share it here?


Broker-1 and Broker-2 are started crashing during end-end test full load test. Earlier we have tested isolation testing with same data with peak load.

Below java dump error:
0SECTION TITLE subcomponent dump routine
NULL ===============================
1TICHARSET UTF-8
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received
1TIDATETIME Date: 2020/03/05 at 12:34:22
1TIFILENAME Javacore filename: /var/mqsi/common/errors/javacore.20200305.123421.32517.0003.txt
1TIREQFLAGS Request Flags: 0x81 (exclusive+preempt)
1TIPREPSTATE Prep State: 0x104 (exclusive_vm_access+trace_disabled)
NULL ------------------------------------------------------------------------
0SECTION GPINFO subcomponent dump routine
NULL ================================
2XHOSLEVEL OS Level : Linux 3.10.0-693.el7.x86_64
2XHCPUS Processors -
3XHCPUARCH Architecture : amd64
3XHNUMCPUS How Many : 1
3XHNUMASUP NUMA is either not supported or has been disabled by user
NULL
1XHERROR2 Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
NULL
NULL ------------------------------------------------------------------------
0SECTION ENVINFO subcomponent dump routine
NULL =================================
1CIJAVAVERSION JRE 1.7.0 Linux amd64-64 build (pxa6470_27sr3fp10-20150708_01(SR3 FP10) )
1CIVMVERSION VM build R27_Java727_SR3_20150630_2236_B255653
1CIJITVERSION tr.r13.java_20150623_94888.01
1CIGCVERSION GC - R27_Java727_SR3_20150630_2236_B255653_CMPRSS
1CIJITMODES JIT enabled, AOT enabled, FSD enabled, HCR enabled
1CIRUNNINGAS Running as a standalone JVM
1CIPROCESSID Process ID: 32517 (0x7F05)
1CICMDLINE
1CIJAVAHOMEDIR Java Home Dir: /opt/IBM/iib-10.0.0.3/common/jdk/jre
1CIJAVADLLDIR Java DLL Dir: /opt/IBM/iib-10.0.0.3/common/jdk/jre/bin
1CISYSCP Sys Classpath: /opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs/jclSC170/vm.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/se-service.jar;/o
pt/IBM/iib-10.0.0.3/common/jdk/jre/lib/math.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/jlm.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmorb.jar;/opt/IBM/iib-10.0.0
.3/common/jdk/jre/lib/ibmorbapi.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmcfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmpkcs.jar;/opt/IBM/iib-10.0.0.3/commo
n/jdk/jre/lib/ibmcertpathfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjgssfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjssefw.jar;/opt/IBM/iib-10.0.0.3/comm
on/jdk/jre/lib/ibmsaslfw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjcefw.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmjgssprovider.jar;/opt/IBM/iib-10.0.0.3/co
mmon/jdk/jre/lib/ibmjsseprovider2.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmcertpathprovider.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/xmldsigfw.jar;/opt/IBM/i
ib-10.0.0.3/common/jdk/jre/lib/xml.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/charsets.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/resources.jar;/opt/IBM/iib-10.0.0.
3/common/jdk/jre/lib/rt.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/dataaccess.jar;/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ibmgpu.jar;
1CIUSERARGS UserArgs:
2CIUSERARG -Xoptionsfile=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs/options.default
2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harm
2CIUSERARG -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harm
ony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple
2CIUSERARG -Xjcl:jclse7b_27
2CIUSERARG -Dcom.ibm.oti.vm.bootstrap.library.path=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64
2CIUSERARG -Dsun.boot.library.path=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64
2CIUSERARG -Djava.library.path=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/compressedrefs:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64:/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/classic:/opt/IBM/iib-10.0.0.3/ie02/lib:/var/mqsi/extensions/10.0.0/server/lib:/var/mqsi/extensions/10.0.0/lib:/opt/IBM/iib-10.0.0.3/server/xml4c/lib:/opt/IBM/iib-10.0.0.3/server/lib:/opt/IBM/iib-10.0.0.3/server/bin:/opt/IBM/iib-10.0.0.3/server/ODBC/drivers/lib:/opt/IBM/iib-10.0.0.3/server/xlxpc/lib:/opt/IBM/iib-10.0.0.3/server/dfdlc/lib:/var/mqsi/extensions/10.0.0/server/lil:/opt/IBM/iib-10.0.0.3/server/lil:/opt/IBM/iib-10.0.0.3/server/jplugin:/opt/mqm/java/lib:/usr/lib
2CIUSERARG -Djava.home=/opt/IBM/iib-10.0.0.3/common/jdk/jre
2CIUSERARG -Djava.ext.dirs=/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/ext
2CIUSERARG -Duser.dir=/var/mqsi
2CIUSERARG -Djava.runtime.version=pxa6470_27sr3fp10-20150708_01 (SR3 FP10)
2CIUSERARG -Xmx268435456
2CIUSERARG -Xss1048576
2CIUSERARG -Xmso1048576
2CIUSERARG -Djava.class.path=/opt/IBM/iib-10.0.0.3/server/classes/bipbroker.jar:/opt/IBM/iib-10.0.0.3/server/classes/cmnCon.jar:
2CIUSERARG -Dmqsi.64bit.build=1
2CIUSERARG -Xjit:exclude={com/ibm/p8/engine/core/types/PHPInteger*}
2CIUSERARG -Dcom.ibm.CORBA.UseSchemaIDR=false
2CIUSERARG -Dbroker.isHAEnabled=Yes
2CIUSERARG -Djava.endorsed.dir=/var/mqsi/common/profiles
2CIUSERARG -Djava.security.manager=com.ibm.broker.plugin.security.PluginSecurityManager
2CIUSERARG -Djava.library.path=/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/mqm/java/lib64:/opt/mqm/lib64/compat:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64:/opt/IBM/iib-10.0.0.3/common/jdk/jre/lib/amd64/classic:/opt/IBM/iib-10.0.0.3/ie02/lib:/var/mqsi/extensions/10.0.0/server/lib:/var/mqsi/extensions/10.0.0/lib:/opt/IBM/iib-10.0.0.3/server/xml4c/lib:/opt/IBM/iib-10.0.0.3/server/lib:/opt/IBM/iib-10.0.0.3/server/bin:/opt/IBM/iib-10.0.0.3/server/ODBC/drivers/lib:/opt/IBM/iib-10.0.0.3/server/xlxpc/lib:/opt/IBM/iib-10.0.0.3/server/dfdlc/lib:/var/mqsi/extensions/10.0.0/server/lil:/opt/IBM/iib-10.0.0.3/server/lil:/opt/IBM/iib-10.0.0.3/server/jplugin:/opt/mqm/java/lib
2CIUSERARG -Xrs:sync
2CIUSERARG -Xdump:heap:events=user
2CIUSERARG -Xdump:snap:events=user
1CIUSERLIMITS User Limits (in bytes except for NOFILE and NPROC)
NULL ------------------------------------------------------------------------
NULL type soft limit hard limit
2CIUSERLIMIT RLIMIT_AS unlimited unlimited
2CIUSERLIMIT RLIMIT_CORE 0 unlimited
2CIUSERLIMIT RLIMIT_CPU unlimited unlimited
2CIUSERLIMIT RLIMIT_DATA unlimited unlimited
2CIUSERLIMIT RLIMIT_FSIZE unlimited unlimited
2CIUSERLIMIT RLIMIT_LOCKS unlimited unlimited
2CIUSERLIMIT RLIMIT_MEMLOCK 65536 65536
2CIUSERLIMIT RLIMIT_NOFILE 10240 10240
2CIUSERLIMIT RLIMIT_NPROC 4096 8192
2CIUSERLIMIT RLIMIT_RSS unlimited unlimited
2CIUSERLIMIT RLIMIT_STACK 8388608 unlimited
2CIUSERLIMIT RLIMIT_MSGQUEUE 819200 819200
2CIUSERLIMIT RLIMIT_NICE 0 0
2CIUSERLIMIT RLIMIT_RTPRIO 0 0
2CIUSERLIMIT RLIMIT_SIGPENDING 63462 63462
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Mar 06, 2020 6:00 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

Quote:
1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received

Your e.g. 's JVM is running out of memory. you will need to check if that is due to message size or memory leak in your JCN.

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
xcuseme123
PostPosted: Fri Mar 06, 2020 6:46 am    Post subject: Reply with quote

Apprentice

Joined: 30 Jun 2013
Posts: 33

mqsichangeproperties <broker> -b agent -n jvmMinHeapSize -o ComIbmJVMManager -v 1073741824

mqsichangeproperties IDS3BRK1 -b agent -n jvmMaxHeapSize -o ComIbmJVMManager -v 3221225472

we have increase broker heap size to 1gb to 3 gb. Still it didnot help and got out of memory error. Server level we have 16GB RAM , it didnot use 100% RAM. We have seen IO wait for long time , may be it was struggling to write to NFS. Any suggestion please?..
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » Clustering » IIB and MQ - performance issue
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.