| Author |
Message
|
| muq1835 |
 Posted: Tue Feb 11, 2014 2:03 pm Post subject: slow performance on message broker 7.0.0.4 |
 |
|
Newbie
Joined: 04 Oct 2012
Posts: 9
|
Hello All,
I am new to MQ/ MB so pleas ehelp me with below issue.
we have recently upgraded software's in our company and is experience a lot of slowness related to message flow processing.
previous landsacpe -->
operating system AIX
Message broker version 6.1.0.8
MQ version 7.0
current landscape
operating system Linux 2.6.32-220.13.1.el6.x86_64
Message broker version 7.0.0.4
MQ version 7.0
i have deployed the same flow which was running in 6.1.0.8 and per my logs it use to process 15000 messages in 40 minson AIX as to 1hr 45 min in linux.
Please let me know if i need any tuning on EG level or broker level.
thank you very much for all you attention. |
|
| Back to top |
|
 |
| Gralgrathor |
| Posted: Wed Feb 12, 2014 12:26 am Post subject: Re: slow performance on message broker 7.0.0.4 |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| muq1835 wrote: |
| Please let me know if i need any tuning on EG level or broker level. |
Well, tuning never hurts, but... it's a bit vague. Broker performance is affected by anything from hardware specs to broker tuning to flow design.
Have the brokers migrated to other (virtual) machines?
Do they get allocated the same amount of CPU, memory and swapspace?
Has anything else changed, such as database versions?
Are any other processes running on the same (virtual) machines that weren't before?
Have any EG's on the broker changed?
Have any been added or moved to other brokers?
Do you have performance issues with any other process on the same hardware/VMs?
Probably best if you can check some of the items on this list before trying to tune the broker. Might be that the difference in speed is due to something not directly related to the broker itself. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Feb 12, 2014 1:06 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Well, if you must change from AIX to Linux (and from P-Series to Intel) ....
Unlikely to be anything in the WMB code, this generally gets faster with newer versions (on the same functions at least).
WMB 7.0.0.6 is the current fixpack level. I personally found 7.0.0.4 a little buggy and would recommend you use a higher version, rather than software that is almost two years out of date.
However your performance issues are much more likely to be environmental that anything else.
Is your new lower-cost system virtualized? What are the CPU usage and page rates etc?
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Feb 12, 2014 5:41 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| zpat wrote: |
| However your performance issues are much more likely to be environmental that anything else. |
I'd look at what's different between the platforms, especially if (as has been metioned) you've gone from physical to virtual.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Feb 12, 2014 9:41 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Start by determining if your flow is CPU bound, I/O bound or network bound. What is it waiting on that is preventing it from going faster - CPU, Disk or Network?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Feb 13, 2014 12:25 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Also memory as some virtualised machines are not given enough real memory so that paging/swapping slows things down.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Thu Feb 13, 2014 12:50 am Post subject: Re: slow performance on message broker 7.0.0.4 |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| muq1835 wrote: |
| it use to process 15000 messages in 40 minson AIX as to 1hr 45 min in linux. |
I know of the rule that AIX is about 70% faster than Linux with the same CPU-Clock (e.g. 1,7 GHZ). Though Linux is cheaper and has other advantages. In practice this means: If you had an AIX cluster with 8 nodes and you switch to Linux... you now need a cluster with what?... 14 nodes? Still better and cheaper probably.
Alternatively tune your flows may help always
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Feb 13, 2014 3:53 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Throughput of a single instance of a flow (a thread) will be limited by the power of one CPU core.
Therefore having more X86 cores may not help (and may cost more in license fees).
AIX has many features that Linux lacks - HA, WLM, self-tuning kernel and so on.
Those AIX development guys know what large Enterprises need.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| muq1835 |
| Posted: Thu Feb 13, 2014 9:54 am Post subject: |
|
|
Newbie
Joined: 04 Oct 2012
Posts: 9
|
Actually i forgot to mention one more thing that we have 2 brokers running parallel on independent machines. these brokers are shared in a cluster to perform round robin mechanism. we observed that the same flow deployed on both the brokers and broker 2 process 7000 requests in like 6mins and broker1 is still holding around 6000 request in the queue during that time. a total of 15000 messages were sent. this never happened in our old environment. if we unshares the queue in broker 1 the wholes process runs in 13 mins while it takes around 45 mins if just process it thru broker 1
This flow has simple EQSL code for converting xml into fixed record. i too beleive its an environment issue but dont know which part may be causing the issue.
New server are in virtual environment.
we have 8 CPU cores each with the below architecture
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
CPU socket(s): 8
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 2700.000
BogoMIPS: 5400.00
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-7
how do i determine this whetehr on what resoruce is the broker waiting on
CPU bound, I/O bound or network bound. What is it waiting on that is preventing it from going faster - CPU, Disk or Network? |
|
| Back to top |
|
|
| muq1835 |
| Posted: Thu Feb 13, 2014 2:49 pm Post subject: |
|
|
Newbie
Joined: 04 Oct 2012
Posts: 9
|
Hello,
I just redeployed the bar file again and things started working lightning fast.
Any idea what could have gone wrong. |
|
| Back to top |
|
|
| mqsiuser |
| Posted: Thu Feb 13, 2014 8:53 pm Post subject: |
|
|
Yatiri
Joined: 15 Apr 2008
Posts: 637
Location: Germany
|
| muq1835 wrote: |
| Any idea what could have gone wrong. |
your initial bar-deployment
zpat, I really like your signature
| Quote: |
| When it really matters - use MQ and not files or web services. |
And your comment(s) on AIX
When developing with broker/mq you are really happy onces the msgs are brought from a file into (an) mq (message).
And web services ... dark age of IT
_________________
Just use REFERENCEs |
|
| Back to top |
|
|
| muq1835 |
| Posted: Fri Feb 14, 2014 5:32 am Post subject: |
|
|
Newbie
Joined: 04 Oct 2012
Posts: 9
|
Actually even if the problem is solved i dont know the reason why it has happened. i have tried to trace all possible places i can and have not find any clue.
i found errors like below when in user.debug.log file
File '/global/site/vendor/MQSI/lil/imbdfjps.lil' could not be loaded; operating system return code '/global/site/vendor/MQSI/lil/imbdfjps.lil:
looks like my team has killed the EG couple of times thinking it would solve the problem which resulted the above error.
Still investigating and waiting for your inputs.  |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Feb 15, 2014 7:43 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
well if the lil file could not be loaded, that might have slowed down things...
A bounce of the broker might have fixed this particular problem... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
