| Author |
Message
|
| neutron |
 Posted: Fri Aug 19, 2011 5:20 am Post subject: MQ 7 Performance Issue |
 |
|
Novice
Joined: 25 Aug 2008
Posts: 21
|
Env: Solaris 10
QMGR : 7.0.1.5 (Linear Logging)
Client : MQ6.0.1 Client
I have upgraded to MQ 7.0.1 for the past 2 months. Recently, we decided to do a load test to test out the performance differences between a MQ6 Client to a MQ6.0.2 Server (Circular Logging) and MQ6 Client to a MQ7 Server
message size : 1kb
However when we increases the load, the performance of MQ7 server was 3 times slower than that of the MQ6.
Initially we suspect that we might have not set the qm.ini properly
hence the following was inserted
1) increase LogFilePages to maximum 65K pages (65535)
2) increase LogBufferPages, maximum value for LogBufferPages is 4096 which is 16MB
3) set LogPrimaryFiles and LogSecondaryFiles to total 511 (maximum).
The performance does not improve.
Next we recreate the QM with circular logging
- only a 5% improvement.
Is there anyone facing the same issue too? Can anyone advise what i can do next? |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Aug 19, 2011 5:24 am Post subject: Re: MQ 7 Performance Issue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| neutron wrote: |
Initially we suspect that we might have not set the qm.ini properly
hence the following was inserted
1) increase LogFilePages to maximum 65K pages (65535)
2) increase LogBufferPages, maximum value for LogBufferPages is 4096 which is 16MB
3) set LogPrimaryFiles and LogSecondaryFiles to total 511 (maximum).
The performance does not improve.
|
That's because nothing much happens. You'll get extra log files if you have transaction / message combinations that require it (assuming you stopped and started the queue manager after changing the ini file) but the other settings can't be changed without deleting and recreating the queue manager.
| neutron wrote: |
| Can anyone advise what i can do next? |
Stop randomly changing queue manager settings and work out exactly where the bottleneck is? For example, why have you fixated on the log as an issue? Why not the file system for the queues? Or the memory available on the 2 servers?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| neutron |
| Posted: Fri Aug 19, 2011 5:32 am Post subject: Re: MQ 7 Performance Issue |
|
|
Novice
Joined: 25 Aug 2008
Posts: 21
|
| Vitor wrote: |
Stop randomly changing queue manager settings and work out exactly where the bottleneck is? For example, why have you fixated on the log as an issue? Why not the file system for the queues? Or the memory available on the 2 servers? |
Thanks for the quick response. Is there a recommended memory for MQ7 on Solaris Sparc?
No of Queues: 300
Avg Connection : 1500
Any settings for Solaris OS required beside adjusting the "proj" for mqm?
To add:
MQ6 env was having the same amount of memory. The logging was changed to linear is because we wanted to set the QM to be able to do a proper DR with logs rollback. However after research and replies from PMR, the answer we gotten was that linear logging has a 60% dropped in performance. Hence with that, we decided to recreate with circular logging to hope that it resolves the 3 times slowness issue. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Aug 19, 2011 5:48 am Post subject: Re: MQ 7 Performance Issue |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
[quote="neutron"...However after research and replies from PMR, the answer we gotten was that linear logging has a 60% dropped in performance. Hence with that, we decided to recreate with circular logging to hope that it resolves the 3 times slowness issue.[/quote]
What??!
Please post the PMR reply here.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 19, 2011 5:49 am Post subject: Re: MQ 7 Performance Issue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| neutron wrote: |
| Is there a recommended memory for MQ7 on Solaris Sparc? |
QB guide for Solaris has the details.
| neutron wrote: |
| Any settings for Solaris OS required beside adjusting the "proj" for mqm? |
Also in the QB, but I'm not familiar with "proj". What's that?
| neutron wrote: |
| However after research and replies from PMR, the answer we gotten was that linear logging has a 60% dropped in performance. |
Research where? How? Was the actual answer from IBM via the PMR that linear logging adds a 60% overhead?
| neutron wrote: |
| Hence with that, we decided to recreate with circular logging to hope that it resolves the 3 times slowness issue. |
I repeat, don't use hope. Don't change stuff to see if it helps, work out where the problem is and fix that. For instance, how have you determined that it's not an I/O bottleneck with the file system?
Also, you keep talking about logging but you've never actually said these messages are persistent (and don't tell me that's what the queues are set to so they must be).
Are you using Zones?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 19, 2011 5:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
And if you are going to continue randomly changing things, why don't you consider randomly changing the mq logs to reside on a separate file system than the queue storage?
Also strongly consider actually reading the Performance Report for Solaris.
Maybe I'm being naive here, but I'd hope that it would tell you about how MQ performs on Solaris. |
|
| Back to top |
|
|
| neutron |
| Posted: Fri Aug 19, 2011 6:43 am Post subject: Re: MQ 7 Performance Issue |
|
|
Novice
Joined: 25 Aug 2008
Posts: 21
|
| Vitor wrote: |
Research where? How? Was the actual answer from IBM via the PMR that linear logging adds a 60% overhead?
|
Reply:
This is are support MP7E WebSphere MQ for Windows V5.3 - Circular and Linear Logging
which have a line that states the following on page 6.
Linear logging reduces performance by up to 60% compared to circular logging, when using SSA disks.
Although this document is not based on the latest version of MQ and not based on Solaris, the comment from our product team states the follow:
"The underlying message is the same for all platforms and releases."
I hope it helps.
Source: http://www-01.ibm.com/support/docview.wss?uid=swg24006436
| Vitor wrote: |
Also, you keep talking about logging but you've never actually said these messages are persistent (and don't tell me that's what the queues are set to so they must be).
Are you using Zones? |
Yes I am using zones for MQ7 Server. But MQ6 server is on the global zone. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 19, 2011 6:45 am Post subject: Re: MQ 7 Performance Issue |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| neutron wrote: |
| Vitor wrote: |
Also, you keep talking about logging but you've never actually said these messages are persistent (and don't tell me that's what the queues are set to so they must be).
Are you using Zones? |
Yes I am using zones for MQ7 Server. But MQ6 server is on the global zone. |
So the WMQv6 server is on the global zone. Where is the WMQv7 server?
And I'm going to keep asking about message persistence until you tell me.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| neutron |
| Posted: Sat Aug 20, 2011 1:15 am Post subject: Re: MQ 7 Performance Issue |
|
|
Novice
Joined: 25 Aug 2008
Posts: 21
|
| Vitor wrote: |
So the WMQv6 server is on the global zone. Where is the WMQv7 server?
And I'm going to keep asking about message persistence until you tell me. |
So sorry i got carried away when you mentioned about the zones. MQ7 is on zones.
Message Persistence is put to yes. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 20, 2011 4:59 am Post subject: Re: MQ 7 Performance Issue |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| neutron wrote: |
This is are support MP7E WebSphere MQ for Windows V5.3 - Circular and Linear Logging which have a line that states the following on page 6.
|
The statement in quotes is NOT in the document. Where exactly did that statement come from?
I am saddened and shocked that you/your organization would rely on a nearly 20yr old document about an 20 yr old unsupported version of MQ, a version that you do not have running.
This statement is from the very beginning of the doc you cite:
| Quote: |
First Edition, June 2003
This edition applies to V1.0 of WebSphere MQ for Windows V5.3 – Performance Evaluations and to all subsequent releases and modifications until otherwise indicated in new editions. |
I repeat Vitor's question: are the messages persistent? NOT the queue attribute - the messages!
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 20, 2011 8:05 am Post subject: Re: MQ 7 Performance Issue |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| neutron wrote: |
| Yes I am using zones for MQ7 Server. But MQ6 server is on the global zone. |
Are you aware of the file system and file speed problems being reported with Solaris zones? Do you have all your V7 files (qmgr, logs, archives) located locally or offloaded to SAN? I suggest you offload them to SAN and report back... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| neutron |
| Posted: Sat Aug 20, 2011 8:56 am Post subject: Re: MQ 7 Performance Issue |
|
|
Novice
Joined: 25 Aug 2008
Posts: 21
|
| bruce2359 wrote: |
The statement in quotes is NOT in the document. Where exactly did that statement come from?
|
From the reply of the support. I believe the support personnel is trying to let me know that there will be performance issue if i have set the MQ to linear logging. This does prompts me to check as my current production environment is in MQ6 with Circular logging. Me/my organization has decided to move on to linear logging due to the fact that only linear logging has the ability to do rollback of logs during a DR.
| bruce2359 wrote: |
I am saddened and shocked that you/your organization would rely on a nearly 20yr old document about an 20 yr old unsupported version of MQ, a version that you do not have running.
|
As mentioned above, the statement came from IBM support. How do i not trust them when i first read the email. Because the results shows otherwise (even after my other test cases), i decided to get help from the forum gurus and experts, like you.
| bruce2359 wrote: |
I repeat Vitor's question: are the messages persistent? NOT the queue attribute - the messages! |
Yes messages are set to persistent. As MQ is used to ensure impt messages are send/consume properly, we set messages to persistent in order to survive any MQ Service Failover due to any issue (H/W / O/S etc).
Nevertheless, i need to thank you for your advice. My team are trying to do more brainstorming and cross-checking (PMR, log case, etc) to ensure that our loadtest are of a certain creditbility (either good or bad) so that we can have more assurance that the LIVE cutover will be a smooth and fearless one. |
|
| Back to top |
|
|
| neutron |
| Posted: Sat Aug 20, 2011 9:01 am Post subject: Re: MQ 7 Performance Issue |
|
|
Novice
Joined: 25 Aug 2008
Posts: 21
|
| fjb_saper wrote: |
Are you aware of the file system and file speed problems being reported with Solaris zones? |
No. Could you advice on any links that i can obtain from anywhere. To my knowledge, there will definitely be performance slowness. (Current Assumption) But i didn't really expect that the performance degradation could be that bad.
| fjb_saper wrote: |
Do you have all your V7 files (qmgr, logs, archives) located locally or offloaded to SAN? I suggest you offload them to SAN and report back... |
SAN is too luxury for a development env. However, i will still try to move MQ7 Server to the global zone to do another round of test to see the difference. PROD env is running MQ on SAN. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 20, 2011 11:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Search the different posts from SAFraser. She is the one who experienced this problem first...
http://www.mqseries.net/phpBB/viewtopic.php?t=55094&highlight=
| Quote: |
Root cause has been determined.
The zfs filesystem for sparse zones does synchronous writes to disk. First data is written to buffer, then to disk. Control is not returned to the application until the write to disk is successful. In zfs, this introduces noticeable latency.
We turned off the synchronous write flag on the sparse zone and the performance issue was completely corrected.
Unfortunately, running without synchronous write protection is not feasible. If the server crashes with messages in the buffer, we believe that the queue manager would not come back up as its logs would not reflect correct information.
For the short term, we may try to move /var/mqm/* to SAN storage. Longer term, we will rebuild everything on global zones.
I am surprised no one has posted about this before. Surely we can't be the only site that has noticed this problem! |
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Sat Aug 20, 2011 12:22 pm Post subject: Re: MQ 7 Performance Issue |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| neutron wrote: |
| bruce2359 wrote: |
The statement in quotes is NOT in the document. Where exactly did that statement come from?
|
From the reply of the support. I believe the support personnel is trying to let me know that there will be performance issue if i have set the MQ to linear logging. |
Please post the exact reply in its entirety from IBM Support on this.
I am stunned that IBM would tell you that nothing about WMQ has improved in 20 years.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
