| Author |
Message
|
| mqdev |
 Posted: Tue Dec 02, 2008 10:29 am Post subject: Multiple QMs on a Windows Server- performance considerations |
 |
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Hello,
We have a Windows 2003 Development server which hosts about a dozen Queue Managers (set of 4 QMs for 3 different environments). The box itself has high horsepower with equally high resources (RAM, Hard disk etc). However the MQ performance has been S-L-O-W. Just to give an idea, MQExplorer takes about 4 minutes to connect to all QMs and display their individual settings. Windows Task Manager shows the box to be ideal 95% of the time. Memory utilization isn't also indicating any heavy usage.
One of the guideline with multiple QMs on same Windows server is to have the individual Queue Manager MQ folders on different drives for better performance which cannot be done here (all the MQ folders for the QMs are on D:\ drive and this cannot be changed).
Given this scenario, how can we get a better MQ performance? We are wondering if there are perhaps other places we can look (say for example, increase heapsize - if yes, how to go about this on Windows). Am fuzzy as to where the bottleneck is, given that System is idle most of the time and there are no noticable memory contentions...any help or pointers as to where to look for signs of trouble will be greatly appreciated.
Thanks
-mqdev |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Dec 02, 2008 11:38 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| mqdev |
| Posted: Tue Dec 02, 2008 2:01 pm Post subject: |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Thanks Bruce. Except for the Drives, I do not find any aspect that has a bearing on the Performance (and like I mentioned earlier, I cannot change this as we have no control on this Application Development Server). One other parameter is regarding the Log Writes - they are TripleWrite at the moment - can change that to SingleWrite. This however has impact (that too just marginal as it effects only the last 4k write) only if there are heavy reads&writes...
I have opened a PMR with IBM...will update the findings.
Thanks
-mqdev |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 02, 2008 2:05 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Is your only symptom the time it takes for WMQ Explorer to connect?
Are there any other symptoms?
What has changed on the o/s?
Anything in the error logs worth mentioning?
Has your Windows performance and tuning specialist identified anything Windows-ish that conicides with this issue?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqdev |
| Posted: Tue Dec 02, 2008 2:21 pm Post subject: |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Bruce,
Anything to do with MQ is slow. Apps are taking a long time to enqueue/dequeue messages just like MQExplorer is taking a long time to connect & display the QM settings.
No smoking guns in MQ logs either nor are any FDCs being generated. The only external symptom we are seeing is that MQ related Apps are slow - but all are functioning normally (if the delay part is ignored that is).
Am checking with the Windows systems engineer to see if something can be found. Also there are no changes in the OS recently.
Thanks
-mqdev |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 02, 2008 2:29 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
So, nothing new/changed in:
o/s?
hardware?
system software?
application software?
databases?
Response/throughput is degraded for all qmgrs on this box?
What else (other applications) are experiencing this symptom?
Is this a VMWare image?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqdev |
| Posted: Tue Dec 02, 2008 2:48 pm Post subject: |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
So, nothing new/changed in:
o/s? ---------------------------> No
hardware? ---------------------> No
system software? --------------> No
application software? -----------> No
databases? -----------------------> No DBs on this server
Response/throughput is degraded for all qmgrs on this box? ---> Yes.
What else (other applications) are experiencing this symptom? This is an MQ Development server - so the only activity on this server is MQ (QMs & MQ Apps running) which are all impacted.
Is this a VMWare image? ---> This is not a VMWare image.
I just heard back from our Windows System Engineer - he finds nothing wrong on the server. He wanted to checkout the Network interface settings which he plans to do tomorrow. Will update if that happens to be the culprit.
Thanks
-mqdev |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 02, 2008 3:09 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I'm guessing that these development qmgrs are lightly used, yes?
After hours (whichever shift developers do the least amount of developing) do the same symptoms persist?
Try telnet-ing into the box. Execute runmqsc. Is this slow, too?
Can you re-boot, then bring up one qmgr at a time, then try WMQ Explorer, to see at what point the symptom appears?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqdev |
| Posted: Tue Dec 02, 2008 3:34 pm Post subject: |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Bruce,
One or two maybe moderately used with the rest being lightly used.
I have already requested a reboot, which hopefully will be done tonight. This being a busy Dev server, I do not have the luxury of other tasks (bringing up the QMs one at a time etc) that you suggested - will try that over the weekend if the problem persists until then. We are in the middle of a tight development schedule of a high-profile, multi-team project which kind of limits my options as to what I can do on this box.
Thanks for your time and suggestions (please keep them coming!) - I greatly appreciate them!
-mqdev |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Dec 02, 2008 4:40 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
I seem to remember other members talking about the Windows O/S having problems with multiple QMs once you try and run more than like 8 or 9. I don't know if this is official, I certainly can't find anything in the documentation. But I do know this has come up multiple times before. Try doing a search on this site. If I recall correctly its nothing that gobs of CPU and/or memory can overcome, its just a limitation unique to Windows.
I hear you when you say that you just can't shut down QMs, but that would be a good test, to see if the performance of the ones you leave up improves when you shut down some of them.
I really hope you don't have 12 QMs each with just a couple of queues. The preferred way is to have different sets of queues on one QM, i.e. MY.QUEUE.DEV, MY.QUEUE.PTE, MY.QUEUE.QA, MY.QUEUE.PROD, if you must share environments on a single server. On the same server, one QM with 1000 queues will perform better than 10 QMs with 100 queues each.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Dec 02, 2008 4:59 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Agreed. I didn't mean to suggest re-booting the Windows box solely to re-boot. Too often it is 'the solution' to Windows problems. Ugggh.
An iterative problem-determination style is what's called-for here.
I agree completely with Mr. Potkay's suggestion that you shut down one qmgr at a time to see at what level of qmgr instances the symptom decreases and/or disappears. If that doesn't yield any significant results, then reboot and bring up one qmgr at a time until the symptom returns.
You indicate that the box has plenty of horsepower. Out of curiousity, how many cpu's? How much RAM? How is this box configured differently than other Windows qmgr boxes?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqdev |
| Posted: Wed Dec 03, 2008 10:01 am Post subject: |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
Peter,
Thanks for your response. Probably the magic number for the QMs is 9 as the performance apparantly nose-dived after the last QM (ie 9th) was created - will check on that one.
I perfectly agree with you about as few QMs as possible - again this isnt something that I can control. If all else fail, this is what I will propose (I did it already but got a push back as it entails a bit of rework on the App teams) again.
Bruce - this box has 16 GB of RAM configured for PAE on an Intel Xeon with 4 CPUs
Thanks
-mqdev |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 03, 2008 3:29 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
If I remember well this has nothing to do with the memory and other stuff of your machine. This is a limitation of windows. You need to modify the Windows desktop stack size in order to go past this. Can't remember which post said that but you might want to do a search about it.
And thank Mr Gates for limiting you beyond your hardware limitations. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| mqdev |
| Posted: Fri Dec 05, 2008 4:21 pm Post subject: Problem resolved.... |
|
|
Centurion
Joined: 21 Jan 2003
Posts: 136
|
All,
When we rebooted the box, the problem got resolved magically by itself without us needing to make any change!
But before hitting the pwer button, I did try to pull down the QMs one by one - checking each time if the problem is fixed. I did stop 3 out of 9 QMs - still without any visible improvement in the response times.
I know this isn't a very satisfactory solution to the problem - but given the circumstances, it was more important to get the box up & running rather then a detailed pathology.
By the way, IBM suggested the same as fjb_saper (to bump up the heap size). However even this wasn't needed (our plan was to reboot the box and if that doesn't fix it, to increase the heap size).
Thanks for all your suggestions & time,
-mqdev |
|
| Back to top |
|
|
|
|
