| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| z/OS WMB Performance Observations |
« View previous topic :: View next topic » |
| Author |
Message
|
| lancelotlinc |
 Posted: Thu Sep 22, 2011 8:37 am Post subject: |
 |
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| sipples wrote: |
| lancelotlinc wrote: |
| I have observed these bottlenecks where the message size is relatively small (a few hundred bytes). With a few messages a second, both platforms perform well (latency ~250 milliseconds). As the TPS ramps past between thirty to around one hundred TPS, the latency explodes on z/OS whereas POWER7 latency creeps up, but stable at 10,000 TPS. I have seen latency go as high as 15 seconds with a steady measurement of 7 seconds on z/OS. The problem is not with Broker, the problem is with the CPU usage per message. z/OS doesn't seem to handle small message sizes well. |
So stop guessing, open a PMR with IBM, follow their instructions about collecting relevant diagnostic information, work the problem, and get back to us with the results, OK?
While you're at it, see if you can find out why that particular I/O-bound run behaved that way in 2008. Maybe AIX has an I/O problem.  |
Well, hopefully the superb resources still left at that client will read this and do the same. I have since moved to a greener pasture.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
 |
| sipples |
| Posted: Thu Sep 22, 2011 8:43 am Post subject: |
|
|
Newbie
Joined: 21 Sep 2011
Posts: 7
|
| zpat wrote: |
| If the WMB message flows are CPU inefficient it's generally cheaper to throw P-series boxes at it. |
No, it's generally cheaper to throw some hours at it and address the inefficient flows. Then, regardless of what platform(s) you run on, you'll have a more cost-effective implementation. For example, a whole person-year devoted to reduce CPU requirements by even one POWER core is probably an excellent investment. |
|
| Back to top |
|
|
| sipples |
| Posted: Thu Sep 22, 2011 9:01 am Post subject: |
|
|
Newbie
Joined: 21 Sep 2011
Posts: 7
|
| lancelotlinc wrote: |
| Well, hopefully the superb resources still left at that client will read this and do the same. I have since moved to a greener pasture. |
Hopefully.
True story: I once spent the better part of a month working with (another) bank troubleshooting a performance problem with a Web application. They were complaining that their z/OS system wasn't providing as much throughput/low latency as an X86 box sitting next to the test mule, both running the same application. Lots of investigation followed, but it turned out z/OS was just fine. However, the network wasn't. The network team swore up and down that the TCP/IP connection between the test mule and z/OS was dedicated and direct. It was anything but. The connection was actually going from the test mule out of the building, over the WAN across half the country, through a firewall or two, back out over another WAN, and then into the mainframe. So we were doing a fantastic job repeatedly demonstrating that our test mule could saturate one of the slow/shared WAN hops. And we had all the z/OS-based evidence you could imagine indicating z/OS wasn't breaking a sweat, but facts and evidence didn't seem to matter. Or maybe they did, because from the bank's past experience if you got the mainframe operators involved they would always eventually find the problem, even if (usually) the problem was somewhere else. I think if the oven in the cafeteria stopped working somebody would probably say, "It must be the mainframe's fault," and the mainframe ops would probably figure out how to fix the oven, too. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Sep 22, 2011 10:59 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| sipples wrote: |
no matter what platform you run on it appears that if you increase the number of message processing instances past about N-1, where N=the number of physical cores available for execution, you shouldn't see much if any benefit for CPU-bound processing.
On mainframes you have considerably broader discretion than on any other platform in selecting how many cores you want for the same (or within a whisker of the same) total price as the single core model.
In other words, if you've got workloads that can benefit from having lots of engines, you might lean more toward the slower but more numerous engine configurations. |
I agree with your statements here. No argument from me. These are sound.
Personally, my character focus is toward the future (ie. I dream alot about possibilities). I am also a realist. This means I know significant investment in legacy technology will not simply go away for the sake of having something newer.
I see the brilliance in POWER7 and the step forward that POWER7 offers as a platform over the legacy z/OS-centric hardware. Every client I have been to in the last two decades has opted to use Broker on AIX after having painfully tried it on z/OS. These decisions to adopt AIX were not mine to make. As an observer, I see a common problem. BMC, State Farm, MasterCard and Bank of America independently have the exact same performance issue.
I would know that they had z/OS performance tuning consultants from IBM consult from time-to-time. Four independent clients with the same performance problem represents to me a trend. To the extent that they abandoned the z/OS platform and went with an AIX Unix solution to solve it. This is truth. Truth speaks volumes.
Not that z/OS couldn't be a good work horse. Just that z/OS is not the ideal for Broker.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Thu Sep 22, 2011 12:06 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
@lancelotlinc
Do you know of anybody running WMB on ZLinux? If yes what is their performance story? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Thu Sep 22, 2011 1:15 pm Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| fjb_saper wrote: |
@lancelotlinc
Do you know of anybody running WMB on ZLinux? If yes what is their performance story? |
The bank was. Same issue. My expertise is not as deep as zpat described a performance expert on z/OS would be, so I am not able to offer definitive causes for the z/OS issues. My hunch is that this issue has more to do with hardware and less to do with the OS.
Again, I can volunteer to assist in developing some methodology that would draw definitive concrete imperical data to point out the root cause. Having observed the problem first hand, I can aid a performance investigative effort that would prove or disprove the problem exists today.
There may already be resolutions that are not yet applied to the client environments at the time I had visibility. The problem may already be solved. Even if it were solved, the production of an apple-to-apple platform performance comparison would benefit IBM customers so they can make good purchasing decisions.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| sipples |
| Posted: Thu Sep 22, 2011 3:19 pm Post subject: |
|
|
Newbie
Joined: 21 Sep 2011
Posts: 7
|
| lancelotlinc wrote: |
| I see the brilliance in POWER7 and the step forward that POWER7 offers as a platform over the legacy z/OS-centric hardware. |
That's jumbled word salad once again. Alternatively, it's so patently obvious it's not particularly interesting.
POWER7 is a CPU, and z/OS is an operating system. I see the brilliance in POWER7 over legacy AIX-centric hardware (POWER4), too. And I see the brilliance in POWER7 over legacy Windows-centric hardware (Pentium II). Newer hardware is great stuff. Throwing newer hardware at a performance issue sometimes solves the performance issue. Sometimes it just papers over it. Your mileage may vary.
Do you have any experience with non-legacy z/OS hardware? Report back to us with z196 data, please. (IBM: You, too.)
And yes, I've seen many, many occasions when otherwise "smart" people haul out a 2003 mainframe and wonder why it doesn't provide compute-centric performance comparable to a machine that's 7 years newer. Like this whole thread. |
|
| Back to top |
|
|
| zpat |
| Posted: Thu Sep 22, 2011 10:51 pm Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
IBM make the hardware for both, and it's based on similar technology which improves all the time. What is different is the operating system. z/OS has had far more development put into it by IBM, than AIX.
Rather pointless comparing the two. Mainframes are the ultimate in reliability, integrity, security, transactional processing and are about processing data. The sophistication of the built-in software features is outstanding.
There is absolutely no reason for them to cost more than P-Series. IBM choose to position them at the top end of the market, because it makes them a lot of money from those who understand and appreciate the advantages of them. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
