| Author |
Message
|
| jonow |
 Posted: Mon Apr 12, 2010 9:01 pm Post subject: MQ Performance question, re channel throughput maxes @25MB/s |
 |
|
Newbie
Joined: 12 Apr 2010
Posts: 8
|
We have a scenario where we have approx 28GByte of data to send from one Qmgr to another Qmgr on AIX. The data is produced at approx 80MByte/s. The definition is as follows
qremote -> xmit -> sdrchl -> recvchl -> qlocal.
parameters:
msg persistence no, maxmsgl 4MByte, batchsz 500
extra tuning parameters
CHANNELS:
MQIBindType=FASTPATH
TuningParameters:
DefaultQBufferSize=10485760
DefaultPQBufferSize=10485760
each MQPut is approx 500KBytes, and 1 MQ Commit every 128 MQPuts.
So our issue is that while we can MQPUT at 80MBytes/s, the channel is only sending about 22MByte/s. This is a gigabit ethernet connection, with only 1ms ping between qmgrs.
We have tried using multiple remoteqs -> xmitq -> chl to a single localq at the second server. I.e.
qremote1 -> xmit1 -> sdrchl1 -> recvchl1 -> qlocal
qremote2 -> xmit2 -> sdrchl2 -> recvchl2 -> qlocal
qremote3 -> xmit3 -> sdrchl3 -> recvchl3 -> qlocal
However we had two issues. We only saw marginal increase in aggregated throughput into the qlocal on the second server. (about 25MByte/s).
- Should we be seeing higher throughput? Or will we need to publish into separate qlocals at the second server to really see any scaling?
The second is issue was that despite the higher aggregated throughput, our application on average spent much more time in MQGET calls, because the messages were not ordered. When messages are not ordered, the application skips to a MQGET based on msgid, but this seemed to hang for periods of a time, presumably because MQ was trying to order stuff under the covers.
- On AIX, I understand there is an implicit index. What is the impact of calling MQGET by msgid rather than an ordered read?
Thanks for your help.
Jono |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Apr 12, 2010 9:32 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
On your first issue, I'd wonder about disc usage as the messages are being put onto the target local queue. It does sound like you have a bottleneck. Unless you've proved conclusively it's the MCA that's chewing resources?
On the second issue, I'm not aware of any index implicit or otherwise on AIX. So on the face of it the queue manager will be scanning the entire queue looking for the msg id. You also refer to the messages being "ordered"; do you mean the messages must be read in order and to do this you're setting the message id to a sequence number or similar?
Setting the message id is never a good idea. Neither is message affinity, where one message relies on another. If you must do this, use message grouping and logical order. And allow the queue manager to set message id.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jonow |
| Posted: Mon Apr 12, 2010 9:51 pm Post subject: |
|
|
Newbie
Joined: 12 Apr 2010
Posts: 8
|
Thanks for the comments. Yes, we generate our own msgid, I should have mentioned that.
I have seen very high disk usage, but I thought this was because the MCA throughput was slow, causing messages in the XMIT to be staged, and not the other way around. I need to look into this more.
Lets assume I have sufficient disk i/o. Should I see a linear increase in throughput as i add multiple MCAs (lets ignore how I try and remove them from the queue at the second server)? Or does it not really scale in such a manner. These are big grunty power6 aix servers, so I expected higher throughput than what I am seeing. |
|
| Back to top |
|
|
| vennela |
| Posted: Mon Apr 12, 2010 10:41 pm Post subject: NPMSPEED ? |
|
|
Jedi Knight
Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India
|
| The NPMSPEED attribute on both the channels are set to FAST I assume |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Apr 13, 2010 4:43 am Post subject: Re: NPMSPEED ? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| vennela wrote: |
| The NPMSPEED attribute on both the channels are set to FAST I assume |
Assuming the messages are non-persistent of course 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 13, 2010 5:24 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jonow wrote: |
| Yes, we generate our own msgid, I should have mentioned that. |
Just assume I'm put in the usual objections & warnings about doing that. 
| jonow wrote: |
| I have seen very high disk usage, but I thought this was because the MCA throughput was slow, causing messages in the XMIT to be staged, and not the other way around. I need to look into this more. |
It should be fairly clear; xmitq i/o is on the sending box, local queue i/o is on the recieving box.
| jonow wrote: |
| Lets assume I have sufficient disk i/o. Should I see a linear increase in throughput as i add multiple MCAs (lets ignore how I try and remove them from the queue at the second server)? Or does it not really scale in such a manner. These are big grunty power6 aix servers, so I expected higher throughput than what I am seeing. |
At an OS level, the MCA is an application writing data to a disc file. So in concept it scales like any other solution where multiple applications are writing to a single disc, i.e. not all that well even with loads of i/o. In this solution you've also got the queue manager trying to keep tabs on the file (like working out which of the physically stored messages is logically next to be delivered), writing out message logs if these messages and persistent and simultaniously beating itself to death scanning the queue of a given message id. That's a lot of i/o activity on a single file system (assuming you've followed best practice and hold the message logs on a different physical disc).
If this implicit index you've heard of that I haven't exists then presumably the queue manager will be rebuilding this constantly as well.
Big grunty servers don't help you in this sort of i/o intensive operation except to move the bottleneck firmly onto the disc.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Tue Apr 13, 2010 5:46 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| What tool/method are you using to support the 22MB throughput claim? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Apr 13, 2010 6:15 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Did I miss a description of the physical channel between the sender and receiver?
Generally, the job of a sysadmin is to remove one bottleneck, and to replace it with another. It's more problem-exchange than problem-resolution.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sumit |
| Posted: Tue Apr 13, 2010 6:22 am Post subject: Re: NPMSPEED ? |
|
|
Partisan
Joined: 19 Jan 2006
Posts: 398
|
| exerk wrote: |
| vennela wrote: |
| The NPMSPEED attribute on both the channels are set to FAST I assume |
Assuming the messages are non-persistent of course |
They are non-persistent.
| rtsujimoto wrote: |
| msg persistence no, maxmsgl 4MByte, batchsz 500 |
What is here-
| rtsujimoto wrote: |
msg persistence no, maxmsgl 4MByte, batchsz 500
extra tuning parameters
CHANNELS:
MQIBindType=FASTPATH
TuningParameters:
DefaultQBufferSize=10485760
DefaultPQBufferSize=10485760
each MQPut is approx 500KBytes, and 1 MQ Commit every 128 MQPuts. |
So each commit is creating a 64MB message but queue manager is handling 4MB. Might be the queue manager is splitting them into smaller messages and sending as a batch. This may cause extra time both at sending and at receiving end. 
_________________
Regards
Sumit |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Apr 13, 2010 6:37 am Post subject: Re: NPMSPEED ? |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| sumit wrote: |
| exerk wrote: |
| Assuming the messages are non-persistent of course |
They are non-persistent.
| rtsujimoto wrote: |
| msg persistence no, maxmsgl 4MByte, batchsz 500 |
|
Good spot! That'll teach me to word-search on a part string next time 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Apr 13, 2010 6:45 am Post subject: Re: NPMSPEED ? |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| exerk wrote: |
| sumit wrote: |
| exerk wrote: |
| Assuming the messages are non-persistent of course |
They are non-persistent.
| rtsujimoto wrote: |
| msg persistence no, maxmsgl 4MByte, batchsz 500 |
|
Good spot! That'll teach me to word-search on a part string next time |
"msg persistence no" doesn't answer the question: are the messages persistent or non-persistent? A dump of an MQMD would be useful, wouldn't it?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| jonow |
| Posted: Tue Apr 13, 2010 8:39 am Post subject: |
|
|
Newbie
Joined: 12 Apr 2010
Posts: 8
|
Thank you all for your helpful insights.
| Quote: |
| What tool/method are you using to support the 22MB throughput claim? |
nmon
| Quote: |
| "msg persistence no" doesn't answer the question: are the messages persistent or non-persistent? A dump of an MQMD would be useful, wouldn't it? |
messages are non-persistent
| Quote: |
At an OS level, the MCA is an application writing data to a disc file. So in concept it scales like any other solution where multiple applications are writing to a single disc, i.e. not all that well even with loads of i/o. In this solution you've also got the queue manager trying to keep tabs on the file (like working out which of the physically stored messages is logically next to be delivered), writing out message logs if these messages and persistent and simultaniously beating itself to death scanning the queue of a given message id. That's a lot of i/o activity on a single file system (assuming you've followed best practice and hold the message logs on a different physical disc).
If this implicit index you've heard of that I haven't exists then presumably the queue manager will be rebuilding this constantly as well.
Big grunty servers don't help you in this sort of i/o intensive operation except to move the bottleneck firmly onto the disc. |
Thanks, I guess I need to consult with the system builder to find out how I can reduce the I/O bottleneck.
| Quote: |
| The NPMSPEED attribute on both the channels are set to FAST I assume |
yes
| Quote: |
| On the second issue, I'm not aware of any index implicit or otherwise on AIX. So on the face of it the queue manager will be scanning the entire queue looking for the msg id. You also refer to the messages being "ordered"; do you mean the messages must be read in order and to do this you're setting the message id to a sequence number or similar? |
Indeed, after contact IBM support, I was informed that there is an index on correlid, not msgid. Bummer! |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Apr 13, 2010 9:27 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jonow wrote: |
| Indeed, after contact IBM support, I was informed that there is an index on correlid, not msgid. Bummer! |
On the plus side, you've told me something I didn't know. Cold comfort for you I accept but it is something. Did they provide a documentation link that explains it / identifies it's existence?
On the minus side, this does also show why you're getting so much i/o reading by message id if the index is on correl id. I repeat my advice to look into logical order and message grouping; work with the product, not against it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Apr 13, 2010 11:12 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Last but not least did you check that your NIC card have the most effective settings? What's the point of a GB ethernet if your NIC cards are configured for 10T (half duplex)? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Apr 13, 2010 12:43 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
I repeat: Did I miss a description of the physical channel between the sender and receiver?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
