ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General Discussion » MQ Performance question, re channel throughput maxes @25MB/s

Post new topic  Reply to topic Goto page 1, 2  Next
 MQ Performance question, re channel throughput maxes @25MB/s « View previous topic :: View next topic » 
Author Message
jonow
PostPosted: Mon Apr 12, 2010 9:01 pm    Post subject: MQ Performance question, re channel throughput maxes @25MB/s Reply with quote

Newbie

Joined: 12 Apr 2010
Posts: 8

We have a scenario where we have approx 28GByte of data to send from one Qmgr to another Qmgr on AIX. The data is produced at approx 80MByte/s. The definition is as follows
qremote -> xmit -> sdrchl -> recvchl -> qlocal.

parameters:
msg persistence no, maxmsgl 4MByte, batchsz 500
extra tuning parameters
CHANNELS:
MQIBindType=FASTPATH
TuningParameters:
DefaultQBufferSize=10485760
DefaultPQBufferSize=10485760

each MQPut is approx 500KBytes, and 1 MQ Commit every 128 MQPuts.

So our issue is that while we can MQPUT at 80MBytes/s, the channel is only sending about 22MByte/s. This is a gigabit ethernet connection, with only 1ms ping between qmgrs.

We have tried using multiple remoteqs -> xmitq -> chl to a single localq at the second server. I.e.

qremote1 -> xmit1 -> sdrchl1 -> recvchl1 -> qlocal
qremote2 -> xmit2 -> sdrchl2 -> recvchl2 -> qlocal
qremote3 -> xmit3 -> sdrchl3 -> recvchl3 -> qlocal

However we had two issues. We only saw marginal increase in aggregated throughput into the qlocal on the second server. (about 25MByte/s).

- Should we be seeing higher throughput? Or will we need to publish into separate qlocals at the second server to really see any scaling?

The second is issue was that despite the higher aggregated throughput, our application on average spent much more time in MQGET calls, because the messages were not ordered. When messages are not ordered, the application skips to a MQGET based on msgid, but this seemed to hang for periods of a time, presumably because MQ was trying to order stuff under the covers.

- On AIX, I understand there is an implicit index. What is the impact of calling MQGET by msgid rather than an ordered read?

Thanks for your help.
Jono
Back to top
View user's profile Send private message
Vitor
PostPosted: Mon Apr 12, 2010 9:32 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

On your first issue, I'd wonder about disc usage as the messages are being put onto the target local queue. It does sound like you have a bottleneck. Unless you've proved conclusively it's the MCA that's chewing resources?

On the second issue, I'm not aware of any index implicit or otherwise on AIX. So on the face of it the queue manager will be scanning the entire queue looking for the msg id. You also refer to the messages being "ordered"; do you mean the messages must be read in order and to do this you're setting the message id to a sequence number or similar?

Setting the message id is never a good idea. Neither is message affinity, where one message relies on another. If you must do this, use message grouping and logical order. And allow the queue manager to set message id.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
jonow
PostPosted: Mon Apr 12, 2010 9:51 pm    Post subject: Reply with quote

Newbie

Joined: 12 Apr 2010
Posts: 8

Thanks for the comments. Yes, we generate our own msgid, I should have mentioned that.

I have seen very high disk usage, but I thought this was because the MCA throughput was slow, causing messages in the XMIT to be staged, and not the other way around. I need to look into this more.

Lets assume I have sufficient disk i/o. Should I see a linear increase in throughput as i add multiple MCAs (lets ignore how I try and remove them from the queue at the second server)? Or does it not really scale in such a manner. These are big grunty power6 aix servers, so I expected higher throughput than what I am seeing.
Back to top
View user's profile Send private message
vennela
PostPosted: Mon Apr 12, 2010 10:41 pm    Post subject: NPMSPEED ? Reply with quote

Jedi Knight

Joined: 11 Aug 2002
Posts: 4055
Location: Hyderabad, India

The NPMSPEED attribute on both the channels are set to FAST I assume
Back to top
View user's profile Send private message Send e-mail Visit poster's website
exerk
PostPosted: Tue Apr 13, 2010 4:43 am    Post subject: Re: NPMSPEED ? Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

vennela wrote:
The NPMSPEED attribute on both the channels are set to FAST I assume


Assuming the messages are non-persistent of course
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Apr 13, 2010 5:24 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

jonow wrote:
Yes, we generate our own msgid, I should have mentioned that.


Just assume I'm put in the usual objections & warnings about doing that.


jonow wrote:
I have seen very high disk usage, but I thought this was because the MCA throughput was slow, causing messages in the XMIT to be staged, and not the other way around. I need to look into this more.


It should be fairly clear; xmitq i/o is on the sending box, local queue i/o is on the recieving box.

jonow wrote:
Lets assume I have sufficient disk i/o. Should I see a linear increase in throughput as i add multiple MCAs (lets ignore how I try and remove them from the queue at the second server)? Or does it not really scale in such a manner. These are big grunty power6 aix servers, so I expected higher throughput than what I am seeing.


At an OS level, the MCA is an application writing data to a disc file. So in concept it scales like any other solution where multiple applications are writing to a single disc, i.e. not all that well even with loads of i/o. In this solution you've also got the queue manager trying to keep tabs on the file (like working out which of the physically stored messages is logically next to be delivered), writing out message logs if these messages and persistent and simultaniously beating itself to death scanning the queue of a given message id. That's a lot of i/o activity on a single file system (assuming you've followed best practice and hold the message logs on a different physical disc).

If this implicit index you've heard of that I haven't exists then presumably the queue manager will be rebuilding this constantly as well.

Big grunty servers don't help you in this sort of i/o intensive operation except to move the bottleneck firmly onto the disc.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
rtsujimoto
PostPosted: Tue Apr 13, 2010 5:46 am    Post subject: Reply with quote

Centurion

Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY

What tool/method are you using to support the 22MB throughput claim?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Apr 13, 2010 6:15 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

Did I miss a description of the physical channel between the sender and receiver?

Generally, the job of a sysadmin is to remove one bottleneck, and to replace it with another. It's more problem-exchange than problem-resolution.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
sumit
PostPosted: Tue Apr 13, 2010 6:22 am    Post subject: Re: NPMSPEED ? Reply with quote

Partisan

Joined: 19 Jan 2006
Posts: 398

exerk wrote:
vennela wrote:
The NPMSPEED attribute on both the channels are set to FAST I assume


Assuming the messages are non-persistent of course

They are non-persistent.
rtsujimoto wrote:
msg persistence no, maxmsgl 4MByte, batchsz 500


What is here-
rtsujimoto wrote:
msg persistence no, maxmsgl 4MByte, batchsz 500
extra tuning parameters
CHANNELS:
MQIBindType=FASTPATH
TuningParameters:
DefaultQBufferSize=10485760
DefaultPQBufferSize=10485760

each MQPut is approx 500KBytes, and 1 MQ Commit every 128 MQPuts.

So each commit is creating a 64MB message but queue manager is handling 4MB. Might be the queue manager is splitting them into smaller messages and sending as a batch. This may cause extra time both at sending and at receiving end.
_________________
Regards
Sumit
Back to top
View user's profile Send private message Yahoo Messenger
exerk
PostPosted: Tue Apr 13, 2010 6:37 am    Post subject: Re: NPMSPEED ? Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 6339

sumit wrote:
exerk wrote:
Assuming the messages are non-persistent of course

They are non-persistent.
rtsujimoto wrote:
msg persistence no, maxmsgl 4MByte, batchsz 500


Good spot! That'll teach me to word-search on a part string next time
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Apr 13, 2010 6:45 am    Post subject: Re: NPMSPEED ? Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

exerk wrote:
sumit wrote:
exerk wrote:
Assuming the messages are non-persistent of course

They are non-persistent.
rtsujimoto wrote:
msg persistence no, maxmsgl 4MByte, batchsz 500


Good spot! That'll teach me to word-search on a part string next time

"msg persistence no" doesn't answer the question: are the messages persistent or non-persistent? A dump of an MQMD would be useful, wouldn't it?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
jonow
PostPosted: Tue Apr 13, 2010 8:39 am    Post subject: Reply with quote

Newbie

Joined: 12 Apr 2010
Posts: 8

Thank you all for your helpful insights.

Quote:
What tool/method are you using to support the 22MB throughput claim?
nmon

Quote:
"msg persistence no" doesn't answer the question: are the messages persistent or non-persistent? A dump of an MQMD would be useful, wouldn't it?
messages are non-persistent


Quote:
At an OS level, the MCA is an application writing data to a disc file. So in concept it scales like any other solution where multiple applications are writing to a single disc, i.e. not all that well even with loads of i/o. In this solution you've also got the queue manager trying to keep tabs on the file (like working out which of the physically stored messages is logically next to be delivered), writing out message logs if these messages and persistent and simultaniously beating itself to death scanning the queue of a given message id. That's a lot of i/o activity on a single file system (assuming you've followed best practice and hold the message logs on a different physical disc).

If this implicit index you've heard of that I haven't exists then presumably the queue manager will be rebuilding this constantly as well.

Big grunty servers don't help you in this sort of i/o intensive operation except to move the bottleneck firmly onto the disc.


Thanks, I guess I need to consult with the system builder to find out how I can reduce the I/O bottleneck.

Quote:
The NPMSPEED attribute on both the channels are set to FAST I assume
yes

Quote:
On the second issue, I'm not aware of any index implicit or otherwise on AIX. So on the face of it the queue manager will be scanning the entire queue looking for the msg id. You also refer to the messages being "ordered"; do you mean the messages must be read in order and to do this you're setting the message id to a sequence number or similar?
Indeed, after contact IBM support, I was informed that there is an index on correlid, not msgid. Bummer!
Back to top
View user's profile Send private message
Vitor
PostPosted: Tue Apr 13, 2010 9:27 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

jonow wrote:
Indeed, after contact IBM support, I was informed that there is an index on correlid, not msgid. Bummer!


On the plus side, you've told me something I didn't know. Cold comfort for you I accept but it is something. Did they provide a documentation link that explains it / identifies it's existence?

On the minus side, this does also show why you're getting so much i/o reading by message id if the index is on correl id. I repeat my advice to look into logical order and message grouping; work with the product, not against it.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Tue Apr 13, 2010 11:12 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY

Last but not least did you check that your NIC card have the most effective settings? What's the point of a GB ethernet if your NIC cards are configured for 10T (half duplex)?
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
bruce2359
PostPosted: Tue Apr 13, 2010 12:43 pm    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.

I repeat: Did I miss a description of the physical channel between the sender and receiver?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General Discussion » MQ Performance question, re channel throughput maxes @25MB/s
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.