ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportIIB v.10 performance issue in MQ Output Node - Txn Mode 'No'

Post new topicReply to topic Goto page 1, 2  Next
IIB v.10 performance issue in MQ Output Node - Txn Mode 'No' View previous topic :: View next topic
Author Message
phil.h
PostPosted: Wed Jan 24, 2018 10:16 pm Post subject: IIB v.10 performance issue in MQ Output Node - Txn Mode 'No' Reply with quote

Newbie

Joined: 24 Jan 2018
Posts: 7

Hi

I have an issue with a subflow used for logging in the MQ Output node where the Transaction Mode is set to 'No'. The reason it is set to 'no' is so that on failure the log message is always generated. In WMB v.8 this function has introduces a 30% overhead to the mainflow's processing. In IIB v.10 this operation introduces a 75% overhead to the mainflow's processing. The requirement is to leave the Transaction Mode set to 'No' - but to reduce the overhead in v.10 so it is comparable to the overhead in WMB v.8. I've logged a case with IBM to determine if this increase is due to a defect in IIB v.10. Was wondering if anyone else had experience this issue?

Cheers
Phil
Back to top
View user's profile Send private message
adubya
PostPosted: Thu Jan 25, 2018 12:48 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 376
Location: GU12, UK

Have you run a user trace to see if you can identify where the overhead is being incurred ?

We use a similar approach for audit and as far as I know don't encounter such a problem in IIBv10.

Which platform/OS ? And what version of MQ ?
_________________
Independent Middleware Consultant
andy@knownentity.com
Back to top
View user's profile Send private message Send e-mail
Andyh
PostPosted: Fri Jan 26, 2018 10:31 am Post subject: Reply with quote

Disciple

Joined: 29 Jul 2010
Posts: 158

Is the message persistent or non-persistent.
Distributed MQ is optimized for persistent messages INSIDE syncpoint.
A set of concurrent processes/threads each calling
MQPUT + MQPMO_SYNCPOIUNT ; MQCMIT
with persistent messages will typically perform FAR better than an equivalent set of processes 'simply' calling
MQPUT + MQPMO_NO_SYNCPOINT
Back to top
View user's profile Send private message
phil.h
PostPosted: Sun Jan 28, 2018 11:39 pm Post subject: Reply with quote

Newbie

Joined: 24 Jan 2018
Posts: 7

Messages are persistent.

New Architecture:
IIB v.10.0.0.9 brokers hosted on Linux RHEL 7.2 servers connect via MQ Endpoint policies to remote MQ 7.5.0.6 QMgrs hosted on AIX 7.1 (which we will be upgrading to MQ 9.0.0.2).

Legacy Architecture we are migrating from:
WMB v.8.0.0.2 brokers hosted on AIX with local MQ v.7.5.0.6 QMgrs.

With our new Architecture - when the following two tests are run with 200,000 msgs being processed by four IIB v.10 Brokers with the only difference being the TransactionMode setting on the MQ Output node we observe the following:
1. When transactionmode='Automatic' - the test completes in 3 minutes 6 seconds
2. When transactionmode='No' - the test completes in 7 minutes 58 seconds

When the same test is run on our Legacy Architecture with transactionmode='No' - the test completes in 2 minutes 53 seconds.

The performance issue seems to only occur under load, so it's difficult to capture in a user trace due to the size of the trace file.
Back to top
View user's profile Send private message
phil.h
PostPosted: Sun Jan 28, 2018 11:59 pm Post subject: Reply with quote

Newbie

Joined: 24 Jan 2018
Posts: 7

Adding additional Linux IIB brokers only leads to a slight increase in performance - which may indicate that the bottleneck is a combination of the TransactionMode='No' setting in our logging subflow and the remote QMgr configuration.
Back to top
View user's profile Send private message
adubya
PostPosted: Mon Jan 29, 2018 12:17 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 376
Location: GU12, UK

As has been mentioned above, MQ is optimsed for persistent messages to be PUT within a syncpoint. With out of syncpoint PUTs then the target queue is locked until the MQ log has been updated with the msg data.

There's a good paper on MQ and persistent messaging (Linux but I'm sure a lot of the principles apply cross-platform).

https://ibm-messaging.github.io/mqperf/mqio_v1.pdf

Section 1.3.1.2 being specifically of interest here. The comparative throughput graphs are interesting showing PUT inside/outside of syncpoint.

Can you repeat the test but using non-persistent messages ?
_________________
Independent Middleware Consultant
andy@knownentity.com


Last edited by adubya on Mon Jan 29, 2018 6:24 am; edited 1 time in total
Back to top
View user's profile Send private message Send e-mail
Andyh
PostPosted: Mon Jan 29, 2018 12:35 am Post subject: Reply with quote

Disciple

Joined: 29 Jul 2010
Posts: 158

Try running with transactionmode='Automatic' and a significantly greater number of broker threads. I would have thought that MQ could process 200,000 messages in a few seconds (rather than the minutes being reported here).
8 minutes is 480,000 milli-seconds. With no concurrency each transaction will typically result in two forced IO's, suggesting an IO latency or around 1.2ms on your system (not great, but not awful). If that's an accurate guess then you can either increase concurrency, or reduce IO latency to meet your performance objectives. At four broker threads a doubling of throughput in a transactional mode is a reasonable expectation (suggests still IO bound, but with each IO carrying 2 transactions).
With significantly more concurrency it should be possible to get close to 1 IO per 'batch' of transactions and each batch could be satisfying multiple MB of transaction data.
See the MQ performance report support pacs for examples of how MQ persistent message scales with concurrency.
Back to top
View user's profile Send private message
joebuckeye
PostPosted: Mon Jan 29, 2018 6:19 am Post subject: Reply with quote

Partisan

Joined: 24 Aug 2007
Posts: 334
Location: Columbus, OH

adubya wrote:
As has been mentioned above, MQ is optimsed for persistent messages to be PUT within a syncpoint. With out of syncpoint PUTs then the target queue is locked until the MQ log has been updated with the msg data.

There's a good paper on MQ and persistent messaging (Linux but I'm sure a lot of the principles apply cross-platform).

https://github.com/ibm-messaging/mqperf/blob/master/mqio_v1.pdf

Section 1.3.1.2 being specifically of interest here. The comparative throughput graphs are interesting showing PUT inside/outside of syncpoint.


It appears that document is not available, I get a 404 from Github trying to access it. I even went to the repo itself and clicked the link there (which matches yours) and got the 404.

I am very curious about this topic as we were recently troubleshooting an issue with a message flow processing a message very quickly but then not pulling the next message for 5-10 seconds. I am wondering if the issue is the persistence outside of a syncpoint. This was also in heavy load conditions.
Back to top
View user's profile Send private message
adubya
PostPosted: Mon Jan 29, 2018 6:23 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 376
Location: GU12, UK

Apologies, I didn't check the URL worked

This one does.

https://ibm-messaging.github.io/mqperf/mqio_v1.pdf

Link fixed in original post too.
_________________
Independent Middleware Consultant
andy@knownentity.com
Back to top
View user's profile Send private message Send e-mail
joebuckeye
PostPosted: Mon Jan 29, 2018 6:28 am Post subject: Reply with quote

Partisan

Joined: 24 Aug 2007
Posts: 334
Location: Columbus, OH

Thanks for the quick response.
Back to top
View user's profile Send private message
adubya
PostPosted: Mon Jan 29, 2018 6:32 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 376
Location: GU12, UK

joebuckeye wrote:


I am very curious about this topic as we were recently troubleshooting an issue with a message flow processing a message very quickly but then not pulling the next message for 5-10 seconds. I am wondering if the issue is the persistence outside of a syncpoint. This was also in heavy load conditions.


Not using any disk replication technology by any chance ? And replicating MQ logs and/or queues ?

I ask as we had a performance issue where message transfer would be fine but then "pause" for 10 seconds or so and then resume. We spent ages trying to track down the issue and eventually found it to be the (async) disk replication technology we were using.
_________________
Independent Middleware Consultant
andy@knownentity.com
Back to top
View user's profile Send private message Send e-mail
fjb_saper
PostPosted: Mon Jan 29, 2018 8:46 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19660
Location: LI,NY

phil h did not specify why they were putting outside of syncpoint. Sometimes you just need to. Interesting would also be the difference of putting outside of syncpoint to a local qmgr vs a remote qmgr....

So may be in his case it might make sense to have a local qmgr for his IIB 10 brokers....
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
adubya
PostPosted: Mon Jan 29, 2018 8:49 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 376
Location: GU12, UK

fjb_saper wrote:
phil h did not specify why they were putting outside of syncpoint.


He did.
_________________
Independent Middleware Consultant
andy@knownentity.com
Back to top
View user's profile Send private message Send e-mail
phil.h
PostPosted: Mon Jan 29, 2018 8:09 pm Post subject: Reply with quote

Newbie

Joined: 24 Jan 2018
Posts: 7

It looks like the resolution to this performance issue was to create separate queues for each of our brokers for the logging subflow - initially our logging subflows for all brokers put to the same queue on a remote QMgr, and this seems to have caused the performance issue due to the serialized put operations. The separate queues meant that our 4 brokers processed the 200,000 txns in 3min 10sec (as opposed to a single queue which took 8min 30sec).
Back to top
View user's profile Send private message
PeterPotkay
PostPosted: Tue Jan 30, 2018 6:27 am Post subject: Reply with quote

Jedi Council

Joined: 15 May 2001
Posts: 7426

Huh.

Did you make 4 separate Transmission Queues as well? Or do the 4 Remote Queues all resolve to the same Transmit Queue?

Is it repeatable - 4 queues always faster, one queue always slower?
_________________
Peter Potkay
Keep Calm and MQ On
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexWebSphere Message Broker SupportIIB v.10 performance issue in MQ Output Node - Txn Mode 'No'
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.