| Author |
Message
|
| phil.h |
 Posted: Wed Jan 24, 2018 10:16 pm Post subject: IIB v.10 performance issue in MQ Output Node - Txn Mode 'No' |
 |
|
Newbie
Joined: 24 Jan 2018
Posts: 7
|
Hi
I have an issue with a subflow used for logging in the MQ Output node where the Transaction Mode is set to 'No'. The reason it is set to 'no' is so that on failure the log message is always generated. In WMB v.8 this function has introduces a 30% overhead to the mainflow's processing. In IIB v.10 this operation introduces a 75% overhead to the mainflow's processing. The requirement is to leave the Transaction Mode set to 'No' - but to reduce the overhead in v.10 so it is comparable to the overhead in WMB v.8. I've logged a case with IBM to determine if this increase is due to a defect in IIB v.10. Was wondering if anyone else had experience this issue?
Cheers
Phil |
|
| Back to top |
|
 |
| adubya |
| Posted: Thu Jan 25, 2018 12:48 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
Have you run a user trace to see if you can identify where the overhead is being incurred ?
We use a similar approach for audit and as far as I know don't encounter such a problem in IIBv10.
Which platform/OS ? And what version of MQ ?
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| Andyh |
| Posted: Fri Jan 26, 2018 10:31 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Is the message persistent or non-persistent.
Distributed MQ is optimized for persistent messages INSIDE syncpoint.
A set of concurrent processes/threads each calling
MQPUT + MQPMO_SYNCPOIUNT ; MQCMIT
with persistent messages will typically perform FAR better than an equivalent set of processes 'simply' calling
MQPUT + MQPMO_NO_SYNCPOINT |
|
| Back to top |
|
|
| phil.h |
| Posted: Sun Jan 28, 2018 11:39 pm Post subject: |
|
|
Newbie
Joined: 24 Jan 2018
Posts: 7
|
Messages are persistent.
New Architecture:
IIB v.10.0.0.9 brokers hosted on Linux RHEL 7.2 servers connect via MQ Endpoint policies to remote MQ 7.5.0.6 QMgrs hosted on AIX 7.1 (which we will be upgrading to MQ 9.0.0.2).
Legacy Architecture we are migrating from:
WMB v.8.0.0.2 brokers hosted on AIX with local MQ v.7.5.0.6 QMgrs.
With our new Architecture - when the following two tests are run with 200,000 msgs being processed by four IIB v.10 Brokers with the only difference being the TransactionMode setting on the MQ Output node we observe the following:
1. When transactionmode='Automatic' - the test completes in 3 minutes 6 seconds
2. When transactionmode='No' - the test completes in 7 minutes 58 seconds
When the same test is run on our Legacy Architecture with transactionmode='No' - the test completes in 2 minutes 53 seconds.
The performance issue seems to only occur under load, so it's difficult to capture in a user trace due to the size of the trace file. |
|
| Back to top |
|
|
| phil.h |
| Posted: Sun Jan 28, 2018 11:59 pm Post subject: |
|
|
Newbie
Joined: 24 Jan 2018
Posts: 7
|
| Adding additional Linux IIB brokers only leads to a slight increase in performance - which may indicate that the bottleneck is a combination of the TransactionMode='No' setting in our logging subflow and the remote QMgr configuration. |
|
| Back to top |
|
|
| adubya |
| Posted: Mon Jan 29, 2018 12:17 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
As has been mentioned above, MQ is optimsed for persistent messages to be PUT within a syncpoint. With out of syncpoint PUTs then the target queue is locked until the MQ log has been updated with the msg data.
There's a good paper on MQ and persistent messaging (Linux but I'm sure a lot of the principles apply cross-platform).
https://ibm-messaging.github.io/mqperf/mqio_v1.pdf
Section 1.3.1.2 being specifically of interest here. The comparative throughput graphs are interesting showing PUT inside/outside of syncpoint.
Can you repeat the test but using non-persistent messages ?
_________________
Independent Middleware Consultant
andy@knownentity.com
Last edited by adubya on Mon Jan 29, 2018 6:24 am; edited 1 time in total |
|
| Back to top |
|
|
| Andyh |
| Posted: Mon Jan 29, 2018 12:35 am Post subject: |
|
|
Master
Joined: 29 Jul 2010
Posts: 239
|
Try running with transactionmode='Automatic' and a significantly greater number of broker threads. I would have thought that MQ could process 200,000 messages in a few seconds (rather than the minutes being reported here).
8 minutes is 480,000 milli-seconds. With no concurrency each transaction will typically result in two forced IO's, suggesting an IO latency or around 1.2ms on your system (not great, but not awful). If that's an accurate guess then you can either increase concurrency, or reduce IO latency to meet your performance objectives. At four broker threads a doubling of throughput in a transactional mode is a reasonable expectation (suggests still IO bound, but with each IO carrying 2 transactions).
With significantly more concurrency it should be possible to get close to 1 IO per 'batch' of transactions and each batch could be satisfying multiple MB of transaction data.
See the MQ performance report support pacs for examples of how MQ persistent message scales with concurrency. |
|
| Back to top |
|
|
| joebuckeye |
| Posted: Mon Jan 29, 2018 6:19 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| adubya wrote: |
As has been mentioned above, MQ is optimsed for persistent messages to be PUT within a syncpoint. With out of syncpoint PUTs then the target queue is locked until the MQ log has been updated with the msg data.
There's a good paper on MQ and persistent messaging (Linux but I'm sure a lot of the principles apply cross-platform).
https://github.com/ibm-messaging/mqperf/blob/master/mqio_v1.pdf
Section 1.3.1.2 being specifically of interest here. The comparative throughput graphs are interesting showing PUT inside/outside of syncpoint. |
It appears that document is not available, I get a 404 from Github trying to access it. I even went to the repo itself and clicked the link there (which matches yours) and got the 404.
I am very curious about this topic as we were recently troubleshooting an issue with a message flow processing a message very quickly but then not pulling the next message for 5-10 seconds. I am wondering if the issue is the persistence outside of a syncpoint. This was also in heavy load conditions. |
|
| Back to top |
|
|
| adubya |
| Posted: Mon Jan 29, 2018 6:23 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
|
| Back to top |
|
|
| joebuckeye |
| Posted: Mon Jan 29, 2018 6:28 am Post subject: |
|
|
Partisan
Joined: 24 Aug 2007
Posts: 365
Location: Columbus, OH
|
| Thanks for the quick response. |
|
| Back to top |
|
|
| adubya |
| Posted: Mon Jan 29, 2018 6:32 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| joebuckeye wrote: |
I am very curious about this topic as we were recently troubleshooting an issue with a message flow processing a message very quickly but then not pulling the next message for 5-10 seconds. I am wondering if the issue is the persistence outside of a syncpoint. This was also in heavy load conditions. |
Not using any disk replication technology by any chance ? And replicating MQ logs and/or queues ?
I ask as we had a performance issue where message transfer would be fine but then "pause" for 10 seconds or so and then resume. We spent ages trying to track down the issue and eventually found it to be the (async) disk replication technology we were using.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Jan 29, 2018 8:46 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
phil h did not specify why they were putting outside of syncpoint. Sometimes you just need to. Interesting would also be the difference of putting outside of syncpoint to a local qmgr vs a remote qmgr....
So may be in his case it might make sense to have a local qmgr for his IIB 10 brokers.... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| adubya |
| Posted: Mon Jan 29, 2018 8:49 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| fjb_saper wrote: |
| phil h did not specify why they were putting outside of syncpoint. |
He did.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| phil.h |
| Posted: Mon Jan 29, 2018 8:09 pm Post subject: |
|
|
Newbie
Joined: 24 Jan 2018
Posts: 7
|
| It looks like the resolution to this performance issue was to create separate queues for each of our brokers for the logging subflow - initially our logging subflows for all brokers put to the same queue on a remote QMgr, and this seems to have caused the performance issue due to the serialized put operations. The separate queues meant that our 4 brokers processed the 200,000 txns in 3min 10sec (as opposed to a single queue which took 8min 30sec). |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Jan 30, 2018 6:27 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
Huh.
Did you make 4 separate Transmission Queues as well? Or do the 4 Remote Queues all resolve to the same Transmit Queue?
Is it repeatable - 4 queues always faster, one queue always slower?
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
|
|
