Author |
Message
|
bef1 |
Posted: Thu Aug 29, 2019 10:31 am Post subject: Using XA on MQ pub/sub |
|
|
Novice
Joined: 29 Aug 2019 Posts: 15
|
We have a java application that is using Camel as its JMS provider.
The application is subscribing to a topic using XA. It consumes 1 message in the queue, and then closes the XA transaction (each message is part of an XA transaction). Then the application re-attaches itself to the topic to start over the same process for each message.
When there are no messages to process, the client waits for 45 seconds (which is the XA transaction timeout) before closing the request and the XA transaction and starting a new iteration.
I have analyzed how this process is actually being done using a tcpdump, and here is what I found:
Code: |
MQ Client --[XA_START]--> MQ Server
MQ Client <--[XA_START_REPLY]-- MQ Server
MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
MQ Client <--[SPI REPLY]-- MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client <--[NOTIFICATION]-- MQ Server
MQ Client --[MQCLOSE]--> MQ Server
MQ Client <--[MQCLOSE_REPLY]-- MQ Server
MQ Client --[XA_END]--> MQ Server
MQ Client <--[XA_END_REPLY]-- MQ Server
MQ Client --[XA_COMMIT]--> MQ Server
MQ Client <--[XA_COMMIT_REPLY]-- MQ Server |
This is process is on a durable subscription, so each of the iterations (even when there is no message to consume) ends up generating I/O on the MQ server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue).
Having a somewhat high number of subscribers doing this process, the MQ server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even outside of business hours, when there are no messages to consume). This process also consumes ~3 CPU outside of business hours.
I am a bit puzzled that using XA on pub/sub scales so bad, and I am wondering if there is any way to implement this solution without doing so much I/O.[/code] |
|
Back to top |
|
 |
fjb_saper |
Posted: Sat Aug 31, 2019 8:51 am Post subject: |
|
|
 Grand High Poobah
Joined: 18 Nov 2003 Posts: 20756 Location: LI,NY
|
Seems to me there is an obvious flaw somewhere in this setup.
You should attach ONCE to the subscription and then use a JMS Listener or JMS receiver to consume the messages in a loop. Every time you are ready (for each message or batch of messages ) you issue either a JMS session commit or a jms context commit.
The same way if using MQ Java Base. You access the topic ONCE, then loop over the gets with syncpoint and issue the qmgr.commit whenever ready to commit...
In other words, you should not reattach to the topic in the loop.
Verify your loop. If you attach to the qmgr and the topic within the loop you are using an anti-pattern!
 _________________ MQ & Broker admin |
|
Back to top |
|
 |
bef1 |
Posted: Tue Sep 03, 2019 6:46 am Post subject: |
|
|
Novice
Joined: 29 Aug 2019 Posts: 15
|
I was also thinking that this is an anti-pattern at first.
However, the tcpdump that I shared shows that by using XA, the subscription operation occurs inside the XA transaction:
Code: |
MQ Client --[XA_START]--> MQ Server
MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client --[MQCLOSE]--> MQ Server
MQ Client --[XA_END]--> MQ Server
MQ Client --[XA_COMMIT]--> MQ Server |
Since the subscription operation itself is inside the XA transaction, I am not sure if there is a way around this...? |
|
Back to top |
|
 |
Vitor |
Posted: Tue Sep 03, 2019 8:06 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
bef1 wrote: |
Since the subscription operation itself is inside the XA transaction, I am not sure if there is a way around this...? |
You need to post your code. Accepting that Java is not my thing and to me "Camel" is a desert dwelling animal, I would expect the code to subscribe outside of the XA transaction unit of work then loop around processing publications.
I wouldn't expect either MQ or the JMS provider to be controlling the XA boundary the way you're describing. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
bef1 |
Posted: Tue Sep 03, 2019 8:55 am Post subject: |
|
|
Novice
Joined: 29 Aug 2019 Posts: 15
|
I'm not sure posting the code would help - it's not very explicit, as it is using the Camel framework. So the code does not have control over how the underlying framework performs its subscriptions... Maybe I have to post this on the Camel support mailing list. |
|
Back to top |
|
 |
Vitor |
Posted: Tue Sep 03, 2019 10:29 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
bef1 wrote: |
Maybe I have to post this on the Camel support mailing list. |
It can't hurt.
Having become an expert on Camel with the aid of Mr. Google ( ) I notice that this page explicitly calls out considerations for using Camel with XA and with durable subscriptions. I wonder if the Virtual Topics they mention in conjunction with durable subscriptions offer a solution to your issue?
The Camel support mailing list may know. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
bef1 |
Posted: Wed Sep 04, 2019 6:53 am Post subject: |
|
|
Novice
Joined: 29 Aug 2019 Posts: 15
|
I have the feeling that the issue might actually not be Camel itself, but a constaint of XA with subscriptions.
It seems like the "scope" of an XA transaction includes the SUBSCRIBE operation. So that each XA transaction that consumes a message needs to do its subscribe and mqclose inside the XA transaction itself, as shown in the tcpdump trace I shared:
Code: |
MQ Client --[XA_START]--> MQ Server
MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
MQ Client --[REQUEST_MSGS]--> MQ Server
MQ Client --[MQCLOSE]--> MQ Server
MQ Client --[XA_END]--> MQ Server
MQ Client --[XA_COMMIT]--> MQ Server |
|
|
Back to top |
|
 |
Vitor |
Posted: Wed Sep 04, 2019 7:13 am Post subject: |
|
|
 Grand High Poobah
Joined: 11 Nov 2005 Posts: 26093 Location: Texas, USA
|
bef1 wrote: |
I have the feeling that the issue might actually not be Camel itself, but a constaint of XA with subscriptions.
It seems like the "scope" of an XA transaction includes the SUBSCRIBE operation. So that each XA transaction that consumes a message needs to do its subscribe and mqclose inside the XA transaction itself, as shown in the tcpdump trace I shared: |
I think that's why that link I shared was talking about Virtual Topics; so you can subscribe and unsubscribe to an in-memory object rather than the "real" topic. _________________ Honesty is the best policy.
Insanity is the best defence. |
|
Back to top |
|
 |
|