| Author |
Message
|
| tof |
 Posted: Fri Oct 03, 2008 1:23 am Post subject: MQ V7.0 Cluster JMS - PUB/SUB - Low Performance |
 |
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
Hi all,
and thanks in advance for any help you might be able to give.
I've finally setup the scripts to automatically configure a big cluster of MQ V7.0 queue managers for distributed PUB/SUB. The "final" cluster is supposed to be quite big (42 nodes) to assess how well the system scales. I completed some experimental runs on the 42 clusters configuration and I get really low throughput even for an entry-level workload.
The figure below shows an example run with 42 nodes (SUBs are submissions, PUBs publications, GETs are delivered messages, and false negatives are messages that should have been delivered but were not).
The experiment setup is as follows:
- I use just one clustered topic
- each QM has ten clients (one Java process running 10 separate threads)
- each client issues 20 subscriptions
- each client publishes roughly 1 message x second (it's a poissonian process) in NON-Persistent mode making it in total around 420 mess/sec input
- Subscription matching is delegated to the broker (factory.setMessageSelection(JMSC.MQJMS_MSEL_BROKER);)
- the replication grade (that is the average recipients per publication) is 10: expected throughput is around 4200 delivered mess/sec
The obtained throughput is instead much lower at ~500 mess/sec.
I checked that messages flow across the whole network and they do, it just seems a configuration for performance issue.
I have taken a look at the performance reports for V7.0 and setup a much smaller cluster (2 nodes) to repeat the experiment. The figure below shows a correct run for the 2 nodes cluster: no messages are "lost".
But as soon as I use a heavier workload things degenerate even on two nodes and half of the messages get lost.
Reading the performance reports I found a list of things I might do differently:
- use TRUSTED connections
- use FASTPATH
- SHARED BINDINGS (?)
- shareCNV(0)
- check "nice" values of listener processes (I'm running on unix_64)
I will try with those and see if things change, but I also have doubts as to whether I just misconfigured the whole thing. I mean, messages go around correctly, but I am not sure about how many listeners / channels I am supposed to have (the documentation isn't really clear concerning config for performance and listeners).
The current setup is just having one listener on one port that I assume is used both for JMS client connections and QM-to-QM connections. (Am I right?) I guess I just have the typical cluster config:
| Code: |
define listener (SYSTEM.DEFAULT.LISTENER.TCP) trptype (tcp) control (qmgr) port (17002) replace
start listener (SYSTEM.DEFAULT.LISTENER.TCP)
ALTER QMGR
DEFINE CHANNEL(TO.S2) CHLTYPE(CLUSRCVR) +
CONNAME('node37(17002)') CLUSTER(WEEVIL) +
DESCR('TCP Cluster-receiver channel for queue manager S2') TRPTYPE(TCP) REPLACE
DEFINE CHANNEL(TO.S1) CHLTYPE(CLUSSDR) +
CONNAME('node36(17001)') CLUSTER(WEEVIL) +
DESCR('TCP Cluster-sender channel from S2 to repository at S1') TRPTYPE(TCP) REPLACE
DEFINE QLOCAL(WEEVILQ) CLUSTER(WEEVIL)
DEFINE TOPIC(WEEVILTOPIC) CLUSTER(WEEVIL) TOPICSTR(WEEVILTOPIC) PUBSCOPE(ALL) SUBSCOPE(ALL)
|
Is there anyone out there who can tell me if there's anything I should do differently apart from trying the other config tweaks (e.g., fastpath and so on) ?
Thanx,
g |
|
| Back to top |
|
 |
| Michael Dag |
| Posted: Fri Oct 03, 2008 2:13 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
interesting graphs!
what are the other object settings used?
your drop rate is very high, almost like you told MQ to drop everything that could not be delivered instantly...
you could try and use seperate the listeners for QM-to-QM and Clients and see if that makes a difference
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| tof |
| Posted: Fri Oct 03, 2008 2:33 am Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| Michael Dag wrote: |
| interesting graphs! |
 Thank you Michael, and thanx for answering...
| Michael Dag wrote: |
what are the other object settings used?
your drop rate is very high, almost like you told MQ to drop everything that could not be delivered instantly... |
This is an interesting statement: is there a way I can tell MQ to hold on to more messages? I still want to use NON-Persistent delivery though, so I guess all I can do is have bigger queues, right?
Anyhow this would not have an impact on the max system throughput, but just take care of peaks in the load.
I'm not sure about what other object settings I can tell you about, the startup script for the QMs looks like this, all the rest is out-of-the-box:
| Code: |
sudo su - mqm -c "crtmqm S2"
sudo su - mqm -c "strmqm S2"
sudo su - mqm -c "runmqsc S2 < /tmp/testpred-webspheremq/WebsphereMQServer/S2/testpred-webspheremq_S2.conf" >/tmp/testpred-webspheremq/WebsphereMQServer/S2/testpred-webspheremq_S2_listener.log
|
The cluster-shared topic is defined only on QM2 (S2), and Java clients use just that one topic.
| Michael Dag wrote: |
| you could try and use seperate the listeners for QM-to-QM and Clients and see if that makes a difference |
I've tried that but got into this:
http://www.mqseries.net/phpBB2/viewtopic.php?t=45653
Thanx again! |
|
| Back to top |
|
|
| Michael Dag |
| Posted: Fri Oct 03, 2008 3:11 am Post subject: |
|
|
Jedi Knight
Joined: 13 Jun 2002
Posts: 2607
Location: The Netherlands (Amsterdam)
|
What is in /tmp/testpred-webspheremq/WebsphereMQServer/S2/testpred-webspheremq_S2.conf ?
Are you setting up subscriptions to seperate queues or the same, if the latter it could be that only 1 publication is writtten to the destination queue instead of the number of subscriptions? or one subscriber is 'eating' all publications from that queue and others don't get it...
_________________
Michael
MQSystems Facebook page |
|
| Back to top |
|
|
| tof |
| Posted: Fri Oct 03, 2008 4:29 am Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| Michael Dag wrote: |
What is in /tmp/testpred-webspheremq/WebsphereMQServer/S2/testpred-webspheremq_S2.conf ?
Are you setting up subscriptions to seperate queues or the same, if the latter it could be that only 1 publication is writtten to the destination queue instead of the number of subscriptions? or one subscriber is 'eating' all publications from that queue and others don't get it... |
Hi again,
testpred-webspheremq_S2.conf contains the MQ commands shown in my first post (listener, channels, and topic definitions).
I am setting up subscriptions to just one topic (not sure how that maps to queues) but I can confirm that with the "slow" workload all messages are delivered to ALL intended recipients (no subscriber consumes all publications). I am assuming the same thing should happen with a faster workload, but the message loss rate I get is really too high.
All I would need to find out is whether the throughput I get makes sense for pub/sub with Websphere MQ or it's just some misconfiguration. I've tried other MQ implementations and the throughput is order of magnitude higher.
Thank you again,
g |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Oct 03, 2008 2:36 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Have you tried with each subscriber specifying it's own queue for receiving the publications. (You need to do this in the topic definition for the subscriber).
Is each subscriber using it's own connection per topic, or is there some connection sharing being done?
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tof |
| Posted: Fri Oct 03, 2008 11:44 pm Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| fjb_saper wrote: |
Have you tried with each subscriber specifying it's own queue for receiving the publications. (You need to do this in the topic definition for the subscriber).
Is each subscriber using it's own connection per topic, or is there some connection sharing being done?
Enjoy |
Hi!
Below is an extract of the code I use for both publishers and subscribers. Each "actor" runs in a separate thread and builds its own connection per topic. I'll try to set topics so that each subscriber has its own queue, although at the moment I'm not really sure how to do it.
Another big issue I have is I read I cannot use the same session both for publishing and subscribing (from the MQ V7 documentation). So the plan was to use two sessions per actor (1 connection) one for pubs and one for subs. The truth is that as soon as I try to create more than one non-durable subscriber on the same session an exception is thrown saying I cannot change subscriptions for a subscriber (which is not what I'm doing). That's the reason why in the code below I have to create multiple sessions in the subAction() method.
| Code: |
public class WebSphereMQActor implements Actor, MessageListener {
public String server_protocol, server_ip, my_identity;
public int server_port;
int index;
MessageConsumer[] sublist = new MessageConsumer[50];
Filter[] contact admin = new Filter[50];
String sub_type, sub_subject, sub_op, sub_value;
// Date subdate;
// long subtime;
//
// Date unsubdate;
// long unsubtime;
//
// //Notification e;
String pub_type, pub_subject, pub_value;
// Date pubdate;
// long pubtime;
//
public int eventid = 0;
String address, user, password;
String subject = "WEEVILTOPIC";
MQTopicConnection connection = null;
MQTopicSession session;
MQTopic topic;
TopicPublisher producer;
LinkedList lastMessages = new LinkedList();
private TopicSession[] tempSessions = new TopicSession[50];
static SimpleDateFormat sdf = new SimpleDateFormat(
"yyyy-MM-dd HH:mm:ss:SSS Z");
public WebSphereMQActor(String arg) {
// parse arg
String[] configparser = arg.split(" ");
server_port = Integer.parseInt(configparser[1]);
my_identity = configparser[0];
String qmgr = configparser[2];
try {
System.out.println(sdf.format(new Date()) + " " + my_identity
+ " creating connection factory ");
JmsFactoryFactory ff = JmsFactoryFactory
.getInstance(JmsConstants.WMQ_PROVIDER);
// Create the connection.
MQTopicConnectionFactory factory = (MQTopicConnectionFactory) ff
.createTopicConnectionFactory();
// Config
factory.setHostName("localhost");
factory.setPort(server_port); factory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
//factory.setChannel("JAVA.CHANNEL");
// message selection performed by broker factory.setMessageSelection(JMSC.MQJMS_MSEL_BROKER); factory.setMQConnectionOptions(JMSC.MQCNO_FASTPATH_BINDING);
connection = (MQTopicConnection) factory.createTopicConnection();
connection.start();
session = (MQTopicSession) connection.createTopicSession(false, Session.DUPS_OK_ACKNOWLEDGE); // can also be
// Session.AUTO_ACKNOWLEDGE
System.out.println("Obtained session: " + session);
topic = (MQTopic) session.createTopic(subject);
topic.setBaseTopicName(subject);
topic.setTargetClient(JMSC.MQJMS_CLIENT_JMS_COMPLIANT);
System.out.println("Obtained topic: " + topic);
// Create the producer.
producer = session.createPublisher(topic);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
} catch (Exception e) {
System.out.println("Caught: " + e);
e.printStackTrace();
}
}
public void subAction(String arg) throws JMSException {
// dunno why but for some reason websphere doesn't let me create more
// than 1 subscriber per session
// so I keep creating sessions although JMS spec below says something different
/*
* A TopicSession allows the creation of multiple TopicSubscriber
* objects per topic. It will deliver each message for a topic to each
* subscriber eligible to receive it. Each copy of the message is
* treated as a completely separate message. Work done on one copy has
* no effect on the others; acknowledging one does not acknowledge the
* others; one message may be delivered immediately, while another waits
* for its subscriber to process messages ahead of it.
*/
TopicSession localSession = connection.createTopicSession(false,
Session.DUPS_OK_ACKNOWLEDGE);
String query = "";
....
sublist[index] = localSession.createSubscriber(topic, query, true);
sublist[index].setMessageListener(this);
tempSessions[index] = localSession;
System.out.println(sdf.format(new Date()) + " " + my_identity
+ " subscribing for " + index + ": "
+ contact admin[index].toString());
}
public void pubAction(String arg) throws JMSException {
Message message = session.createMessage();
...
producer.publish(message);
}
public void stop() {
System.out.println(my_identity + " stop() method invoked");
for (int i = 0; i < index; i++) {
if (sublist[i] != null) {
try {
sublist[i].close();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
if (tempSessions[i] != null) {
try {
tempSessions[i].close();
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
try {
if (connection != null) {
connection.close();
}
} catch (Throwable ignore) {
}
System.out.println(sdf.format(new Date()) + " " + my_identity
+ " shutdown complete");
}
public void onMessage(Message arg0) {
try{
arg0.acknowledge();
synchronized (this) {
eventid++;
local = eventid;
}
System.out.println(sdf.format(new Date()) + " " + my_identity
+ " getting " + local + ": "
+ arg0.getStringProperty("msgId"));
} catch (JMSException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
|
Sorry for the formatting but the form ain't helping much... 
Thank you again for your suggestion!
g |
|
| Back to top |
|
|
| tof |
| Posted: Fri Oct 03, 2008 11:58 pm Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| fjb_saper wrote: |
Have you tried with each subscriber specifying it's own queue for receiving the publications. (You need to do this in the topic definition for the subscriber).
Is each subscriber using it's own connection per topic, or is there some connection sharing being done?
Enjoy |
Just one more thing: the application has to support any number of subscribers, so if I would have to administratively create a queue for each subscriber this would not do for me...
Thank you again,
g |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Oct 04, 2008 7:35 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Well a few things... If your messages are not persistent and your subscriptions are not durable use a model queue to get a temporary dynamic queue for your subscriptions.
The subscription queue is part of the topic uri. Read up in the manual and experiment with rfhutil to see how to build the right uri: ( Alternative hint use the file based JNDI to specify the topic and look at how it looks once built into a uri). If your subscriptions are durable or messages are persistent, you should have a fixed queue by subscriber anyways. I would expect the subscriber to request the queue from the admin before subscribing...
Now to the juicier part. Review your design. Sharing a connection is not a good thing as messages are single threaded through a connection. This is why the J2EE appservers have connection pooling. I understand that your standalone JMS does not, so you'll just have to take the performance hit of opening a new connection.
This means as well no session sharing as the session partakes of the same connection.
This being out of the way, another hint: If you search some of the papers written by Hursley on MQ you will find one where it states that with multiple consumers hitting the same queue, you will have a definitive performance advantage to have your calls transacted. So please set the transaction attribute on the session to true and commit the session every so often. (Every message, every 10 messages or every 100 messages?). Keep in mind the max uncommitted messages settings for the queue manager which will play against all consumers/producers. I see that you used dups_ok_acknowledge for your session... don't know if that matters on a transacted session, I just use auto_acknowledge...
Hope this helps clear up some of the performance pitfalls you're seeing right now.
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| tof |
| Posted: Sat Oct 04, 2008 7:50 am Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| fjb_saper wrote: |
Well a few things... If your messages are not persistent and your subscriptions are not durable use a model queue to get a temporary dynamic queue for your subscriptions.
The subscription queue is part of the topic uri. Read up in the manual and experiment with rfhutil to see how to build the right uri: ( Alternative hint use the file based JNDI to specify the topic and look at how it looks once built into a uri). If your subscriptions are durable or messages are persistent, you should have a fixed queue by subscriber anyways. I would expect the subscriber to request the queue from the admin before subscribing...
Now to the juicier part. Review your design. Sharing a connection is not a good thing as messages are single threaded through a connection. This is why the J2EE appservers have connection pooling. I understand that your standalone JMS does not, so you'll just have to take the performance hit of opening a new connection.
This means as well no session sharing as the session partakes of the same connection.
This being out of the way, another hint: If you search some of the papers written by Hursley on MQ you will find one where it states that with multiple consumers hitting the same queue, you will have a definitive performance advantage to have your calls transacted. So please set the transaction attribute on the session to true and commit the session every so often. (Every message, every 10 messages or every 100 messages?). Keep in mind the max uncommitted messages settings for the queue manager which will play against all consumers/producers. I see that you used dups_ok_acknowledge for your session... don't know if that matters on a transacted session, I just use auto_acknowledge...
Hope this helps clear up some of the performance pitfalls you're seeing right now.
Enjoy |
Thank you again for the hints!
I'll try your suggestions and get back to the forum hopefully with much better performance
g |
|
| Back to top |
|
|
| tof |
| Posted: Sun Oct 05, 2008 12:26 am Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
Ok,
just for completeness here's what I did.
I am now starting a different connection and a different session for each subscriber.
| Code: |
public void subAction(String arg) throws JMSException {
subConnections[index] = (MQTopicConnection) factory.createTopicConnection();
subConnections[index].start();
MQTopicSession localSession = (MQTopicSession) subConnections[index].createTopicSession(false,
Session.DUPS_OK_ACKNOWLEDGE); // can also be
// Session.AUTO_ACKNOWLEDGE
MQTopic localTopic = (MQTopic) localSession.createTopic(subject);
localTopic.setBaseTopicName(subject);
localTopic.setTargetClient(JMSC.MQJMS_CLIENT_JMS_COMPLIANT);
localTopic.setReadAheadAllowed(JMSC.MQJMS_READ_AHEAD_ALLOWED_ENABLED);
localTopic.setPutAsyncAllowed(JMSC.MQJMS_PUT_ASYNC_ALLOWED_ENABLED);
...
sublist[index] = localSession.createSubscriber(localTopic, query, true);
sublist[index].setMessageListener(this);
|
I seem not to be able to find out how to configure a "managed queue" (a separate queue for each subscriber) using JMS in MQ V7. The documentation seems clear up to here:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqnar.doc/ps22750_.htm
But a clear mapping between properties such as MQSO_MANAGED and properties that I can set through JMS is nowhere to be found. All I found was this and the properties I can play with are just a few:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzaw.doc/jm25070_.htm
This leaves me without the option of setting the subscriber queue when I create a new subscriber. I even tried to create a temporary queue and set it at topic creation, but all I can do is invoke methods to set queues for DURABLE subscribers (and I want to be fast and non-durable).
Next thing I'm going to try is transactions as suggested by fjb_saper, but in theory this should only complicate things at least the way I see it (adds yet another layer of indirection for operations).
Concerning the auto-acknowledge vs dups_ok_acknowledge, the latter should be faster as it relies on clients to detect duplicates relieving the QM from the burden of doing it.
I now setup MQs to be "performant" as they suggest at Hursley: http://hursleyonwmq.wordpress.com/2007/02/03/creating-a-new-performance-queue-manager/
I looked around to see whether I could find some performance results concerning PUB/SUB in WS MQ and I stumbled into this:
http://www3.informatik.uni-wuerzburg.de/TR/tr380.pdf
They seem to be getting really low INPUT throughput (just as I do) although they say that message replication (that is the number of recipients per message) has surprisingly no effect on websphere MQ.
I see they get better results by having "complex" subscriptions instead of many simple ones. I am going to try that now and then I think I'll give up fiddling with it.
Thank you all for the help!
g |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Oct 05, 2008 7:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| tof wrote: |
| Code: |
MQTopicSession localSession = (MQTopicSession) subConnections[index].createTopicSession(false,
Session.DUPS_OK_ACKNOWLEDGE); // can also be
// Session.AUTO_ACKNOWLEDGE
MQTopic localTopic = (MQTopic) localSession.createTopic(subject);
localTopic.setBaseTopicName(subject);
localTopic.setTargetClient(JMSC.MQJMS_CLIENT_JMS_COMPLIANT);
localTopic.setReadAheadAllowed(JMSC.MQJMS_READ_AHEAD_ALLOWED_ENABLED);
localTopic.setPutAsyncAllowed(JMSC.MQJMS_PUT_ASYNC_ALLOWED_ENABLED);
...
|
I seem not to be able to find out how to configure a "managed queue" (a separate queue for each subscriber) using JMS in MQ V7. The documentation seems clear up to here:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.amqnar.doc/ps22750_.htm
But a clear mapping between properties such as MQSO_MANAGED and properties that I can set through JMS is nowhere to be found. All I found was this and the properties I can play with are just a few:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzaw.doc/jm25070_.htm
This leaves me without the option of setting the subscriber queue when I create a new subscriber. I even tried to create a temporary queue and set it at topic creation, but all I can do is invoke methods to set queues for DURABLE subscribers (and I want to be fast and non-durable).
|
Did you try setting up the subscription with a topic defined in JNDI using the file context? I believe there is a place there to setup the subscription queue. This should show you how to set up the subscription using a uri Destination....(you'd have to print out the destination acquired by JNDI to show the URI form)
| tof wrote: |
Next thing I'm going to try is transactions as suggested by fjb_saper, but in theory this should only complicate things at least the way I see it (adds yet another layer of indirection for operations).
Concerning the auto-acknowledge vs dups_ok_acknowledge, the latter should be faster as it relies on clients to detect duplicates relieving the QM from the burden of doing it.
I now setup MQs to be "performant" as they suggest at Hursley: http://hursleyonwmq.wordpress.com/2007/02/03/creating-a-new-performance-queue-manager/
I looked around to see whether I could find some performance results concerning PUB/SUB in WS MQ and I stumbled into this:
http://www3.informatik.uni-wuerzburg.de/TR/tr380.pdf
They seem to be getting really low INPUT throughput (just as I do) although they say that message replication (that is the number of recipients per message) has surprisingly no effect on websphere MQ.
I see they get better results by having "complex" subscriptions instead of many simple ones. I am going to try that now and then I think I'll give up fiddling with it.
Thank you all for the help!
g |
I hope that you understand that the test done by the University of Wuerzburg only took the JMS aspect of MQ into consideration. There is no test being done cross platform/language, neither do they specify whether they used the native implementation form of the correlation Id for their filtering test. (this can have a significant impact on performance in MQ). Also the V7 filtering model is quite different and performance enhanced compared to the V6 filtering the test was conducted with.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| moe |
| Posted: Mon Oct 06, 2008 1:05 am Post subject: |
|
|
Apprentice
Joined: 05 Sep 2006
Posts: 33
Location: Sydney, Australia
|
Im surprised that you are not getting a dynamically created queue per subscriber by default, I'm working on pubsub with mq7 now and I can see that without setting any special options in the topic connection factory, my durable subscriber has a unique queue created for it.
In my case it is called SYSTEM.MANAGED.DURABLE.48E9D......
The only time I have seen it not dynamically create a queue per subsciber is when the subscription has been non-durable.
If you are still having problems, try setting the subscriber queue by calling the setBrokerCCSubQueue or setBrokerSubQueue functions from the MQTopicConnectionFactory class. When setting the queue name, try setting it to something like "SYSTEM.MANAGED.*" (include the asterisk). |
|
| Back to top |
|
|
| tof |
| Posted: Mon Oct 06, 2008 4:17 am Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| moe wrote: |
Im surprised that you are not getting a dynamically created queue per subscriber by default, I'm working on pubsub with mq7 now and I can see that without setting any special options in the topic connection factory, my durable subscriber has a unique queue created for it.
In my case it is called SYSTEM.MANAGED.DURABLE.48E9D......
The only time I have seen it not dynamically create a queue per subsciber is when the subscription has been non-durable.
If you are still having problems, try setting the subscriber queue by calling the setBrokerCCSubQueue or setBrokerSubQueue functions from the MQTopicConnectionFactory class. When setting the queue name, try setting it to something like "SYSTEM.MANAGED.*" (include the asterisk). |
Thank you Moe,
I was also expecting MQ V7 to create dynamic queues for each subscriber by itself. The difference btw your set-up and mine though is that I'm using NON-DURABLE subscribers.
I tried your second suggestion (plus all the suggestions above: transactions, "composed subscriptions", one connection per subscriber) and maximum IN throughput has doubled.
I now use the code below to (supposedly) create a temp queue for each JMS subscriber:
| Code: |
factory.setBrokerSubQueue("SYSTEM.JMS.ND.*");
factory.setBrokerCCSubQueue("SYSTEM.JMS.ND.*");
|
Performance has indeed improved since my first post, but still I get massive (50%) message loss above the 800 mess/sec threshold. See figure below:
 |
|
| Back to top |
|
|
| tof |
| Posted: Mon Oct 06, 2008 7:09 am Post subject: |
|
|
Novice
Joined: 29 Sep 2008
Posts: 14
|
| fjb_saper wrote: |
Did you try setting up the subscription with a topic defined in JNDI using the file context? I believe there is a place there to setup the subscription queue. This should show you how to set up the subscription using a uri Destination....(you'd have to print out the destination acquired by JNDI to show the URI form)
|
Thank you fjb_saper,
I think setting up the subscriber queues worked in the way that Moe suggested (I get a good performance improvement 2x!). I even found out how to set the ShareConvAllowed parameter to 0 so that no connections are shared for clients in the same process.
I know it's not best practice but I'd rather not use JNDI and the JMSAdmin tool as setting up the configuration for the full-size experiment (42 nodes) is complicated enough as is. Plus using JNDI should not impact on performances.
The client initialization code I use now for the 2 QM experiment sets all these properties:
| Code: |
JmsFactoryFactory ff = JmsFactoryFactory .getInstance(JmsConstants.WMQ_PROVIDER);
// Create the connection.
factory = (MQTopicConnectionFactory) ff .createTopicConnectionFactory();
// Config
factory.setHostName("localhost");
factory.setPort(server_port);
factory.setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
factory.setMessageSelection(JMSC.MQJMS_MSEL_BROKER);
factory.setMQConnectionOptions(JMSC.MQCNO_FASTPATH_BINDING);
factory.setChannel("SYSTEM.DEF.SVRCONN");
factory.setBrokerSubQueue("SYSTEM.JMS.ND.*");
factory.setBrokerCCSubQueue("SYSTEM.JMS.ND.CC.*");
factory.setShareConvAllowed(0);
factory.setDirectAuth(JMSC.MQJMS_DIRECTAUTH_BASIC);
factory.setUseConnectionPooling(true);
System.out.println(sdf.format(new Date()) + " " + my_identity
+ " connection factory created");
connection = (MQTopicConnection) factory.createTopicConnection();
connection.start();
session = (MQTopicSession) connection.createTopicSession(true,
Session.DUPS_OK_ACKNOWLEDGE);
// can also be
// Session.AUTO_ACKNOWLEDGE
System.out.println("Obtained session: " + session);
topic = (MQTopic) session.createTopic(subject);
topic.setBaseTopicName(subject);
topic.setPersistence(JMSC.MQJMS_PER_NON);
topic.setTargetClient(JMSC.MQJMS_CLIENT_JMS_COMPLIANT);
topic.setReadAheadAllowed(JMSC.MQJMS_READ_AHEAD_ALLOWED_ENABLED);
topic.setPutAsyncAllowed(JMSC.MQJMS_PUT_ASYNC_ALLOWED_ENABLED); |
If anybody sees anything wrong with that I'll be really glad for any suggestion...
Most of the properties can be set on the connection factory, while I was trying to set them directly on the topic (this is why I couldn't find them).
Here's the (somewhat convoluted) page where properties are mapped to JMS:
http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzaw.doc/jm10910_.htm
| fjb_saper wrote: |
I hope that you understand that the test done by the University of Wuerzburg only took the JMS aspect of MQ into consideration. There is no test being done cross platform/language, neither do they specify whether they used the native implementation form of the correlation Id for their filtering test. (this can have a significant impact on performance in MQ). Also the V7 filtering model is quite different and performance enhanced compared to the V6 filtering the test was conducted with. |
I completely agree with you that the results from the guys at Wuerzburg only took JMS in consideration, but that's exactly what I need at the moment: we are comparing the throughputs of different JMS implementations and it is only fair to use Java and JMS, no other interface.
I also understand that a real comparison would consider cross-platform and all the possible system tweaks only a pro ($) would be able to for each MQ implementation.
To be honest I think no one has the time and money (and I might add real interest) to make a complete "fair" comparison.
Thank you again,
g |
|
| Back to top |
|
|
|
|
