| Author |
Message
|
| boos |
 Posted: Mon Mar 18, 2013 6:14 am Post subject: REQ: best practices on pub/sub in MQ 7.5 |
 |
|
Apprentice
Joined: 27 Jan 2004
Posts: 37
Location: Netherlands
|
Hello,
I’m looking for (some) best practices around pub/sub (in MQ v7.5 where broker pub/sub is integrated in core MQ product) And I’m looking for some information about the governance of pub/sub (mostly how to governance topics and subscriptions). I’ve already done some googling but couldn’t find anything (except for the standard IBM documentation of MQ v7.5). I have not yet worked with de pub/sub in MQ (I do have a lot experience with ‘plain’ MQ)
A few questions I have are:
- What is the limitation of #topics & #subscriptions ? (is 16 mil. a lot ?)
- What is the strain on hardware with a lot of (durable) subscriptions ? (is it just storage and memory or does it effect CPU & performance as well)
- is there tooling available to ease the governance of topics and subscribers ? (I’ve seen the commands but there’s probably someone somewhere that made an easy to use interface/program)
- Are topicnames case sensitive ?
- what is the difference between a topic and a stream ?
- I’m looking for some best practices for the following example:
16 mil. customers in 50+ applications, not every application is interested in every customers address. I could create a topic ‘ADRESS’ to which every application subscribes (but then every application collects a lot more messages/adresses than it needs, and has to validate them before it can discard or process them) Or I could create topics per customer i.e. customernumbr/ADRESS to which only applications subscribe that know that customer (and are interested in its address). But that will result in a lot off topics and subscriptions (16 mil.). Is that a problem or is it a number that is common practice ?
If anyone could direct me to some page/document with best practices around pub/sub (with MQ v 7.5). I would be much obliged.
Thanks,
Edde |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Mon Mar 18, 2013 7:23 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9472
Location: US: west coast, almost. Otherwise, enroute.
|
Have you done any research at all? Did you actually read the pub/sub documentation? Did you actually read about the API calls to find out if topics are case-sensitive?
Did you go to google to search for 'wmq pub/sub case-sensitive'?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Mar 18, 2013 2:28 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
A pub/sub engine was first integrated into MQ v7.0.0.
Are your design requirements a proper fit for the pub/sub paradigm (ie. ephemeral context sensitive data), or would a Database be more appropriate?
Don't implement a large scale design on an inappropriate technology, you will get seriously bitten further down the track.
FYI, there is a Pub/Sub performance SupportPac
http://www-01.ibm.com/support/docview.wss?uid=swg24026456
_________________
Glenn |
|
| Back to top |
|
|
| boos |
| Posted: Mon Mar 25, 2013 6:01 am Post subject: |
|
|
Apprentice
Joined: 27 Jan 2004
Posts: 37
Location: Netherlands
|
thanks (both of you).
@bruce2359: I did do my research but when I used your search query I ended up on a page that qlearly stated:
| Quote: |
| "Furthermore, the topic strings are case sensitive" |
It also stated that if you don't use quotes in your definition
| Quote: |
| "Because they will be folded to uppercase" |
http://www-01.ibm.com/support/docview.wss?uid=swg27016146
Which makes me decide to only use uppercase topicnames (just like our queuenames).
@gbaddeley: thanks for your reference to the IBM supportpacs page (there is newer version of the document you referred to, which is adjusted for MQ v7.5 (MP0C) http://www-01.ibm.com/support/docview.wss?uid=swg24033638 ). It helped a lot in getting some insights in pub/sub performance. For which I oringinally asked.
| Quote: |
| Are your design requirements a proper fit for the pub/sub paradigm (ie. ephemeral context sensitive data), or would a Database be more appropriate? |
That was the main reason for my questions. We currently have an application (with database) that does the job but is expensive to maintain. It would be cheaper (more efficient) if we could use the 'built in' functionality of MQ. But only if it is a proper fit and that is what I'm trying to establish. It seems we're going towards a hybrid solution where personlised subscriptions (a subscription per client) is still done by the appliaction. But more general subscriptions (and its publications) is done via the topics solution of MQ.
Thanks for your help. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Mar 25, 2013 6:57 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
You really need to attend the IBM course on pub/sub.
You have to strike a balance between what is in the topic string and what is in the message data. The topic string just classifies the data - it does not contain all of it!
E.g. for publishing address changes you could have a topic string which ended (make sure to structure the string more than just these levels with some sort of prefix)
/corpname/customer/address/create
/corpname/customer/address/update
/corpname/customer/address/delete
Interested subscribers could then pick up whatever they wanted (all customer messages, just address messages, just new address etc) using suitable topic string wildcards on the subscriptions.
The actual details of the customers' name/address would be in the message itself. It would be quite wrong to have every customer's name or address (16 million) in the topic tree!
Choose the topic tree carefully - allow for future expansion and remember it's hard to change it once in place. Structure it left to right, most generic to most specific. Bear in mind any security requirements may affect your choice of topic tree hierarchy.
IBM recommend lower case (or at least did on the course I went on). |
|
| Back to top |
|
|
| boos |
| Posted: Mon Mar 25, 2013 8:14 am Post subject: |
|
|
Apprentice
Joined: 27 Jan 2004
Posts: 37
Location: Netherlands
|
| Quote: |
| You really need to attend the IBM course on pub/sub. |
Or maybe talk to an IBM specialist/consultant on the subject to help with the architectural/design part of it all.
All your advice is sound (and logical).
It's not that I want (or try) to put as much information in the Topic as possible (the address and name info are in the message itself). I Don't see another way to get the message to exactly the right applications that needs to do something with that information (because there's just a few applications that know a specific customer). Other applications may also be interested in the ADDRESS topic but may not now that specific customer (and are therefore not interested in that specific ADDRESS). If they would subscibe to a topic with wildcard '#/ADDRESS' the get a lot more addresses then they want because they have to validate every message if it contains a customer that is in its database.
So, to describe my original dilema another way:
I've got 1 publish application which publishes (a change of) address(es). And I've got 50+ applications that have (partly the same) customers in their database (luckily the same customer in different applications have the same unique customer-id). Every application is interested in a change in address of their OWN customers. But the publishing application doesn't know which application contains which customer. How can I solve the problem of getting the right customer address change to only those applications that know that customer (within the bounds of pub/sub performance)
The only way I see is to create Topics that contain <customer-id>/ADDRESS (to which only applications subscribe that know that specific customer). But again that would result in 16+ million topics. Judging by the comments that would a bit to much (and overall not a good idea). Anyone another good idea to solve this dilema (with pub/sub)?
And I don't mean putting all addresses in 1 database/application because we've already done that, the address in above example is just that, an example that should make the dilema clear to anyone. Our topics are a bit more business specific but the dilema is the same. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 25, 2013 9:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
If you need to perform pub/sub routing based on the contents of the message rather than the merely the topic, you need to purchase and use Message Broker rather than merely MQ.
Even with message broker, your applications will either need to register a new subscription for every customer that they know about, or be prepared to filter all incoming messages to determine if they are supposed to use it or not.
Unless you have some method of scoping customer numbers. |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Mar 25, 2013 9:50 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
[quote="boos"]
| Quote: |
| . Other applications may also be interested in the ADDRESS topic but may not now that specific customer (and are therefore not interested in that specific ADDRESS). If they would subscibe to a topic with wildcard '#/ADDRESS' the get a lot more addresses then they want because they have to validate every message if it contains a customer that is in its database. |
I was thinking more of the wildcard at the right hand side like /corpname/customer/address/# to see all address messages.
The whole point of pub/sub is to de-couple producers from consumers so you can't expect to receive messages precisely targetted. You can simply ignore the ones you don't want.
If there is a way of making the publication assist in this filtering process by including some other value - then of course try to include that value in the topic string (e.g. country code). It depends what you are looking for - but you can't expect it to know what's in your database and what isn't - that would be tight coupling. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 25, 2013 11:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| boos wrote: |
| It seems we're going towards a hybrid solution where personlised subscriptions (a subscription per client) is still done by the appliaction. But more general subscriptions (and its publications) is done via the topics solution of MQ. |
I think you're right to say this. 16m subscriptions is a ponderously large number to administer even if the software can manage it. As others have rightly said, you're not going to get WMQ (or possibly even WMB) to filter addresses based on which of your 50+ applications are interested in the change without something in the topic to discriminate; country code has been suggested, business unit might be another possibility.
But to do what you want requires the publisher to know which of the possible subscribers will want the publication; this is not what pub/sub is intended to manage and not the use case it fits.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Mar 25, 2013 11:53 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| But to do what you want requires the publisher to know which of the possible subscribers will want the publication; this is not what pub/sub is intended to manage and not the use case it fits. |
Well, again, this is where Broker could value add.
The publisher could send messages to non-customer specific topics.
The consumer apps could all register customer-specific subscriptions that used the location of the customer ID in the message itself.
But it's still rife with performance and security issues.... |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Mar 25, 2013 12:59 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| But it's still rife with performance and security issues.... |
Hence my comment of "possibly even WMB". You might be able to, but then again you might not. And as you correctly point out, if you got it working you might later wish you hadn't....
And the OP did ask for "best practices" rather than a solution. You could do this with WMB, and with a huge amount of dancing WMQ, but you'd struggle to describe either as "best".
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Mar 26, 2013 3:23 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| And the OP did ask for "best practices" rather than a solution. You could do this with WMB, and with a huge amount of dancing WMQ, but you'd struggle to describe either as "best". |
Using WMB is always best. |
|
| Back to top |
|
|
|
|
