| Author |
Message
|
| HubertKleinmanns |
 Posted: Thu Aug 15, 2019 4:22 am Post subject: MQ Pub/Sub - Topic re-routing |
 |
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
Hi everybody,
I've got a simple question - but may be a complex answer:
Is it possible (and if so, how) to re-route a Pub/Sub topic?
What I mean is:
- I get informations from a customer - let's say some user data. This information comes via Pub/Sub published to a topic named "userdata".
- I get the same information from another customer - also via Pub/Sub and also published to a topic named "userdata".
What I want/need is, to separate the information and to extend the topic tree like this:
- Company 1: userdata -> comp1/userdata
- Company 2: userdata -> comp2/userdata
Is there a (simple) way, to extend the topic tree?
Thanks in advance
_________________
Regards
Hubert |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Aug 15, 2019 5:15 am Post subject: Re: MQ Pub/Sub - Topic re-routing |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| HubertKleinmanns wrote: |
| Is there a (simple) way, to extend the topic tree? |
Assume the existence of a magic bullet for a moment; some kind of topic mapper somewhere in the MQ topology?
What criteria would it be using? Seriously. It's not clear from your post how, looking at a publication on the topic "userdata", the magic bullet dohicky would determine if it's from company 1 or company 2.
What data points distinguish them?
For the record, the simple way to extend the topic tree is to get the publishers to add it. I'm further assuming you wouldn't be posting if that was an option; I'm sure you've be told (probably in a hysterical scream) tweaking the publishers to add a parent root to the topic will take 20 years, cost $100,000,000,000,000 and need to be authorized by the unanimous vote of the UN General Assembly, The Grey Council & The Great Old Ones.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Thu Aug 15, 2019 6:26 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
Hi Vitor,
good question  . I'm working for a new employer and he asked me about re-routing of topics. Separate customers would come across different channels, but on a closer examination this wouldn't be helpful, right  ?
My employer wants to separate the customers without giving any guidelines about the name of the topics. They all deliver the same kind of data via Pub/Sub and could choose the same name for the topic  .
I'm looking for a solution to distinguish the data of our coustomers (and our own data, because we have also "userdata").
I think, we need to specify a prefix of the topics for our customers (that's what my employer didn't want).
Or do you have another idea?
_________________
Regards
Hubert |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 15, 2019 7:13 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| HubertKleinmanns wrote: |
| Separate customers would come across different channels, but on a closer examination this wouldn't be helpful, right ? |
You're in the murky world of channel exits where the receiver MCA intercepts the message and prepends something contextual to the topic string.
My hair is standing on end at the thought, which given the amount of hair I have left after decades of pulling it out isn't as impressive as it sounds.
| HubertKleinmanns wrote: |
| My employer wants to separate the customers without giving any guidelines about the name of the topics. They all deliver the same kind of data via Pub/Sub and could choose the same name for the topic . |
Could you try some kind of meta-subscriber, that subscribes to "userdata" then introspects the payload to determine who sent it? Re-publishing it with the appropriate topic? Scaled to a point it's not a bottleneck?
| HubertKleinmanns wrote: |
| I think, we need to specify a prefix of the topics for our customers (that's what my employer didn't want). |
This is certainly the solution least likely to blow up in your face. And the one I'd recommend to your employer in your place.
But I'm a bad person.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| hughson |
| Posted: Thu Aug 15, 2019 4:10 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
There are several things I can think of, but if your current applications are already directly specifying the topic string in their application then you can't easily change that.
- Have each customer use their own Topic Object - either on it's own, or as the root of their little portion of the tree.
- Have them use an alias queue pointing to a topic, then you can set the topic string however you want.
- Have them set their company name as a property on the published message
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Fri Aug 16, 2019 1:04 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
Hi Vitor,
| Vitor wrote: |
| ... You're in the murky world of channel exits where the receiver MCA intercepts the message and prepends something contextual to the topic string. |
I've suspected getting such an answer .
| Vitor wrote: |
| My hair is standing on end at the thought, which given the amount of hair I have left after decades of pulling it out isn't as impressive as it sounds. |
Sorry about any hair you've lost answering my question  .
| Vitor wrote: |
| Could you try some kind of meta-subscriber, that subscribes to "userdata" then introspects the payload to determine who sent it? Re-publishing it with the appropriate topic? Scaled to a point it's not a bottleneck? |
Seems to be just another Kind of "Exit" .
| Vitor wrote: |
...
And the one I'd recommend to your employer in your place. |
I'll try my very best. 
_________________
Regards
Hubert |
|
| Back to top |
|
|
| markt |
| Posted: Fri Aug 16, 2019 4:31 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
Interestingly (perhaps) the AMQP channels into MQ do have a TPROOT attribute which allows something like this - each channel can get a different prefix on messages published through it. See https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ref.con.doc/q123909_.htm.
But that's because the AMQP listener process could be thought of as acting as a kind of intermediate broker or exit component - rewriting formats etc - so it's doing the kind of work that is being suggested you could do here for regular MQ apps. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Fri Aug 16, 2019 5:00 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
Hi Morag,
my problem is as follows:
- The publishing applications are at the customer's side. These applications don't want to care about the subscribers. Maybe they publish a message to another application at the customer side - now the topic (in my sample "userdata") would be unique.
- An additional subscription to that topic then could route this data to us as well - fine so far.
- But: Lot's of customers share the same kind of data to us and could use (accidentally) the same topic name. Now on our side this topic ist not unique.
- We don't want to prescribe the rules at customer's side.
At the Moment I'm working on a PoC. My first test was:
- Publish a message with the sample program "amqspub" on QM1.
- Create an administrative subscription (PSMODE is "RFH2") on QM1 to that topic, which points to a remote queue.
- The remote queue addresses a stream queue on the target QMgr QM2.
- Get the message on the receiving QMgr using the sample program "amqssub".
- This works fine and publisher and subscriber don't need to care about the infrastructure. Even the topic may verifiy and could be recovered, because it is part of the message (in the RFH2 header).
- When I'm at the receiving end I could create individual stream queues for every sender and so identify the customer and separate them.
But how could I extend the topic and add the customer's name?
I found some description for V6 to v7 here:
https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.pla.doc/q005080_.htm
But this seems not to be functional in v9
_________________
Regards
Hubert |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Aug 16, 2019 5:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| HubertKleinmanns wrote: |
| Vitor wrote: |
| Could you try some kind of meta-subscriber, that subscribes to "userdata" then introspects the payload to determine who sent it? Re-publishing it with the appropriate topic? Scaled to a point it's not a bottleneck? |
Seems to be just another Kind of "Exit" . |
But an "exit" not subject to language restrictions that doesn't run in-line with the MCA and offer the possibility of serious performance degredation / instability that a real exit does.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Fri Aug 16, 2019 5:05 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
Hi markt,
this seems to be something special on AMQP. In my scenario messages come across a QMgr - QMgr Connection.
_________________
Regards
Hubert |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sat Aug 17, 2019 8:59 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
For each customer make an administrative subscription with delivery to a queue. Have that queue be an alias to a topic and the topic object underlying that alias can then have any topic string in it. You will need to define a topic by provider/client and subscription; and there you can specify a hierarchy of type /client/userdata.
As this information gets republished in your system you'll have the prepended client information.
Hope this helps.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hughson |
| Posted: Sun Aug 18, 2019 12:49 am Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
| fjb_saper wrote: |
| For each customer make an administrative subscription .... |
This subscription, what topic would it subscribe to?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Mon Aug 19, 2019 3:50 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| fjb_saper wrote: |
| For each customer make an administrative subscription with delivery to a queue. ... |
Hmm  , how should this work when all customers publish to the same topic (on their side) ?
The question is, how to separate publications to the same topic, but coming across differents channels?
But may be the customer (producer) on his side would implement such a mechanism. Then we would get a uniqe topic on our side.
_________________
Regards
Hubert
Last edited by HubertKleinmanns on Mon Aug 19, 2019 5:36 am; edited 1 time in total |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Mon Aug 19, 2019 4:41 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| hughson wrote: |
| fjb_saper wrote: |
| For each customer make an administrative subscription .... |
This subscription, what topic would it subscribe to? |
This subscription would subscribe to the "userdata" topic at the customer queue manager. This means that if you have for instance 10 different customers, you would have 10 different subscriptions to "userdata" going to 10 different "queues" (topic alias)...
The other possibility would be to subscribe to the "userdata" topic administratively, inspect the message in the queue and republish according to the client. This supposes that you are able to distinguish the client once the original publication is received...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| HubertKleinmanns |
| Posted: Mon Aug 19, 2019 5:44 am Post subject: |
|
|
Shaman
Joined: 24 Feb 2004
Posts: 732
Location: Germany
|
| fjb_saper wrote: |
... This subscription would subscribe to the "userdata" topic at the customer queue manager. This means that if you have for instance 10 different customers, you would have 10 different subscriptions to "userdata" going to 10 different "queues" (topic alias)...
The other possibility would be to subscribe to the "userdata" topic administratively, inspect the message in the queue and republish according to the client. This supposes that you are able to distinguish the client once the original publication is received...
|
I was able to subscribe to "userdata" and put the message to a queue alias, but I was not able, to put it to a topic alias. I got the error
| Quote: |
| AMQ8470E: The object <Name of the alias> is not a valid subscription destination. |
_________________
Regards
Hubert |
|
| Back to top |
|
|
|
|
