ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » General IBM MQ Support » MQ Pub/Sub - Topic re-routing

Post new topic  Reply to topic Goto page 1, 2  Next
 MQ Pub/Sub - Topic re-routing « View previous topic :: View next topic » 
Author Message
HubertKleinmanns
PostPosted: Thu Aug 15, 2019 4:22 am    Post subject: MQ Pub/Sub - Topic re-routing Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

Hi everybody,

I've got a simple question - but may be a complex answer:

Is it possible (and if so, how) to re-route a Pub/Sub topic?

What I mean is:

- I get informations from a customer - let's say some user data. This information comes via Pub/Sub published to a topic named "userdata".

- I get the same information from another customer - also via Pub/Sub and also published to a topic named "userdata".

What I want/need is, to separate the information and to extend the topic tree like this:

- Company 1: userdata -> comp1/userdata

- Company 2: userdata -> comp2/userdata

Is there a (simple) way, to extend the topic tree?

Thanks in advance
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Thu Aug 15, 2019 5:15 am    Post subject: Re: MQ Pub/Sub - Topic re-routing Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

HubertKleinmanns wrote:
Is there a (simple) way, to extend the topic tree?


Assume the existence of a magic bullet for a moment; some kind of topic mapper somewhere in the MQ topology?

What criteria would it be using? Seriously. It's not clear from your post how, looking at a publication on the topic "userdata", the magic bullet dohicky would determine if it's from company 1 or company 2.

What data points distinguish them?

For the record, the simple way to extend the topic tree is to get the publishers to add it. I'm further assuming you wouldn't be posting if that was an option; I'm sure you've be told (probably in a hysterical scream) tweaking the publishers to add a parent root to the topic will take 20 years, cost $100,000,000,000,000 and need to be authorized by the unanimous vote of the UN General Assembly, The Grey Council & The Great Old Ones.


_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
HubertKleinmanns
PostPosted: Thu Aug 15, 2019 6:26 am    Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

Hi Vitor,

good question . I'm working for a new employer and he asked me about re-routing of topics. Separate customers would come across different channels, but on a closer examination this wouldn't be helpful, right ?

My employer wants to separate the customers without giving any guidelines about the name of the topics. They all deliver the same kind of data via Pub/Sub and could choose the same name for the topic .

I'm looking for a solution to distinguish the data of our coustomers (and our own data, because we have also "userdata").

I think, we need to specify a prefix of the topics for our customers (that's what my employer didn't want).

Or do you have another idea?
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Thu Aug 15, 2019 7:13 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

HubertKleinmanns wrote:
Separate customers would come across different channels, but on a closer examination this wouldn't be helpful, right ?


You're in the murky world of channel exits where the receiver MCA intercepts the message and prepends something contextual to the topic string.

My hair is standing on end at the thought, which given the amount of hair I have left after decades of pulling it out isn't as impressive as it sounds.

HubertKleinmanns wrote:
My employer wants to separate the customers without giving any guidelines about the name of the topics. They all deliver the same kind of data via Pub/Sub and could choose the same name for the topic .


Could you try some kind of meta-subscriber, that subscribes to "userdata" then introspects the payload to determine who sent it? Re-publishing it with the appropriate topic? Scaled to a point it's not a bottleneck?

HubertKleinmanns wrote:
I think, we need to specify a prefix of the topics for our customers (that's what my employer didn't want).


This is certainly the solution least likely to blow up in your face. And the one I'd recommend to your employer in your place.

But I'm a bad person.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
hughson
PostPosted: Thu Aug 15, 2019 4:10 pm    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

There are several things I can think of, but if your current applications are already directly specifying the topic string in their application then you can't easily change that.
  • Have each customer use their own Topic Object - either on it's own, or as the root of their little portion of the tree.
  • Have them use an alias queue pointing to a topic, then you can set the topic string however you want.
  • Have them set their company name as a property on the published message
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
HubertKleinmanns
PostPosted: Fri Aug 16, 2019 1:04 am    Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

Hi Vitor,

Vitor wrote:
... You're in the murky world of channel exits where the receiver MCA intercepts the message and prepends something contextual to the topic string.
I've suspected getting such an answer .

Vitor wrote:
My hair is standing on end at the thought, which given the amount of hair I have left after decades of pulling it out isn't as impressive as it sounds.
Sorry about any hair you've lost answering my question .

Vitor wrote:
Could you try some kind of meta-subscriber, that subscribes to "userdata" then introspects the payload to determine who sent it? Re-publishing it with the appropriate topic? Scaled to a point it's not a bottleneck?
Seems to be just another Kind of "Exit" .

Vitor wrote:
...
And the one I'd recommend to your employer in your place.
I'll try my very best.
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
markt
PostPosted: Fri Aug 16, 2019 4:31 am    Post subject: Reply with quote

Knight

Joined: 14 May 2002
Posts: 502

Interestingly (perhaps) the AMQP channels into MQ do have a TPROOT attribute which allows something like this - each channel can get a different prefix on messages published through it. See https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.1.0/com.ibm.mq.ref.con.doc/q123909_.htm.

But that's because the AMQP listener process could be thought of as acting as a kind of intermediate broker or exit component - rewriting formats etc - so it's doing the kind of work that is being suggested you could do here for regular MQ apps.
Back to top
View user's profile Send private message
HubertKleinmanns
PostPosted: Fri Aug 16, 2019 5:00 am    Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

Hi Morag,

my problem is as follows:

  • The publishing applications are at the customer's side. These applications don't want to care about the subscribers. Maybe they publish a message to another application at the customer side - now the topic (in my sample "userdata") would be unique.

  • An additional subscription to that topic then could route this data to us as well - fine so far.

  • But: Lot's of customers share the same kind of data to us and could use (accidentally) the same topic name. Now on our side this topic ist not unique.

  • We don't want to prescribe the rules at customer's side.


At the Moment I'm working on a PoC. My first test was:

  • Publish a message with the sample program "amqspub" on QM1.

  • Create an administrative subscription (PSMODE is "RFH2") on QM1 to that topic, which points to a remote queue.

  • The remote queue addresses a stream queue on the target QMgr QM2.

  • Get the message on the receiving QMgr using the sample program "amqssub".

  • This works fine and publisher and subscriber don't need to care about the infrastructure. Even the topic may verifiy and could be recovered, because it is part of the message (in the RFH2 header).

  • When I'm at the receiving end I could create individual stream queues for every sender and so identify the customer and separate them.


But how could I extend the topic and add the customer's name?

I found some description for V6 to v7 here:

https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_7.5.0/com.ibm.mq.pla.doc/q005080_.htm

But this seems not to be functional in v9
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
Vitor
PostPosted: Fri Aug 16, 2019 5:01 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

HubertKleinmanns wrote:
Vitor wrote:
Could you try some kind of meta-subscriber, that subscribes to "userdata" then introspects the payload to determine who sent it? Re-publishing it with the appropriate topic? Scaled to a point it's not a bottleneck?
Seems to be just another Kind of "Exit" .


But an "exit" not subject to language restrictions that doesn't run in-line with the MCA and offer the possibility of serious performance degredation / instability that a real exit does.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
HubertKleinmanns
PostPosted: Fri Aug 16, 2019 5:05 am    Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

Hi markt,

this seems to be something special on AMQP. In my scenario messages come across a QMgr - QMgr Connection.
_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
fjb_saper
PostPosted: Sat Aug 17, 2019 8:59 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

For each customer make an administrative subscription with delivery to a queue. Have that queue be an alias to a topic and the topic object underlying that alias can then have any topic string in it. You will need to define a topic by provider/client and subscription; and there you can specify a hierarchy of type /client/userdata.
As this information gets republished in your system you'll have the prepended client information.

Hope this helps.
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
hughson
PostPosted: Sun Aug 18, 2019 12:49 am    Post subject: Reply with quote

Padawan

Joined: 09 May 2013
Posts: 1914
Location: Bay of Plenty, New Zealand

fjb_saper wrote:
For each customer make an administrative subscription ....

This subscription, what topic would it subscribe to?
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
HubertKleinmanns
PostPosted: Mon Aug 19, 2019 3:50 am    Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

fjb_saper wrote:
For each customer make an administrative subscription with delivery to a queue. ...


Hmm , how should this work when all customers publish to the same topic (on their side) ?

The question is, how to separate publications to the same topic, but coming across differents channels?

But may be the customer (producer) on his side would implement such a mechanism. Then we would get a uniqe topic on our side.
_________________
Regards
Hubert


Last edited by HubertKleinmanns on Mon Aug 19, 2019 5:36 am; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website
fjb_saper
PostPosted: Mon Aug 19, 2019 4:41 am    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

hughson wrote:
fjb_saper wrote:
For each customer make an administrative subscription ....

This subscription, what topic would it subscribe to?

This subscription would subscribe to the "userdata" topic at the customer queue manager. This means that if you have for instance 10 different customers, you would have 10 different subscriptions to "userdata" going to 10 different "queues" (topic alias)...

The other possibility would be to subscribe to the "userdata" topic administratively, inspect the message in the queue and republish according to the client. This supposes that you are able to distinguish the client once the original publication is received...


_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
HubertKleinmanns
PostPosted: Mon Aug 19, 2019 5:44 am    Post subject: Reply with quote

Shaman

Joined: 24 Feb 2004
Posts: 732
Location: Germany

fjb_saper wrote:
... This subscription would subscribe to the "userdata" topic at the customer queue manager. This means that if you have for instance 10 different customers, you would have 10 different subscriptions to "userdata" going to 10 different "queues" (topic alias)...

The other possibility would be to subscribe to the "userdata" topic administratively, inspect the message in the queue and republish according to the client. This supposes that you are able to distinguish the client once the original publication is received...



I was able to subscribe to "userdata" and put the message to a queue alias, but I was not able, to put it to a topic alias. I got the error
Quote:
AMQ8470E: The object <Name of the alias> is not a valid subscription destination.

_________________
Regards
Hubert
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic  Reply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum Index » General IBM MQ Support » MQ Pub/Sub - Topic re-routing
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.