| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Pub Sub with CSV |
« View previous topic :: View next topic » |
| Author |
Message
|
| Bichu |
 Posted: Thu Sep 22, 2016 4:12 am Post subject: Pub Sub with CSV |
 |
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Hi Guys,
This is the first time I am trying Pub Sub scenario and I am working in IIB v9.0.0.1 and WMQ v7.5.
I need to get a CSV file from an FTP server and to create a pub - sub. When I tried samples with xml, all are working fine.
But I am not able to perform this with CSV data.
I have created a Topic with Topic name TestData. Then I created a subscription with subscription name as sub1, topic name as TestData and Destinationname as
TEST.SUB1. My flow is like FileINput -> Compute -> Publish. In the compute node, I tried setting the MQRFH2 header as below.
| Code: |
CALL CopyMessageHeaders();
SET OutputRoot.MQRFH2.psc.Topic = 'TestData';
CALL CopyEntireMessage(); |
But MQRFH2 header is getting disappeared when CopyEntireMessage procedure was invoked.
TestData is the name of the first column in the header of my CSV file.
Could you please let me know where am I going wrong. What change should I do?[/code] |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Thu Sep 22, 2016 4:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Swap the order of your statements.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Bichu |
| Posted: Thu Sep 22, 2016 4:33 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
| i tried that Jeff, but I got an exception BIP6062, which states payload should be the last one |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Sep 22, 2016 6:08 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Bichu wrote: |
| i tried that Jeff, but I got an exception BIP6062, which states payload should be the last one |
Sure. Because you're putting the RHF2 header in the wrong place.
When you do CopyEntireMessage, you replace everything in OutputRoot with everything in InputRoot.
Once you've done that, you have to add fields in the right places.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Bichu |
| Posted: Fri Sep 23, 2016 1:48 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Thanks Jeff for pointing out my mistake. I am able to add MQRFH2 header now. But still I am not getting message in output queues and I wonder where these messages are going. I examined the queue depth of system queues too but those are not incrementing as well.
Do i miss anything in ESQL code. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Sep 23, 2016 1:53 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
What does your topic string look like?
i.e. show us the dmpmgcfg output for the topic.
Your topic in the RFH2 header has to match it exactly. If there is no topis for the publish then your messages (AFAIK) go to that great bit bucket in the sky never to be seen again.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Bichu |
| Posted: Fri Sep 23, 2016 2:37 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
My input is a CSV file. I am using the first column name in the header , which is Site, as the topic String.
This is the output of dmpmqcfg command.
| Code: |
* Queue manager name: TEST1
* Queue manager platform: Windows
* Queue manager command level: (750/750)
* Command issued: dmpmqcfg -m TEST1 -t topic
*****************************************************************************
**********
DEFINE TOPIC('SYSTEM.BASE.TOPIC') +
TOPICSTR('') +
DESCR('Base topic for resolving attributes') +
DURSUB(YES) +
PUB(ENABLED) +
SUB(ENABLED) +
DEFPSIST(NO) +
DEFPRTY(0) +
DEFPRESP(SYNC) +
* ALTDATE(2016-08-23) +
* ALTTIME(12.01.03) +
PMSGDLV(ALLDUR) +
NPMSGDLV(ALLAVAIL) +
PUBSCOPE(ALL) +
SUBSCOPE(ALL) +
MDURMDL('SYSTEM.DURABLE.MODEL.QUEUE') +
MNDURMDL('SYSTEM.NDURABLE.MODEL.QUEUE') +
MCAST(DISABLED) +
COMMINFO('SYSTEM.DEFAULT.COMMINFO.MULTICAST') +
USEDLQ(YES) +
REPLACE
DEFINE TOPIC('SYSTEM.BROKER.ADMIN.STREAM') +
TOPICSTR('SYSTEM.BROKER.ADMIN.STREAM') +
DESCR('Admin stream for queued Pub/Sub interface') +
* ALTDATE(2016-08-23) +
* ALTTIME(12.01.04) +
WILDCARD(BLOCK) +
MCAST(DISABLED) +
REPLACE
DEFINE TOPIC('SYSTEM.BROKER.DEFAULT.STREAM') +
TOPICSTR('') +
DESCR('Default stream for queued Pub/Sub interface') +
* ALTDATE(2016-08-23) +
* ALTTIME(12.01.04) +
WILDCARD(BLOCK) +
MCAST(DISABLED) +
REPLACE
DEFINE TOPIC('SYSTEM.BROKER.DEFAULT.SUBPOINT') +
TOPICSTR('') +
DESCR('Default RFH2 subscription point for queued Pub/Sub interface') +
* ALTDATE(2016-08-23) +
* ALTTIME(12.01.04) +
WILDCARD(BLOCK) +
MCAST(DISABLED) +
REPLACE
DEFINE TOPIC('SYSTEM.BROKER.MB.TOPIC') +
TOPICSTR('$SYS/Broker') +
* ALTDATE(2016-08-31) +
* ALTTIME(11.06.26) +
REPLACE
DEFINE TOPIC('SYSTEM.DEFAULT.TOPIC') +
TOPICSTR('') +
* ALTDATE(2016-08-23) +
* ALTTIME(12.01.03) +
REPLACE
DEFINE TOPIC('data') +
TOPICSTR('Site') +
* ALTDATE(2016-09-22) +
* ALTTIME(11.15.29) +
REPLACE
SET AUTHREC +
PROFILE('SYSTEM.BASE.TOPIC') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BASE.TOPIC') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.ADMIN.STREAM') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.ADMIN.STREAM') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.DEFAULT.STREAM') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.DEFAULT.STREAM') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.DEFAULT.SUBPOINT') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.DEFAULT.SUBPOINT') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.MB.TOPIC') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.MB.TOPIC') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.BROKER.MB.TOPIC') +
GROUP('mqbrkrs@mid') +
OBJTYPE(TOPIC) +
AUTHADD(PUB,SUB)
SET AUTHREC +
PROFILE('SYSTEM.DEFAULT.TOPIC') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('SYSTEM.DEFAULT.TOPIC') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('TT_XML_EXAMPLE_TOPIC') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('data') +
PRINCIPAL('user@org') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
SET AUTHREC +
PROFILE('data') +
GROUP('grp@mid') +
OBJTYPE(TOPIC) +
AUTHADD(CHG,CLR,DLT,DSP,PASSALL,PASSID,SETALL,SETID,CTRL,PUB,SUB,RESUME)
|
[/code] |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Sep 23, 2016 3:06 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Perhaps my eyesight is failing me but I don't see a topic to match the topic with a 'TOPICSTR' that you are trying to use.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Bichu |
| Posted: Fri Sep 23, 2016 3:42 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
I hope you are looking for this.
| Code: |
DEFINE TOPIC('data') +
TOPICSTR('Site') +
* ALTDATE(2016-09-22) +
* ALTTIME(11.15.29) +
REPLACE |
I have now altered the flow like
FileInput -> MQOutput
MQInput -> Publish
and in the MqInput node properties, I have specified the topic staring and now I am able to see the messages in my subscriber queues.
But this costs me an extra message flow.
Any option for a simple one.
 |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 23, 2016 3:44 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| smdavies99 wrote: |
| Perhaps my eyesight is failing me but I don't see a topic to match the topic with a 'TOPICSTR' that you are trying to use. |
You don't need one.
The only requirement is that the subscriber is using a topic that matches the publisher's topic exactly or matches with a wildcard.
TOPIC objects are only conveniences and points of security.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Fri Sep 23, 2016 4:03 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| mqjeff wrote: |
| smdavies99 wrote: |
| Perhaps my eyesight is failing me but I don't see a topic to match the topic with a 'TOPICSTR' that you are trying to use. |
You don't need one.
The only requirement is that the subscriber is using a topic that matches the publisher's topic exactly or matches with a wildcard.
TOPIC objects are only conveniences and points of security. |
That is true but isn't it better when testing to have one defined then you know that it exists.
The OP is having a problem with messages disappearing. Whenever I have used the Publish nodes and had this happen to me, the root cause what not getting the topic string right.
The alternative to the Publish node would be to use an ALIAS Queue setup to publish to a defined topic. Easy to test outside a flow and script for source control.
I prefer not to leave things to chance. As the saying goes, 'what can go wrong, will'.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 23, 2016 5:16 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| smdavies99 wrote: |
| mqjeff wrote: |
| smdavies99 wrote: |
| Perhaps my eyesight is failing me but I don't see a topic to match the topic with a 'TOPICSTR' that you are trying to use. |
You don't need one.
The only requirement is that the subscriber is using a topic that matches the publisher's topic exactly or matches with a wildcard.
TOPIC objects are only conveniences and points of security. |
That is true but isn't it better when testing to have one defined then you know that it exists. |
It doesn't really make sure it exists. It merely say "If someone publishes to this topic property of this object,then apply these rules". It also says "If someone uses this topic object, then prepend the topicstr to their publish topic".
Yes, the usual cause of pub/sub messages "disappearing" is that the publisher and the subscriber don't agree on the topic.
Yes, if both used a topic object, then both sides would be using the same topic.
But that doesn't really help, particularly because it leads to an MQ administrator having to create a lot of topic objects.
Better idea is to have config files (or something nicer, like UDPs or node overrides) that holds the topic, and that makes sure they both match.
Also easy enough to check substatus to see what topics are being subscribed to.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Bichu |
| Posted: Fri Sep 23, 2016 6:02 am Post subject: |
|
|
Centurion
Joined: 16 Oct 2011
Posts: 124
Location: London
|
Hi all,
I have solved it by just adding the topic to the properties header and its working like a charm and modelling the input csv data
| Code: |
CALL CopyMessageHeaders();
SET OutputRoot.Properties.Topic = 'Site';
SET OutputRoot.DFDL.modelCSV_Msg = InputRoot.DFDL.modelCSV_Msg; |
Thanks all for your support. |
|
| Back to top |
|
|
| timber |
| Posted: Fri Sep 23, 2016 6:47 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1280
|
<twitch>
| Quote: |
| its working like a charm and modelling the input csv data |
Well...actually *you* did the modelling of the data when you created the DFDL schema. IIB is parsing the data using your model. But I'm glad it's working, and apologies for being picky about the terminology. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
