ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportCOA correlation issue

Post new topicReply to topic Goto page 1, 2  Next
COA correlation issue View previous topic :: View next topic
Author Message
ziliboba
PostPosted: Tue Sep 25, 2018 12:51 pm Post subject: COA correlation issue Reply with quote

Newbie

Joined: 25 Sep 2018
Posts: 3

Hello MQ Gurus, best wishes and seeking for advise on the below faced COA correlation issue.

Background: there is an application A which is feeding data to an application B via MQ (nothing special - remote queue def pointing to the local q def on remote QM). Where the sending app A is requesting COAs. That is a stable setup working for years: App A -> QM.A[Q1] -channel-> QM.B[Q2] -> App B
Here Q1 is a remote q def pointing to the Q2.

Problem: there is an application C which requires exactly the same data feed which A is sending to B via MQ. => it is required to duplicate data feed considering the following constraint.

Constraint: neither code, nor app config of applications A and B could be changed - duplication of the data feed from A to B should be transparent for applications A and B - A puts messages to the same queue Q1 on QM.A; B gets messages from the same queue Q2 on the QM.B

Proposed solution: duplicate the feed on MQ layer by creation of Topic/subscirbers configuration on the QM of the app B:

App A -> QM.A[Q1] -channel-> QM.B[QA->T->{S2,S3}->{Q2,Q3}] -> {App B, QM.C[Q4] -> App C}
Here:
Q1 - has the rname property updated to point to the QA for Topic instead of Q2
QA - Queue Alias for Topic T
T - Topic
S2, S3 - subscribers publishing data to the Q2 and Q3
Q2 - unchanged, the same local queue definition where App B consumes from
Q3 - remote queue definition pointing to the Q4
Q4 - local queue definition on the QM.C, the queue with copy of messages sent from A to B

With this set up duplication of the messages from the app A to the app B and C works fine.
But ... there is an issue

Issue: application A is not able to correlate COAs and that is the problem - after switch destination from queue to queue alias for topic, the COA messages (sent by the QM) are getting queued up on the reply queue.

I'm not sure if app A is not able to correlate COAs at all, or (what is more likely guess) it is not able to correlate additional COAs e.g. from the QM.C
And my MQ knowledge is not sufficient to easily understand the reason, can only guess and play around, but very limited in time to get the new config work or acknowledge that this won't work.

Any hint or advise is very much appreciated.


Last edited by ziliboba on Tue Sep 25, 2018 1:46 pm; edited 1 time in total
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Sep 25, 2018 1:25 pm Post subject: Re: COA correlation issue Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8221
Location: US: west coast, almost. Otherwise, enroute.

ziliboba wrote:
But application A is not able to correlate COAs and that is the problem.
I'm not sure if app A is not able to correlate COAs at all, or (what is more likely guess) it is not able to correlate additional COAs e.g. from the QM.C

"not able" is insufficient description of the problem. What messages arrive in the reply-to-queue? I'd expect to see a reply msg from app B. Did B's reply message arrive? Did app B move the inbound MsgId to the CorrelId of the reply message before it MQPUT the reply msg?

Did a COA message arrive in the reply-to-queue? If so, did it contain the original MsgId? If it did, then perhaps app A didn't set the match options when doing an MQGET from the reply-to-queue.

Again, you will need to provide more information than 'not able."
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
ziliboba
PostPosted: Tue Sep 25, 2018 1:44 pm Post subject: Reply with quote

Newbie

Joined: 25 Sep 2018
Posts: 3

Thanks, Bruce

Quote:
What messages arrive in the reply-to-queue?

-> COA messages which App A can't consume due to mismatch of correlation id e.g. App A can't correlate COAs with request messages
I only have a guess that now there are 2 COAs being sent back to the App A (instead of 1) - one from QM.B and another from QM.C. Where the first received COA is getting matched and marking the respective request record in database as successfully processed, so that when the 2nd COA with the same CorrID received - there is already nothing in database to match with. I don't have source codes of the App A, so can't check.

Quote:
I'd expect to see a reply msg from app B. Did B's reply message arrive?

-> COA is generated by the QM.B, not by the App B itself. Yes, COAs are arriving.

Quote:
Did app B move the inbound MsgId to the CorrelId of the reply message before it MQPUT the reply msg?

-> No, since COA is generated by the B's Queue Manager and not by the App B itself

Quote:
Did a COA message arrive in the reply-to-queue?

Yes, COAs arrives to the reply-to-queue

Quote:
If so, did it contain the original MsgId?

-> Not sure if it should pertain original MsgId or CorrId, but anyways here is the issue - App A can't correlate COAs because of it can't match the correlation id with that it has stored in the database. Probably because of duplicated COA e.g. first COA marked request record in database as replied => when another COA with the same CorrId arrives - there is already nothing in the database to match with.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Sep 25, 2018 7:08 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8221
Location: US: west coast, almost. Otherwise, enroute.

So, app A successfully MQGETs both COA messages. Fix the portion of thie DB process to tolerate 2 COAs. If your design asks for 2 COAs, you get 2 COAs.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
ziliboba
PostPosted: Tue Sep 25, 2018 8:32 pm Post subject: Reply with quote

Newbie

Joined: 25 Sep 2018
Posts: 3

bruce2359 wrote:
So, app A successfully MQGETs both COA messages. Fix the portion of thie DB process to tolerate 2 COAs. If your design asks for 2 COAs, you get 2 COAs.

That's the guess - I don't know for sure how many COAs are generated and how many consumed.

Quote:
Constraint: neither code, nor app config of applications A and B could be changed
- duplication of the data feed from A to B should be transparent for applications A and B - A puts messages to the same queue Q1 on QM.A; B gets messages from the same queue Q2 on the QM.B without any code or config changes in the Apps A, B [/quote]
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Sep 25, 2018 8:49 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8221
Location: US: west coast, almost. Otherwise, enroute.

Dont guess. You need to understand how this design works.

Fire up the IBM MQ Application Activity Trace (or equivalent) for both apps.
_________________
There are two types of people in this world:
1) Those that can extrapolate from incomplete data
Back to top
View user's profile Send private message
exerk
PostPosted: Wed Sep 26, 2018 12:37 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5961

To help me better understand: App A sends a request message and specifies a COA is to be returned - what options does App A require in the COA, for example MQRO_EXCEPTION_WITH_DATA, MQRO_PASS_CORREL_ID?

I suspect you may be hitting the issue of each of the outbound messages having different MsgId values, therefore, only one COA will match the CorrelId.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
hughson
PostPosted: Wed Sep 26, 2018 3:06 am Post subject: Reply with quote

Sentinel

Joined: 09 May 2013
Posts: 860
Location: Bay of Plenty, New Zealand

Remember that when a message is published to a topic, the Message ID is unique for each copy of the message sent (i.e. one to each publisher).

Since there are multiple messages, and thus multiple message IDs, the putting/publishing application is NOT returned the message ID of the (or any of the) messages.

COA relies on matching the message ID of the message you sent with the message ID of the returned report. You don't have the message ID of the message(s) you sent, so none of the returned reports will match.

Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
exerk
PostPosted: Wed Sep 26, 2018 3:10 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5961

hughson wrote:
...COA relies on matching the message ID of the message you sent with the message ID of the returned report. You don't have the message ID of the message(s) you sent, so none of the returned reports will match...

So not even one MsgId is returned? Another school day for me!
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
hughson
PostPosted: Wed Sep 26, 2018 3:25 am Post subject: Reply with quote

Sentinel

Joined: 09 May 2013
Posts: 860
Location: Bay of Plenty, New Zealand

exerk wrote:
hughson wrote:
...COA relies on matching the message ID of the message you sent with the message ID of the returned report. You don't have the message ID of the message(s) you sent, so none of the returned reports will match...

So not even one MsgId is returned? Another school day for me!
If memory serves, a unique ID is generated and returned in the MQMD to the MQPUT, but it is not the message ID of any of the published message copies.
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
exerk
PostPosted: Wed Sep 26, 2018 3:26 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5961

hughson wrote:
exerk wrote:
hughson wrote:
...COA relies on matching the message ID of the message you sent with the message ID of the returned report. You don't have the message ID of the message(s) you sent, so none of the returned reports will match...

So not even one MsgId is returned? Another school day for me!
If memory serves, a unique ID is generated and returned in the MQMD to the MQPUT, but it is not the message ID of any of the published message copies.

Thank you for the clarification
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
hughson
PostPosted: Thu Sep 27, 2018 2:27 am Post subject: Reply with quote

Sentinel

Joined: 09 May 2013
Posts: 860
Location: Bay of Plenty, New Zealand

exerk wrote:
hughson wrote:
exerk wrote:
hughson wrote:
...COA relies on matching the message ID of the message you sent with the message ID of the returned report. You don't have the message ID of the message(s) you sent, so none of the returned reports will match...

So not even one MsgId is returned? Another school day for me!
If memory serves, a unique ID is generated and returned in the MQMD to the MQPUT, but it is not the message ID of any of the published message copies.

Thank you for the clarification
I will add a further clarification after trying it (instead of relying upon my memory!)

If your MQPUT uses MQPMO_NEW_MSG_ID then a message ID will be generated and returned on output to the caller of the MQPUT. As noted above, this is not the message ID of any of the copies sent to subscribers.

If your MQPUT does not use that option, then the message ID will be all zeroes.

Cheers,
Morag

P.S. I think I've found a subject for the latest in my series of MQuirks blog posts!
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software


Last edited by hughson on Thu Sep 27, 2018 3:10 am; edited 1 time in total
Back to top
View user's profile Send private message Visit poster's website
exerk
PostPosted: Thu Sep 27, 2018 2:43 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5961

hughson wrote:
...If your MQPUT uses MQPMO_NEW_MSG_ID then a message ID will be generated and returned on output to the caller of the MQPUT. As noted above, this is not the message ID of any of the copies sent to subscribers.

If your MQPUT does not use that option, then the message ID will be all zeroes...

And still the RFE for all pub/sub messages to carry the MsgId of the 'original' is still outstanding. I know it breaks the paradigm, but in MQ-terms of native message duplexing, it would be a bonus to have for applications that are not (strictly speaking) pub/sub.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
hughson
PostPosted: Thu Sep 27, 2018 3:12 am Post subject: Reply with quote

Sentinel

Joined: 09 May 2013
Posts: 860
Location: Bay of Plenty, New Zealand

exerk wrote:
And still the RFE for all pub/sub messages to carry the MsgId of the 'original' is still outstanding.
Do you have the number to hand? I could add it to the blog post I have just written, maybe get some more votes on it.
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Back to top
View user's profile Send private message Visit poster's website
exerk
PostPosted: Thu Sep 27, 2018 5:02 am Post subject: Reply with quote

Jedi Council

Joined: 02 Nov 2006
Posts: 5961

hughson wrote:
exerk wrote:
And still the RFE for all pub/sub messages to carry the MsgId of the 'original' is still outstanding.
Do you have the number to hand? I could add it to the blog post I have just written, maybe get some more votes on it.

Morag, it's still 'Under Consideration', so not a definite NO at the moment...
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys.

Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexGeneral IBM MQ SupportCOA correlation issue
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.