| Author |
Message
|
| hal |
 Posted: Tue May 31, 2011 10:28 am Post subject: mirrorq and amqsaxe MQPMO Options MQPMO_NEW_MSG_ID |
 |
|
Acolyte
Joined: 07 Dec 2005
Posts: 67
Location: New York City, New York
|
I have identified a behavioral change in WebSphere MQ 7 on distributed platforms that negatively impacts MsgId message tracking, including applications based upon API exits (mirrorq) and pub/sub. From http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzal.doc/fg10820_.htm
| Quote: |
| Globally unique MsgIds improve serviceability because you can track messages across queue managers... |
The MQPMO Options MQPMO_NEW_MSG_ID attribute is being set internally after messages are put to queues in WebSphere MQ 7 but not in WebSphere MQ 6. The result is that WebSphere MQ 7 MQMD MsgId values change when messages traverse between queue managers on the first hop of a message flow. MQPMO Options differences can be validated by running the amqsput and amqsaxe IBM sample programs on both WebSphere MQ 6 and 7.
1.) Configure the amqsaxe sample exit as documented in the Information Center at http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzal.doc/fg17960_.htm
You need to set the MQAPI_TRACE_LOGFILE environment variable, for example on UNIX systems,
| Code: |
| export MQAPI_TRACE_LOGFILE=/var/mqm/logs/MqiTrace |
2.) For convenience, you may modify the amqsaxe0.c sample exit by adding debug details that clearly print the values of the MQPMO Options in the PutBefore and PutAfter MQPUT EntryPoints. Otherwise, you must decipher the existing amqsaxe hex dumps. For example,
| Code: |
/* Debug */
MQPMO pmo = {MQPMO_DEFAULT};
memcpy(&pmo,*ppPutMsgOpts, sizeof(MQPMO));
…
fprintf(fp, "PUT AFTER pmo.Options decimal %d\n", pmo.Options); |
API exit compilation details are at http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/index.jsp?topic=/com.ibm.mq.csqzal.doc/fg14160_.htm
3.) Note that in amqsput0.c, MQPMO Options is assigned to the default value for WebSphere MQ 6, but is assigned to “MQPMO_NO_SYNCPOINT | MQPMO_FAIL_IF_QUIESCING” in WebSphere MQ 7.
From /opt/mqm/inc/cmqc.h,
| Code: |
#define MQPMO_NONE 0x00000000
#define MQPMO_NO_SYNCPOINT 0x00000004
#define MQPMO_NEW_MSG_ID 0x00000040
#define MQPMO_FAIL_IF_QUIESCING 0x00002000 |
4.) Put a message to a local queue via amqsput in WebSphere MQ 6. Observe the expected MQPMO Options behavior in the amqsaxe logs:
| Quote: |
PUT BEFORE pmo.Options decimal 0
…
PUT AFTER pmo.Options decimal 0 |
5.) Put a message to a local queue via amqsput in WebSphere MQ 7. Observe the unexpected MQPMO Options behavior in the amqsaxe logs. The MQPMO_NEW_MSG_ID option is being set sometime between the PutBefore and the PutAfter invocations:
| Quote: |
PUT BEFORE pmo.Options decimal 8196
8196 decimal = 2004 hexadecimal => MQPMO_NO_SYNCPOINT | MQPMO_FAIL_IF_QUIESCING
PUT AFTER pmo.Options decimal 8260
8260 decimal = 2044 hexadecimal => MQPMO_NO_SYNCPOINT | MQPMO_FAIL_IF_QUIESCING | MQPMO_NEW_MSG_ID |
I have opened a PMR with IBM. Until a fix is provided, the mirrorq workaround is to explicity mask off the MQPMO_NEW_MSG_ID in both the PutAfter and Put1After MQPUT entry points:
| Code: |
/* Used to remove the MQPMO_NEW_MSG_ID MQPMO Options attribute */
MQLONG delPMOOptions = 0; |
...
Just after the pmo is memcpyed:
| Code: |
/********************************************************/
/* Remove the MQPMO_NEW_MSG_ID MQPMO Options attribute */
/* because the MsgId must be preserved, not regenerated.*/
/********************************************************/
delPMOOptions = MQPMO_NEW_MSG_ID;
pmo.Options =~ pmo.Options;
pmo.Options |= delPMOOptions;
pmo.Options =~ pmo.Options; |
.
I do not have a fix for the convoluted pub/sub mirrorq replacement described at
http://www.mqseries.net/phpBB2/viewtopic.php?t=51319
Hal |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue May 31, 2011 10:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I think you haven't properly identified the issue, but presumably the PMR will resolve that.
Please do provide an update when you get a resolution or explanation from IBM through the PMR. |
|
| Back to top |
|
|
| hal |
| Posted: Tue May 31, 2011 10:47 am Post subject: |
|
|
Acolyte
Joined: 07 Dec 2005
Posts: 67
Location: New York City, New York
|
Please clarify
| Quote: |
| haven't properly identified the issue |
I do not know the root cause (something in the bowels of WebSphere MQ 7) but I have indeed identified the issue. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue May 31, 2011 10:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| hal wrote: |
Please clarify
| Quote: |
| haven't properly identified the issue |
I do not know the root cause (something in the bowels of WebSphere MQ 7) but I have indeed identified the issue. |
What I mean is that you appear to have identified a change in behavior. I do not, however, believe that you have necessarily identified an issue, nor that you have necessarily isolated the change to the right thing.
It could just, for example, be a change in the default MQPMR record. Or something else similar, that is showing up here rather than somewhere else.
And again, this may actually be a documented change in behavior - but not where you went looking, because it's a consequence of something else.
Among other things, it could be a side effect of the way you compiled amqsaxe to add logging.
Do you see the same options change reflected when you use the precompiled amqsaxe and use MQ tracing to debug it?
But again, I expect that the PMR process will help you resolve this. |
|
| Back to top |
|
|
| hal |
| Posted: Tue May 31, 2011 11:26 am Post subject: |
|
|
Acolyte
Joined: 07 Dec 2005
Posts: 67
Location: New York City, New York
|
| Quote: |
| Do you see the same options change reflected when you use the precompiled amqsaxe and use MQ tracing to debug it? |
Yes. Please try to reproduce the procedure as I have described using the original amqsaxe API sample exits. You will need to translate the hexadecimal values to extract the MQPMO Options. You will notice that in between the PutBefore and the PutAfter calls the MQPMO Options change. This is not the expected behavior. Are you aware of any changes in WebSphere MQ 7 that explains this? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue May 31, 2011 11:37 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| hal wrote: |
| Quote: |
| Do you see the same options change reflected when you use the precompiled amqsaxe and use MQ tracing to debug it? |
Yes. Please try to reproduce the procedure as I have described using the original amqsaxe API sample exits. You will need to translate the hexadecimal values to extract the MQPMO Options. You will notice that in between the PutBefore and the PutAfter calls the MQPMO Options change. This is not the expected behavior. Are you aware of any changes in WebSphere MQ 7 that explains this? |
As I said, I expect the PMR process will resolve this.
And, no, I'm not aware of any changes in v7 that would explain this, but that is merely a lack of effort on my part and not a statement on whether such a change exists or not. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue May 31, 2011 12:35 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The initial value of the OPTIONS field of the PMO structure is zero for both v6 and v7. MQPMO_NEW_MSGID is an application-defined option.
It is possible (and likely) that sample program behaviors may have changed between v6 and v7.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| hal |
| Posted: Thu Jun 02, 2011 1:28 pm Post subject: |
|
|
Acolyte
Joined: 07 Dec 2005
Posts: 67
Location: New York City, New York
|
As I have already said,
| Quote: |
| 3.) Note that in amqsput0.c, MQPMO Options is assigned to the default value for WebSphere MQ 6, but is assigned to “MQPMO_NO_SYNCPOINT | MQPMO_FAIL_IF_QUIESCING” in WebSphere MQ 7. |
I may be missing something but I am unable to find anything in amqsaxe0.c or included header files that explicitly assigns the MQMPO Options MQPMO_NEW_MSG_ID attribute. (The amqsaxe binary is built with the libmqm, libmqmcs, libmqmzse, libmqmzf and libmqz libraries. Perhaps someone from IBM who has access to library source code could perform a similar search and enlighten us. ) And I am using the exact same mirrorq.c source code for both WebSphere MQ 6 and 7.
I have also observed that the pub/sub mirrorq "replacement" described at http://www.mqseries.net/phpBB2/viewtopic.php?t=51319 does not clone MsgIds. Try changing the queue that an MQPUT application uses into an alias queue which points to a topic by following that procedure. You will see that the MsgId values in the original and mirrored queues (Q1 and Q2) are different. Not sure if this is a coincidence or a WebSphere MQ 7 MQPMO_NEW_MSG_ID pattern. In any case, pub/sub cannot be used to track messages by MsgId. |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Jun 03, 2011 6:15 am Post subject: Re: mirrorq and amqsaxe MQPMO Options MQPMO_NEW_MSG_ID |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| hal wrote: |
| The MQPMO Options MQPMO_NEW_MSG_ID attribute is being set internally after messages are put to queues in WebSphere MQ 7 but not in WebSphere MQ 6. The result is that WebSphere MQ 7 MQMD MsgId values change when messages traverse between queue managers on the first hop of a message flow. MQPMO Options differences can be validated by running the amqsput and amqsaxe IBM sample programs on both WebSphere MQ 6 and 7. |
amqsput supplies MQMI_NONE in the MD.MsgId
This causes the queue manager to generate a new message id, just the same as if you had included MQPMO_NEW_MSG_ID in PMO.Options.
So I am not surprised you get a new MsgId for your message when using amqsput.
However it does seem to be a change that the altered PMO.Options, including the MQPMO_NEW_MSG_ID implied by your empty input MD.MsgId, are reflected back as output to the app. I do not know if this is "by design".
<< UPDATE: The above paragraph is not strictly correct; see next posts. >>
The manual here http://publib.boulder.ibm.com/infocenter/wmqv7/v7r0/topic/com.ibm.mq.csqzak.doc/fr14060_.htm says that PMO.Options is an input field. From the evidence here it appears to be operating as an input/output field, and that might be a bug in the manual page, or in the code.
Hope this helps
Last edited by mvic on Fri Jun 03, 2011 7:37 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 03, 2011 7:07 am Post subject: Re: mirrorq and amqsaxe MQPMO Options MQPMO_NEW_MSG_ID |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| mvic wrote: |
| However it does seem to be a change that the altered PMO.Options, including the MQPMO_NEW_MSG_ID implied by your empty input MD.MsgId, are reflected back as output to the app. I do not know if this is "by design". |
Remember that the 'app' that is seeing the altered PMO Options is actually an exit.
So it may not be that any "app" would see this as an output field. |
|
| Back to top |
|
|
| mvic |
| Posted: Fri Jun 03, 2011 7:32 am Post subject: Re: mirrorq and amqsaxe MQPMO Options MQPMO_NEW_MSG_ID |
|
|
Jedi
Joined: 09 Mar 2004
Posts: 2080
|
| mqjeff wrote: |
Remember that the 'app' that is seeing the altered PMO Options is actually an exit.
So it may not be that any "app" would see this as an output field. |
I'm glad you're here to spot my mistakes.
I just added a couple of printf's to my amqsput, and you are quite right.
So, as far as the app is concerned,
pmo.Options in = 0x00002004
pmo.Options out = 0x00002004
No change.
So the OP has found a change somewhere lower than the MQPUT, that still affects amqsaxe or (I would assume) any API exit. It looks like the behaviour of MQPUT "at the interface" is still ok.
Unless I missed something. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 03, 2011 7:46 am Post subject: Re: mirrorq and amqsaxe MQPMO Options MQPMO_NEW_MSG_ID |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| mvic wrote: |
| mqjeff wrote: |
Remember that the 'app' that is seeing the altered PMO Options is actually an exit.
So it may not be that any "app" would see this as an output field. |
I'm glad you're here to spot my mistakes. |
I'm always happy to help.
Especially when all I'm doing is returning the favor. |
|
| Back to top |
|
|
| hal |
| Posted: Fri Jun 03, 2011 9:07 am Post subject: |
|
|
Acolyte
Joined: 07 Dec 2005
Posts: 67
Location: New York City, New York
|
Anyway, WebSphere MQ 7 pub/sub and API exits both degrade MsgId message tracking serviceability. But I found another example that indicates it may not be systemic.
WebSphere Message Broker provides multiple facilities to clone messages. (Sorry about mentioning Broker details in
a WebSphere MQ forum, but I believe those details are relevant.) Using WebSphere Message Broker 7.0.2, I deployed a very simple message flow with one MQInput node that connects to two MQOutput nodes. I put a message on the queue associated with the input node. That message was cloned to two distinct queues associated with the output nodes. The MsgIds of the two output queue messages were identical. Next I altered the message flow to chain additonal MQOutput nodes whose properties were linked to more queues; I connected MQOutput node 2 to MQOutput node 3 which in turn connected to MQOutput node 4. When I put a message to the queue linked to the input node, messages arrived on four separate queues all with the same MsgId.
Last edited by hal on Fri Jun 03, 2011 9:21 am; edited 1 time in total |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Jun 03, 2011 9:17 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| hal wrote: |
| When I put a message to the queue linked to the input node, messages arrived on four separate queues all with the same MsgId. |
Were any of the MQOutput nodes configured to request a new message id (in the Advanced tab)?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| hal |
| Posted: Fri Jun 03, 2011 9:46 am Post subject: |
|
|
Acolyte
Joined: 07 Dec 2005
Posts: 67
Location: New York City, New York
|
| The "New message ID" option is not set in any of my output nodes and the behavior is as expected. The API exits do not explicitly assign MQPMO_NEW_MSG_ID and the behavior is unexpected. Not sure about pub/sub. |
|
| Back to top |
|
|
|
|
