| Author |
Message
|
| DAS |
 Posted: Sat Dec 17, 2005 7:44 am Post subject: Still flaws in MirrorQ |
 |
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
I have read through about each and every topic about the MirrorQ application, but I am still having a major problem.
Platform and used software:
- Windows XP
- Websphere Message Broker 5.0.6
- MQSeries 5.3
- Latest mirrorq package from IBM site
The problem:
When I put a message onto a queue which I added to the namelist mirrorq uses the message is getting copied when I put it using RFH Util.
Though, when the broker itself puts a message onto the queue, the message is not copied.
I have read the post regarding this problem, in which the person said you simply had to remove the part where the SET_ALL_CONTEXT option is set as an PMO option, so I changed the properties of my MQOutput node to all possible options (Pass All, Set All, Default, None), but with no result.
Indeed, when I set the Set All Context option in RFH Util, the mirrorq exit does not copy the message, so I bet it has something to do with the Context fields. Really, how stupid would it be if MQ skips an API Exit based on header contents??
Can someone help me out here?
Thanks in advance! |
|
| Back to top |
|
 |
| DAS |
| Posted: Sun Dec 18, 2005 6:16 am Post subject: |
|
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
|
| Back to top |
|
|
| wschutz |
| Posted: Sun Dec 18, 2005 6:37 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| DAS wrote: |
| Solved it already. |
For the benefit of others, what was the problem?
_________________
-wayne |
|
| Back to top |
|
|
| DAS |
| Posted: Sun Dec 18, 2005 9:06 am Post subject: |
|
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
In some post (I don't remember which one anymore) I read about the fact that my problem occurs, because the MirrorQ Exit uses the MQOO_OUTPUT | MQOO_FAIL_IF_QUIESCING Options for the MQOPEN of the target queue. By simply adding the MQOO_SET_ALL_CONTEXT flag, the problem is gone. I simply assumed (yes I know, assumption is the mother of all ***) that C. Lindsey had already fixed this in the latest package. As this was not true I made the change myself and the problem was gone.
As Charly suggested, it would be wiser to save the open options used to open the source queue and use those options to open the target queue as well. I tried this (adding an attribute to the mirrorqueuelist struct) and saved the options in the MQ_OPEN_AFTER exit and setting them at the place the target queue is opened, though...that didn't work very well.
So, in short...simply adding the SET_ALL_CONTEXT when opening the target queue solved my problem.
Thanks to the person who came across this problem in the first place and fixed it. |
|
| Back to top |
|
|
| clindsey |
| Posted: Sun Dec 18, 2005 9:08 am Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
In the next update to mirrorq, it saves the open options of the original open and uses these options to open the target queue. This solves the context problem.
I hope to have this update on the web site soon.
Charlie |
|
| Back to top |
|
|
| DAS |
| Posted: Sun Dec 18, 2005 9:23 am Post subject: |
|
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
Charlie, doing this gave me problems. I saved the options in the MQ_OPEN_AFTER and passed them at the place where the target queue is opened (if it wasn't opened already). The code I added in the MQ_OPEN_AFTER worked like this:
if the source queue which was opened is in the list of queues to mirror, I get the particular mirrorqueuelist entry and set an extra attribute (MQLONG with name oOptions) to hold the open options (*pOptions).
I assume this is the correct way, but it does not give me the expected results. Well, I will experiment with this some more and will post my findings on this forum. |
|
| Back to top |
|
|
| clindsey |
| Posted: Sun Dec 18, 2005 2:27 pm Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
That sounds very close to the change I made. I have not seen any problems yet. I will send you my latest code and let you try it.
Charlie |
|
| Back to top |
|
|
| DAS |
| Posted: Mon Dec 19, 2005 4:25 am Post subject: |
|
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
Thanks for your code Charlie. It is essentially the same indeed.
But, silly me...it was not the API Exit that didn't function now, it was me setting the Message Context to None in my MQOutput node 
But now I am noticing other problems which made me decide to remove the change and simply add the SET_ALL_CONTEXT flag always to the open options (NOT saving the original open options!).
The problem:
Simple message flow, one MQInput, one MQOutput, Message Context of MQOutput is Pass All. Both queues are being mirrored by mirrorq
Now, when I put a message on the input queue with either the Set All Context or Set Iden Context (in RFH Util) the message from the OUTPUT queue is correctly mirrored, but the message from the INPUT queue is not mirrored at all.
Quick testing with always setting MQOO_OUTPUT|MQOO_FAIL_IF_QUIESCING|MQOO_SET_ALL_CONTEXT fixes this issue, so saving the Open options does not work.
Maybe this has negative consequences, but if this isn't the case I think your latest package should always set the MQOO_SET_ALL_CONTEXT flag. |
|
| Back to top |
|
|
| billybong |
| Posted: Tue Dec 20, 2005 12:58 pm Post subject: |
|
|
Disciple
Joined: 22 Jul 2005
Posts: 150
Location: Stockholm, Sweden
|
Sorry for not reporting when I solved the problem earlier. This is the options I use on the target queue.
pmo is the old pmo from the message.
pmo.Options=~pmo.Options;
pmo.Options|=MQPMO_PASS_IDENTITY_CONTEXT;
pmo.Options|=MQPMO_PASS_ALL_CONTEXT;
pmo.Options|=MQPMO_SET_IDENTITY_CONTEXT;
pmo.Options=~pmo.Options;
pmo.Options|=MQPMO_SET_ALL_CONTEXT; |
|
| Back to top |
|
|
| DAS |
| Posted: Wed Dec 21, 2005 7:43 am Post subject: |
|
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
Thanks for you reply!
Though, I was only having the problem when opening the target queue. I see you are clearing the PASS_IDENT, PASS_ALL and SET_IDENT flags and you only set the SET_ALL_CONTEXT. Could you tell me why you clear out the others? Doesn't SET_ALL_CONTEXT automatically override the other context flags?
And can you please tell me why you change the Put options too? As you are setting the SET_ALL_CONTEXT flag I am assuming that character fields from the context are filled in with spaces and numerical fields with zeroes, or am I mistaken?
I think that for a pure 100% copy you should do the following:
- Save open options from source queue
- Use open option from source queue to open target queue
- Save put options from source queue
- Use put options from source queue to write to target queue
Doesn't this give you a good copy (in theory it does I think)? I think this way you will always put messages in the same way on the target queue as they arrived on the source queue, thus effectively mirroring the actions which were done to the source queue. I think this will make the tampering with context flags unneccesary.
Charlie -> can you give me your view on this? |
|
| Back to top |
|
|
| clindsey |
| Posted: Wed Dec 21, 2005 4:06 pm Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
| Quote: |
- Save open options from source queue
- Use open option from source queue to open target queue
- Save put options from source queue
- Use put options from source queue to write to target queue
|
Das,
I agree with you that this seems to be the correct strategy. I thought this is what I as doing with the latest code that I sent you. It could be that I need to hang on to some of the context information in the MQOD as well.
It would be helpful if you could try that code again that I sent you with mq trace running so I can see why it fails for your scenario.
From a command prompt, enter strmqtrc -t ALL , run the scenario, and enter endmqtrc. Zip up the *.trc files in <mq>\errors and email them to me. Just reply to the email I sent you.
Charlie |
|
| Back to top |
|
|
| DAS |
| Posted: Thu Dec 22, 2005 8:02 am Post subject: |
|
|
Apprentice
Joined: 15 Aug 2005
Posts: 30
|
Charlie,
I have figured out the problem already and it has to do with the caching of the handle of the target queue.
Because you are caching the handle of the target queue and therefore the open options too, you are not effectively mirroring, but you are only setting the Open options only ONE time per source queue!!
So in concreto this means that the open options used for opening the very first source queue will (during the lifetime of the MirrorQ exit) always be used for opening the target queue belonging to that source queue too!! All my findings have confirmed this to be the problem (reason code 2095 when enabling tracing of the MirrorQ exit).
Of course I had to confirm this to be true, so I did the following:
- Use my standard message flow also used for previous tests
- Run your latest MirrorQ exit (without modification)
- Use RFH Util for a series of test rolls
Now I fired up RFH Util, did nothing with the context options and I wrote a message to the input queue. Of course, the message from the input AND the output queue were perfectly mirrored by MirrorQ.
Under the hood though, the handle of the target queue got cached AND
the open options WITHOUT the SET_ALL_CONTEXT flag got stored too for the source queue (the Input queue)!
Now I checked the Set All Context flag in RFH Util and with the new settings I wrote a new message in the Input queue. I think you will imagine by now that the Input message did NOT get copied by MirrorQ, though the Output message did (precisely according to my problem). And why is that?
First put in RFH Util
Open options do NOT include SET_ALL_CONTEXT!
What happens in MirrorQ:
- open options of Input queue are stored (no SET_ALL_CONTEXT!)
- put options of Input queue are stored
- target queue is opened with the stored open options
- target queue is written to with the stored put options
Second put in RFH Util
Open options DO include SET_ALL_CONTEXT
What happens in MirrorQ:
- open options of Input queue are NOT stored, as the target queue handle was cached and the open options too.
- put options of Input queue are stored (they include SET_ALL_CONTEXT!)
- target queue is written to with the stored put options (which DO include SET_ALL_CONTEXT)
Obviously at this point, a RC of 2095 occurs(MQRC_NOT_OPEN_FOR_SET_ALL)
It's very easy to see now that this happens, the RC perfectly describes the problem. We opened the target queue without SET_ALL_CONTEXT, but we are trying to write to the queue with the SET_ALL_CONTEXT flag enabled. Obviously this is an illegal act 
Now for the solution:
Caching of the target queue handle is of course good as far as it goes for speeding things up. With your code only one minor change must be performed:
When it's time to get the cached target queue handle and you received the cached handle you must first check if the open options used for opening the source queue (which actions you are now mirroring) are still equal to the cached open options. If not, close the target queue and reopen it using the correct open options. This gives you the benefits of handle caching along with a more solid mirroring approach.
Are you going to make the needed change and post me the altered source?
Thanks in advance!
Btw....I was wondering if the MirrorQ application is also able to mirror messages from source queue, where the source queue is a remote queue. I know the target queue can be located remotely, but what about remote source queues? Is the PutAfter callback triggered by MQPUT(1) operations on a remote queue? |
|
| Back to top |
|
|
| clindsey |
| Posted: Thu Dec 22, 2005 2:57 pm Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
Excellent find DAS!
Thanks. You saved me a lot of time on this one. I will make the change and send you a new copy.
| Quote: |
Btw....I was wondering if the MirrorQ application is also able to mirror messages from source queue, where the source queue is a remote queue. I know the target queue can be located remotely, but what about remote source queues? Is the PutAfter callback triggered by MQPUT(1) operations on a remote queue?
|
This works as long as you do NOT export the environment var MIRRORQ_NO_INTERNAL. Setting that variable prevents the channel agent from loading the api exit and it is the channel agent that does the put at the remote queue manager. Of course, mirrorq has to be configured on the qmgr where the remote queue is.
Charlie
Last edited by clindsey on Thu Dec 22, 2005 3:02 pm; edited 1 time in total |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Dec 22, 2005 2:59 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| clindsey wrote: |
| Of course, mirrorq has to be configured on the qmgr where the remote queue is. |
Would you mind clarifying this a bit?
If I have a QRemote on QMGRA, that points to a QLOCAL on QMGRB, then where do I need MirrorQ configured, to get this to work?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| clindsey |
| Posted: Thu Dec 22, 2005 3:58 pm Post subject: |
|
|
Knight
Joined: 12 Jul 2002
Posts: 586
Location: Dallas, Tx
|
| Quote: |
Would you mind clarifying this a bit?
If I have a QRemote on QMGRA, that points to a QLOCAL on QMGRB, then where do I need MirrorQ configured, to get this to work?
|
Jeff, I am glad you asked because I made the assumption that since DAS said he knew you could mirror to a remote queue from the local qmgr, that he was looking at this from the perspective of the remote queue manager. I think it is easier to think of the remote qmgr (QMGRB) in local terms.
If you want to mirror messages that arrive on QLOCAL at QMGRB, then you set up mirrorq on QMGRB with the source queue as LOCALQ and the target as some other local queue on QMGRB. The messages are mirrored if the application is local, but also if the application is on QMGRA and it puts to QRemote.
To go a step further though with your scenario, it depends on where you want the copy to go. It could be that you want a local copy on QMGRA of all messages that are put to QRemote. Then you would set up mirrorq on QMGRA with the source queue being QRemote and the target being some other local queue. It is also possible for the target to also be another remote queue definition on QMGRA. This would be an alternative to what I suggested to DAS, as you could copy to a target queue on QMGRB with mirrorq only defined on QMGRA. I would only do that in a case where you could not define mirrorq on QMGRB for some reason.
I hope that clears it up.
Charlie |
|
| Back to top |
|
|
|
|
