| Author |
Message
|
| ewilliams |
 Posted: Wed Oct 12, 2005 10:36 pm Post subject: Queue as a queryable datasource? |
 |
|
Apprentice
Joined: 27 Nov 2002
Posts: 30
Location: Portland
|
This might be a way out there but are there any third party APIs or available source code out there so that one could run a 'query' against a queue? I'm thinking in terms of simple select type questions. It sure would be nice to Select MesssageID from Q.TEST.IN WHERE PutDateTime > 'three hours ago'. I know this isn't what queues are designed for. Just thinking it would be one of those nice to haves.
I have to write a little .Net Windows Service to remove messages from a queue (vendor app will be logging all outbound messages to this queue) that is x hours old and thought it would be nice to get a quick list of MessageIDs instead of having to go through each and every message in the queue. Would there be a better way of doing this?
Just thought I would touch base with all the experts here and see what options came out of this post.
Thanks,
Eric |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Thu Oct 13, 2005 3:18 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Basically, in order to do this, any program would have to browse the entire queue, build a set of indices, and then allow you to run your queries against those criteria.
And hope that the queue hasn't changed in the meantime.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| wschutz |
| Posted: Thu Oct 13, 2005 3:32 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
| I have to write a little .Net Windows Service to remove messages from a queue (vendor app will be logging all outbound messages to this queue) that is x hours old |
Are you removing and discarding these messages, or passing them on to the vendor's app (sorry, if its a stupid question, its because I'm still a little low on coffee here...)
If the former, why not use Expiry?
_________________
-wayne |
|
| Back to top |
|
|
| ewilliams |
| Posted: Thu Oct 13, 2005 7:52 am Post subject: |
|
|
Apprentice
Joined: 27 Nov 2002
Posts: 30
Location: Portland
|
So the vendor app (actually any application that we plug into the message bus) are going to be required to log their outbound messages to a normal processing queue, this is where a transaction broker will deal with them, as well as a second queue for logging purposes. This is so we can see all their messages in the order that they put them there for support, DR and when they are backed up they can be used to rerun actually transactions down the road for development etc.
I thought about expiry but the nature of the applications are that they may put in several thousand messages one hour and two or three the next. So when my service checks the queues to clean up messages it will do an initial check to make sure the depth isn't getting to close to full (maybe use a percentage threshold) and then starting cycling through and do a destructive get to clean up messages older than x hours.
This leads me to another question. What is the 'proper' way to check the depth of a queue, open inquiry or PCF? Both the MQ Server and the .Net service will be on the same machine so there is not need to connect remotely. Speaking of PCF, are there any good examples of how to use this in .Net? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 13, 2005 7:58 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Expiry works from the time the message is put, not at fixed intervals. So the frequency of the message output doesn't matter.
I think the decision to use Inquiry vs PCF depends on what you're trying to do. And also whether there is a command server running all the time or not...
And you might look at the Monitoring manual to see if you'd rather handle Queue Depth Events instead of periodically polling for queue depths.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Sat Oct 15, 2005 11:53 am Post subject: Re: Queue as a queryable datasource? |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
| ewilliams wrote: |
| This might be a way out there but are there any third party APIs or available source code out there so that one could run a 'query' against a queue? I'm thinking in terms of simple select type questions. It sure would be nice to Select MesssageID from Q.TEST.IN WHERE PutDateTime > 'three hours ago'. |
Hummm, I started a pet project 3 or 4 years called MQ/SQL to do exactly that. I built all the front-end stuff & keywords that I thought were important but everybody who I discussed it with said "why" or "why bother". You are the first person in all this time to request a "database-like" (i.e. SQL) interface for MQ.
Interesting, makes me think - I've always kinda thought it was a good idea.  
Anyway, on a totally different note, my MQ Visual Edit and MQ Batch Tooklit have the ability to:
- Delete messages "older than" a particular time
- Forward (i.e. move) messages to a different queue.
It would be a no brainer to add a new function that could Forward messages "older than" a particular time.
Let me know and I can add the feature because I am in the middle of updating MQ Visual Edit for the next release.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| ewilliams |
| Posted: Tue Dec 27, 2005 10:20 pm Post subject: |
|
|
Apprentice
Joined: 27 Nov 2002
Posts: 30
Location: Portland
|
Ok so hopefully I am a few months wiser. I have stumbled up the world of the exit MirrorQ and see the potential, actually it does everything I need it to do. All without having to try to convince vendors to add multiple queue outputs to their applications.
Although it isn't playing nice with the Biztalk MQ Adapter but I can post on that else where (some feel free to chime in on that one if interested).
Anyway, the mirroring of messages to my logging queues has pointed my down another road of the Perf Event. Wow that is some cool stuff, tells you the Queue Manager and the Queue that throws the high event but then I hit a road block or maybe IBM documentation road block (please someone at IBM give us better examples and API docs!! think ...MSDN...). How the heck are you supposed to read that thing? I've looked at the Java and C samples from IBM (mo01) but when trying to implement in C# and that convoluted namespace ... arrggg.... so many constants, no good way to know what they mean ...
So to my question? Has anyone gotten it to work? Code sample? I see a single application (Win Service) listening the Perf Event queue and going to work on the logging queue when the qmgr informs it via the event message, per jefflowrey comments. Date time isn't the right answer for me either. If there is no activity, like over the weekend, then come Monday I have an empty log queue! Thanks for that eye opener and suggestions guys/gals!
Hey Roger, how about a "clear" queue by size? Pass it a depth to purge at?
Oh and I would really be interested in "forward message to queue by x criteria" similar to all the clear functions you have built already into your product, which are stellar by the way!
Eric |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Tue Dec 27, 2005 10:35 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
| ewilliams wrote: |
| Hey Roger, how about a "clear" queue by size? Pass it a depth to purge at? |
Do you mean by message size?
| ewilliams wrote: |
| Oh and I would really be interested in "forward message to queue by x criteria" |
What criteria are you referring to? Are you referring to the "older than" criteria that I listed above?
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 28, 2005 4:36 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
OK so for your time problem
a) Assumption: It is safe to assume that the messages will be in the "mirror" queue by date time order of the put.
b) When triggered (x % full) just run a simple get against the queue.
As you retrieve the messages in the order they were put, you will start removing the oldest first... I guess you get the drift there. Just stop if date time of put later than desired cutoff.
c) I would not just discard the message but serialize it to some file for backup if needed.
d) Be careful with the Biztalk adapter. We evaluated it a few years back, and I sure do hope it's been enhanced because I was not too happy with the performance... With CSD > 05 and the .NET API you could if needed write your own.
Enjoy  |
|
| Back to top |
|
|
| ewilliams |
| Posted: Wed Dec 28, 2005 6:57 am Post subject: |
|
|
Apprentice
Joined: 27 Nov 2002
Posts: 30
Location: Portland
|
Roger - sorry for not being clearer. For the purge I meant a count. Lets say there is a queue with a current depth of 1200 messages and you only want 1000 in there. Pass the application 1000 for that particular function and the first 200 messages are deleted.
For the criteria thing. I was referring to your current clear functions. Clear Queue, Clear Queue by ID, Clear Queue by Date/Time, Clear Queue by Matched String. If you made that a submenu (just to save UI space) and added a sibling submenu that would be identical except that is was a Forward Message by instead of the listed Clear Queue by items above.
--
saper - we are on the same page there. The only thing I was hoping for was to use an event to start the process instead of having to poll a list of logging queues every x seconds or minutes. It is the whole waste of cycles argument, but if I can't figure out wow to make sense of the event message I'll end up going that route anyway.
As for the Biztalk adapter, not sure what version you tested but they actually rewrote it for Biztalk 2004. We were beta testers for it and got to work out a lot of the bugs in it. Without using a FIFO (correlation set) we were seeing about 50 messages/second - about 10 when the FIFO logic is there. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 28, 2005 2:08 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| ewilliams wrote: |
--
saper - we are on the same page there. The only thing I was hoping for was to use an event to start the process instead of having to poll a list of logging queues every x seconds or minutes. It is the whole waste of cycles argument, but if I can't figure out wow to make sense of the event message I'll end up going that route anyway. |
Totally I was thinking of a triggered process.
The trigger could come from a listener on the event queue evaluating the 80% full event on the queue...
| ewilliams wrote: |
As for the Biztalk adapter, not sure what version you tested but they actually rewrote it for Biztalk 2004. We were beta testers for it and got to work out a lot of the bugs in it. Without using a FIFO (correlation set) we were seeing about 50 messages/second - about 10 when the FIFO logic is there. |
Those numbers still looks somewhat low to me on the correl id check(FIFO) unless you are talking about really huge messages. The degradation should not be that big...
(I was thinking that it might be worth to write a little program that does nothing more than take from MQ and put to MSMQ and the other way round. Could be written in J# and be JMS compliant ?.... but I guess this new improved version makes this a moot point...)
Overall it looks like they managed quite a performance improvement since 2002.
Enjoy |
|
| Back to top |
|
|
| ewilliams |
| Posted: Wed Dec 28, 2005 2:22 pm Post subject: |
|
|
Apprentice
Joined: 27 Nov 2002
Posts: 30
Location: Portland
|
and them to the million dollar question. Has anybody cracked the MQEvent message with .Net? If so could maybe post/send some sample code on how to do it? I am mostly interested in the Q Full, High and Low events.
Thanks in advance! |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 28, 2005 3:12 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| ewilliams wrote: |
and them to the million dollar question. Has anybody cracked the MQEvent message with .Net? If so could maybe post/send some sample code on how to do it? I am mostly interested in the Q Full, High and Low events.
Thanks in advance! |
Have you looked at the MS0B package ? (PCF)
It is java and you might need a J# wrapper around it...
Enjoy |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Dec 28, 2005 5:37 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| ewilliams wrote: |
Roger - sorry for not being clearer. For the purge I meant a count. Lets say there is a queue with a current depth of 1200 messages and you only want 1000 in there. Pass the application 1000 for that particular function and the first 200 messages are deleted.
|
Do anything based on the q depth is pure folly. Expired Messages, Uncommitted messages and Priority confuse the issue to much. In your example: what if messages 1-200 are not committed yet, or what if new higher priority mesages sneak in ahead of these 200, or what if there are 1200 messages there, you want 1000, and by the time you click OK, 1000 more messages arrive (now you've deleted the first 1200, not 200), or what if messages keep arriving as this too keeps trying to get the number down to 1000, in effect making it a bottemless pit, or what if all 1200 are expired, and then you curse the tool for "deleting" 1200 not 200 messages, etc, etc, etc.....
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| ewilliams |
| Posted: Fri Dec 30, 2005 11:40 am Post subject: |
|
|
Apprentice
Joined: 27 Nov 2002
Posts: 30
Location: Portland
|
| PeterPotkay wrote: |
| Do anything based on the q depth is pure folly. Expired Messages, Uncommitted messages and Priority confuse the issue to much. In your example: what if messages 1-200 are not committed yet, or what if new higher priority mesages sneak in ahead of these 200, or what if there are 1200 messages there, you want 1000, and by the time you click OK, 1000 more messages arrive (now you've deleted the first 1200, not 200), or what if messages keep arriving as this too keeps trying to get the number down to 1000, in effect making it a bottemless pit, or what if all 1200 are expired, and then you curse the tool for "deleting" 1200 not 200 messages, etc, etc, etc..... |
Or it might behave like the application already does and deletes the messages that it is aware of when it opens up the queue with the message criteria selected. I do see your point what but what I do not see is a helpful suggestion or a better way to accomplish the result of our discussion |
|
| Back to top |
|
|
|
|
