| Author |
Message
|
| Leigh Kendall |
 Posted: Sun Apr 09, 2006 6:59 pm Post subject: Need recommendation: Queue Listener |
 |
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Setup: MQ 5.3 for Windows & VB.NET using MQ server bindings.
I'm developing a queue listener (Windows service) that upon detecting a new message, will fire an event to a dispatcher; similar to the new .NET Monitor in MQ 6.0.
After reading the MQ docs, searching the forum here and coding a few tests, I've come up with the following choices for listening on a queue, but would like a more experienced MQ developer to validate my decision or suggest a more appropriate solution. So to check/listen for a new message, I'm thinking:
1. Get/Backout under syncpoint
2. Check CurrentDepth
3. Browse first
Seems like option 1 or 3 would be the most appropriate solution; 3 probably being less CPU intensive than 1? From what I can gather, option 2 is NOT recommended for a variety of reasons.
I can't do the actual final get in the listening thread since I would like to let a dispatcher class know (via an event) that there is a message to be processed. In turn, the dispather will place a thread in the thread pool which will use COM+ get the message and do some db work under the DTC and this transaction needs to be in the same COM+ thread. Since the transaction may take anywhere up to 10 seconds I don't want to tie up the listener with this work, since my incoming requests will likely be in excess of 1 per 10 secs.
So is checking for the existince of a message using browse first a viable solution for a listener?
TIA...
_________________
Leigh Kendall |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Mon Apr 10, 2006 12:15 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Why reinvent the wheel ?
You can use plain old triggering with a frequency of every.
Let the trigger program kick off some process in the app server that does the work. Make sure to limit the number of threads allowed to the worker program in the appserver.
MQ on queue trigger -->triggered program --> appserver triggered worker prog --> MQ servicing triggered queue.
The only components to write are - the triggered program (standalone .NET talking to the app server)
- the worker process int the app server that can be triggered by the previous part. It must service the queue until it is empty (rc 2033)
- the resource limitation that gets invoked by the worker process on start up, it needs to be able to communicate across (keep track of) all instances of worker processes
Of course you could also write a connection pool and emulate JMS using the XMS package.... But then that would probably take you months longer than the above suggested solution.
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Apr 10, 2006 2:48 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Of course, Trigger On Every is not a best practice, and is not actually guaranteed to generate a trigger message for EVERY message on the triggered queue.
So it would be better to use triggering to initiate a "traffic cop" pattern, which is close to the original design.
But again, it's better to emulate the JMS behavior in which the subordinate thread is actually passed the message object, rather than having to hold an MQ connection and get messages on it's own. This allows you to better control the number of connections you have, and also handle transactionality in a manner that is abstracted from the message consumer.
It may, indeed, be better to simply use IA9H and see what facilities this gives you for handling this problem.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Leigh Kendall |
| Posted: Tue Apr 11, 2006 6:53 am Post subject: |
|
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Thanks for the feeback and I'll take it into consideration. However, given the fact that I would like/need to use the method I described, are there any issues with using one of the three methods I described outside of what I've already highlighted? I think BROWSE_FIRST is my best option of the three, correct?
Note: I DON'T have the luxury of using MQ 6.0 which includes the new built-in .NET Monitor which upon reading the very limited and not in-depth docs on it, sounds like it would work and this is what I'm trying to emulate in a way. On top of that, I need to be able to handle from start to finish within a single thread, simultaneous requests. They need to be self-contained because of the way COM+ works with threading, so the transaction root must establish all the connections to MQ and the DB on it's own COM+ controlled threading.
To respond to your suggestions -
fjb_saper:
Triggering; that was my initial thought, but I didn't see that it bought me much. And I felt if I could detect new incoming messages easily, I'll just trigger myself.
Jeff:
"...better to use triggering to initiate a "traffic cop" pattern, which is close to the original design." Whose design? Mine? Can you clarify?
"...subordinate thread is actually passed the message object..."
I can't do this for COM+ transactions. The root COM+ object must be able commit/rollback on it's own thread and since I'm putting this work into the thread pool, the object that would be passing the object in, would be gone and thus can't be the root COM+ object.
"IA9H"
Not familiar with this. What is it?
Thanks!...
_________________
Leigh Kendall |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Apr 11, 2006 6:59 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Support Pack IA9H, Message Service for .NET (XMS).
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 11, 2006 7:49 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Leigh Kendall wrote: |
Jeff:
"...better to use triggering to initiate a "traffic cop" pattern, which is close to the original design." Whose design? Mine? Can you clarify?
|
The q is triggered on First. When a message arrives, the listener is triggered to start. The listener then goes a keeps getting messages, handing them off to the dispatcher class. It has logic built in to determine how many instances of the dispatcher class it will start up. Once it hits that limit (limited for whatever reason) additional messages queue as they arrive would queue up.
But, what I am wondering is, why don't you just fire up n instances of the dispatcher to listen directly on the queue with a long Get with Wait, as soon as a message lands on the q, one of the n instances will pick it up immediatly. As long as you have n or fewer messages arriving, no queueing. If more arrive, they queue up. If that becomes a chronic problem, increase performance by bigger hardware, leaner code, more instances to begin with, etc.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Leigh Kendall |
| Posted: Tue Apr 11, 2006 9:11 am Post subject: |
|
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Jeff - checked out the XMS and it looks interesting. Unfortunately, we're already underway with the .NET MQ classes and since the .NET XMS is "as-is" that would be a no go here by the powers that be. Also, I saw in the supportpac notes, that it's has NOT been tested/verified to work with Local queues, only remote?? We also have to support "legacy" VMS/Cobol apps running MQ on VMS Alpha and other mainframes, so wide coverage I think is better with the MQI API according to IBM. More importantly though (for us) is distributed transaction support and I could not verify support for distributed transactions (XA compliant) with XMS. No where in the docs; only references local transactions. I currenly have a post on the IBM newsgroup asking for verification. Does anyone here know??
Hey Peter, another Hartford, CT guy!
"But, what I am wondering is, why don't you just fire up n instances of the dispatcher to listen directly on the queue with a long Get with Wait, as soon as a message lands on the q, one of the n instances will pick it up immediatly. As long as you have n or fewer messages arriving, no queueing. If more arrive, they queue up. If that becomes a chronic problem, increase performance by bigger hardware, leaner code, more instances to begin with, etc."
I thought of this pattern as well... using a cofig file setting for the number of listeners/threads to assign to the queue with a long wait; this would work. However, by using the approach of a single queue listener that does nothing but notify a dispatcher (event) whenever there is at least one message (browse_first) to handle, I'm thinking that that listener will be able to process messages as fast as they can come in. Then the Dispatcher class would then put into the .NET threadpool the actual worker thread request(s) to get the message and do work. This way, as long as I'm handling incoming requests without any bottleneck, I don't have to explicity create threads to manage the incoming queue; I just need to notify the dispatcher that a message exists, dispatcher submits requests into the threadpool, and .NET will manage the number of threads (threadpool) and management of them for me.
If you see holes in this, let me know... I appreciate all the input!
_________________
Leigh Kendall |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Apr 11, 2006 10:32 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| Leigh Kendall wrote: |
| I saw in the supportpac notes, that it's has NOT been tested/verified to work with Local queues, only remote?? |
That doesn't make any sense. Unless they haven't implemented anything that GETs messages. Which also doesn't make sense. Publishing is the *easy* part of JMS.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Leigh Kendall |
| Posted: Tue Apr 11, 2006 11:22 am Post subject: |
|
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Jeff - scratch that. I agree; of course I'm trying to find where I read that so I can REREAD it to make sure I'm not misquoting. Of course I can't find it now...  If I find the xref I'll post it...
_________________
Leigh Kendall |
|
| Back to top |
|
|
| Leigh Kendall |
| Posted: Tue Apr 11, 2006 11:51 am Post subject: |
|
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Found it... 
From:
ftp://ftp.software.ibm.com/software/integration/support/supportpacs/individual/ia9h.readme.txt
"Note 2: This release of XMS has been tested with IBM WebSphere MQ v6 install,
connecting to local or remote v6 queue managers and remote v5.3 queue managers.
Other combinations may work, but might not have been fully tested."
V6 local/remote; 5.3 remote. How 'bout 5.3 local? I can't imagine this was not tested, but it reads that way.
Anyhow, still "as-is" with no XA transactions makes it a no-go for us. I'll keep an eye on it down the road as it does look interesting. Thanks for the lead Jeff...
_________________
Leigh Kendall |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 11, 2006 12:12 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
I read that as they tested client connections to 5.3 QMs, but not local bindings mode on 5.3 QMs.
If you client connect to the 5.3 QM, you can still use both local and remote q defs defined on that QM.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Leigh Kendall |
| Posted: Tue Apr 11, 2006 12:27 pm Post subject: |
|
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Peter - just curious if you saw my response to you a few posts up? Just looking for someone to confirm whether browse_first would be an accurate way to check for the existense of 1 or more messages.
_________________
Leigh Kendall |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue Apr 11, 2006 12:33 pm Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
between the 3 choices you provided, I guess that's the best. make sure your app can handle a 2033 no message found error gracefully. Just because the listener browsed a message successfully is no guarantee the message will still be available a split second later.
Personally, I would still prefer plain old triggering, or a few instances of the consuming app directly hovering on the queue (get with waits) if the volume is higher and more consistent. Triggering is best for sporadic arrivals of messages.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Apr 11, 2006 12:35 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The difficulty with using browse is that you can't guarantee the synchronization between the two GETTERs. That is, you can't guarantee that between when your Listener Browses the message and when the dispatched connection tries to GET the message, that the message hasn't already been grabbed by someone else.
It's also a very intensive use of a queue - JMS MDB Listener Ports have to browse the queue continuously, and it causes a lot of overhead.
On the flip side, this is essentially exactly what the JMS MDB Listener Ports do - so you can say that it's the "right" thing to do.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Leigh Kendall |
| Posted: Tue Apr 11, 2006 1:23 pm Post subject: |
|
|
Acolyte
Joined: 10 Apr 2003
Posts: 66
Location: Hartford, CT
|
Peter -
"Personally, I would still prefer plain old triggering, or a few instances of the consuming app directly hovering on the queue (get with waits) if the volume is higher and more consistent. Triggering is best for sporadic arrivals of messages."
I understand the mechanics of triggering and all that, but I'm not sure how I'd effectively use it to create class instances. The trigger (on Windows) needs to launch an .exe right? My app is an .exe (Windows service), but I don't want to spawn multiple instances of the .exe; I really just want the trigger another instance of a class. How would I do this using a trigger?
_________________
Leigh Kendall |
|
| Back to top |
|
|
|
|
