| Author |
Message
|
| nickkirk |
 Posted: Tue Oct 07, 2008 8:53 am Post subject: Control when messages are taken off the Input Queue |
 |
|
Apprentice
Joined: 26 Sep 2008
Posts: 48
|
Is there a way to control when messages are taken off an Input Queue?
Requirement:
- I need to be able to take one message off the Input Queue if a 'Failure' Queue is empty. Otherwise, do not take messages off the Input Queue.
Can a timer be used with the Input queue so that it polls the Input Queue at regular intervals ? |
|
| Back to top |
|
 |
| hopsala |
| Posted: Tue Oct 07, 2008 9:24 am Post subject: Re: Control when messages are taken off the Input Queue |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| nickkirk wrote: |
| Can a timer be used with the Input queue so that it polls the Input Queue at regular intervals ? |
Yes. Use a TimeOutNotification node, coupled with an MQGET node for the Failure queue, and another MQGET for the Input queue. You cannot control message processing using a regular MQInput node.
So technologically speaking, this is possible. But I must say your design sounds curious at best. Are you sure such a functionality is what you need? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 07, 2008 11:20 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Reeks of "message affinity" to me.   enough said 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nickkirk |
| Posted: Tue Oct 07, 2008 12:20 pm Post subject: |
|
|
Apprentice
Joined: 26 Sep 2008
Posts: 48
|
| fjb_saper, I hope all your other posts are more intelligent than this latest one ! |
|
| Back to top |
|
|
| bower5932 |
| Posted: Tue Oct 07, 2008 12:29 pm Post subject: |
|
|
Jedi Knight
Joined: 27 Aug 2001
Posts: 3023
Location: Dallas, TX, USA
|
| nickkirk wrote: |
| fjb_saper, I hope all your other posts are more intelligent than this latest one ! |
Do a search on "message affinity". This has been discussed numerous times with reasons given why it is not something to do. |
|
| Back to top |
|
|
| JosephGramig |
| Posted: Tue Oct 07, 2008 12:45 pm Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Are you asking if you can read the message under sync point and then commit the work if you succeed?
Yes, read the programmers guide and see. |
|
| Back to top |
|
|
| nickkirk |
| Posted: Tue Oct 07, 2008 1:11 pm Post subject: |
|
|
Apprentice
Joined: 26 Sep 2008
Posts: 48
|
hopsala, thankyou.
This will work a treat !
Unfortunately, this is the requirement.
There is data coming from a legacy application, getting processed nightly via a message flow. The incomming data (only one large message per night) is data that has 'changed' since the previous day. Therefore, this data must be processed in order so that the 'state' change can be recorded.
Cheers |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Oct 07, 2008 2:42 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| nickkirk wrote: |
hopsala, thankyou.
This will work a treat !
Unfortunately, this is the requirement.
There is data coming from a legacy application, getting processed nightly via a message flow. The incomming data (only one large message per night) is data that has 'changed' since the previous day. Therefore, this data must be processed in order so that the 'state' change can be recorded.
Cheers |
And the way to do away with message affinity here, is instead of sending just the changed fields, send the whole object if something changed. Make sure the message/object has a timestamp. This could lead you to split your objects into more manageable parts...
On the receiving side record the timestamp of the change and discard messages with an older timestamp as no longer relevant.... byebye message affinity...
and see I was right you had message affinity in your design... and did not get back to the drawing board to try and get rid of it...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| nickkirk |
| Posted: Tue Oct 07, 2008 10:07 pm Post subject: |
|
|
Apprentice
Joined: 26 Sep 2008
Posts: 48
|
fjb_saper, not quite as easy as this.
As I said, the interfaces to the broker are legacy apps. They can NOT be modified. They are working fine.
Part of the idea for using the broker was so that the existing apps wouldn't need to be changed.
However, thanks for you ideas. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Oct 08, 2008 2:18 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| nickkirk wrote: |
fjb_saper, not quite as easy as this.
As I said, the interfaces to the broker are legacy apps. They can NOT be modified. They are working fine.
Part of the idea for using the broker was so that the existing apps wouldn't need to be changed.
However, thanks for you ideas. |
Agreed. However most of the time "cannot be changed" is a cost benefit ratio.
And most of the "canned" legacy apps that are smart enough to generate a delta message can also generate a full message...
The tough analysis is to determine what changes often and if it warrants a separate object to be implemented in the interface... The rest is mostly processing ease and error handling vs bandwidth... as I said cost/benefit and you may end up being on the wrong side of the stick stuck with message affinity...
Message affinity is such a bad proposition because it prevents you from true load balancing between multiple processing instances (no vertical and no horizontal scaling).
The other way to alleviate it is to simply stick the messages into a DB and let some process on the DB determine whether it can continue processing or must wait for a message to be resolved (the msg may be on a DLQ along the path) ...
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| dkeister |
| Posted: Wed Oct 08, 2008 5:28 am Post subject: |
|
|
Disciple
Joined: 25 Mar 2002
Posts: 184
Location: Purchase, New York
|
As I think I now understand the requirement (in middle of thread):
Unfortunately, this is the requirement.
There is data coming from a legacy application, getting processed nightly via a message flow. The incomming data (only one large message per night) is data that has 'changed' since the previous day. Therefore, this data must be processed in order so that the 'state' change can be recorded.
I had what I think is a similar situation where I needed to compare the incoming data from today with the 'last successful message' and look for changes. My solution was to take the data from the message and put it into a database. Then I issued a SQL search to find records that were new, changed, deleted and do the appropriate processing for each case. (In my case, it was to put an output message to another application for processing.)
Since this happened only once a day and the message arrived 'after hours' the database performance was not really an issue.
_________________
Dean Keister |
|
| Back to top |
|
|
| hopsala |
| Posted: Wed Oct 08, 2008 12:02 pm Post subject: |
|
|
Guardian
Joined: 24 Sep 2004
Posts: 960
|
| dkeister wrote: |
Unfortunately, this is the requirement.
There is data coming from a legacy application, getting processed nightly via a message flow. The incomming data (only one large message per night) is data that has 'changed' since the previous day. Therefore, this data must be processed in order so that the 'state' change can be recorded. |
I see. So that's why you need the failure queue checking. A strange, but interesting, design. I think you'll be better off with fjb's offer, though.
Assuming you're going to go ahead with your design, note that you should always have a TimeOutControl connected to some MQ control queue somewhere in your broker, so you can control execution time in case of failures. Otherwise, if the TimeOutNotification didn't start the flow at the expected time (broker failure, for example) you can start it without having to redeploy.
Search the forum or the lit, this is amply specified. For example http://www.mqseries.net/phpBB2/viewtopic.php?t=45310&highlight=timeoutcontrol. |
|
| Back to top |
|
|
|
|
