| Author |
Message
|
| JoeConnolly |
 Posted: Thu Oct 24, 2013 1:11 pm Post subject: Time based queue in .NET |
 |
|
Newbie
Joined: 10 Jul 2012
Posts: 2
|
What is the best way to implement a time based queue in MQ on .NET?
Each message must not be processed until a specific time.
Each message has an execution time between now and 60 minutes from now.
Assume multiple writers and readers.
We tried this in an Oracle database and it could not handle the load which is over 400 reads and writes per second.
The exact time at which the message can be processed is stored as a long message property of the form yyyymmddhhmmss.
Options considered include: MQ .NET API Browse and XMS
MQ .NET API using Browse
Write code snippet
| Code: |
MQMessage message = new MQMessage();
message.Persistence = MQC.MQPER_PERSISTENT;
message.WriteObject(object); //around 50k
message.SetInt8Property("RetryDateTime", GetTimestampAsLong(RetryDateTime));
MQPutMessageOptions putMessageOptions = new MQPutMessageOptions();
putMessageOptions.Options = MQC.MQPMO_FAIL_IF_QUIESCING | MQC.MQPMO_SYNCPOINT;
mqConnection.Queue.Put(message, putMessageOptions); |
Read code snippet
| Code: |
MQMessage message = null;
try
{
mqConnection = GetConnection();
MQGetMessageOptions getMessageOptions = new MQGetMessageOptions();
getMessageOptions.Options = MQC.MQGMO_FAIL_IF_QUIESCING | MQC.MQGMO_NO_WAIT | MQC.MQGMO_BROWSE_FIRST | MQC.MQGMO_LOCK;
long utcNow = GetTimestampAsLong(DateTime.UtcNow);
// this loop will either find a message and break or throw a no msg found exception.
while (true)
{
message = new MQMessage();
mqConnection.Queue.Get(message, getMessageOptions);
long nextRetryDateTime = message.GetInt8Property("RetryDateTime");
if (nextRetryDateTime < DateTime.UtcNow)
{
getMessageOptions.Options = MQC.MQGMO_FAIL_IF_QUIESCING | MQC.MQGMO_NO_WAIT | MQC.MQGMO_MSG_UNDER_CURSOR;
mqConnection.Queue.Get(message, getMessageOptions);
break;
}
getMessageOptions.Options = MQC.MQGMO_FAIL_IF_QUIESCING | MQC.MQGMO_NO_WAIT | MQC.MQGMO_BROWSE_NEXT | MQC.MQGMO_LOCK;
}
myObject = (MyObject) message.ReadObject();
}
catch (MQException mqException)
{
if (mqException.Reason == 2033) myObject = null;//MQRC_NO_MSG_AVAILABLE
else throw;
} |
XMS using Message Selector
| Code: |
public override void Send(MyObject myObject)
{
IObjectMessage objectMessage = session.CreateObjectMessage();
objectMessage.Object = myObject;
objectMessage.SetLongProperty("RetryDateTime", GetTimestampAsLong(myObject.NextRetryDateTime));
producer.Send(objectMessage);
}
public override myObject Receive(int timeout)
{
MyObject myObject = null;
string selector = "RetryDateTime < " + GetTimestamp(DateTime.UtcNow);
using(IMessageConsumer consumer = session.CreateConsumer(queue, selector))
{
IMessage message = consumer.Receive(timeout);
if (message != null)
{
IObjectMessage objectMessage = (IObjectMessage) message;
myObject = (MyObject) objectMessage.Object;
}
}
return myObject;
} |
The MQ .NET API Browse option suffers from to many reads of the message when
there are many messages that are not ready to process.
Is there a way to just read the properties without reading the whole message?
The XMS version seems to work better but the rest of our MQ code uses MQ .NET and we use a connection string
containing multiple queue managers to provide a limited form of high availability.
Does XMS support a multiple queueManager connection string or do we have to implement it ourselves? |
|
| Back to top |
|
 |
| gbaddeley |
| Posted: Thu Oct 24, 2013 2:54 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
No, MQ is designed to process messages as quickly as possible. You can code a delay using browse and intermediate databases but it is not likely to be very efficient or have high throughput.
Can the message producing application only put messages after the specified time, and let the normal MQ speedy processing occur after that?
_________________
Glenn |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Fri Oct 25, 2013 5:33 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You could of course place the messages on a dummy queue that is being browsed with an expiry with full content that sends them to the correct queue for immediate processing.
This would ensure that they are not being processed before expiry. It would not ensure that they are processed right after expiry... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Oct 27, 2013 2:29 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| fjb_saper wrote: |
You could of course place the messages on a dummy queue that is being browsed with an expiry with full content that sends them to the correct queue for immediate processing.
This would ensure that they are not being processed before expiry. It would not ensure that they are processed right after expiry... |
fjb_saper,
That's a very creative use of Expiry and Report features of MQ. However, I wouldn't trust Expiry on my mission critical messages and then have Report message processing fail for some reason. If it fails, the mesasge is gone!
_________________
Glenn |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Oct 27, 2013 7:22 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
It was never an ideal solution in my mind. I think that the requirements should be revisited and the design changed... MQ is not a timer that allows you to store and process at a specific time...., I'm thinking here about a DB being browsed and acted upon at the critical time... Again I don't see this design working correctly in any volume based scenario...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Mon Oct 28, 2013 2:19 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| Quote: |
| What is the best way to implement a time based queue in MQ on .NET? |
Joe, I think the answer is that there is no best way to implement this using MQ. For high performance I would look at using structure arrays or linked lists in application memory that could be ordered by time. This is the classic "dispatcher" approach used by operating systems. For reliability and recoverability, this should be backed up by a database.
_________________
Glenn |
|
| Back to top |
|
|
|
|
