MQSeries.net :: View topic

jh72i · Posted: Wed Aug 30, 2006 2:05 am Post subject: Message Processing Design Ideas?

Hi,
I'm using MQ from a .NET application and so far all is going fine. But I would like to know what your thoughts are on my design - particularly one aspect. Thing is the volume, number of apps using my components, etc. are increasing and I need to act now if I can better my approach.

Heres generally what happens:

.i have a queue processor Windows service that attaches/subscribes to any configured comms protocols (config in database and/or .config files)

.one of the protocols will be a WebSphere queue on which some client app expects to get messages from external systems. All WebSphere Q access is wrapped in my own .NET component that implements a wellknown interface so that the Window service can create and interact with it without knowing that it is, in fact, a WebSphere Queue/MSMQ/FTP/ etc.

Ok, in terms of processing a single message off the queue here is what happens:

.i issue a Get with MQC.MQWI_UNLIMITED

.when a message is available it is read off the queue and immediately persisted to a raw message table

.if this works the message is committed off the queue and the WebSphere wrapper asynchronously raises an notification to any listeners that content/data has come

.loop back to the .get and keep doing this 24/7

point of this is that it is one single task to get a message from the queue and persist it. Only after it has been successful brought into our world do we start to manipulate the contents of the message from a business perspective.

So, an actual business specific component would then:

.receive a notification that a message has arrived together with the message wrapped in our own persisted communication package (a business object in itself).

.process the contents in whatever way it needs to. Incidentally, I use XMLSerialization for XML messages which is another performance question for later.

.finally saves the internal communication package with a Processed flag together with a reason for failure if it fails.

So if the business processing fails we can fix whatever is needed (might just be an our of sequence message) and simply replay the raw message from the persisted storage.

The main motivation for this, I suppose, was at the time of developing I couldn't trust the external systems to provide good data/good messages/etc. I couldn't either trust the internal systems to process the message contents correctly 100% of the time. Added to this the fact that I couldn't trust either of them to take responsbility for their own mistakes.

In effect I needed to protect my own processes and provide an audit trail of sorts so that I could show with 100% accuracy the raw message that came off any given queue at any given time. I actually provide a searchable web interface to the raw messages should any of the participants want to question the contents.

This method has been very, very effective in eliminating the too-ing and fro-ing when some status/value/etc. doesn't end up in the state that was expected - we can direct to the participants to the raw data that was received and pretty much figure who and what were the points of failure.

Now, while this all works well there is no doubt that persisting those messages adds some degree of overhead to the processing of a single message. With more systems wanting to use my components and with existing systems wanting to push a lot more messages through I'm wondering if there is an alternative approach I should be considering?

Any ideas, experiences, thoughts, etc. would be great.