| Author |
Message
|
| Goeff |
 Posted: Wed May 02, 2007 8:16 pm Post subject: Message flow design |
 |
|
Newbie
Joined: 17 Apr 2007
Posts: 9
|
Can anyone suggest a message flow design for the following situation?
Client -- (messages) --> [MQ > Convert format > MQ] > Websphere application server processing > [MQ > Convert format > MQ] > Legacy backend system
Flow:
1) Messages put into Message Broker through MQ
2) Message Broker pickup messages and does convertion
3) Websphere application server pickup converted messages and processes the data
4) Processed data put into another MQ, a legacy backend system pickup the message for further processing
Major problem:
The sequence of messages must be maintained between the very first input Q and the very last output Q.
Thanks!  |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu May 03, 2007 12:47 am Post subject: Re: Message flow design |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Goeff wrote: |
Major problem:
The sequence of messages must be maintained between the very first input Q and the very last output Q.
|
You're right - this is message affinity and a very serious problem!
If you need to absolutely guarantee message sequence you can create a reply on successful message read, triggering the transmission of the next message. This will ensure that messages are only ever processed in sequence, but is a bad design in so many ways..... 
Anything else & you end up in the tangled world of trying to read messages in sequence when they're sent in arbitary packets over TCP/IP. You need to think about are what to do if a message is missed out or delayed; where to put the subsequent messages. Aside from the performance implications of browsing a queue looking for the next one.
The best solution is to remove message affinity at target and store messages received out of sequence in a database or similar until the "next" one arrives. Databases are much better at holding & searching data than queues.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Maximreality |
| Posted: Thu May 03, 2007 3:02 am Post subject: Re: Message flow design |
|
|
Acolyte
Joined: 04 Jun 2004
Posts: 65
Location: Copenhagen
|
| Goeff wrote: |
Can anyone suggest a message flow design for the following situation?
Client -- (messages) --> [MQ > Convert format > MQ] > Websphere application server processing > [MQ > Convert format > MQ] > Legacy backend system
Flow:
1) Messages put into Message Broker through MQ
2) Message Broker pickup messages and does convertion
3) Websphere application server pickup converted messages and processes the data
4) Processed data put into another MQ, a legacy backend system pickup the message for further processing
Major problem:
The sequence of messages must be maintained between the very first input Q and the very last output Q.
Thanks! |
You should consider if the sequence is forever or just for groups of messages.
Also what to do in case of error?, should the error message stop the entire data stream og let the data continue to go through the system.
We have an interface with "sequencing groups" since the sequence is only required pr. batch of messages we receive. This allows us to let the data flow even though one batch fails.
/Rasmus |
|
| Back to top |
|
|
| jsware |
| Posted: Thu May 03, 2007 4:09 am Post subject: Re: Message flow design |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
| Goeff wrote: |
Major problem:
The sequence of messages must be maintained between the very first input Q and the very last output Q. |
This implies that the order of processing by message broker and application server is not sequence dependant - correct?
Message ordering can be solved in a number of ways. As already mentioned it helps to think about failure scenarios and whether absolute sequencing is required, or whether the sequence only occurs within groups (e.g. ensure that an update occurs after an add, but the sequence of adds does not matter).
Though its focus is on publications, http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp?topic=/com.ibm.mq.amqnar.doc/amq622g.htm might help.
You can also use MQMD.MsgSeqNo and match on an increasing number (dealing with seq wrap).
You can also "chain" messages together - ie. Message n has a correlation ID that is the msg id returned in the MQMD after message n-1 was put. Again match on correlation ID in the receiver. Use a blank correlID to indicate a start/end of a sequence.
See also:
http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/topic/com.ibm.mq.csqzal.doc/msgordr.htm
As a lowest, quickest dirtiest (beat myself with a stick for mentioning) option - see http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/topic/com.ibm.mq.csqzae.doc/ic11080_.htm - Doing this will not scale well as you can only single thread any application that performs an MQGET and can't use separate queues and have to remove dead letter queues/backout requeues to stop messages getting "parked".
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| Goeff |
| Posted: Thu May 03, 2007 6:28 pm Post subject: |
|
|
Newbie
Joined: 17 Apr 2007
Posts: 9
|
Thank you very much mates.
As I said, message sequence between the very first Q and the very last Q must be maintained.
However, I didn't mention that it only applies to individual senders. ie. message seqnece must be maintained within messages from individual senders (which is identified by MQMD-UserIdentifier).
To make the message flow scales, I've decided to use multiple instances of a message flow, which is ordered by MQMD-UserIdentifier. And to make sure the message sequence can be kept after WAS processing, I've decided to take the CorrelationID way.
As a result, my design looks like this:
Client ---> [ MQ-1 > Convert > MQ-2 ] > WAS (get from MQ2, put to MQ3, WAS does not keep message seq.) > [ MQ-3 (get by CorrelationID) > Convert > MQ-4 ] > Legacy backend system
Comments are highly welcomed! |
|
| Back to top |
|
|
| Goeff |
| Posted: Thu May 03, 2007 6:29 pm Post subject: Re: Message flow design |
|
|
Newbie
Joined: 17 Apr 2007
Posts: 9
|
| scottj2512 wrote: |
| This implies that the order of processing by message broker and application server is not sequence dependant - correct? |
Correct!  |
|
| Back to top |
|
|
| Goeff |
| Posted: Thu May 03, 2007 6:30 pm Post subject: Re: Message flow design |
|
|
Newbie
Joined: 17 Apr 2007
Posts: 9
|
| Goeff wrote: |
| scottj2512 wrote: |
| This implies that the order of processing by message broker and application server is not sequence dependant - correct? |
Correct! |
ie: Sequence must be maintained within Message Broker message flows. Application server will not take care of any sequencing. |
|
| Back to top |
|
|
| Goeff |
| Posted: Fri May 04, 2007 12:32 am Post subject: |
|
|
Newbie
Joined: 17 Apr 2007
Posts: 9
|
I've implemented the design in message flow and tested it with messages.
It seems works well but there is an annoying problem.
The MQGet node is always unable to get the last message left on the MQ. ie: the last message will remain on the MQ, until new messages come. Is there any crack? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri May 04, 2007 3:23 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You should be looping until the MQGet node tells you that there are no more messages.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
