| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Multiple message types but only one message flow |
« View previous topic :: View next topic » |
| Author |
Message
|
| sjclark1 |
 Posted: Fri Apr 22, 2005 4:04 am Post subject: Multiple message types but only one message flow |
 |
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
Hi all.
I'm after a bit of msg flow design advice.
I am currently looking at migrating a project based on MQSI v1 (NEON) to Message Broker v5.
In v1 we have 29 input queues (therefore 29 brokers) which take delivery of 7 different message types (i.e. each of the 7 message types can arrive on each of the 29 queues).
We have taken ONE of the 7 message types and constructed a very complex message flow to process messages of this type. But if I wanted a flow to process the other 6 message types it would get very complicated - I am thinking 7 different subflows with a RouteToLabel at the start to pick out the relevant subflow. Doing this 29 times over is ringing alarm bells in my head as it sounds like a huge deployment. Anyone agree / disagree with this?
Would it be better if I routed to one of 7 output queues rather than a subflow? I could then have 7 new flows, each reading off one of the queues. But then I'd have to have 29 + 7 = 36 flows. Plus I'd be worried the messages wouldn't be procssed as quickly if 7 flows are expected to do the work that used to be done by 29 v1 brokers.
Or are there other options? Only requirement is that message order within each of the individual 29 input streams MUST be preserved (but doesn't have to be preserved across different input streams, if you see what I mean). It may even be possible to cut down the number of input queues from 29 down to 10 (i.e. 3 times less queues, but 3 times more messages arriving on each queue).
Any thoughts, experiences or suggestions would be highly appreciated.
Broker runs on Solaris 8, MB v5 FP4, Oracle 9.
Thanks in advance,
Steven |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Fri Apr 22, 2005 4:24 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There are performance impacts for using too many queues, and not complex enough flows.
There are also performance impacts for using too complex a flow, with too many nodes.
You will find that the performance of v5 is significantly improved over the performance of v1. Are you really using version 1? wow!
You will also have the ability to run multiple instances of a message flow to process data faster.
How many output queues do you have? 29?
Except for the bit about maintaining order, I would recommend building 8 flows, or maybe 9. Total. One flow to route input (possibly with 29 input nodes!), one flow to route output(using one output node with a one element destination list), and then 7 flows for each message type. And maybe a flow to write processing information to a database for logging.
Your requirement to maintain the order of each stream doesn't completely eliminate this design. You just need to take some care and run some tests with it.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Fri Apr 22, 2005 4:49 am Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
Hi Jeff
| jefflowrey wrote: |
How many output queues do you have? 29?
|
Not exactly sure but it's round about that figure - basically we have 29 instances of our "supplier" system (for 29 geographical areas in the UK) that send us events - we then transform the messages and send to interested subscribers. So the number of output queues varies from release to release.
| jefflowrey wrote: |
Except for the bit about maintaining order, I would recommend building 8 flows, or maybe 9. Total. One flow to route input (possibly with 29 input nodes!), one flow to route output(using one output node with a one element destination list), and then 7 flows for each message type. And maybe a flow to write processing information to a database for logging.
Your requirement to maintain the order of each stream doesn't completely eliminate this design. You just need to take some care and run some tests with it. |
That's a helpful suggestion. Could you possibly expand on the message order thing - I need to make sure that, within each of the 29 input queues, the messages from that geographical area are processed in the order they arrive. what sort of things do I need to watch out for to ensure message order is maintained?
Re the 7 flows doing the bulk of the work, could I run multiple instances of these and still preserve message order??
Thanks,
Steven. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Apr 22, 2005 6:05 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
There are two basic issues with message order in the scenario where you are using multiple queues instead of multiple subflows.
One is that it could take less time to process message type 1 than say message type 3. This would lead to a fairly good chance that a message of type 1 could be sent after a message of type 3, but end up on the output queue first.
There is also the chance that one message of type 3 could take less time to process than a different message of type 3 (maybe because of size or whatever). With multiple instances, the message that takes less time to process will end up on the output queue before the message that takes longer.
You run the same risk of the second problem if you are using multiple instances of a single flow using sub-flows.
This is one of the reasons that message affinity is to be avoided wherever possible. Maybe you can use this opportunity to revisit the affinity/ordering requirements of the sending/receiving apps...
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sjclark1 |
| Posted: Fri Apr 22, 2005 7:14 am Post subject: |
|
|
Apprentice
Joined: 06 Jan 2003
Posts: 35
Location: Scotland
|
OK - regrettably message order matters to the subscriber systems and I would have to move heaven and earth to change that, but that's something I will bear in mind for the future.
I notice the following in the help for MQInput node:
| IBM wrote: |
Select Order Mode from the drop-down list to determine the order in which messages are retrieved from the input queue. This property has an effect only if the message flow property Additional Instances is set to greater than 0, that is, if multiple threads read the input queue. Valid values are:
Default. Messages are retrieved in the order defined by the queue attributes, but this order is not guaranteed as the messages are processed by the message flow.
By User ID. Messages that have the same UserIdentifier in the MQMD are retrieved and processed in the order defined by the queue attributes; this order is guaranteed to be preserved when the messages are processed. A message associated with a particular UserIdentifier that is being processed by one thread is completely processed before the same thread, or another thread, can start to process another message with the same UserIdentifier. No other ordering is guaranteed to be preserved.
By Queue Order. Messages are retrieved and processed by this node in the order defined by the queue attributes; this order is guaranteed to be preserved when the messages are processed. This behavior is identical to the behavior exhibited if the message flow property Additional Instances is set to 0.
|
I am wondering therefore if I could get the 29 supplier systems to populate the MQMD user identifer field with their unique identifier so that messages could be processed in order for each of the supplier systems, which is exactly what I need.
Does that seem like a good idea?? I could run the 7 main flows each with, say, 3 instances each (if i ran the 7 in separate exe groups, each of the 7 exe groups would take up a total of about 420MB memory, including 3 x 50M = 150MB for the message flows, the rest is message sets plus whatever else is in a exe group). So straight away I would need 7 x 420MB = 3 GB memory spare to play with, plus a minimal amount to run the simple flow to do the routing in the first place.
I'm still concerned that 29 seems like a lot of userid's to cooridinate the order for (?) , but anything's better than having 29 gigantic flows on the one machine.
Regards
Steven |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Apr 22, 2005 7:39 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Neither of the issues I spelled out dealt with messages from different input streams interfering with the order of other input streams.
So separating them by input stream won't really help.
Another thing to keep in mind is that in version 5, you can override the InputQueue and OutputQueue properties at deployment. So you can create one set of flows, and put them in 29 different bar files, with different input and output queues in each bar file.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
