| Author |
Message
|
| SKAT |
 Posted: Mon Aug 15, 2005 2:11 am Post subject: Messaging Architecture |
 |
|
Novice
Joined: 15 Aug 2005
Posts: 24
|
I have a requirement to group input messages with common field values.
The input messages are of different message instances distributed over a month sales. This is an extension of an existing message flow.
An illustration
A monthly sale file contains records of daily sales. Daily sale records are retrieved and sent to the InputQueue one after the other. Each daily record is processed and written into the OutputQueue. It all works fine. The monthly output file which contains records of processed daily sales have duplicate records. Mr A appears twice and Mr B appears three times. The new requirement should have one Mr A with salesfield = sum of all sales to A. Likewise Mr B. Instead of 5 daily sales records, there should be only two daily sales records of montlhy output file.
Plan to Design
I have planned to use a database for this, storing the last value of salesfield and Mr. For each daily sales message instance, I will query the database for matching Mr and add his current salesfield to the one from the database. All field items are the same in name and number for each daily sales record.
Question:
How do I replace the previous sales record in the OutputQueue with the new daily sales record of same Mr A whose salesfield (the only change) is now different in value to the previous one. If written to the Queue, I will still have two records of Mr A.
Has any one an alternative idea without using a database ? |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Aug 15, 2005 3:14 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You mention "message flow", so I'm going to assume you're actually using WBIMB, instead of just WebSphere MQ.
If you were just using WebSphere MQ, you could avoid using a database by leaving your program running all the time, and do the aggregation in memory. But then if your program crashed, or had to shutdown, you would lose data. This is why people use databases in the first place.
Regardless, you should forget the notion of "replacing the previous sales record in the OutputQueue." Don't write data to the output except the final data.
A message flow does not have to write anything out to a queue, or a file, or anywhere else. It can simply consume the message and stop - a flow consisting of only an MQInput node is perfectly valid (although of very limited usefulness).
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| SKAT |
| Posted: Mon Aug 15, 2005 3:31 am Post subject: |
|
|
Novice
Joined: 15 Aug 2005
Posts: 24
|
Hi Jeff,
| jefflowrey wrote: |
Regardless, you should forget the notion of "replacing the previous sales record in the OutputQueue." Don't write data to the output except the final data.
|
Thanks for the reply. I have misused the word OutputQueue to denote OutputNode.
Is there any possible way getting rid of the previous message in the output node ?
Regards
Skat |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Aug 15, 2005 3:34 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I don't really understand what your question is, or what it means.
The only things that ever get "output" are the things that you tell the system to "output". If you don't want something to go to an OutputNode or any other form of output, then don't instruct the system to send them to the output.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| SKAT |
| Posted: Mon Aug 15, 2005 3:47 am Post subject: |
|
|
Novice
Joined: 15 Aug 2005
Posts: 24
|
Let me explain a bit further:
Monthly File
-> retrieve a message (daily sales message) from file -> MQInput Node
->process data for Mr A -> MQOutput Node
-> retrieve another message (daily sales message) from file ->MQInput Node->process new data for Mr A -> MQOutput Node
What would you suggest if the data should not be written to the MQOutput Node ?
Regards
Skat |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Aug 15, 2005 3:55 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Either - Don't include any output node, and use another flow to retrieve from the database and write output
- Include ESQL that makes sure that the only thing that goes to that MQOutputNode is the final output and not any intermediate output. A Compute node that ends at
will cause processing to halt.
I think you need to do some analysis of the different things that need to occur in this process - at a high level - and then try and map those back to the logic you need to implement in the flow, and then try and map that logic to nodes.
It's just like any other kind of programming.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| SKAT |
| Posted: Mon Aug 15, 2005 4:14 am Post subject: |
|
|
Novice
Joined: 15 Aug 2005
Posts: 24
|
| jefflowrey wrote: |
Either - Don't include any output node, and use another flow to retrieve from the database and write output
- Include ESQL that makes sure that the only thing that goes to that MQOutputNode is the final output and not any intermediate output. A Compute node that ends at
will cause processing to halt.
|
Thank you Jeff,
all these discussions are considered a high level analysis. I have not started programing yet. Please understand my quest for new ideas and more inputs.
I will buy your first suggestion with the database.
To stop the processing with return false, you mean I should load all messages in the file to the WBI. This is exactly what I'll like to avoid because the messages in file is just too big.
It is only possible to write to the ouput only when the whole file message is processed but caching in WBI till end of file is reached ? What do think?
Regards
Stephen |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Aug 15, 2005 4:28 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
This is what I would suggest:
Process each record in the file one at a time, as an individual message.
Do something to indicate the end of all records - either another message or a flag on another message or etc.
Insert each record into the database.
When the end of file comes, produce the report messages you need.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| SKAT |
| Posted: Mon Aug 15, 2005 4:41 am Post subject: |
|
|
Novice
Joined: 15 Aug 2005
Posts: 24
|
Hi Jeff,
this is a matured solution and I think it has over ruled my plan to use the queue as a database.
Following the forum, I came accross one of your comments:
| jefflowrey wrote: |
Also, treating a queue as a database is an anti-pattern. That means it seems
like a reasonable idea at first, and can be done. But it leads to horrible
performance later on and creeping contact admin problems that could have been avoided. |
I had initially thought of using another flow, triggered by a control, EOF in a queue, and start doing sql selection on the output node taking my group criteria into consideration.
Your comment has given me a second thought and I am sure you still maintain that statement.
Regards
Skat. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Aug 15, 2005 4:43 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Yes, I do still stand by this, for all sorts of reasons. One good one is that databases are usually highly optimized for performing SELECTS. It is not very possible, without a lot of work before hand, to optimize a select style operation on a queue so that it doesn't involve an O(n = number of messages on the queue) operation.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| SKAT |
| Posted: Mon Aug 15, 2005 4:53 am Post subject: |
|
|
Novice
Joined: 15 Aug 2005
Posts: 24
|
Thank you very much for now Jeff .
It was nice discussing this issue with you and I appreciate your experienced input.
Regards
Skat |
|
| Back to top |
|
|
|
|
