| Author |
Message
|
| kayhansefat |
 Posted: Thu Apr 12, 2007 2:15 am Post subject: Message Flow Design Question |
 |
|
Acolyte
Joined: 18 Oct 2006
Posts: 65
|
We have a situation where we will receive a large file once a month. The file contains multiple lines/records. Each line is 2858 characters in length with each "field" in a certain position within the file which makes things easier (e.g. idNumber is at position 30-45 in the line).
Every record here will belong to the same message format but the first field (the first 3 characters, 030,020,010 etc.) will determine which output message format it will have to adhere to. So all the 030 records will have to be written to ONE file for message format 1 (for example), all 020 records to a separate file which apply to a different message format 2. There are up to 17 different output message formats.
I thought about it working as follows:
Flow 1 - Get the file, propagate each of the lines to a queue using the BLOB domain.
TimeoutNotificationNode --> JavaComputeNode --> MQOuputNode
Flow 2 - Create a message set that defines each of the fields in the message from their positions, route each of the messages (depending on 010,020,030 etc) to their own output queue.
MQInputNode --> ComputeNode --> RouteToLabelNode
Label1Node --> MQOutputNode (for 010)
Label2Node --> MQOutputNode (for 020)
Label3Node --> MQOutputNode (for 030) etc etc.
Flow 3 (One for each Outpute Node in Flow 2) - Map this message to the new message type e.g. 010 only. Then write the message to the end of the 010 Output File.
MQInput --> JavaCompute
Any advice? Thanks. |
|
| Back to top |
|
 |
| elvis_gn |
| Posted: Thu Apr 12, 2007 3:02 am Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi kayhansefat,
How are you planning to pick the file....this will play an important role in your design....Are you going to use the File Extender nodes, or Custom nodes, or use an external App to pick the file etc...
If you use your RouteToLabel design you will have 17 RouteToLabels itself in the flow....this seems to be getting out of hand....
Perhaps you should pick record by record, find out what type it is, parse it and put it into the respective queue by looking up some Db reference or promoted property....
Else read the whole file, parse against a message set where you have all the 17 types defined, and then select and seperate in a compute node based on Type....the propagate them as above...
Regards. |
|
| Back to top |
|
|
| kayhansefat |
| Posted: Thu Apr 12, 2007 7:08 am Post subject: |
|
|
Acolyte
Joined: 18 Oct 2006
Posts: 65
|
Thanks for replying elvis_gn. If by picking you mean selecting the file in the first place then it will be dropped/put, maybe by ftp into a certain directory, ready with all the records in the file.
If you mean how am I going to pick the records out of the file, then I instantly thought about using the JavaComputeNode to create a new msg for each of the records in this huge file.
In the JCN I could determine which type the records are, but how would I then propagate it to a specific queue without a RouteToLabel Node? I guess you can do this by promoting the Queue Name property of an MQOutputNode??? Then setting it within the JCN? If so do you know of any links to help me with this?
Will I still then have to have a further 17 flows and queues for each type of message? Then in each flow map it to the new message and write to file?
Or have I completely gone off track and there is a more efficient way to do it? |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Thu Apr 12, 2007 9:12 pm Post subject: |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi kayhansefat,
| kayhansefat wrote: |
| If you mean how am I going to pick the records out of the file, then I instantly thought about using the JavaComputeNode to create a new msg for each of the records in this huge file. |
You cannot trigger a flow using the Java compute node, if i'm right...you will have to trigger the flow on a timer if not an input message from the application which drops/puts the file in the location...
| kayhansefat wrote: |
| In the JCN I could determine which type the records are, but how would I then propagate it to a specific queue without a RouteToLabel Node? I guess you can do this by promoting the Queue Name property of an MQOutputNode??? Then setting it within the JCN? If so do you know of any links to help me with this? |
Whether you use a Java compute or a simple ESQL compute, the issue of propagating will remain...that is where you will need either 17 UDPs promoted...or some reference table in the database...or perhaps pub/sub...
| kayhansefat wrote: |
| Will I still then have to have a further 17 flows and queues for each type of message? Then in each flow map it to the new message and write to file? |
You should be able to do this in the same flow and put it into the new file right....this will get rid of your routing and queue issue..
Regards. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Apr 13, 2007 3:45 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Don't write code to read the file yourself - if you need to ever read more than one file from more than one directory, you probably want the File Extender.
It sounds like your message data can be parsed as TDS, and so there's no reason to break the file up into individual records - unless the file is really big.
If the file is really big, the File Extender can break it up FOR you.
You need to evaluate the business requirements and the rate of change for the transformation code against the maintenance complexity and overhead of using different queues (and a table of some kind to store the association of queues to message types).
If you do decide to do queues, you can still only use ONE MQOutputNode and NO RouteToLabels if you set the MQOutput node to use a DestinationList.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| kayhansefat |
| Posted: Fri Apr 13, 2007 4:58 am Post subject: |
|
|
Acolyte
Joined: 18 Oct 2006
Posts: 65
|
Ok thanks guys that does help quite a bit. We have not purchased the File Extender for the MB so the only option I have (and have done previously) is to read and write via the JCN. And yes I would have to trigger it by a TimeoutNotification.
I did realise afterwards and jeff pointed out that I could output to a single MQOutput node specifying all the queues (retrieved from a DB table) in the Destination List array.
Are you saying jeff that when I read all the records from the file and individually split them into messages to put on their specifc queue I should create each of them under the TDS parser? I will still then have to have 17 flows that read off these queues and write them to the specific files.
Would it not be quicker to do it all in one flow, ie. to do the mapping manually in the JCN for each of the 17 and write to the files there? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Apr 13, 2007 6:12 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Yes, it's faster to do it all in one flow. You could even do it all inside a single compute node (or JCN).
But as I said
| Quote: |
| You need to evaluate the business requirements and the rate of change for the transformation code against the maintenance complexity and overhead of using different queues (and a table of some kind to store the association of queues to message types). |
There may be good reason for splitting it into multiple flows.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
