| Author |
Message
|
| next |
 Posted: Tue Jun 14, 2011 1:53 am Post subject: FileOutput node design question |
 |
|
Voyager
Joined: 02 May 2010
Posts: 75
|
Hi
I have a requirement like the messages will be read from the queue and written to multiple files.
I am setting the name of the file in the LocalEnvironment.Destination.File.name and writing the files. But when i try to finish the files, i have to remember the names of the files written for finishing them.
Finish file terminal doesnt take wild card matches for finishing the file.
Is there any way i can finish all the files in one directory at once? |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Jun 14, 2011 2:00 am Post subject: Re: FileOutput node design question |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| next wrote: |
| Is there any way i can finish all the files in one directory at once? |
No.
That would kind of be like telling a database to commit all outstanding transactions. |
|
| Back to top |
|
|
| next |
| Posted: Tue Jun 14, 2011 2:20 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
Thanks for your reply.
Having said we cant finish all the files at once, is it a good design to write the names of the files in a message put in a queue and check the message for the file names for each transction using MQGet node and finish all the files finally? |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 14, 2011 2:23 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's a good design, provided that other design options aren't better.
First design question: Why are you leaving the files open in the first place?
Second design question: How do you know when any file or all files are supposed to be closed?
Third design question: How *sure* are you that a specific file is really ready to be closed?
Fourth design question: Do you need to use a separate message flow to close the files? |
|
| Back to top |
|
|
| next |
| Posted: Tue Jun 14, 2011 2:35 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
First design question: Why are you leaving the files open in the first place?
The messages are coming from different stores and i create files for each store. We need to append the messages to the file for a day.
Second design question: How do you know when any file or all files are supposed to be closed?
The files have to be closed at store closure. May be a particular time every night.
Third design question: How *sure* are you that a specific file is really ready to be closed?
At store closure, there wont be any messages.
Fourth design question: Do you need to use a separate message flow to close the files?
In the same flow, i can put another MQInput node to trigger the finish file job.
Please let me know your thoughts. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jun 14, 2011 4:13 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Why not accumulate the messages in a queue(s) during the day, and use a scheduled job to read & produce each of the files in a single go, finishing the file at end of messages?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jun 14, 2011 4:33 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I would prefer to have the stores send an explicit "end of day" message.
That way you don't have to worry about if a store happens to be running slow one day or etc. etc. etc.
But I'd also prefer to accumulate data into a database rather than a file, and then use the end of day to trigger a flow that pulls from the database. |
|
| Back to top |
|
|
| next |
| Posted: Tue Jun 14, 2011 4:38 am Post subject: |
|
|
Voyager
Joined: 02 May 2010
Posts: 75
|
I am not sure if accumulating messages in queues is good...considering the performance and the QM may go down in case of heavy load.
Is it a right idea to read and write message in a queue ? or is it good to use database to avoid reading and writing of the same message? |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jun 14, 2011 4:46 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| next wrote: |
| I am not sure if accumulating messages in queues is good...considering the performance and the QM may go down in case of heavy load. |
How many messages are you talking about here? What platform is the queue manager running on? A lot of of people use WMQ for message volumes in the tens of thousands at very high transmission rates. Likewise WMB doesn't balk at working hard.
Clearly if you've got a queue manager & broker running on a small Windows box using the default configuration with no attempt at sizing then you could hit a problem. Until you correctly size them.
| next wrote: |
| Is it a right idea to read and write message in a queue ? or is it good to use database to avoid reading and writing of the same message? |
If you have access to a database, why are you writing to files at all? Accumulate the messages into the database on arrival and use a scheduled job to extract information for whatever end-of-day processing these files go into. Not only does this eliminate your file problem, but allows for future end-of-month & end-of-year processing.
Keeping business data in a database? Radical but I'm in the mood to push the envelope... 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
