| Author |
Message
|
| EternalLearner |
 Posted: Fri Sep 14, 2012 2:21 am Post subject: Creating a csv file from from mq messages and FTP It |
 |
|
Novice
Joined: 20 Jul 2012
Posts: 16
|
Hi,
I've got a requirement where I need to create a csv file at a parrticular time near the end of the day from all the messsages in a queue and the FTP it. In case of no message in a day still an empty csv file with header needs to be produced. File name will be like 'Update_yyyymmdd.csv'.
Regarding th design, I've planned the following
MQIn ---> Comput ---> Timer Control
Timer Notification --> Compute1-->( MQGet --> Compute2)---> FileOutput
Loop Through messages between MQGet and Comput2
The 'No Messsage' terminal of MQ Get node will be connected to Fileoutput Node.
On Fileoutput node 'Append if file exists' option selected.
Can I have views on this design approach.
One isssue I can see is that the file is produced at each iteration instead of at the end of the enitire loop. Connecting to the 'Finish File' terminal doesn't help. I've also selected option of 'Stage in mqsitransit directory and move on finish file'.
Since it will be FTpd when it is created, that is potentially a problem.
Can I please have your views on this design approacch and alternative suggestions.
Thanks. |
|
| Back to top |
|
 |
| EternalLearner |
| Posted: Fri Sep 14, 2012 2:52 am Post subject: |
|
|
Novice
Joined: 20 Jul 2012
Posts: 16
|
Hi ,
I could get rid of immediate file wrting without waiting for finish file with changing 'Record definition' property' on FileOutput. I should've got it earlier.
However would like to know your views and suggestions regarding the design approach.
Thanks |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Sep 14, 2012 4:42 am Post subject: Re: Creating a csv file from from mq messages and FTP It |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| EternalLearner wrote: |
| Can I please have your views on this design approacch and alternative suggestions. |
Looping round a flow with an MQGet should always be viewed with caution for reasons often discussed in this forum. The short version is that it can break your execution group.
You can eliminate that and the Timer node (which isn't the best way of scheduling to a given time) by writing a simple flow which drains the queue with an MQInput node, producing a file as you indicate, and is started on a schedule by whatever OS level facilities are available. Such scheduling is another often discussed topic in here, some of which discussions you may find of assistance.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 14, 2012 5:08 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
My preference would be to use a database rather than file I/O. You might get the file I/O version to work for a short time, but under load I think you will find that it breaks.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Sep 14, 2012 5:21 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| lancelotlinc wrote: |
| My preference would be to use a database rather than file I/O. You might get the file I/O version to work for a short time, but under load I think you will find that it breaks. |
Breaks how?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Sep 14, 2012 5:52 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| bruce2359 wrote: |
| lancelotlinc wrote: |
| My preference would be to use a database rather than file I/O. You might get the file I/O version to work for a short time, but under load I think you will find that it breaks. |
Breaks how? |
regardless of breaks how, one's business partner may not be willing to accept "read your data out of this database hosted in our DMZ" in place of "Here's a flat file that was securely copied to your servers".
The file in question in this thread is being used as an aggregation point for messages stored on a queue. I might tend to prefer to store the messages on a database rather than a queue - but there's very likely no room to change the aggregation point from a file to anything else....
if there was room to change the aggregation point, I'd eliminate it entirely and merely forward messages in real time. |
|
| Back to top |
|
|
| EternalLearner |
| Posted: Fri Sep 14, 2012 6:07 am Post subject: |
|
|
Novice
Joined: 20 Jul 2012
Posts: 16
|
| mqjeff wrote: |
I might tend to prefer to store the messages on a database rather than a queue.
|
Would be interested to know why would you prefer that if the messages are persistant.
Thanks. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 14, 2012 6:29 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
| bruce2359 wrote: |
| lancelotlinc wrote: |
| My preference would be to use a database rather than file I/O. You might get the file I/O version to work for a short time, but under load I think you will find that it breaks. |
Breaks how? |
Breaks when more than one message flow is trying to read/write the file.
Use of an in-memory solution would be the best.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| EternalLearner |
| Posted: Fri Sep 14, 2012 6:51 am Post subject: |
|
|
Novice
Joined: 20 Jul 2012
Posts: 16
|
| lancelotlinc wrote: |
Breaks when more than one message flow is trying to read/write the file.
Use of an in-memory solution would be the best. |
Here it shouldn't be the isssue as only one flow instance will write to a file only once in a schedule time of the day and the file will be FTPd immediately.
Thanks. |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Sep 14, 2012 6:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| bruce2359 wrote: |
| lancelotlinc wrote: |
| My preference would be to use a database rather than file I/O. You might get the file I/O version to work for a short time, but under load I think you will find that it breaks. |
Breaks how? |
Breaks when more than one message flow is trying to read/write the file. |
a) Nothing in this solution is attempting to read this file & I agree with @mqjeff that it's being FTP'd to a external party that will be reading it in their own sweet way
b) Why would you want or need more than 1 flow to write to the file? In this kind of "end of day reporting" response time is hardly going to be a factor
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 14, 2012 7:09 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
The OP has left alot out of the requirement, so its hard to design blindly. In a few years, it could be that someone will increment the additional message flows parm to improve performance. If you stick with this design, I would make a note in the code about the requirement to disallow additional flow instances.
In the end, whatever works is what is best. If stuck with the file approach, I would prefer to use cron over timer nodes. I have never liked leaving file handles open for a long time. If the design requires this, I would defensively code around that requirement.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| Vitor |
| Posted: Fri Sep 14, 2012 8:22 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| lancelotlinc wrote: |
| . In a few years, it could be that someone will increment the additional message flows parm to improve performance. If you stick with this design, I would make a note in the code about the requirement to disallow additional flow instances. |
How many additional instances do you need to break the FileOutput node? Also if there's a need to improve performance, i.e make the output file available earlier, wouldn't the obvious first step be to start the process running & reading the queue earlier? It's not like the first message on the queue will change between when it was written & the end of the day!
| lancelotlinc wrote: |
| I would prefer to use cron over timer nodes. |
As I indicated above, as assuming the OP is using Unix. If not, use the cron-like feature all OS have.
| lancelotlinc wrote: |
| I have never liked leaving file handles open for a long time. If the design requires this, I would defensively code around that requirement. |
What's bad about long duration open file handles that isn't a potential problem with any open file handle? What kind of "defensive" code are you indicating?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Sep 14, 2012 8:49 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| lancelotlinc wrote: |
| bruce2359 wrote: |
| lancelotlinc wrote: |
| My preference would be to use a database rather than file I/O. You might get the file I/O version to work for a short time, but under load I think you will find that it breaks. |
Breaks how? |
Breaks when more than one message flow is trying to read/write the file.
Use of an in-memory solution would be the best. |
I asked how it breaks. You responded with when it breaks. Exactly what error condition arises that indicates that it is broken?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Fri Sep 14, 2012 10:39 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
One defensive code that could be used would be mutex, semaphore, or the like to protect the file from attempting to be written to by multiple instances or from being read while it is being written. I reiterate that I don't like the whole approach and I would take a whirl at it from a different angle anyway. How it could break would be if more than one process is writing to the file, you may not have record integrity as the records could be interrupted in mid-stream.
While this is a good discussion, I'm thinking the OP should step in here and clarify the details of the design.
_________________
http://leanpub.com/IIB_Tips_and_Tricks
Save $20: Coupon Code: MQSERIES_READER |
|
| Back to top |
|
|
| EternalLearner |
| Posted: Mon Sep 17, 2012 2:36 am Post subject: |
|
|
Novice
Joined: 20 Jul 2012
Posts: 16
|
Hi,
Based on the discussions above some more information.
The OS is windows 2008.
There won't probably be any performance enhancement requirement in future as the process is asynchronous in nature and it can take some time to create and FTP the file.
But I'm still interested in knowing why accumulating the messages in a db will be preferred over on a queue if they are persistent.
Thanks. |
|
| Back to top |
|
|
|
|
