| Author |
Message
|
| angka |
 Posted: Sun Jul 19, 2009 11:14 pm Post subject: How does Fileinput node process the multiple files? |
 |
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
If I set the Fileinput node to poll a directory and if 3 files were found, in what sequence (base on filename?) the flow will process the files?
Thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Sun Jul 19, 2009 11:36 pm Post subject: Re: How does Fileinput node process the multiple files? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| angka wrote: |
| If I set the Fileinput node to poll a directory and if 3 files were found, in what sequence (base on filename?) the flow will process the files? |
AFAIK (and I'm ready to be corrected here) they will be processed in the order they're presented by the OS. I don't believe (again ready to be corrected) the order of files can be guaranteed.
Obvious question - what happens when you try it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| angka |
| Posted: Mon Jul 20, 2009 12:13 am Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
I tried it, it is based on creation of the file. is there anyone to confirm on this? btw i am running on Windows server.
Thanks |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jul 20, 2009 12:29 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| I would be cautious about relying on this order unless it is a documented feature. |
|
| Back to top |
|
|
| angka |
| Posted: Mon Jul 20, 2009 1:25 am Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
Can't find anywhere.. now asking IBM to get back.
Thanks |
|
| Back to top |
|
|
| angka |
| Posted: Wed Jul 22, 2009 12:46 am Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
I tried picking files from remote server via FTP then the files are read in the order of the filename. this is different from polling the local directory. mayb this is due to the file is transfered over to local directory and the creation time is the same. Anyone got any ideas on how to read file from remote server in the order of the creation time??
Thanks. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Jul 22, 2009 2:04 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Why not make the filename a timestamp?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| angka |
| Posted: Wed Jul 22, 2009 2:16 am Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
How can a filename with timestamp help? think i may need to do some processing on my side rather then depending on the fileinput node |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Jul 22, 2009 4:49 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Perhaps I should have given an example
200907211234.dat
200907211321.dat
As the name of the file is a timestamp that would order the files in the directory in the order that they were created.
The oldest file is at the top of the list. The newest is at the bottom.
If the order of processing was important then this is what I would propose.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Jul 22, 2009 7:42 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
To be clear.
The FileInput node processes files in the order that the OS level file subsystems present them through directory traversing APIs.
This may have no relationship to the name of the file, nor to the date that a given file was created, nor to any other human sensible ordering.
It is entirely dependant on the OS itself.
In general, you should not develop Broker flows that have message affinity, regardless of where the message arrives from - either a queue or a file or an HTTP input. If you do need to do so, you should make sure that you can handle the affinity within your flow. |
|
| Back to top |
|
|
| angka |
| Posted: Wed Jul 22, 2009 6:44 pm Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
Ya I know that there shld not be message affinity in the design however it is the customer requirement. think i will have to handle it in the flow.
Thanks |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Jul 23, 2009 1:16 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| angka wrote: |
| however it is the customer requirement. think i will have to handle it in the flow. |
It's worth pushing back on the customer to see if this is a real requirement or just something they think they need. What this sequence actually adds to the process.
It's surprising how many requirements in a new system are based on "the old system worked that way".
My 2 cents, other views valid, etc.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| angka |
| Posted: Tue Sep 01, 2009 7:57 pm Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
Message affinity is really required in our business process. e.g. "create profile" has to come before "update profile". a "update profile" may exist or not exist.. is there any best or standard design to handle message affinity? my input will be file and output MQ.
Any suggestion?
Thank you. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Tue Sep 01, 2009 10:15 pm Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| angka wrote: |
| "create profile" has to come before "update profile". a "update profile" may exist or not exist.. |
As I see it, you have two main choices.
1) If there is all the information required to create the profile in the update message then create it(if the business rules allow this)
2) Send the error back to the oroginating process. They are at fault not you.
There is obviously a bug in the originating process that would allow an update to a non existant record if the business rules state that this is not allowed.
Don't try to cover every aspect of every possible error to account for the mistakes made by others (bangs head against brick wall). Their apps should be as well behaved as yours.
Now I can get off my soapbox and have some brekkie.
( I thought there was an emocion for head banging?)
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| angka |
| Posted: Tue Sep 01, 2009 11:38 pm Post subject: |
|
|
Chevalier
Joined: 20 Sep 2005
Posts: 406
|
Hi,
No the Sender put the file to their folder in the correct sequence i.e. "create profile" follow by "update profile", however, the file input node picks up from the folder with multiple files in it not base on the sequence the file is written. so this is the problem... any one have implement something similar before?
Thanks |
|
| Back to top |
|
|
|
|
