| Author |
Message
|
| gs |
 Posted: Fri Aug 22, 2014 3:57 am Post subject: FileNotFoundException in receiving application |
 |
|
Master
Joined: 31 May 2007
Posts: 254
Location: Sweden
|
We've got a a message flow which basically looks like this:
FileInput node -> compute node -> FileOutput node.
The file nodes are connected to remote windows servers through windows file sharing.
The messages are XML based and nodes are configured to use the whole file (no records). Output node is configured to use a staging directory (mqsitransit).
Now this works great most of the time but every now and then the receiving (java) application reports errors. They do a listFiles() every 5 seconds on the input directory and then go through every file returned from the method. Sometimes they get a FileNotFoundException for one of these files. Later when they try again it works just fine for the very same file.
Any hints, ideas, comments, similar experiences on this issue?
Could WMB have any part in this or is it strictly an OS issue?
I see two solutions to this which are:
1) have the application wait for a short while between listFiles() and opening the file
2) have the application catch the exception and retry later.
WMB 8.0.0.4 on Windows Server 2008 R2 |
|
| Back to top |
|
 |
| McueMart |
| Posted: Fri Aug 22, 2014 4:12 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
Maybe add a exists() and canRead() check when iterating through the list of files returned from listFiles()?
I expect this is a OS/Windows file sharing issue whereby the file becomes visible before it's actually readable.
Just a guess! |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 22, 2014 5:21 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's almost like you're discovering that using files to exchange data between application is a really bad idea.
Knowing whether or not a file is "ready" to be procesesd is THE HALTING PROBLEM.
It is unsolvable!
You can take a *lot* of steps within your code to make sure, EVEN UNDER THE CONTROLLED CONDITIONS OF LOCAL FILE SYSTESM, that you cover all of the eventualities.
Or you can *skip* ALL of that work, and rewrite the java app to accept MQ messages instead of Files.
REALLY. USE MESSAGES NOT FILES.
You're only scratching the surface of the problems you'll have to deal with. "The file wasn't there, except it said it was", "well, we read *most* of the file... "
 
YOU ARE STARTING DOWN A HARD ROAD. TURN BACK NOW. |
|
| Back to top |
|
|
| gs |
| Posted: Fri Aug 22, 2014 5:33 am Post subject: |
|
|
Master
Joined: 31 May 2007
Posts: 254
Location: Sweden
|
mqjeff,
Different customers have different integration maturity, budget, target architecture etc. Very often the ideal solution doesn't correlate with the reality you work within. |
|
| Back to top |
|
|
| gs |
| Posted: Fri Aug 22, 2014 5:34 am Post subject: |
|
|
Master
Joined: 31 May 2007
Posts: 254
Location: Sweden
|
| McueMart wrote: |
Maybe add a exists() and canRead() check when iterating through the list of files returned from listFiles()?
I expect this is a OS/Windows file sharing issue whereby the file becomes visible before it's actually readable.
Just a guess! |
Thanks, good points to take into account for the application team. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Aug 22, 2014 7:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| gs wrote: |
mqjeff,
Different customers have different integration maturity, budget, target architecture etc. Very often the ideal solution doesn't correlate with the reality you work within. |
So let's review.
Your customer is willing to have you spend what will be *at least a week* of interative changes to make their file based approach more reliable.
But they're not willing to have you spend three days changing it to use MQ?
Customer has Broker. Customer therefore has MQ. Customer therefore can use MQ to receive data at endpoint applications.
I repeat.
You are going to have spend a lot more time fixing this problem than you think you will.
And the fact that "exists" and "canRead" will return true does not remotely tell you that the file is complete and written. |
|
| Back to top |
|
|
| gs |
| Posted: Wed Aug 27, 2014 5:23 am Post subject: |
|
|
Master
Joined: 31 May 2007
Posts: 254
Location: Sweden
|
| mqjeff wrote: |
| gs wrote: |
mqjeff,
Different customers have different integration maturity, budget, target architecture etc. Very often the ideal solution doesn't correlate with the reality you work within. |
So let's review.
Your customer is willing to have you spend what will be *at least a week* of interative changes to make their file based approach more reliable.
But they're not willing to have you spend three days changing it to use MQ?
Customer has Broker. Customer therefore has MQ. Customer therefore can use MQ to receive data at endpoint applications.
I repeat.
You are going to have spend a lot more time fixing this problem than you think you will.
And the fact that "exists" and "canRead" will return true does not remotely tell you that the file is complete and written. |
It's not me you have to convince (in a rather harsh fashion) but the customer.
Let's just say this...what you prefer and recommend as an external partner doesn't always resonate with the customer. Some counter-arguments valid, some not. |
|
| Back to top |
|
|
|
|
