| Author |
Message
|
| matuwe |
 Posted: Fri Mar 17, 2017 4:11 am Post subject: FileReadNode read line by line |
 |
|
Master
Joined: 05 Dec 2007
Posts: 296
|
Hi Please help.. I am using FileRead Node to read a file line by line... I need to stop processing if one of the lines fails validation..
I noticed that it still picks up the next line from the file. I tried using throw node, but still doesn't work.
My requirement is to read a 10MB file which is too big to read all at once.. So I am reading one line at the time. If one line fails I need to discard the whole file.. and make sure the Outputfile sitting on transit doesn't get processed. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Fri Mar 17, 2017 4:27 am Post subject: Re: FileReadNode read line by line |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| matuwe wrote: |
| I noticed that it still picks up the next line from the file. I tried using throw node, but still doesn't work. |
No, because you're throwing an error about that line.
| matuwe wrote: |
| My requirement is to read a 10MB file which is too big to read all at once.. |
What are you running broker on? An 1980s pocket calculator?
That's a tenth of the default maximum of 100Mb, and my personal record for a file is 185Mb (on a sizeable Unix machine). There's specific guidance in the InfoCenter for techniques to process large files in a memory efficient way, which is important for larger files, don't get me wrong.
| matuwe wrote: |
| If one line fails I need to discard the whole file.. and make sure the Outputfile sitting on transit doesn't get processed. |
You need to set a shared variable or other transaction-surviving flag which you can check at the start of line processing to see if it should be processed or rejected out of hand because a previous line failed.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Mar 17, 2017 5:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
You're using a FileRead node.
You should get an exception from that node if the current record fails validation (I think?).
If you get that exception, and you don't hook up the exception terminal, then it will roll back to the input node. You then just need to make sure that the file gets moved, and doesn't get processed again. And throw an exception to make the flow halt.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| matuwe |
| Posted: Wed Mar 22, 2017 12:00 am Post subject: |
|
|
Master
Joined: 05 Dec 2007
Posts: 296
|
Thanks for the response..
@mqjeff that's the behavior i am getting.. My Flow is
FILEINPUT >> Compute >> XSLT >> Compute >> FILEOUT (Append)
Only the fileInput catch is wired.. So when I get an exception, I see it goes back to FIleRead, then to my Exception subFlow.. But then it still goes to pick up the next record? How can I delete the file sitting in mqsitransitin folder
I am using IIB 9.0.0.3..
@Vitor Thanks for the response.. The problem is not IIB, the big file reads fine, but as soon as it hits the XSLT node, then I basically wait forever.
If I use shared variable then I cannot multi thread as this will affect other files processed around same time.
I just need to stop reading the file incase of error encounter. |
|
| Back to top |
|
|
| adubya |
| Posted: Wed Mar 22, 2017 12:46 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
| matuwe wrote: |
Thanks for the response..
@mqjeff that's the behavior i am getting.. My Flow is
FILEINPUT >> Compute >> XSLT >> Compute >> FILEOUT (Append)
Only the fileInput catch is wired.. So when I get an exception, I see it goes back to FIleRead, then to my Exception subFlow.. But then it still goes to pick up the next record? How can I delete the file sitting in mqsitransitin folder
,
|
Add a throw node at the end of your exception handler. Currently, if you're catching the exception and not rethrowing it then as far as the flow is concerned processing can continue as you're managing the error.
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Mar 22, 2017 1:26 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Reading each line may well be a separate unit of work. Thus the error that is rolled back only affects that line and not the whole file.
The only way to be sure is
1) a lot of experimantation
2) Raise a PMR and get the word from the horses mouth so to speak.
This is yet another reason why files are yesterdays technology and really should be avoided if at all possible.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
|
|
