ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportFileReadNode read line by line

Post new topicReply to topic
FileReadNode read line by line View previous topic :: View next topic
Author Message
matuwe
PostPosted: Fri Mar 17, 2017 4:11 am Post subject: FileReadNode read line by line Reply with quote

Master

Joined: 05 Dec 2007
Posts: 251

Hi Please help.. I am using FileRead Node to read a file line by line... I need to stop processing if one of the lines fails validation..

I noticed that it still picks up the next line from the file. I tried using throw node, but still doesn't work.

My requirement is to read a 10MB file which is too big to read all at once.. So I am reading one line at the time. If one line fails I need to discard the whole file.. and make sure the Outputfile sitting on transit doesn't get processed.
Back to top
View user's profile Send private message
Vitor
PostPosted: Fri Mar 17, 2017 4:27 am Post subject: Re: FileReadNode read line by line Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 24127
Location: Ohio, USA

matuwe wrote:
I noticed that it still picks up the next line from the file. I tried using throw node, but still doesn't work.


No, because you're throwing an error about that line.

matuwe wrote:
My requirement is to read a 10MB file which is too big to read all at once..


What are you running broker on? An 1980s pocket calculator?

That's a tenth of the default maximum of 100Mb, and my personal record for a file is 185Mb (on a sizeable Unix machine). There's specific guidance in the InfoCenter for techniques to process large files in a memory efficient way, which is important for larger files, don't get me wrong.

matuwe wrote:
If one line fails I need to discard the whole file.. and make sure the Outputfile sitting on transit doesn't get processed.


You need to set a shared variable or other transaction-surviving flag which you can check at the start of line processing to see if it should be processed or rejected out of hand because a previous line failed.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Fri Mar 17, 2017 5:06 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 16959

You're using a FileRead node.

You should get an exception from that node if the current record fails validation (I think?).

If you get that exception, and you don't hook up the exception terminal, then it will roll back to the input node. You then just need to make sure that the file gets moved, and doesn't get processed again. And throw an exception to make the flow halt.
_________________
Read, Think, Try, Repeat
Back to top
View user's profile Send private message
matuwe
PostPosted: Wed Mar 22, 2017 12:00 am Post subject: Reply with quote

Master

Joined: 05 Dec 2007
Posts: 251

Thanks for the response..

@mqjeff that's the behavior i am getting.. My Flow is

FILEINPUT >> Compute >> XSLT >> Compute >> FILEOUT (Append)

Only the fileInput catch is wired.. So when I get an exception, I see it goes back to FIleRead, then to my Exception subFlow.. But then it still goes to pick up the next record? How can I delete the file sitting in mqsitransitin folder

I am using IIB 9.0.0.3..

@Vitor Thanks for the response.. The problem is not IIB, the big file reads fine, but as soon as it hits the XSLT node, then I basically wait forever.



If I use shared variable then I cannot multi thread as this will affect other files processed around same time.

I just need to stop reading the file incase of error encounter.
Back to top
View user's profile Send private message
adubya
PostPosted: Wed Mar 22, 2017 12:46 am Post subject: Reply with quote

Partisan

Joined: 25 Aug 2011
Posts: 339
Location: GU12, UK

matuwe wrote:
Thanks for the response..

@mqjeff that's the behavior i am getting.. My Flow is

FILEINPUT >> Compute >> XSLT >> Compute >> FILEOUT (Append)

Only the fileInput catch is wired.. So when I get an exception, I see it goes back to FIleRead, then to my Exception subFlow.. But then it still goes to pick up the next record? How can I delete the file sitting in mqsitransitin folder
,


Add a throw node at the end of your exception handler. Currently, if you're catching the exception and not rethrowing it then as far as the flow is concerned processing can continue as you're managing the error.
_________________
Independent Middleware Consultant
andy@knownentity.com
Back to top
View user's profile Send private message Send e-mail
smdavies99
PostPosted: Wed Mar 22, 2017 1:26 am Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6046
Location: Somewhere over the Rainbow this side of Never-never land.

Reading each line may well be a separate unit of work. Thus the error that is rolled back only affects that line and not the whole file.
The only way to be sure is
1) a lot of experimantation
2) Raise a PMR and get the word from the horses mouth so to speak.

This is yet another reason why files are yesterdays technology and really should be avoided if at all possible.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportFileReadNode read line by line
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.