ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportGetting total file record count before processing a record

Post new topicReply to topic
Getting total file record count before processing a record View previous topic :: View next topic
Author Message
wmbfrz
PostPosted: Tue Nov 22, 2016 6:04 am Post subject: Getting total file record count before processing a record Reply with quote

Apprentice

Joined: 08 Jan 2010
Posts: 28

Hello,
We have requirement where each line of a flat file needs to be written as separate xml to a queue IF the number of records in the file is equal to the field value in the header. The LocalEnvironment.File.Record gives us the total record count after it has processed all the messages. Is there a way the flow can identify the total record count before the messages are processed? The flat file will have millions of records.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Nov 22, 2016 6:07 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Explain to the sender of the file that millions of records is bad.

Explain to the person giving you the requirement that millions of records is bad.

Read the file one record at a time. Do something with each record. Stick them somewhere. When you're done, do something to get them out of wherever you sent them, and then send them where/how they need to go.

Explain to your technical manager that millions of records is bad.

Explain to your technical manager that reading files is the Halting Problem.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
wmbfrz
PostPosted: Tue Nov 22, 2016 7:24 am Post subject: Reply with quote

Apprentice

Joined: 08 Jan 2010
Posts: 28

mqjeff wrote:
Explain to the sender of the file that millions of records is bad.

Explain to the person giving you the requirement that millions of records is bad.

Explain to your technical manager that millions of records is bad.

Explain to your technical manager that reading files is the Halting Problem.

I wish it would be so easy in the real world. I did speak but thats how it will be and ESB will need to solution it.
And i am looking for a solution where i can reject an invalid message before reading all records.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Nov 22, 2016 7:31 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

So you're looking to compare the number of actual records in the file with the value of a field at the top of the file?

You might be able to read a chunk of the file, and then see if IIB gives you the size of the file. Then you can potentially calculate the number of records in the file... if your records are fixed length.

If your records are something like XML or JSON, then you'll have to read all the records, count them and then compare.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
wmbfrz
PostPosted: Tue Nov 22, 2016 10:08 pm Post subject: Reply with quote

Apprentice

Joined: 08 Jan 2010
Posts: 28

An update on the message strucutre.
1) File will have only a body and footer.
2) Footer will have a filed to give total number of records.
3) Body will consist of multiple lines wherein each line is an xml.
Back to top
View user's profile Send private message
smdavies99
PostPosted: Wed Nov 23, 2016 12:01 am Post subject: Reply with quote

Jedi Council

Joined: 10 Feb 2003
Posts: 6077
Location: Somewhere over the Rainbow this side of Never-never land.

So, read the file and write the messages to a temporary queue until the footer is read.
Do the comparison and then do the right thing.
The temporary queue must have sufficient resources (depth) to let you write all the records to it.

As has been said, this is a frankly awful and very outdated design. Perhaps this is output from a legacy system?
What is the likelyhood of the records count not matching the count in the trailer?
If it does not match then the application/system writing the file has a bug that needs to be fixed.
Is this requirement just to get detect bugs in the old system?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995

Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions.
Back to top
View user's profile Send private message
zpat
PostPosted: Wed Nov 23, 2016 12:31 am Post subject: Reply with quote

Jedi Council

Joined: 19 May 2001
Posts: 5669
Location: UK

Files are sort of like queues, records are sort of like messages.

Combining multiple records into one message is OK up to a point, but it does introduce limits that have to coded around.

You can use groups of messages if you want them to be processed together.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Wed Nov 23, 2016 5:06 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

wmbfrz wrote:
An update on the message strucutre.
1) File will have only a body and footer.
2) Footer will have a filed to give total number of records.
3) Body will consist of multiple lines wherein each line is an xml.


So you don't actually have to check that the footer contains the same number of "records" as the body?

Also, whoever designed this message structure does not understand what XML is, nor how it works.

If the footer is the last record in the message tree, then you could use partial parsing ("on demand") to access the footer without parsing the rest of the message data.

But that *still* doesn't let you count the records in the body. If you don't want to parse them, or don't want to break them out in fixed length pieces, then you are out of luck.

You could change your model to make it a bit faster - treat each "xml" as a string, and only parse the footer.

This *only* works, in fact the whole thing only works, if you have some way of saying "this is the body, and now we have the footer" - that is, you have some *delimiter* or "indicator" when the footer starts.

If, properly, the entire document is a single XML document, then you will have to do something more complicated to handle this as strings for records and a separate string for a body.
_________________
chmod -R ugo-wx /
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Nov 23, 2016 5:44 am Post subject: Reply with quote

Grand Poobah

Joined: 18 Nov 2003
Posts: 19923
Location: LI,NY

The design reminds me of old times and unreliable ftp.
Having the expected number of records, (total debit / credit amounts, etc...) in a header or footer will allow you to check a file before processing to make sure that the file is complete... and you don't have part of it missing.

May I suggest that you might want to handle the file in a 2 step process.
One of these steps -- the integrity check -- could be made outside of IIB. Thus IIB would only see files that are complete and no longer need to do the integrity check.

Hope this helps
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
timber
PostPosted: Wed Nov 23, 2016 2:21 pm Post subject: Reply with quote

Guardian

Joined: 25 Aug 2015
Posts: 947

- Message flow #1
Read the file using a FileInput node with Record Detection set to Delimited and the delimiter set to the appropriate single character or pair of characters. Write the lines to an output file with a different name, counting the number of records processed.
When the end of file terminal is triggered, check that the value of the footer field matches your record count.

- Message flow #2
Process the file that was created by the first message flow.
Back to top
View user's profile Send private message
wmbfrz
PostPosted: Wed Nov 23, 2016 11:14 pm Post subject: Reply with quote

Apprentice

Joined: 08 Jan 2010
Posts: 28

Thanks for all the solutions provided. I was actually looking for something where ESB can avoid processing the file if it were not a valid one. It seems that it cannot be done. So next approach will be to minimise processing. Some approaches have already been suggested and i have thought of another non-verified one. Can we issue Queue commands like 'Get Inhibit' and 'Purge/Read' (Excluding MQGet node) from the flow? So the processing goes as follows
1) Flow Starts and get inhibits the queue A.
2) Reads the file and writes the records on the queue A.
3) The record validation is done at 'End of file' terminal after the last record.
4) if valid then 'Get Enable' queue A. So msg go to destination.
5) If invalid then purge the queue.
All this depends on the ability to issue queue commands from ESB. Please suggest if anyone has worked on any similar implementation.
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker SupportGetting total file record count before processing a record
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.