| Author |
Message
|
| wmbfrz |
 Posted: Tue Nov 22, 2016 6:04 am Post subject: Getting total file record count before processing a record |
 |
|
Apprentice
Joined: 08 Jan 2010
Posts: 28
|
Hello,
We have requirement where each line of a flat file needs to be written as separate xml to a queue IF the number of records in the file is equal to the field value in the header. The LocalEnvironment.File.Record gives us the total record count after it has processed all the messages. Is there a way the flow can identify the total record count before the messages are processed? The flat file will have millions of records. |
|
| Back to top |
|
 |
| mqjeff |
| Posted: Tue Nov 22, 2016 6:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Explain to the sender of the file that millions of records is bad.
Explain to the person giving you the requirement that millions of records is bad.
Read the file one record at a time. Do something with each record. Stick them somewhere. When you're done, do something to get them out of wherever you sent them, and then send them where/how they need to go.
Explain to your technical manager that millions of records is bad.
Explain to your technical manager that reading files is the Halting Problem.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| wmbfrz |
| Posted: Tue Nov 22, 2016 7:24 am Post subject: |
|
|
Apprentice
Joined: 08 Jan 2010
Posts: 28
|
| mqjeff wrote: |
Explain to the sender of the file that millions of records is bad.
Explain to the person giving you the requirement that millions of records is bad.
Explain to your technical manager that millions of records is bad.
Explain to your technical manager that reading files is the Halting Problem. |
I wish it would be so easy in the real world. I did speak but thats how it will be and ESB will need to solution it.
And i am looking for a solution where i can reject an invalid message before reading all records. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Nov 22, 2016 7:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
So you're looking to compare the number of actual records in the file with the value of a field at the top of the file?
You might be able to read a chunk of the file, and then see if IIB gives you the size of the file. Then you can potentially calculate the number of records in the file... if your records are fixed length.
If your records are something like XML or JSON, then you'll have to read all the records, count them and then compare.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| wmbfrz |
| Posted: Tue Nov 22, 2016 10:08 pm Post subject: |
|
|
Apprentice
Joined: 08 Jan 2010
Posts: 28
|
An update on the message strucutre.
1) File will have only a body and footer.
2) Footer will have a filed to give total number of records.
3) Body will consist of multiple lines wherein each line is an xml. |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Wed Nov 23, 2016 12:01 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
So, read the file and write the messages to a temporary queue until the footer is read.
Do the comparison and then do the right thing.
The temporary queue must have sufficient resources (depth) to let you write all the records to it.
As has been said, this is a frankly awful and very outdated design. Perhaps this is output from a legacy system?
What is the likelyhood of the records count not matching the count in the trailer?
If it does not match then the application/system writing the file has a bug that needs to be fixed.
Is this requirement just to get detect bugs in the old system?
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Nov 23, 2016 12:31 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Files are sort of like queues, records are sort of like messages.
Combining multiple records into one message is OK up to a point, but it does introduce limits that have to coded around.
You can use groups of messages if you want them to be processed together.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 23, 2016 5:06 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| wmbfrz wrote: |
An update on the message strucutre.
1) File will have only a body and footer.
2) Footer will have a filed to give total number of records.
3) Body will consist of multiple lines wherein each line is an xml. |
So you don't actually have to check that the footer contains the same number of "records" as the body?
Also, whoever designed this message structure does not understand what XML is, nor how it works.
If the footer is the last record in the message tree, then you could use partial parsing ("on demand") to access the footer without parsing the rest of the message data.
But that *still* doesn't let you count the records in the body. If you don't want to parse them, or don't want to break them out in fixed length pieces, then you are out of luck.
You could change your model to make it a bit faster - treat each "xml" as a string, and only parse the footer.
This *only* works, in fact the whole thing only works, if you have some way of saying "this is the body, and now we have the footer" - that is, you have some *delimiter* or "indicator" when the footer starts.
If, properly, the entire document is a single XML document, then you will have to do something more complicated to handle this as strings for records and a separate string for a body.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Nov 23, 2016 5:44 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
The design reminds me of old times and unreliable ftp.
Having the expected number of records, (total debit / credit amounts, etc...) in a header or footer will allow you to check a file before processing to make sure that the file is complete... and you don't have part of it missing.
May I suggest that you might want to handle the file in a 2 step process.
One of these steps -- the integrity check -- could be made outside of IIB. Thus IIB would only see files that are complete and no longer need to do the integrity check.
Hope this helps 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| timber |
| Posted: Wed Nov 23, 2016 2:21 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
- Message flow #1
Read the file using a FileInput node with Record Detection set to Delimited and the delimiter set to the appropriate single character or pair of characters. Write the lines to an output file with a different name, counting the number of records processed.
When the end of file terminal is triggered, check that the value of the footer field matches your record count.
- Message flow #2
Process the file that was created by the first message flow. |
|
| Back to top |
|
|
| wmbfrz |
| Posted: Wed Nov 23, 2016 11:14 pm Post subject: |
|
|
Apprentice
Joined: 08 Jan 2010
Posts: 28
|
Thanks for all the solutions provided. I was actually looking for something where ESB can avoid processing the file if it were not a valid one. It seems that it cannot be done. So next approach will be to minimise processing. Some approaches have already been suggested and i have thought of another non-verified one. Can we issue Queue commands like 'Get Inhibit' and 'Purge/Read' (Excluding MQGet node) from the flow? So the processing goes as follows
1) Flow Starts and get inhibits the queue A.
2) Reads the file and writes the records on the queue A.
3) The record validation is done at 'End of file' terminal after the last record.
4) if valid then 'Get Enable' queue A. So msg go to destination.
5) If invalid then purge the queue.
All this depends on the ability to issue queue commands from ESB. Please suggest if anyone has worked on any similar implementation. |
|
| Back to top |
|
|
|
|
