| Author |
Message
|
| MarkOates |
 Posted: Thu Nov 06, 2008 7:01 am Post subject: FileInput Node - record count problem |
 |
|
Newbie
Joined: 24 Sep 2007
Posts: 6
|
We have a requirement to process a text file which has the following structure:
header
data 1
data 2
data 3
..
data n
footer
The problem we have is that the number of data lines in the text file is variable, but is stored in the footer - the footer stores the number of data lines plus a further 2 lines (1 each for the header and footer).
Is there an easy way of using a message set to determine the number of data lines in the file in the message set definition? If not what is the best way to achieve this.
The number of data lines in the message is always a positional field and will never change.
Any help would be much appreciated.
Rgds,
Mark |
|
| Back to top |
|
 |
| Vitor |
| Posted: Thu Nov 06, 2008 7:19 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
Are the header & trailer identified by anything other than their postion? For instance, another poster had a file with "0" in the 1st postion for the header, "9" in the footer and "5" in the data records. Is there anything like that you could use, to remove for the need for the record count.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| elvis_gn |
| Posted: Thu Nov 06, 2008 8:03 am Post subject: Re: FileInput Node - record count problem |
|
|
Padawan
Joined: 08 Oct 2004
Posts: 1905
Location: Dubai
|
Hi MarkOates,
Is your question related to FileInput node or is mainly about creating a message set for the input fetched ? The topic seems to throw me off.
| MarkOates wrote: |
| Is there an easy way of using a message set to determine the number of data lines in the file in the message set definition? If not what is the best way to achieve this. |
Yes you can. The repeat(or a property named like that  ) can be set to reference a field within the message defn. and use it's value as the number of occurrences.
You can ofcourse only do the above if the message set can differentiate the header and footer in the first place...either by position as Vitor asked or by some identifier.
But if you are sure that the first and the last row are the header and footer, then you don't really need the occurrence, do you ? You can simply ask all rows except first and last to be considered as 'data'.
Regards. |
|
| Back to top |
|
|
| MarkOates |
| Posted: Thu Nov 06, 2008 8:26 am Post subject: Re: FileInput Node - record count problem |
|
|
Newbie
Joined: 24 Sep 2007
Posts: 6
|
We are reading a file using a fileinput node and then using a message set to read the file, so the question is related to creating the message set.
The input message footer is identifiable by having a 'F' in character position 1, but we aren't sure how to reference this in the message set |
|
| Back to top |
|
|
| MarkOates |
| Posted: Thu Nov 06, 2008 8:45 am Post subject: Re: FileInput Node - record count problem |
|
|
Newbie
Joined: 24 Sep 2007
Posts: 6
|
Something else:
The reason we need to do this is to change part of each data row in the message, and then pass on the remaining content out to an MQ or file output node, including the message header and footer. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Nov 06, 2008 8:47 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| This seems like a basic TDS message, depending on how the header and data records are identified or layed out. |
|
| Back to top |
|
|
| MarkOates |
| Posted: Thu Nov 06, 2008 9:06 am Post subject: Re: FileInput Node - record count problem |
|
|
Newbie
Joined: 24 Sep 2007
Posts: 6
|
How can this be treated as a simple TDS message? I think I must be missing something! n example of the file we are trying to process looks like:
D00011225446469a0198020081031104749003
EIMSTD00002231000062YZ044 001 000000001145X
EIMSTD00002231100062YZ054 001 000000000000X
EIMSTD00002231200062YZ056 001 000000000000X
EIMSTD00002231300062YZ066 001 000000000000X
EIMSTD00002231400062YZ067 001 000000001150X
EIMSTD00002231500062YZ082 001 000000000000X
EIMSTD00002231600062ZGO008 001 000000000000X
EIMSTD00002231700062ZRR005 001 000000000000X
F000000010
As can be seen in the above example, the total number of rows in the message is 10, including the header and the footer. The footer also shows the total number of lines in the message. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Nov 06, 2008 9:12 am Post subject: Re: FileInput Node - record count problem |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| MarkOates wrote: |
| How can this be treated as a simple TDS message? I think I must be missing something! |
Possibly that the messages are Tagged by the first character, and Delimited by a CR/LF?
Doesn't seem too tricky. What problem do you encounter when modelling it?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| kimbert |
| Posted: Thu Nov 06, 2008 2:57 pm Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Hi Mark,
Your message consists of lines of text, each starting with a single-character tag. Message Broker calls that a 'Tagged Delimited' message with a delimiter of CR/LF ( or whatever you line feed character happens to be ).
You need to
- Create a message set
- Add a TDS physical format
- Create a new message looking like this:
| Code: |
Message
Complex Type Data Element Separation='Tagged Delimited', Delimiter='<CR><LF>', Tag Length=1
Element Name='Header' Tag='D'
Element Name='Line' Tag='E'
Element Name='Footer' Tag='F'
|
Obviously, the complex types which define Header, Line and Trailer will need a different Data Element Separation ( probably Fixed Length, or possibly All Elements Delimited ) but I cannot advise on that as you have not given any details about the internal structure of these elements.
Don't worry about the 'number of lines' element in the footer. You message flow is not adding or removing lines, so it does not need to change
elvis_gn: Two reasons why Repeat Reference is not the correct answer here. It only works when the counter comes before the repeating item, and it is not automatically populated when a message is written. |
|
| Back to top |
|
|
| Muhammad Haider |
| Posted: Tue Apr 05, 2011 10:11 pm Post subject: FileInput Node: Code to get File Name and record count |
|
|
Apprentice
Joined: 14 Mar 2011
Posts: 43
|
Here are my findings to get file name and reocrd count for FileInput node in broker 7. I used a java compute node at 'End of Data' terminal and used following code:
MbElement localEnv = contact admin.getLocalEnvironment().getRootElement();
String directory = localEnv.getFirstChild().getFirstChild().getValueAsString();
String fileName1 = localEnv.getFirstChild().getFirstChild().getNextSibling().getValueAsString();
logger1.info("File Name :"+fileName1);
String records = localEnv.getFirstChild().getFirstChild().getNextSibling().getNextSibling().getNextSibling().getNextSibling().getNextSibling().getValueAsString();
String propertyName = localEnv.getFirstChild().getFirstChild().getNextSibling().getNextSibling().getNextSibling().getNextSibling().getNextSibling().getName().toString(); // Record property
logger1.info("property :"+propertyName+" : "+recordNumber);
logger1.info("Record :"+records); |
|
| Back to top |
|
|
|
|
