| Author |
Message
|
| apmohan |
 Posted: Wed Jul 31, 2019 3:15 am Post subject: Logging Data When Using FileInput Node |
 |
|
Apprentice
Joined: 28 Dec 2012
Posts: 27
|
Hello All,
I am using IIB V9.0 version where my requirement is to read a file line by line, where I need to cache the records line by line and when the End of Data is received those records should be written to a database as part of logging process.
I tried using Shared Row Variable to store the records line by line which is not working. Can some one help here 
Thanks |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Jul 31, 2019 5:13 am Post subject: Re: Logging Data When Using FileInput Node |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| apmohan wrote: |
| I tried using Shared Row Variable to store the records line by line which is not working. |
Are you saying that the Shared Row Variable is not working? Because that's a PMR issue.
Or are you saying that the code you wrote is not achieving the desired result? Because that means your design and/or code is faulty.
We can help with neither of these; for the first, we're not IBM, for the second we can see neither the code you're using nor the effect you're getting.
| apmohan wrote: |
| Can some one help here |
Well on the face of it I'd store the individual records in a Shared Row Variable (or the global cache - Shared Row is probably easier) and write them out on End Of Data.
An alternative (and not necessarially a good one) would be to write the transactions to the database as they arrive in a temporary table. On End Of Data copy the contents of this table to the logging table and drop/truncate the temporary table. On Failure / Error / Other Reason To Stop truncate the temporary table.
Consider using stored procedures if you do go this route.
Or post your code and the problem you're having 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 31, 2019 5:18 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
One variation would be to capture the inbound records with monitoring events (configured to include message payload) and have A.N.Other application handle them.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| apmohan |
| Posted: Wed Jul 31, 2019 5:37 am Post subject: |
|
|
Apprentice
Joined: 28 Dec 2012
Posts: 27
|
Thanks Vitor for the reply
| Quote: |
| Well on the face of it I'd store the individual records in a Shared Row Variable (or the global cache - Shared Row is probably easier) and write them out on End Of Data. |
I tried with the above one, please find the Input File which is been read:
| Quote: |
0120190730EFT REPORTS 0001NAMWERF1001D0001DATAOUT20190730TEST03500001
NAM NCE009 11H06 N A M C L E A R (PROPRIETARY) L I M I T E D 2019/07/30 PAGE:- 1
REG. NO. 2003/0645.
MAGTAPE SERVICE PROCESSING DATE :- 19/07/30
WORKUNIT NO : NS009D FILENAME : ESNS009D USER :- 1401
T D A / I N S T A L L A T I O N / H E A D E R / T R A I L E R R E P O R T
INSTALLATION CONTACT NAME & TEL NO
------------ ---------------------
ENHANCED BUSINESS ONLINE-COLLECTIONS 1401 ZANE KOTZE
P.O.BOX 3327 HEAD PAYMENTS
WINDHOEK 2942876
TAPE DETAILS
------------
TAPE NUMBER : 00000000
CREATION DATE : 19/07/30
DATE SUBMITTED : 19/07/30
----------------------------------------------------------------------------------------------------------------------------------
|
| Code: |
DECLARE InFile SHARED ROW;
DECLARE OutFile SHARED ROW;
CREATE COMPUTE MODULE VET_REPORT_transformation
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET data = InputRoot.BLOB.BLOB;
SET Environment.data = data;
SET InFile = InFile || InputRoot.BLOB.BLOB;
SET data = NULL;
SET Environment.data = NULL;
On the Another Compute Node for the End of Data, below code is written
SET Environment.Events.Event.DataLog.DataBody.Details.Payload = InFile;
|
When checked on the Database (we have a separate flow where the Payload will be written to the Database), it is showing only the first line of the file and not the entire content of the file !!
Content logged in Database
| Quote: |
0120190730EFT REPORTS 0001NAMWERF1001D0001DATAOUT20190730TEST03500001
|
|
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jul 31, 2019 6:48 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Code: |
DECLARE InFile SHARED ROW;
DECLARE OutFile SHARED ROW;
CREATE COMPUTE MODULE VET_REPORT_transformation
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN
SET data = InputRoot.BLOB.BLOB;
SET Environment.data = data;
SET InFile = InFile || InputRoot.BLOB.BLOB;
SET data = NULL;
SET Environment.data = NULL;
|
Well firstly:
What exactly is that dance with data supposed to be doing? You set it to the input BLOB, add it to the Environment tree, then null it and the Environment out, deleting it.
Secondly:
That's not how you use a ROW variable. At all. The first line of the description says this:
| Quote: |
| The ROW data type holds a tree structure |
I'm struggling to work out how you thought concatenating it like it was CHARACTER was going to work.
| apmohan wrote: |
On the Another Compute Node for the End of Data, below code is written |
| Code: |
SET Environment.Events.Event.DataLog.DataBody.Details.Payload = InFile;
|
We assume this somewhat pointless piece of code gets the data to the database in a way you've not posted. We further assume the SHARED ROW OutFile is part of this process, another logging structure for an Output file, or just something you added for no reason, like that thing with the data.
At this point (in your design) InFile contains a concatinatied string of all the Input BLOBs. Are you saying that you want the entire file contents, no matter the length, held as a single column in a database?
[/list]
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jul 31, 2019 8:48 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
You do not do Justice to your design.
Reading a file Row by row will happen over multiple invocations of the FileInputNode, in fact one invocation for each row.
What you are looking for with the SHARED ROW is for a structure that spans the multiple invocations of the input node.
None of the Environment structures that you are effectively using does that.
Your concatenation of the Infile Shared Row is sooo wrong!. Read up in the manuals on how to use a shared row...
You also need to set a limit to the number of records that you are going to accept before writing out to the DB. You cannot hold a 100 GB file in memory!
Have fun 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| apmohan |
| Posted: Thu Aug 01, 2019 4:19 am Post subject: |
|
|
Apprentice
Joined: 28 Dec 2012
Posts: 27
|
| Quote: |
| We further assume the SHARED ROW OutFile is part of this process, another logging structure for an Output file, |
You are right OutFile is to store the output file which is happening as expected.
| Quote: |
| Are you saying that you want the entire file contents, no matter the length, held as a single column in a database? |
Yes this is what I am expecting,
Database log should be logging both the req & response for debugging purpose.
something like this
| Quote: |
| <Details><Payload>333030373230313935303030303030303132383236373230303030303030303030303030303030303030303030303030303036323235323539393920202020202020202020202020202020202020202020202020202020202020202020202020202020202036323735373339303030303030303030303030303139332e3030303020202020202020202020202020202020202020202020202020202020202020202020202020202020202020564554534133303132</Payload></Details> |
|
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 01, 2019 5:01 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| apmohan wrote: |
| Quote: |
| Are you saying that you want the entire file contents, no matter the length, held as a single column in a database? |
Yes this is what I am expecting, |
That's contact admin. What happens when you get a really big message.
| apmohan wrote: |
| Database log should be logging both the req & response for debugging purpose. |
It's slightly less contact admin if this is a non-prod testing construct, but even so this is a very large and unwieldy sledgehammer to crack a fairly small nut.
As I and others have said, your use of the ROW data type is way, way off. Given this comment:
| apmohan wrote: |
| You are right OutFile is to store the output file which is happening as expected. |
I suggest you compare the 2 pieces of code to determine your error.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
