ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Logging Data When Using FileInput Node

Post new topic  Reply to topic
 Logging Data When Using FileInput Node « View previous topic :: View next topic » 
Author Message
apmohan
PostPosted: Wed Jul 31, 2019 3:15 am    Post subject: Logging Data When Using FileInput Node Reply with quote

Apprentice

Joined: 28 Dec 2012
Posts: 27

Hello All,

I am using IIB V9.0 version where my requirement is to read a file line by line, where I need to cache the records line by line and when the End of Data is received those records should be written to a database as part of logging process.

I tried using Shared Row Variable to store the records line by line which is not working. Can some one help here

Thanks
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Wed Jul 31, 2019 5:13 am    Post subject: Re: Logging Data When Using FileInput Node Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

apmohan wrote:
I tried using Shared Row Variable to store the records line by line which is not working.



Are you saying that the Shared Row Variable is not working? Because that's a PMR issue.

Or are you saying that the code you wrote is not achieving the desired result? Because that means your design and/or code is faulty.

We can help with neither of these; for the first, we're not IBM, for the second we can see neither the code you're using nor the effect you're getting.

apmohan wrote:
Can some one help here


Well on the face of it I'd store the individual records in a Shared Row Variable (or the global cache - Shared Row is probably easier) and write them out on End Of Data.

An alternative (and not necessarially a good one) would be to write the transactions to the database as they arrive in a temporary table. On End Of Data copy the contents of this table to the logging table and drop/truncate the temporary table. On Failure / Error / Other Reason To Stop truncate the temporary table.

Consider using stored procedures if you do go this route.

Or post your code and the problem you're having
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Vitor
PostPosted: Wed Jul 31, 2019 5:18 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

One variation would be to capture the inbound records with monitoring events (configured to include message payload) and have A.N.Other application handle them.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
apmohan
PostPosted: Wed Jul 31, 2019 5:37 am    Post subject: Reply with quote

Apprentice

Joined: 28 Dec 2012
Posts: 27

Thanks Vitor for the reply

Quote:
Well on the face of it I'd store the individual records in a Shared Row Variable (or the global cache - Shared Row is probably easier) and write them out on End Of Data.


I tried with the above one, please find the Input File which is been read:
Quote:
0120190730EFT REPORTS 0001NAMWERF1001D0001DATAOUT20190730TEST03500001
NAM NCE009 11H06 N A M C L E A R (PROPRIETARY) L I M I T E D 2019/07/30 PAGE:- 1

REG. NO. 2003/0645.
MAGTAPE SERVICE PROCESSING DATE :- 19/07/30

WORKUNIT NO : NS009D FILENAME : ESNS009D USER :- 1401
T D A / I N S T A L L A T I O N / H E A D E R / T R A I L E R R E P O R T
INSTALLATION CONTACT NAME & TEL NO
------------ ---------------------
ENHANCED BUSINESS ONLINE-COLLECTIONS 1401 ZANE KOTZE
P.O.BOX 3327 HEAD PAYMENTS
WINDHOEK 2942876


TAPE DETAILS
------------
TAPE NUMBER : 00000000
CREATION DATE : 19/07/30
DATE SUBMITTED : 19/07/30
----------------------------------------------------------------------------------------------------------------------------------

Code:

DECLARE InFile SHARED ROW;
DECLARE OutFile SHARED ROW;

CREATE COMPUTE MODULE VET_REPORT_transformation
   CREATE FUNCTION Main() RETURNS BOOLEAN
   BEGIN

      SET data = InputRoot.BLOB.BLOB;
      SET Environment.data = data;
      SET InFile = InFile || InputRoot.BLOB.BLOB;
      SET data = NULL;
      SET Environment.data = NULL;
      
On the Another Compute Node for the End of Data, below code is written      
      SET Environment.Events.Event.DataLog.DataBody.Details.Payload = InFile;


When checked on the Database (we have a separate flow where the Payload will be written to the Database), it is showing only the first line of the file and not the entire content of the file !!

Content logged in Database

Quote:
0120190730EFT REPORTS 0001NAMWERF1001D0001DATAOUT20190730TEST03500001
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Wed Jul 31, 2019 6:48 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

Code:

DECLARE InFile SHARED ROW;
DECLARE OutFile SHARED ROW;

CREATE COMPUTE MODULE VET_REPORT_transformation
   CREATE FUNCTION Main() RETURNS BOOLEAN
   BEGIN

      SET data = InputRoot.BLOB.BLOB;
      SET Environment.data = data;
      SET InFile = InFile || InputRoot.BLOB.BLOB;
      SET data = NULL;
      SET Environment.data = NULL;


Well firstly:
What exactly is that dance with data supposed to be doing? You set it to the input BLOB, add it to the Environment tree, then null it and the Environment out, deleting it.

Secondly:
That's not how you use a ROW variable. At all. The first line of the description says this:

Quote:
The ROW data type holds a tree structure


I'm struggling to work out how you thought concatenating it like it was CHARACTER was going to work.

apmohan wrote:

On the Another Compute Node for the End of Data, below code is written

Code:
SET Environment.Events.Event.DataLog.DataBody.Details.Payload = InFile;


We assume this somewhat pointless piece of code gets the data to the database in a way you've not posted. We further assume the SHARED ROW OutFile is part of this process, another logging structure for an Output file, or just something you added for no reason, like that thing with the data.

At this point (in your design) InFile contains a concatinatied string of all the Input BLOBs. Are you saying that you want the entire file contents, no matter the length, held as a single column in a database?
[/list]
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Wed Jul 31, 2019 8:48 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20696
Location: LI,NY

You do not do Justice to your design.
Reading a file Row by row will happen over multiple invocations of the FileInputNode, in fact one invocation for each row.
What you are looking for with the SHARED ROW is for a structure that spans the multiple invocations of the input node.

None of the Environment structures that you are effectively using does that.
Your concatenation of the Infile Shared Row is sooo wrong!. Read up in the manuals on how to use a shared row...
You also need to set a limit to the number of records that you are going to accept before writing out to the DB. You cannot hold a 100 GB file in memory!

Have fun
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
apmohan
PostPosted: Thu Aug 01, 2019 4:19 am    Post subject: Reply with quote

Apprentice

Joined: 28 Dec 2012
Posts: 27

Quote:
We further assume the SHARED ROW OutFile is part of this process, another logging structure for an Output file,


You are right OutFile is to store the output file which is happening as expected.

Quote:
Are you saying that you want the entire file contents, no matter the length, held as a single column in a database?
Yes this is what I am expecting,

Database log should be logging both the req & response for debugging purpose.

something like this
Quote:
<Details><Payload>333030373230313935303030303030303132383236373230303030303030303030303030303030303030303030303030303036323235323539393920202020202020202020202020202020202020202020202020202020202020202020202020202020202036323735373339303030303030303030303030303139332e3030303020202020202020202020202020202020202020202020202020202020202020202020202020202020202020564554534133303132</Payload></Details>
Back to top
View user's profile Send private message Send e-mail
Vitor
PostPosted: Thu Aug 01, 2019 5:01 am    Post subject: Reply with quote

Grand High Poobah

Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA

apmohan wrote:
Quote:
Are you saying that you want the entire file contents, no matter the length, held as a single column in a database?
Yes this is what I am expecting,


That's contact admin. What happens when you get a really big message.

apmohan wrote:
Database log should be logging both the req & response for debugging purpose.


It's slightly less contact admin if this is a non-prod testing construct, but even so this is a very large and unwieldy sledgehammer to crack a fairly small nut.

As I and others have said, your use of the ROW data type is way, way off. Given this comment:

apmohan wrote:
You are right OutFile is to store the output file which is happening as expected.


I suggest you compare the 2 pieces of code to determine your error.
_________________
Honesty is the best policy.
Insanity is the best defence.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Logging Data When Using FileInput Node
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.