| Author |
Message
|
| rajeevreddy |
 Posted: Sat Jan 24, 2009 11:44 pm Post subject: File Processing in WMB V6.1 |
 |
|
Apprentice
Joined: 07 Sep 2004
Posts: 39
|
|
| Back to top |
|
 |
| smdavies99 |
| Posted: Sun Jan 25, 2009 12:10 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
There have been several posts in this forum over the past few months about just this very topic.
Performance has been an issue mentioned in them as well.
That siad, without knowing the basic structure of your file and what you intend to do with it plus how large is large we can't give you any specific help.
Some time ago I had a discussion with a customer about this issue. They were outputting one large file 'because the App Supplier said so'. After a little investigation, it turned out that it was simple for the App to output lots of much smaller files.
The result was that the solution was
1) easier to implement
2) faster ( the end app started receiving the transformed msgs sooner)
Which meant for a happier customer.... (& me for that matter)
Procesing large files can result in a long UOW, high CPU usage and unless the sysadmins are aware of this, sometimes they will 'kill' the offending process because they think it is stuck in a loop. 
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| rajeevreddy |
| Posted: Sun Jan 25, 2009 9:00 am Post subject: |
|
|
Apprentice
Joined: 07 Sep 2004
Posts: 39
|
We are planning todo below tasks,
Read the file from FTP location. Parse record by record and map to a target format using a mapping node . Finally write to a queue or to a file. (and vise versa)
File size may from vary from 100MB to few GB's.
Thanks. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Sun Jan 25, 2009 9:55 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
As long as you read the file record by record, and remember that you don't have any transactionality with the FileInput node, you'll be fine.
You may want to write to a queue first, then map and re-output from a separate flow just for transactional recovery.
Also, do not use the Trace node with any large data. It'll abend your EG.  |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Jan 25, 2009 10:41 am Post subject: |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
Whats the betting that the file is XML with lots of embedded records...
(see an entry here from abouyt a week or so ago)
Sigh. People think that one humoungous XML tree struture is the answer to life the univers and everything.
Bring back the old days when this would be written as records to a flat file in Cobol Copybook format.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jan 25, 2009 10:50 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| smdavies99 wrote: |
Bring back the old days when this would be written as records to a flat file in Cobol Copybook format.
|
These were the batch processing days  . Nowadays you'd have to create a single message for each atomic transaction that you used to batch into that huge file and yes most of the time it is xml. However a lot of people just have not switched their thinking mode and just send the huge batch file with a huge XML transaction in it... Grouping of the atomic transactions with a batch indicator would be much more efficient... This would allow each system to carry the batch indicator, verify the batch totals (# and $), and ultimately run a back-out if needed on a particular batch...
Enough said 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| rajeevreddy |
| Posted: Sun Jan 25, 2009 8:19 pm Post subject: |
|
|
Apprentice
Joined: 07 Sep 2004
Posts: 39
|
We are dealing with all mainframe data....we need to get the file from file server and process the main frame data.Data will be parsed and validated record by record againest the COBOL copy which will loaded into the message set.
Thanks. |
|
| Back to top |
|
|
|
|
