| Author |
Message
|
| Spidyctf |
 Posted: Tue Jan 09, 2007 12:11 am Post subject: Batch Processing using WBIMB V6.0 |
 |
|
Newbie
Joined: 09 Dec 2006
Posts: 8
|
Hi
We want to do batch processing using WBIMB V6.0. The situation is like this.
We would be getting huge number of files in a particular folder. The size of the files can be in GBs.
I need MB to pick up the files from that folder, transform them and proceed for further processing.
For example: A folder named "Input" contains files a.xml, b.xml, c.xml.
I need MB to pick up these files and transform them.
I am aware of FTP plugin for MB. But i think that FTP node wont be able to handle huge data files.
Can anyone suggest some plugin node or adapter for MB which can do this task.
Spidy |
|
| Back to top |
|
 |
| mvarghese |
| Posted: Tue Jan 09, 2007 12:19 am Post subject: |
|
|
Centurion
Joined: 27 Sep 2006
Posts: 141
|
We can to use Jtext Adapter for this case.
Please make sure that ur MQ setup is ready to process each file ..i mean size.
_________________
Jain Varghese |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jan 09, 2007 1:37 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| We would be getting huge number of files in a particular folder. The size of the files can be in GBs. |
Does each file contain a single, huge XML message. Or can one file contain several XML messages? |
|
| Back to top |
|
|
| Spidyctf |
| Posted: Tue Jan 09, 2007 1:48 am Post subject: |
|
|
Newbie
Joined: 09 Dec 2006
Posts: 8
|
ok... JText adapter might serve the purpose of picking up files from the folder.
But then in that case the transformation part has to be done in MB.
In that case we end up we writing transformation logic in MB for Messages 1 GB in sizes.
Also in certain cases it becomes difficult to make message sets and therefore we thot of using Datastage Transformation Extender Plugin which can support various types of industry standard messages format and can handle transformation easily using MAP's.
With Datastage Transformation Extender the issue is we can only pick a specific file for e.g. say xyz.xml it can't pick all the file with particular extention say by using *.xml.
So does anyone know such type of plugin for MB which can do batch processing and transformation both at the same time.
@ Kimbert
Each file contains huge xml message. |
|
| Back to top |
|
|
| kimbert |
| Posted: Tue Jan 09, 2007 1:54 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
| Quote: |
| In that case we end up we writing transformation logic in MB for Messages 1 GB in sizes. |
What kind of transformation logic? Are you converting to a non-XML format? Are you generating multiple XML messages for each input XML message? |
|
| Back to top |
|
|
| Spidyctf |
| Posted: Tue Jan 09, 2007 2:02 am Post subject: |
|
|
Newbie
Joined: 09 Dec 2006
Posts: 8
|
We would be doing some calculations based on some particular values.
As of now we are taking it as XML to XML transformation.
For each input file there will be corresponding output file.
(for ex: suppose input file is abc.xml then the output file will be something as abc_timeStamp.xml) |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Jan 09, 2007 2:36 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You are going to have many issues dealing with 1GB of data as a single "message" using WMB.
You won't be able to use the JText adapter, as the JText Adapter communicates with WMB using WMQ messages, and those can only be 100MB in size.
This means you won't be able to output the data as a single WMQ message, even if you can get it into WMB as a single piece of data.
Also, you are going to have to do a lot of performance tuning and allocate a lot of memory to WMB for this to work.
You should look very closely at your data and see if there is any way to process it in smaller pieces.
You should also look at the developerworks article on writing WMB flows that perform well.
And you should look at the File Extender product extension. This will allow you to read and write files from WMB locally, and can break the file up into smaller pieces.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mqmatt |
| Posted: Tue Jan 09, 2007 2:57 am Post subject: |
|
|
Grand Master
Joined: 04 Aug 2004
Posts: 1213
Location: Hursley, UK
|
| FWIW, the Message Broker development team understands well the requirement to parse huge messages such as these. |
|
| Back to top |
|
|
|
|
