| Author |
Message
|
| dertel |
 Posted: Wed May 20, 2009 12:53 pm Post subject: Broker 6.1 performance processing 100MB xml files |
 |
|
Novice
Joined: 17 Dec 2003
Posts: 15
|
Environment: WMB 6.1.0.3 on Z/linux
Is anyone having success at processing large (>100MB) xml files through the file processing nodes in broker 6.1? We have a flow to convert xml to flat files; it works for small files. We found apar IC58202 that allowed us to set MQSI_FILENODES_MAXIMUM_RECORD_LENGTH so that the fileinput node can receive the larger messages. We’re having difficulty getting acceptable performance. Our 250MB file finished in 5 hours today. Our z/linux server is sized at 4GB, and we have a 1.5GB heapsize on the execution group. Within about 20 seconds of starting the file processing, virtually all the free memory on the server is gone, and the cpu usage approachs the limit that VM allows. It sits that way for 5 hours; little free memory, maximum cpu consumption; during that time we see the output file growing slowly. We are making some flow changes including eliminating cardinality and moving references closer to the leaf level elements; we expect those will help. We would like to know if anyone is doing this successfully; we need to soon make a decision about going forward with this design. |
|
| Back to top |
|
 |
| paranoid221 |
| Posted: Thu May 21, 2009 10:06 pm Post subject: |
|
|
Centurion
Joined: 03 Apr 2006
Posts: 101
Location: USA
|
100MB/250MB XML files!!!   
Seems like you are hell-bent on tonking broker on its head & whacking the brains out of it. Just kidding brotha.
We have been using the file processing nodes in broker for smaller files. But I do know another project effort at our client where they use another tool called Itemfield ContentMaster (now called Informatica ContentMaster) to create flat files out of large messages.
Since the file processing nodes are a recent addition to broker, I'm not too sure if slamming large files at it is a good bet at this point. But that's just me.
_________________
LIFE is a series of complex calculations, somewhere multiplied by ZERO. |
|
| Back to top |
|
|
| Maximreality |
| Posted: Mon May 25, 2009 3:34 am Post subject: Re: Broker 6.1 performance processing 100MB xml files |
|
|
Acolyte
Joined: 04 Jun 2004
Posts: 65
Location: Copenhagen
|
Consider splitting the input file into smaller chunks...., you can always append data to the output file.
My experience is that you get best performance with files/messages less than 10 Mb.
As you mention the mem usage is the main problem, decreasing complexity of the message flow will give you better performance, but as long as you exceed the memory capacity you will get poor performance. |
|
| Back to top |
|
|
| mqpaul |
| Posted: Tue May 26, 2009 1:23 am Post subject: Re: Broker 6.1 performance processing 100MB xml files |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
| dertel wrote: |
Environment: WMB 6.1.0.3 on Z/linux
Is anyone having success at processing large (>100MB) xml files through the file processing nodes in broker 6.1? We have a flow to convert xml to flat files; it works for small files. We found apar IC58202 that allowed us to set MQSI_FILENODES_MAXIMUM_RECORD_LENGTH so that the fileinput node can receive the larger messages. We’re having difficulty getting acceptable performance. |
The FileInput node has no problems with that records of that size. If you are having difficulty, I suspect problems in the downstream flow processing.
| Quote: |
| Our 250MB file finished in 5 hours today. Our z/linux server is sized at 4GB, and we have a 1.5GB heapsize on the execution group. Within about 20 seconds of starting the file processing, virtually all the free memory on the server is gone, and the cpu usage approachs the limit that VM allows. It sits that way for 5 hours; little free memory, maximum cpu consumption; during that time we see the output file growing slowly. |
That will not be the result of the FileInput node. It opens the file and creates an input stream, and passes that stream to the parses and downstream nodes. This means the FileInput node uses very little system resource. It looks like something is happening downstream.
For a simple FileInput node, 1.5GB is way more than needed - for a simple whole file input the default heap (something like 64MB) is fine. Of course, the FileInput node shares the Java heap with all other Java nodes and Broker Java components. Depending on the other tasks running on your zSeries server,you may have over-committed main memory and be suffering paging. Paging always causes Java performance issue.
You also mention an output file. Is that 250MB as well? That might cause problems. The FileOutput node has to generate the entire output record as a Broker bit stream, and that will require 250 MB contiguous space in the JVM, and therefore probably cause heavy load through the garbage collector.
The message tree for a 250MB message might be much larger (I think up to 6 times as large, though not as a single object) depending on how many elements the tree contains. That might be why you are using 1.5GB heap. That's an awfully large tree. If that applies, you will probably need the processing techniques described in the Large Message Processing sample.
You say you see the output file growing. Are you writing multiple records to the output file - that should perform well provided the records are not large.
| Quote: |
| We are making some flow changes including eliminating cardinality and moving references closer to the leaf level elements; we expect those will help. We would like to know if anyone is doing this successfully; we need to soon make a decision about going forward with this design. |
If you want to be sure whether the FileInput node is part of the problem, copy the file contents to a queue, and try your flow with an MQInput node instead.
_________________
Paul |
|
| Back to top |
|
|
| happyj |
| Posted: Fri May 29, 2009 3:12 am Post subject: |
|
|
Voyager
Joined: 07 Feb 2005
Posts: 87
|
|
| Back to top |
|
|
| Gralgrathor |
| Posted: Fri Feb 12, 2010 4:06 am Post subject: Re: Broker 6.1 performance processing 100MB xml files |
|
|
Master
Joined: 23 Jul 2009
Posts: 297
|
| Maximreality wrote: |
| you can always append data to the output file |
How, on broker 6.1? Previously I worked with the MBFE plugin, but the FileInput node on 6.1 does not have an option "append to existing file". |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Feb 12, 2010 4:30 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The Fileoutput node can't append to an EXISTING file, no.
It can append to a NEW file, until you tell it that the new file is FINISHED. |
|
| Back to top |
|
|
| moonwalker |
| Posted: Mon Apr 19, 2010 3:00 am Post subject: |
|
|
Apprentice
Joined: 02 Jun 2009
Posts: 42
|
| paranoid221 wrote: |
But I do know another project effort at our client where they use another tool called Itemfield ContentMaster (now called Informatica ContentMaster) to create flat files out of large messages.
|
Is a free version for this available for trial purpose? I tried everywhere on the net and still searching. |
|
| Back to top |
|
|
| paranoid221 |
| Posted: Mon Apr 19, 2010 1:25 pm Post subject: |
|
|
Centurion
Joined: 03 Apr 2006
Posts: 101
Location: USA
|
I believe it is now called Informatica B2B Data Exchange.
Perhaps if you have friends working for Informatica, they may be able to get you from their software portal.
Good Luck finding it.
_________________
LIFE is a series of complex calculations, somewhere multiplied by ZERO. |
|
| Back to top |
|
|
|
|
