| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| A way to handle really big xmls? |
View previous topic :: View next topic |
| Author |
Message
|
| Frik |
 Posted: Wed Nov 02, 2011 6:32 am Post subject: A way to handle really big xmls? |
 |
|
Acolyte
Joined: 25 Nov 2009
Posts: 69
|
Hello,
I have 2 questions about big sized xml files:
1. I have xml which represent part of some db.
The xml contains thousand records about customers, and its size is about
25 MB to 1000 GB (yes, you read it just right).
Is there a way to reformat this file to more readable format? (CSV, that can be handled in the broker is a good example, but I search something else)
Can it be done with the broker?
2. Do you know a program or application that can make such a xml to be read comfortably? (without reformat it)
Appreciate your help a lot.
F |
|
| Back to top |
|
 |
| Vitor |
| Posted: Wed Nov 02, 2011 6:49 am Post subject: Re: A way to handle really big xmls? |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Frik wrote: |
1. I have xml which represent part of some db.
The xml contains thousand records about customers, and its size is about
25 MB to 1000 GB (yes, you read it just right). |
That's not an XML document, it's a database expressed in an XML format.
| Frik wrote: |
| Is there a way to reformat this file to more readable format? (CSV, that can be handled in the broker is a good example, but I search something else) |
If I wanted to read (human read) 1000Gb of data I'd want it in an XML format so I can use XPath to find things. If I wanted to read it with broker I'd be likewise inclined to stick with XML to exploit the parser.
| Frik wrote: |
| Can it be done with the broker? |
Broker can do XML to CSV conversion if you want it to.
| Frik wrote: |
| 2. Do you know a program or application that can make such a xml to be read comfortably? (without reformat it) |
XMLSpy on a really big 64 bit machine?
Seriously, why are you not leaving the bulk of this data in the database where it belongs, & passing XML of changes / updates / processing requests / whatever into broker & getting it to select whatever's not in the XML from the database? Moving 1000Gb over a network is going to kill it.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| MrSmith |
| Posted: Wed Nov 02, 2011 8:19 am Post subject: |
|
|
Master
Joined: 20 Mar 2008
Posts: 215
|
Not only that but depending on the XML Parser (broker parses on demand by default) any DOM based parser that will try to load the whole doc will cream your memory use.
Why are these "XML files" so large sounds to me a bit of a design issue at 1000GB! 
_________________
-------- *
“Outside of a dog, a book is man's best friend. Inside of a dog it's too dark to read.” |
|
| Back to top |
|
|
| Frik |
| Posted: Sat Nov 05, 2011 11:38 pm Post subject: oops |
|
|
Acolyte
Joined: 25 Nov 2009
Posts: 69
|
oops, I meant - 25MB to 1000MB...
any other format but csv that can help me read this file?
Thanks |
|
| Back to top |
|
|
| smdavies99 |
| Posted: Sun Nov 06, 2011 12:47 am Post subject: Re: oops |
|
|
Jedi Council
Joined: 10 Feb 2003
Posts: 6076
Location: Somewhere over the Rainbow this side of Never-never land.
|
| Frik wrote: |
oops, I meant - 25MB to 1000MB...
any other format but csv that can help me read this file?
Thanks |
XML is supposed to be more 'human readable' than csv/tds formats
Are you suggesting that a Human is going to read every line in a CSV file this size and interpret it correctly? Are you mad?
The lines/rows will son send the person crazy. This sort of thing is what we have errr.... Computers for.
If you want to move this data somewhere else then I'd look at compressing it first but so far you have not really explained what the ultimate goal of this excericse.
To me, this sounds like a PHB has asked you to do this thinking it would be a good idea.
As has been said, even 1Gb of data is a lot of data. As these are customer records then there are lots of repeating groups so you could spilit it up by customer. This would give you items in a more manageable size.
_________________
WMQ User since 1999
MQSI/WBI/WMB/'Thingy' User since 2002
Linux user since 1995
Every time you reinvent the wheel the more square it gets (anon). If in doubt think and investigate before you ask silly questions. |
|
| Back to top |
|
|
| Vitor |
| Posted: Sun Nov 06, 2011 6:39 am Post subject: Re: oops |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| Frik wrote: |
| oops, I meant - 25MB to 1000MB... |
This doesn't alter any of my comments.
| Frik wrote: |
| any other format but csv that can help me read this file? |
If I had even 25Mb of data I'd prefer it in XML so I can use XPath. I can personally confirm that XMLSpy will quite happily cope with 25Mb of XML data (WinXP, 4MB RAM, 2ish GHz processor, cheap laptop) and even with 500 Mb of XML (Windows 7, dual core 3ish & big memory).
I echo the comments of my associate. The only way I want to look at a 25Mb csv file is if I'd imported it into Excel first and made a chart, i.e. not looking at the CSV!
But as I said above, broker will convert XML to CSV if that's what you want.
Again echoing, I'd be interested in the requirement here because I too think this sounds like a PHB deciding he wants a human readable database report without really thinking how or why.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
|
|
|
|
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
