| Author |
Message
|
| prasunad |
 Posted: Mon Aug 18, 2014 8:12 am Post subject: |
 |
|
Novice
Joined: 10 Jul 2014
Posts: 22
|
Yes Vitor, This has been asked already, but it is a case i have been assigned to which has been pre-agreed.
Been saying, for Bulk data use ETL and not ESB..  |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Aug 18, 2014 8:36 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| prasunad wrote: |
| it is a case i have been assigned to which has been pre-agreed. |
Tell them to look up "rework" in a dictionary.
| prasunad wrote: |
| Been saying, for Bulk data use ETL and not ESB.. |
It's a valid point; really depends on the payload of the XML.
You should at least mention your life will be greatly simplified if they abandon XML so you can use a streaming parser. Whatever they've got at the other end (and I assume some kind of unload utility based on your example XML) shouldn't find that much of a problem.
Failing that, and to return to your original point, you've no option but to slice this by hand or used parsed record sequence (which I don't think will work in your particular case because you need to record container headers) and watch your resource usage climb.
Increasing of course the risk that the flow will abend with an out of resource error halfway through processing..... 
_________________
Honesty is the best policy.
Insanity is the best defence.
Last edited by Vitor on Mon Aug 18, 2014 8:39 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 18, 2014 8:38 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| XMLNSC is a streaming parser. |
|
| Back to top |
|
|
| Vitor |
| Posted: Mon Aug 18, 2014 8:40 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| mqjeff wrote: |
| XMLNSC is a streaming parser. |
I apologise if I've offended with the wrong terminology, and I await (as does the OP) a solution that allows the streaming of a 2Gb+ XML file stanza by stanza.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Aug 18, 2014 9:13 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| Vitor wrote: |
| mqjeff wrote: |
| XMLNSC is a streaming parser. |
I apologise if I've offended with the wrong terminology, and I await (as does the OP) a solution that allows the streaming of a 2Gb+ XML file stanza by stanza. |
None of the streaming parsers support more than 2gb. That's the RFE.
XMLNSC will let you use parsed record sequence to stream out stanza by stanza. As long as the file is less than 2Gb.
 |
|
| Back to top |
|
|
| prasunad |
| Posted: Tue Aug 19, 2014 12:39 am Post subject: |
|
|
Novice
Joined: 10 Jul 2014
Posts: 22
|
| Got a doubt, if i dont need to send the record headers to the target, then will the slicing work.. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 19, 2014 4:49 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| prasunad wrote: |
| Got a doubt, if i dont need to send the record headers to the target, then will the slicing work.. |
Using parsed record slicing never "works", it functions while consuming huge amounts of resource.
But you should be able to parse such that the <pl> and </pl> forms a single record (as one of my worthy associates mentioned earlier). I hope you a) have a lot of memory availble to the broker b) don't have a tight SLA c) don't have very large stanza and d) are ready for disapointment when you find it doesn't actually work out.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 19, 2014 5:09 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| mqjeff wrote: |
XMLNSC will let you use parsed record sequence to stream out stanza by stanza. As long as the file is less than 2Gb.
|
I thought the whole point of parsed record was to overcome that 2GB limit.
My understanding was that the single message in parsed record had to be under 2GB. So if you eliminate the xml root here (Data), you'd have multiples of <pl>....</pl>. Using the pl tag as root xml, are you saying that the file still needs to be under 2GB? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
|
|
