MQSeries.net :: View topic - On demand parsing BLOB/RCD

MQSeries.net

Tech Exchange

Education

Certifications

Library

Info Center

SupportPacs

FAQÂ Â

Usergroups

RSS Feed - WebSphere MQ Support

RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » On demand parsing BLOB/RCD

On demand parsing BLOB/RCD

« View previous topic :: View next topic »

Author

Message

longhorn

Posted: Thu May 12, 2016 11:53 am Post subject: On demand parsing BLOB/RCD

Novice

Joined: 22 May 2014
Posts: 14

I think I may be missing a subtlety here...

I'm observing an unexpected scenario with parsing. I have a flow (FileInputNode -> Compute -> MQOutput) within which I am processing a 200MB+ file with Records and Elements set to whole file.

I'm using the large messaging technique in the compute node which is working very well.

If I set the domain to DFDL on the FIN, memory usage only goes up by a miniscule amount (~60MB). However, if I set the domain to BLOB on the FIN and then use an RCD before the Compute node, memory usage goes up by about 700MB. Is this simply to be expected?

mqjeff

Posted: Thu May 12, 2016 12:07 pm Post subject:

Grand Master

Joined: 25 Jun 2008
Posts: 17447

The FIN is probably doing parse on demand. The RCD probably not.
_________________
chmod -R ugo-wx /

longhorn

Posted: Thu May 12, 2016 12:14 pm Post subject:

Novice

Joined: 22 May 2014
Posts: 14

Thanks for your response. That's what appears to be happening. However, I thought that RCD does not actually instigate an immediate parse. The code in the compute module has not changed so should still be on-demand...

timber

Posted: Thu May 12, 2016 2:04 pm Post subject:

Grand Master

Joined: 25 Aug 2015
Posts: 1292

See my reply on your other thread : http://www.mqseries.net/phpBB2/viewtopic.php?t=72226&sid=101056d138d10b5ef4060698630b8af9

A streaming parser does not load the entire input bitstream into memory before starting to parse. It loads small chunks, and when each chunk is exhausted it discards it and loads another.

BLOB is not a streaming parser. DFDL is. Does that explain it?

Display posts from previous:

Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » On demand parsing BLOB/RCD

Jump to:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum

Protected by Anti-Spam ACP