ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker SupportBase64 Decode - Efficiency

Post new topicReply to topic Goto page 1, 2  Next
Base64 Decode - Efficiency View previous topic :: View next topic
Author Message
goffinf
PostPosted: Tue Jan 31, 2012 6:45 am Post subject: Base64 Decode - Efficiency Reply with quote

Chevalier

Joined: 05 Nov 2005
Posts: 401

version: 6.1.0.9

I have a flow which retrieves emails from an server. Some of these have attachments. The attachments are Base64 encoded.

Thats not a big problem EXCEPT when the attachments are large (ish) lets say 5MB.

I am using this code to do the Base64 Decode but it can take a couple of seconds (admittedly on a laptop, but even still) ... taking a user trace the line below is easily the most compute intensive part

Is there a more efficient method that I could try ???

... (I have noted that the XMLNSC parser supports Base64 encoding/decoding but I couldn't find an example of Decoding - not sure if that would help me but ...)

Code:


-- Replace the existing Part data with the Base64 DEcoded version
-- WOW this can take a bunch of time on a 5MB attachment !

SET Part.Data.BLOB.BLOB = base64Decode(CAST(Part.Data.BLOB.BLOB AS CHAR CCSID InputProperties.CodedCharSetId));

...
CREATE PROCEDURE base64Decode(IN msgChar CHARACTER) RETURNS BLOB
LANGUAGE JAVA
EXTERNAL NAME "com.ibm.broker.javacompute.Base64.decode";
Back to top
View user's profile Send private message
kimbert
PostPosted: Tue Jan 31, 2012 6:56 am Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:
I have noted that the XMLNSC parser supports Base64 encoding/decoding but I couldn't find an example of Decoding
No example required, because it is all done by the parser. All you need to do is tell the parser that the element in question is of type xsd:base64Binary. You do that using an xsd. When you parse the XML document the XMLNSC parser recognises the element, automatically decodes the base64 string and puts a BLOB into the message tree. Of course, in order to exploit this feature, you must
a) be using data that is in the XML format
b) have an xsd that describes the XML
c) switch on validation in your message flow
d) tell the XMLNSC parser to build the message tree using XML Schema types ( which is not the default setting, unlike in the SOAP domain ).

Not sure whether your email message is an XML message, but at least you have the facts at your disposal now. You may be able to use CREATE..PARSE provided that at least the base64 part of the email is XML.
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jan 31, 2012 6:57 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

Separate the call to the Java code from the CAST as character, just for performance analysis testing.

Take note of the fact that this operation does have to touch every single byte of the blob value, and thus scales linearly with the size of that blob.
Back to top
View user's profile Send private message
goffinf
PostPosted: Tue Jan 31, 2012 7:00 am Post subject: Reply with quote

Chevalier

Joined: 05 Nov 2005
Posts: 401

kimbert wrote:

Not sure whether your email message is an XML message, but at least you have the facts at your disposal now. You may be able to use CREATE..PARSE provided that at least the base64 part of the email is XML.


Thanks. Unfortunately the attachments are mostly jpeg and such like, so no dice there I guess.
Back to top
View user's profile Send private message
kimbert
PostPosted: Tue Jan 31, 2012 7:49 am Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:
Unfortunately the attachments are mostly jpeg and such like, so no dice there I guess.
Well...of course. That's why the sender is encoding them as base64! What I meant was "You may be able to use CREATE..PARSE provided that at least the base64 part of the email is wrapped in XML".
Back to top
View user's profile Send private message
goffinf
PostPosted: Tue Jan 31, 2012 8:54 am Post subject: Reply with quote

Chevalier

Joined: 05 Nov 2005
Posts: 401

kimbert wrote:
Quote:
Unfortunately the attachments are mostly jpeg and such like, so no dice there I guess.
Well...of course. That's why the sender is encoding them as base64! What I meant was "You may be able to use CREATE..PARSE provided that at least the base64 part of the email is wrapped in XML".


Maybe I'm just being a bit thick ... the email is a multipart related MIME message, each attachment in its own boundary and is just a bunch of Base64 encoded binary as a representation of a PDF, JPEG, etc ...

No XML AFAIK ??
Back to top
View user's profile Send private message
goffinf
PostPosted: Tue Jan 31, 2012 9:03 am Post subject: Reply with quote

Chevalier

Joined: 05 Nov 2005
Posts: 401

mqjeff wrote:
Take note of the fact that this operation does have to touch every single byte of the blob value, and thus scales linearly with the size of that blob.


Good idea. For the following code :-

Code:

SET attachmentAsChar = CAST(Part.Data.BLOB.BLOB AS CHAR CCSID InputProperties.CodedCharSetId);

SET attachmentB64Decoded = base64Decode(attachmentAsChar);

...
SET Part.Data.BLOB.BLOB = attachmentB64Decoded;


The CAST is about 30% of the total time
The DECODE about 40%

The rest is other stuff before and after.

mqjeff wrote:
Take note of the fact that this operation does have to touch every single byte of the blob value, and thus scales linearly with the size of that blob.


Sure I see this when I send 3 attachment of different sizes :-

att1 (500K) :- Total time around 1 sec (CAST: 0.3, DECODE: 0.4)

att2 (2500K): Total time around 4.5 sec (CAST: 1.8, DECODE: 2.3

att3: (<1K) Total time : 0.003 secs
Back to top
View user's profile Send private message
mqjeff
PostPosted: Tue Jan 31, 2012 9:11 am Post subject: Reply with quote

Grand Master

Joined: 25 Jun 2008
Posts: 17447

goffinf wrote:
kimbert wrote:
Quote:
Unfortunately the attachments are mostly jpeg and such like, so no dice there I guess.
Well...of course. That's why the sender is encoding them as base64! What I meant was "You may be able to use CREATE..PARSE provided that at least the base64 part of the email is wrapped in XML".


Maybe I'm just being a bit thick ... the email is a multipart related MIME message, each attachment in its own boundary and is just a bunch of Base64 encoded binary as a representation of a PDF, JPEG, etc ...

No XML AFAIK ??

He's suggesting you do something like
Code:
Create Last Child of OutputRoot DOMAIN(XMLNSC) PARSE('<xml><body>'||Part.Data.BLOB.BLOB||'</body>');

but of course aligned to a message set that indicated that Body was a base64 encoded field.

I can't say as this is particularly faster than a raw base64 decode. Kimbert might know, but he might not have tested it either.
Back to top
View user's profile Send private message
mgk
PostPosted: Tue Jan 31, 2012 9:38 am Post subject: Reply with quote

Padawan

Joined: 31 Jul 2003
Posts: 1591
Location: IBM Hursley, UK

This may not help you now, but in V7 ESQL has a native BASE64DECODE function built in...

Kind regards
_________________
MGK
IBM Global Blockchain Enablement
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Back to top
View user's profile Send private message
kimbert
PostPosted: Tue Jan 31, 2012 11:29 am Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

Quote:
Maybe I'm just being a bit thick
No - more likely I'm not explaining myself very well. However, mqjeff has successfully worked out what I was trying to say - we'll put that down to long and bitter experience!
re: the performance of that option...you would have to try it. I'm not making any predictions at all about the relative performance of the Java and C implementations of base64 decode. What I do know is that it's not a good idea to make bold assertions about the relative performance of Java and C++ unless you have the evidence in your hand
Back to top
View user's profile Send private message
xvigo
PostPosted: Thu Nov 08, 2012 6:37 pm Post subject: Reply with quote

Newbie

Joined: 08 Nov 2012
Posts: 4

kimbert wrote:
Quote:
I have noted that the XMLNSC parser supports Base64 encoding/decoding but I couldn't find an example of Decoding
No example required, because it is all done by the parser. All you need to do is tell the parser that the element in question is of type xsd:base64Binary. You do that using an xsd. When you parse the XML document the XMLNSC parser recognises the element, automatically decodes the base64 string and puts a BLOB into the message tree. Of course, in order to exploit this feature, you must
a) be using data that is in the XML format
b) have an xsd that describes the XML
c) switch on validation in your message flow
d) tell the XMLNSC parser to build the message tree using XML Schema types ( which is not the default setting, unlike in the SOAP domain ).

Not sure whether your email message is an XML message, but at least you have the facts at your disposal now. You may be able to use CREATE..PARSE provided that at least the base64 part of the email is XML.



Hi Thanks to your post I was able to have my xml attribute from the soap request decoded, unfortunately the result is an hexadecimal string.
(I put a trace just after the soap input node).
Do you have any suggestion to get the ascii representation of my request message?
(I tested the hex cod from the trace in a hex2ascii converted and I can see my ascii string (that actually is another xml message).


I think it's related to the type of WMB message (BLOB instead of character), but I don't know how to set that in the SOAP Input node, where the parser oerate.

(0x03000060:PCDataField+base64Binary)http://www.curamsoftware.com/WorkspaceServices/IntakeApplication:applicationData = X'3c3f786d6c2076657... Truncated by me' (BLOB)


Thanks, Giovanni
Back to top
View user's profile Send private message
mgk
PostPosted: Fri Nov 09, 2012 12:04 am Post subject: Reply with quote

Padawan

Joined: 31 Jul 2003
Posts: 1591
Location: IBM Hursley, UK

Quote:
Do you have any suggestion to get the ascii representation of my request message?


Yes, CAST it with a CCSID to a CHARACTER string.

Kind regards,
_________________
MGK
IBM Global Blockchain Enablement
The postings I make on this site are my own and don't necessarily represent IBM's positions, strategies or opinions.
Back to top
View user's profile Send private message
xvigo
PostPosted: Fri Nov 09, 2012 12:58 am Post subject: Reply with quote

Newbie

Joined: 08 Nov 2012
Posts: 4

mgk wrote:
Quote:
Do you have any suggestion to get the ascii representation of my request message?


Yes, CAST it with a CCSID to a CHARACTER string.

Kind regards,



Hey thanks for the quick answer.
I'm definitely new with Borker development, locking the infocenter and other resources, I wrote this piece of esql code, but I didn't get any result (at least in the debugger).
I guess that's the wrong way to add a new element to the output message.

Any advice?
thanks giovanni


CREATE COMPUTE MODULE C2B_Compute
CREATE FUNCTION Main() RETURNS BOOLEAN
BEGIN

CALL CopyMessageHeaders();
CALL CopyEntireMessage();

DECLARE requestIntakeOut CHARACTER '';
SET requestIntakeOut = CAST (InputRoot.XMLNSC.ns:receiveApplication.ns:applicationData AS CHARACTER CCSID InputRoot.Properties.CodedCharSetId);
SET OutputRoot.XMLNSC.Message.Type.requestIntakeOut = requestIntakeOut;


RETURN TRUE;
END;
Back to top
View user's profile Send private message
kimbert
PostPosted: Fri Nov 09, 2012 1:10 am Post subject: Reply with quote

Jedi Council

Joined: 29 Jul 2003
Posts: 5542
Location: Southampton

It looks as if you are base64-decoding the attribute and then putting the resulting content directly into OutputRoot.XMLNSC. Are you sure that this is a safe thing to do? Maybe there was a good reason why the sender base64-encoded the attribute contents.

If the debugger does not show you want you need, you can use a Trace node and/or take a user trace ( command-line utilities mqsichangetrace, mqsireadlog, mqsiformatlog. Search this forum for example command lines ).
Back to top
View user's profile Send private message
xvigo
PostPosted: Fri Nov 09, 2012 1:21 am Post subject: Reply with quote

Newbie

Joined: 08 Nov 2012
Posts: 4

The filed InputRoot.XMLNSC.ns:receiveApplication.ns:applicationData is already decoded (by the XMLNSC), what I need is to have the result as an XML (in order to map the content to another schema). Unfortunately the XMLNSC parser decode the message as hex.. so I need that as ascii (it's now a readable xml) and then do something to have that as xml message 'mappable" by a Borker mapper (graphical is better for me)

Hope this gives a bit of context.

Giovanni
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Goto page 1, 2  Next Page 1 of 2

MQSeries.net Forum IndexWebSphere Message Broker SupportBase64 Decode - Efficiency
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.