| Author |
Message
|
| eammnsh |
 Posted: Sat Dec 31, 2016 9:55 am Post subject: EmailOutput Node MIME Binary Attachment Encoding Issues |
 |
|
Apprentice
Joined: 22 Aug 2016
Posts: 29
|
We have IIB V10.0.0.7 running on Linux Redhat X86-64.
we are trying to send an Email through EmailOutput node with MIME attachment, the main issue is that we are reeving the attachment but corrupted, and that's because we can see extra X'0A' and X'0D' are added to our attachment.
The extra X'0A and X'0D' are not affecting the tex, csv, ..etc files, but for sure corrupt the Images, docs, pdfs, etc..
Below are our code, and the comparison between the original file and the received one by Email:
| Code: |
BEGIN
CALL CopyMessageHeaders();
SET OutputRoot.EmailOutputHeader.To = InputRoot.XMLNSC.MSG.Body.sendEmailRq.emailTo;
SET OutputRoot.EmailOutputHeader.Cc = InputRoot.XMLNSC.MSG.Body.sendEmailRq.emailCC;
SET OutputRoot.EmailOutputHeader.Subject = InputRoot.XMLNSC.MSG.Body.sendEmailRq.subject;
-- Create a new MIME message body, which will be sent as the main text of the email,
-- including an attachment.
CREATE FIELD OutputRoot.MIME TYPE Name;
DECLARE M REFERENCE TO OutputRoot.MIME;
-- Create the Content-Type child of MIME explicitly to ensure the correct order. If we set
-- the ContentType property instead, the field could appear as the last child of MIME.
CREATE FIELD M."Content-Type" TYPE NameValue VALUE 'multipart/Mixed; boundary=myBoundary';
CREATE FIELD M."Content-ID" TYPE NameValue VALUE 'new MIME document';
CREATE LASTCHILD OF M TYPE Name NAME 'Parts';
CREATE LASTCHILD OF M.Parts TYPE Name NAME 'Part';
DECLARE P1 REFERENCE TO M.Parts.Part;
-- First part:
-- Create the body of the email.
-- The body of the email has the text 'This is the main body of the email.'.
CREATE FIELD P1."Content-Type" TYPE NameValue VALUE 'text/plain; charset=utf-8';
CREATE FIELD P1."Content-Transfer-Encoding" TYPE NameValue VALUE '8bit';
CREATE LASTCHILD OF P1 TYPE Name NAME 'Data';
CREATE LASTCHILD OF P1.Data DOMAIN('BLOB') PARSE(CAST(InputRoot.XMLNSC.MSG.Body.sendEmailRq.body AS BLOB CCSID 1208));
CREATE LASTCHILD OF M.Parts TYPE Name NAME 'Part';
DECLARE P2 REFERENCE TO M.Parts.Part[2];
-- Second part:
-- Create the attachment of an email.
CREATE FIELD P2."Content-Type" TYPE NameValue VALUE InputRoot.XMLNSC.MSG.Attachment.*[<].MIME_Headers."Content-Type";
CREATE FIELD P2."Content-Transfer-Encoding" TYPE NameValue VALUE InputRoot.XMLNSC.MSG.Attachment.*[<].MIME_Headers."Content-Transfer-Encoding";
CREATE FIELD P2."Content-Id" TYPE NameValue VALUE InputRoot.XMLNSC.MSG.Attachment.*[<].MIME_Headers."Content-Id";
CREATE LASTCHILD OF P2 TYPE Name NAME 'Data';
CREATE LASTCHILD OF P2.Data DOMAIN('BLOB');
SET P2.Data.BLOB.BLOB = InputRoot.XMLNSC.MSG.Attachment.*[<].BLOB.BLOB;
RETURN TRUE;
END;
|
To see the image, use the below HTML
| Code: |
<img src='data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABRoAAAJOCAYAAADYnbtEAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAA
DsMAAA7DAcdvqGQAAP+lSURBVHhe7L2/byU7ri7qP+YGN+zkBA5f+NJJnE5wYKCzB+zQyTXQOKkxkeHAwE09B9gwMIAjw5lvMMD1P2A0OtvxROckG6gnUlIVxa
JYlFbV+uHFb/BNe5UkiiJV1eK3a7UvhhXxr3/9C/m//tf/Gn92Op1Op9PpdDqdTqfT6XQ6nV+fLjQ6nU6n0+l0Op1Op9PpdDqdzp3pQqPT6XQ6nU6n0+l0Op1Op9Pp
3JkuNDqdTqfT6XQ6nU6n0+l0Op3OnelCo9PpdDqdTqfT6XQ6nU6n0+ncmS40Op1Op9PpdDqdTqfT6XQ6nc6d6UKj0+l0Op1Op9PpdDqdTqfT6dyZxys0/udfh4uLv
w7/Sa79518vwrXIf/uPj7I/EMb89T/Dzx/Df/zb1Pev/8n6mGwo82k2BL+Bk61/G/7jo2wbWRlbvU7J+nz8x7+l+Si5DTlOtrH/Ofy1aKfrSm0pjpzRvhKHHu4Qd97Hsn
7s82//MXyQa5Gw9pXXtil5Htn9sjmVeH38x/Bvq8ayttZTy5nT6XQ6e6meC+AsQc8us7OFcr7MJDaO5hyZWJ5vcj/DmgLlsYHaWpD8717qf2o7gfNi8/pNcVl5bZuydo
baF5V4+XnR6XQ6j45HKDSmA89f/xoe5uQv8vCXyH+Mf6FJD3oYl6/95/CfuS/+5ZPtwDj6s2KjOl/NRsXvQDi8yIeMzNrYus3lsSXhgDT3oRankrWxNH54ABuFt9j2b
2M+KKEt+DuLfS93ibutz3z9aQ2rCY09Y9bivufm8+1z/tpch4y/0+l0OvdF/e98OE...... |
|
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Sat Dec 31, 2016 3:38 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Looks like you define the image as hex encoded whereas the content is defined as per the image tag as base 64 encoded....
Base 64 has certain rules as to the length of a line... and will introduce as many line breaks as necessary. Reading the data with a base64 decoder can then drop the new lines and reassemble the data as one stream...
You need to define the container according to the data it contains and not to your wishes... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| eammnsh |
| Posted: Sun Jan 01, 2017 11:29 am Post subject: |
|
|
Apprentice
Joined: 22 Aug 2016
Posts: 29
|
| No, The image is in binary, but the image attached in this post is a screenshot of the hex comparison between the original image and the received one by email, dont mix things:) |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jan 01, 2017 12:09 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| eammnsh wrote: |
| No, The image is in binary, but the image attached in this post is a screenshot of the hex comparison between the original image and the received one by email, dont mix things:) |
Never said that the image wasn't in binary. However the MIME part you defined is defined as
| Code: |
CREATE FIELD P1."Content-Type" TYPE NameValue VALUE 'text/plain; charset=utf-8';
CREATE FIELD P1."Content-Transfer-Encoding" TYPE NameValue VALUE '8bit'; |
whereas it should probably be defined as 'application/64bit encoded'...
This is probably the reason why the CRLF(0d0a) that base64 encoding inserts when representing the binary values. Defining the content correctly as 64 bit encoded would then strip the inserted CRLF before rendering.
Have fun
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| eammnsh |
| Posted: Tue Jan 03, 2017 3:29 am Post subject: |
|
|
Apprentice
Joined: 22 Aug 2016
Posts: 29
|
Part 1 of MIME tree contains the Email body, which you are referring to in your reply, not the attachment. Part 2 does contain the attachment with the proper information.
But let me ask my question in different way based on your reply, does the EmailOutput node only support Base64 content type for Binary attachments?
Below is my MIME attachment values for Part 2 (attachment)
Content-Type: application/octet-stream; name=IMAGE.PNG
Content-Transfer-Encoding: binary
Content-ID: <IMAGE.PNG>
Content-Disposition: attachment; name="IMAGE.PNG"; filename="IMAGE.PNG"
Thanks. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Jan 03, 2017 6:28 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| eammnsh wrote: |
Part 1 of MIME tree contains the Email body, which you are referring to in your reply, not the attachment. Part 2 does contain the attachment with the proper information.
But let me ask my question in different way based on your reply, does the EmailOutput node only support Base64 content type for Binary attachments?
Below is my MIME attachment values for Part 2 (attachment)
Content-Type: application/octet-stream; name=IMAGE.PNG
Content-Transfer-Encoding: binary
Content-ID: <IMAGE.PNG>
Content-Disposition: attachment; name="IMAGE.PNG"; filename="IMAGE.PNG"
Thanks. |
Well this definition is fine for strict binary... I'd go with Base64 encoded though. With Mime this is a nice way to display binary as characters and a little bit more compact than strict hex...
What happens when you try it?
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| eammnsh |
| Posted: Tue Jan 03, 2017 7:44 am Post subject: |
|
|
Apprentice
Joined: 22 Aug 2016
Posts: 29
|
Ok, this is the hex representation, I am expecting :
| Code: |
| 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 |
and this is what I am getting
| Code: |
| 89 50 4E 47 0D 0A 1A 0D 0A 00 00 00 0D 0A 49 |
As you can notice, There is an Extra X'0D' and X'0A' in what I get.
I tried to send it as Base64
Content-Type: text/plain; charset=Cp1252; name=IMAGE.PNG
Content-Transfer-Encoding: base64
Content-ID: <IMAGE.PNG>
but still I am getting the same response with the corrupted data above.
_________________
WMB 7 and 8 IIB 9 and 10
MQ 7, 7.5, and 8
Since 2012 |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 03, 2017 7:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It looks like you are only getting an extra X'0A' as the X'0D' is in both expected and received.
This is unix-windows line endings.
Are you sure the ' charset=Cp1252' should be Cp1252 and not UTF-8 or etc?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 03, 2017 7:51 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| eammnsh wrote: |
| As you can notice, There is an Extra X'0D' and X'0A' in what I get. |
You're using the wrong CCSID. You're not getting an extra X'0D', you're getting an extra X'0A'. Unix platforms (and their CCSIDs) represent end-of-line with X'0D', Windoze represents end-of-line with a X'0D' X'0A' pair.
Functioning as coded. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| timber |
| Posted: Tue Jan 03, 2017 8:06 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
I'm not convinced yet. 0x0D and 0x0A are separate characters. Windows just happens to use a pair of characters to represent an end-of-line. Using the wrong CCSID can cause individual characters to be parsed/written incorrectly but it does not magically cause extra characters to appear.
@eammnsh: This looks like some kind of EOL character normalization. Is there anything in the processing pathway that could be doing that? |
|
| Back to top |
|
|
| eammnsh |
| Posted: Tue Jan 03, 2017 8:08 am Post subject: |
|
|
Apprentice
Joined: 22 Aug 2016
Posts: 29
|
Nop, this is not true  here is is another sample of the data:
original data:
| Code: |
| F5 F1 0A 5F CC 05 7F 1B C4 8D 08 CC D6 17 |
What I am receiving
| Code: |
| F5 F1 0D 0A 5F CC 05 7F 1B C4 8D 08 CC D6 17 |
In this sample you will find extra '0D' not '0A' )
_________________
WMB 7 and 8 IIB 9 and 10
MQ 7, 7.5, and 8
Since 2012 |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 03, 2017 8:14 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
... This ESQL is confusing
| Code: |
| SET P2.Data.BLOB.BLOB = InputRoot.XMLNSC.MSG.Attachment.*[<].BLOB.BLOB |
Does the XML document really have a field named "MSG.Attachment.<something>.BLOB.BLOB" ?
Or are you building that BLOB.BLOB from somewhere ?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| eammnsh |
| Posted: Tue Jan 03, 2017 8:21 am Post subject: |
|
|
Apprentice
Joined: 22 Aug 2016
Posts: 29
|
@mqjeff: I am receiving the attachment in MIME part with SOAP request, so I am just taking the incoming BLOB and putting it inside my XMLNSC tree:
| Code: |
CREATE LASTCHILD OF OutBodyRef.MSG.Attachment.*[<] DOMAIN 'BLOB';
SET OutBodyRef.MSG.Attachment.*[<].BLOB.BLOB = InBodyRef.Attachment.*[<].BLOB.BLOB;
|
@timber: even when I leave my content encoding to Binary, and I dont touch the incoming BLOB within the SOAP request, I get the same results.
Do you think SOAP-UI client can be doing this to the attachment?
For your information my server is linux redhat x86-64 and I am using SOAP-UI from a windows machine.
_________________
WMB 7 and 8 IIB 9 and 10
MQ 7, 7.5, and 8
Since 2012 |
|
| Back to top |
|
|
| adubya |
| Posted: Tue Jan 03, 2017 8:54 am Post subject: |
|
|
Partisan
Joined: 25 Aug 2011
Posts: 377
Location: GU12, UK
|
Have you verified that the extra characters are being added in Broker and not by SOAP-UI ?
_________________
Independent Middleware Consultant
andy@knownentity.com |
|
| Back to top |
|
|
| timber |
| Posted: Tue Jan 03, 2017 2:47 pm Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
| That was going to be my next question. At this point, I would be putting Trace nodes into the flow and inspecting the byte arrays that are being received/produced. |
|
| Back to top |
|
|
|
|
