| Author |
Message
|
| petervh1 |
 Posted: Thu Jul 21, 2022 1:33 am Post subject: Encoding when processing a .jpg file |
 |
|
Centurion
Joined: 19 Apr 2010
Posts: 135
|
Hi
ACE 12.0.2 on Linux
I'm trying the following:
Get a .jpg file from an external website and write it to an external location. as follows:
Compute node to set up the request -> HTTPRequest -> Compute node to read the .jpg and set up a BLOB so that I can write this file to another remote location.
If I set the Response Message Parsing tab on HTTPRequest to BLOB, the data that leaves the HTTPRequest Out terminal is not the correct bitstream. I know this because if I save it to the intended destination, I can't open it using MS Paint. It also appears different when viewed in an editor.
I then tried to create a "basic" DFDL structure to map the bitstream returned from the HTTPRequest. I believe this DFDL .xsd is correct because if I run Test Parse Model on a copy of the file that I know is a valid .jpg it parses successfully. Note: to get this to work, I changed the encoding in the .xsd to"ISO-8859-1".
However, if I set the Response Message Parsing tab on HTTPRequest to DFDL and set the Message to the message name in the .xsd the flow fails with the following:
| Code: |
BIP5807 - An error occurred whilst parsing with DFDL.
CTDP3011E: Character decoding error. The sequence of bytes starting at byte offset '0' is not a valid character in the encoding '1208'
|
Why is the message indicating an encoding of 1208? Do I need to set a different value?
TIA |
|
| Back to top |
|
 |
| timber |
| Posted: Thu Jul 21, 2022 2:29 am Post subject: |
|
|
Grand Master
Joined: 25 Aug 2015
Posts: 1292
|
There are at least 4 steps involved here:
1. Receive the file contents from the external website
2. Decode the response to create a BLOB (an array of raw bytes) in the message tree.
3. Encode the BLOB correctly for the receiving application. You have not specified what the output format is (hexBinary, base64, raw bytes etc) so it's hard to advise.
4. Send the resulting stream of bytes/characters to the target application.
| Quote: |
| Response Message Parsing tab on HTTPRequest to BLOB, the data that leaves the HTTPRequest Out terminal is not the correct bitstream. |
This is step 2 above. You have not stated what the format of the incoming data is, so it's hard to give advice. Are you receiving a raw stream of bytes, or a base64-encoded stream of characters, or something else?
Similarly, you have not stated what format the target application wants to receive, so we will struggle to advise re: step 3. |
|
| Back to top |
|
|
| petervh1 |
| Posted: Thu Jul 21, 2022 6:44 am Post subject: |
|
|
Centurion
Joined: 19 Apr 2010
Posts: 135
|
Thanks for the assistance.
I have managed to complete steps 1 and 2. I was initially using the wrong encoding value. If I use CCSID 819 then I can see the correct data in the debug message tree.
The problem is now in step 3 - I am trying to encode the BLOB for writing to an AWS S3 bucket (the target application).
I used CCSID 1208 for CASTing the payload to a BLOB in preparation for writing to AWS S3 - this writes to the output but produces incorrect data when compared to my "good" sample.
If I use CCSID 819 the PUT process to AWS S3 fails.
I suspect this is now an issue of sorting out the AWS S3 PUT process. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jul 21, 2022 8:17 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
A BLOB is a BLOB - Binary, not character, not string. Jpgs, gifs, audio, video, and other BLOB types are not associated with a CCSID, and should not be converted.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Sun Jul 24, 2022 12:01 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
For AWS3 to take it, verify if you need to 64 bit encode it...
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Sun Jul 24, 2022 3:09 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| petervh1 wrote: |
Thanks for the assistance.
I have managed to complete steps 1 and 2. I was initially using the wrong encoding value. If I use CCSID 819 then I can see the correct data in the debug message tree.
The problem is now in step 3 - I am trying to encode the BLOB for writing to an AWS S3 bucket (the target application).
I used CCSID 1208 for CASTing the payload to a BLOB in preparation for writing to AWS S3 - this writes to the output but produces incorrect data when compared to my "good" sample.
If I use CCSID 819 the PUT process to AWS S3 fails.
I suspect this is now an issue of sorting out the AWS S3 PUT process. |
You may be completely on the wrong track. JPEG files are binary data, they are not encoded in any particular character set. Changing CCSID should not influence the JPEG binary blob data at all. You need a byte-perfect copy to be passed through, with no conversion.
_________________
Glenn |
|
| Back to top |
|
|
| petervh1 |
| Posted: Tue Jul 26, 2022 3:37 am Post subject: |
|
|
Centurion
Joined: 19 Apr 2010
Posts: 135
|
I managed to put the image to the S3 bucket successfully, but only after first base64-encoding the BLOB that I get from the external website.
I can then download this file, base64-decode it, and then display the jpg image correctly. |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Jul 26, 2022 3:20 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| petervh1 wrote: |
I managed to put the image to the S3 bucket successfully, but only after first base64-encoding the BLOB that I get from the external website.
I can then download this file, base64-decode it, and then display the jpg image correctly. |
That seems like a band-aid solution, and the file will be significantly larger. It may be worth the investment in doing it properly. At some point in the future, your approach will probably be questioned, and the increased overhead of encode/decode and larger files may cause issues. Have a nice day. 
_________________
Glenn |
|
| Back to top |
|
|
| petervh1 |
| Posted: Wed Jul 27, 2022 2:39 am Post subject: |
|
|
Centurion
Joined: 19 Apr 2010
Posts: 135
|
| Quote: |
That seems like a band-aid solution, and the file will be significantly larger. It may be worth the investment in doing it properly. At some point in the future, your approach will probably be questioned, and the increased overhead of encode/decode and larger files may cause issues. |
Fair comment. I've added part of the AWS SDK to my Java project in ACE (just the relevant classes, because the SDK is pretty large) and redone the code in Java. Works perfectly now. |
|
| Back to top |
|
|
|
|
