ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Encoding when processing a .jpg file

Post new topic  Reply to topic
 Encoding when processing a .jpg file « View previous topic :: View next topic » 
Author Message
petervh1
PostPosted: Thu Jul 21, 2022 1:33 am    Post subject: Encoding when processing a .jpg file Reply with quote

Centurion

Joined: 19 Apr 2010
Posts: 122

Hi

ACE 12.0.2 on Linux

I'm trying the following:

Get a .jpg file from an external website and write it to an external location. as follows:

Compute node to set up the request -> HTTPRequest -> Compute node to read the .jpg and set up a BLOB so that I can write this file to another remote location.

If I set the Response Message Parsing tab on HTTPRequest to BLOB, the data that leaves the HTTPRequest Out terminal is not the correct bitstream. I know this because if I save it to the intended destination, I can't open it using MS Paint. It also appears different when viewed in an editor.

I then tried to create a "basic" DFDL structure to map the bitstream returned from the HTTPRequest. I believe this DFDL .xsd is correct because if I run Test Parse Model on a copy of the file that I know is a valid .jpg it parses successfully. Note: to get this to work, I changed the encoding in the .xsd to"ISO-8859-1".

However, if I set the Response Message Parsing tab on HTTPRequest to DFDL and set the Message to the message name in the .xsd the flow fails with the following:


Code:
BIP5807 - An error occurred whilst parsing with DFDL.
CTDP3011E: Character decoding error. The sequence of bytes starting at byte offset '0' is not a valid character in the encoding '1208'


Why is the message indicating an encoding of 1208? Do I need to set a different value?

TIA
Back to top
View user's profile Send private message
timber
PostPosted: Thu Jul 21, 2022 2:29 am    Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1280

There are at least 4 steps involved here:
1. Receive the file contents from the external website
2. Decode the response to create a BLOB (an array of raw bytes) in the message tree.
3. Encode the BLOB correctly for the receiving application. You have not specified what the output format is (hexBinary, base64, raw bytes etc) so it's hard to advise.
4. Send the resulting stream of bytes/characters to the target application.

Quote:
Response Message Parsing tab on HTTPRequest to BLOB, the data that leaves the HTTPRequest Out terminal is not the correct bitstream.
This is step 2 above. You have not stated what the format of the incoming data is, so it's hard to give advice. Are you receiving a raw stream of bytes, or a base64-encoded stream of characters, or something else?

Similarly, you have not stated what format the target application wants to receive, so we will struggle to advise re: step 3.
Back to top
View user's profile Send private message
petervh1
PostPosted: Thu Jul 21, 2022 6:44 am    Post subject: Reply with quote

Centurion

Joined: 19 Apr 2010
Posts: 122

Thanks for the assistance.

I have managed to complete steps 1 and 2. I was initially using the wrong encoding value. If I use CCSID 819 then I can see the correct data in the debug message tree.

The problem is now in step 3 - I am trying to encode the BLOB for writing to an AWS S3 bucket (the target application).

I used CCSID 1208 for CASTing the payload to a BLOB in preparation for writing to AWS S3 - this writes to the output but produces incorrect data when compared to my "good" sample.

If I use CCSID 819 the PUT process to AWS S3 fails.

I suspect this is now an issue of sorting out the AWS S3 PUT process.
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Jul 21, 2022 8:17 am    Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 9392
Location: US: west coast, almost. Otherwise, enroute.

A BLOB is a BLOB - Binary, not character, not string. Jpgs, gifs, audio, video, and other BLOB types are not associated with a CCSID, and should not be converted.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Sun Jul 24, 2022 12:01 pm    Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20695
Location: LI,NY

For AWS3 to take it, verify if you need to 64 bit encode it...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
gbaddeley
PostPosted: Sun Jul 24, 2022 3:09 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2491
Location: Melbourne, Australia

petervh1 wrote:
Thanks for the assistance.
I have managed to complete steps 1 and 2. I was initially using the wrong encoding value. If I use CCSID 819 then I can see the correct data in the debug message tree.
The problem is now in step 3 - I am trying to encode the BLOB for writing to an AWS S3 bucket (the target application).
I used CCSID 1208 for CASTing the payload to a BLOB in preparation for writing to AWS S3 - this writes to the output but produces incorrect data when compared to my "good" sample.
If I use CCSID 819 the PUT process to AWS S3 fails.
I suspect this is now an issue of sorting out the AWS S3 PUT process.

You may be completely on the wrong track. JPEG files are binary data, they are not encoded in any particular character set. Changing CCSID should not influence the JPEG binary blob data at all. You need a byte-perfect copy to be passed through, with no conversion.
_________________
Glenn
Back to top
View user's profile Send private message
petervh1
PostPosted: Tue Jul 26, 2022 3:37 am    Post subject: Reply with quote

Centurion

Joined: 19 Apr 2010
Posts: 122

I managed to put the image to the S3 bucket successfully, but only after first base64-encoding the BLOB that I get from the external website.

I can then download this file, base64-decode it, and then display the jpg image correctly.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Jul 26, 2022 3:20 pm    Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2491
Location: Melbourne, Australia

petervh1 wrote:
I managed to put the image to the S3 bucket successfully, but only after first base64-encoding the BLOB that I get from the external website.
I can then download this file, base64-decode it, and then display the jpg image correctly.

That seems like a band-aid solution, and the file will be significantly larger. It may be worth the investment in doing it properly. At some point in the future, your approach will probably be questioned, and the increased overhead of encode/decode and larger files may cause issues. Have a nice day.
_________________
Glenn
Back to top
View user's profile Send private message
petervh1
PostPosted: Wed Jul 27, 2022 2:39 am    Post subject: Reply with quote

Centurion

Joined: 19 Apr 2010
Posts: 122

Quote:

That seems like a band-aid solution, and the file will be significantly larger. It may be worth the investment in doing it properly. At some point in the future, your approach will probably be questioned, and the increased overhead of encode/decode and larger files may cause issues.


Fair comment. I've added part of the AWS SDK to my Java project in ACE (just the relevant classes, because the SDK is pretty large) and redone the code in Java. Works perfectly now.
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic  Reply to topic Page 1 of 1

MQSeries.net Forum Index » WebSphere Message Broker (ACE) Support » Encoding when processing a .jpg file
Jump to:  



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP
 
 


Theme by Dustin Baccetti
Powered by phpBB © 2001, 2002 phpBB Group

Copyright © MQSeries.net. All rights reserved.