ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexWebSphere Message Broker Supportcompressing data before sending to Trace node

Post new topicReply to topic
compressing data before sending to Trace node View previous topic :: View next topic
Author Message
syedimtiyaz1234
PostPosted: Fri Nov 06, 2020 9:51 am Post subject: compressing data before sending to Trace node Reply with quote

Apprentice

Joined: 22 Jan 2018
Posts: 29

Hey,

Currently we are logging base64 encoded string in Trace file. but data size is the concern.

And I am trying to find a solution to reduce the size it logs and developer can easily convert the data back to original.

If any one has suggestion , please let me know.

Thanks in advance.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Sun Nov 08, 2020 11:35 am Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

Can you not base64 encode the string? It will then 75% of the base64 encoded size.
_________________
Glenn
Back to top
View user's profile Send private message
syedimtiyaz1234
PostPosted: Mon Nov 09, 2020 2:13 pm Post subject: Reply with quote

Apprentice

Joined: 22 Jan 2018
Posts: 29

if we log plain simply as it is , then data is not formatted , the data could be json/xml or plain string so it does not looks good.

when we do encode it the logging format is good i.e. we can follow generic logging format
Back to top
View user's profile Send private message
timber
PostPosted: Mon Nov 09, 2020 4:02 pm Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1148

You need to decide whether you want readability or compression. You cannot have both. If this log format is for developers, you should be able to think up other solutions (e.g. log4j) that allow more flexibility. The base64 encoding is extremely easy to decode using any scripting language or Java.
Back to top
View user's profile Send private message
syedimtiyaz1234
PostPosted: Tue Nov 10, 2020 10:36 am Post subject: Reply with quote

Apprentice

Joined: 22 Jan 2018
Posts: 29

I am looking for both readability and compression as well as.

We are using IIB Monitoring events to receive the data that we need to log so even we use log4j , at the end storing data will be same right or am I missing anything.

so we don't have any way to reduce the data size when we log to file.

lets consider we want to log plain xml data to file , is there any way we can compress this before logging to file.
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Nov 10, 2020 3:13 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

Note that base64 does not compress data. It actually makes it longer....

To reduce the size, you could try squeezing out all the white space.

A logging frame work should able to cope with large messages and have room to scale. Disk space is cheap. I am not 100% sure of your concern.
_________________
Glenn
Back to top
View user's profile Send private message
syedimtiyaz1234
PostPosted: Fri Nov 13, 2020 12:50 pm Post subject: Reply with quote

Apprentice

Joined: 22 Jan 2018
Posts: 29

Yes base64 does not compress the data.

if we log the original data then my concern is readability and that's why I thought of doing base64 encoding so that at-least it solves my readability concern and so i am kind of looking if there is other way where i can maintain my readability and along with that it takes less data size than base64 encoded string.

For me Disk space is an issue and I agree we should have room for scale but I am trying to see if I can do something about this from reducing size perspective.
Back to top
View user's profile Send private message
fjb_saper
PostPosted: Fri Nov 13, 2020 3:24 pm Post subject: Reply with quote

Grand High Poobah

Joined: 18 Nov 2003
Posts: 20423
Location: LI,NY

Have you thought about encrypting the data with a key? In the long run you might have less data to ship across when encrypted than when using base64...

You could also look at compression on the fly? Have the agent compress the data before sending it to your backend...
_________________
MQ & Broker admin
Back to top
View user's profile Send private message Send e-mail
gbaddeley
PostPosted: Sun Nov 15, 2020 1:44 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

syedimtiyaz1234 wrote:
Yes base64 does not compress the data.

if we log the original data then my concern is readability and that's why I thought of doing base64 encoding so that at-least it solves my readability concern and so i am kind of looking if there is other way where i can maintain my readability and along with that it takes less data size than base64 encoded string.

For me Disk space is an issue and I agree we should have room for scale but I am trying to see if I can do something about this from reducing size perspective.

Have you looked at RLE, LZ or ZIP compression algorithms?
I'm not sure what you mean by readability. Does this mean storing a shortened form of the data, but being able to easily read the original data?
_________________
Glenn
Back to top
View user's profile Send private message
timber
PostPosted: Mon Nov 16, 2020 8:27 am Post subject: Reply with quote

Grand Master

Joined: 25 Aug 2015
Posts: 1148

Just to be clear
- You cannot safely put raw bytes into a text file, so logging the 'raw data' or a compressed version of it is not advisable.
- The simple (naive) solution is to encode each byte value using two characters (so 0x61 becomes "3631" which occupies two bytes (in a single-byte character set or UTF-.
- A better solution is to encode the bytes using base64 encoding. This uses 4 characters for every 3 bytes, so the text data is 33% larger than the raw data. Yes, it is a little larger than the 'raw' data - but it is also safe for all text formats.

You can apply base64 encoding to any sequence of bytes, so you can compress using any of the popular compression algorithms and then base64-encode the resulting byte stream. It will be harder to read, but if your main objective is to reduce disk space...
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Mon Nov 16, 2020 2:27 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2195
Location: Melbourne, Australia

syedimtiyaz1234 wrote:
if we log plain simply as it is , then data is not formatted , the data could be json/xml or plain string so it does not looks good.
when we do encode it the logging format is good i.e. we can follow generic logging format

Can you clarify what you mean by 'does not look good' ?

Are you looking for a generic logging format that converts JSON, XML or other string data into a common format? eg. Convert everything to JSON, and using only printable characters.

Our IIB flows process many different formats of data, and logs them as-is. This leaves no doubt about the content or format of the original data. There is a learning curve for being able to visually read XML, JSON, CSV, fixed columns, etc. but its not very steep.
_________________
Glenn
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexWebSphere Message Broker Supportcompressing data before sending to Trace node
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.