| Author |
Message
|
| jeevan |
 Posted: Tue Sep 22, 2009 7:57 pm Post subject: How to achieve message compression in MQ? |
 |
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
I have been asked to figure out how can we achieve message compression in MQ. In fact, this is not an administrator's job however, this is how it works in my company. I found the follwoing two discussions.
http://www.mqseries.net/phpBB2/viewtopic.php?p=179055&sid=19d502753b67e8c2262a0ffaf2de6c28.
http://www.mqseries.net/phpBB2/viewtopic.php?p=222903&sid=191eb0147d4dbb18f06dc67eba906c38
I also reviewed the inforcentre on channel compression.
http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp?topic=/com.ibm.mq.csqzaw.doc/uj11200_.htm
However, I still have some basic questions.
Is MQ compression the same thing as channel compression? But according to the inforcentre, the channel compression also has some code changes . Also, it talks about channel exits.
I am really confused. Can someone help me to understand exactly what channel compression requires? Where does the channel eixts come into picture?
Our scenario is like this:
The message is put connecgting to a gateway queue manager. In fact the message travel to another queue managers ( which hold cluster queues) via cluster channels.
Our requirements( we would liek to do) is to compress the message where it originates not when the message arrives at mq (channel). We are concerned about the reuesting application. There is not problem for processing application.
In the current setup, both, requesting and processing app use server connection channels.
Our Env is as follows:
MQ version: mq 6.0.2.5
platform : Windows and Unix
Client : base JAVA API
Thanks
Last edited by jeevan on Thu Oct 01, 2009 9:31 pm; edited 3 times in total |
|
| Back to top |
|
 |
| jeevan |
| Posted: Wed Sep 23, 2009 7:55 am Post subject: Re: How to achieve message compression in MQ? |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| jeevan wrote: |
I have been asked to figure out how can we achieve message compression in MQ. I found the follwoing discussion
http://www.mqseries.net/phpBB2/viewtopic.php?p=179055&sid=19d502753b67e8c2262a0ffaf2de6c28.
http://www.mqseries.net/phpBB2/viewtopic.php?p=222903&sid=191eb0147d4dbb18f06dc67eba906c38
Also, I reviewed the inforcentre on compression.
http://publib.boulder.ibm.com/infocenter/wmqv6/v6r0/index.jsp?topic=/com.ibm.mq.csqzaw.doc/uj11200_.htm
However, I still have some basic questions.
Are there two different kinds of compressions technique in MQ as said in the first post? client and MQ compression?
Is MQ compression is same as channel compression? But according to the inforcentre, the channel compression also has some code compression. Also, it talks about channel exits.
I am really confused. Can someone help me to understand exactly what the compression requires? Where does the channel eixts come into picture?
Also, when do these channel attributes come into picture?
COMPHDR
COMPMSG
The message is put connecgting to a gateway queue manager. In fact the message travel to another queue managers ( which hold cluster queues) via cluster channel.
Our need is to compress the message where it originates not when the message arrives at mq (channel).
Both, requesting and processing app use server connection channels.
Our Env is as follows:
MQ version: mq 6.0.2.5
platform : Windows and Unix
Client : base JAVA API
thanks |
After some googling, and going through this forum I come up wtih some further understanding.
When we say client compression, there is not any role of MQ whatsoever. At the beginning, I was thinking that MQ is involve anyway, whether it is channel or client compression. But now I came to know that ( i may still be wrong) that a client compression means, MQ does not know anything or does not do anything. This is the language feature and capability.
When we say MQ or Channel compression, it involves two ( may be 3 ) steps:
1. some code changes eg
| Quote: |
Collection msgComp = new Vector();
msgComp.add(new Integer(MQC.MQCOMPRESS_RLE));
msgComp.add(new Integer(MQC.MQCOMPRESS_ZLIBHIGH));
MQEnvironment.msgCompList = msgComp;
MQQueueManager qMgr = new MQQueueManager(QM);
|
2. Setting the two channel attributes
COMPHDR
COMPMSG
But I am still not sure:
1. How and where does the channel exits come into picture?
2. Does it require to set these parameters of the channel used by
processing application ?
3. Do we need changes in the code of the processing application like in 1
I would really appreciate if someone can give some idea.
Last edited by jeevan on Wed Sep 23, 2009 8:59 am; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 23, 2009 8:19 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
Compression happens on the data passing over a MQ channel of any kind.
All MQ channels have two sides.
You should know this by now.
You should know how to configure the CLNTCONN side of a client channel.
You should know that regardless of what the MQ channels do, the application is fully able to whatever it wants to do to the message buffer, including compress it. |
|
| Back to top |
|
|
| jeevan |
| Posted: Wed Sep 23, 2009 8:50 am Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| mqjeff wrote: |
Compression happens on the data passing over a MQ channel of any kind.
All MQ channels have two sides.
You should know this by now.
|
I know that and that is not my question.
| Quote: |
You should know how to configure the CLNTCONN side of a client channel.
|
I know that and that was not my question.
| Quote: |
You should know that regardless of what the MQ channels do, the application is fully able to whatever it wants to do to the message buffer, including compress it.
|
I learned that too and that was not again my questions.
So, in my second post, I said, this clearly. I understood that client app ( regardless of MQ ) can do their own compression. And my question is not how to do it.
Only my question is if we choose to implement the compression facility provided by MQ, what does it involve? Whatever I found and read did not have much about process. Again, I am not looking for a cook book. I just wanted some high level steps: create client channel, set the attribute of the channle etc.
Is not this fair to ask such a question in a MQ forum?
Last edited by jeevan on Thu Oct 01, 2009 9:34 pm; edited 1 time in total |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Sep 23, 2009 8:58 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| jeevan wrote: |
| Quote: |
You should know how to configure the CLNTCONN side of a client channel.
|
I know that and that was not my question. |
Yes, it was. |
|
| Back to top |
|
|
| jeevan |
| Posted: Wed Sep 23, 2009 11:37 am Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| mqjeff wrote: |
| jeevan wrote: |
| Quote: |
You should know how to configure the CLNTCONN side of a client channel.
|
I know that and that was not my question. |
Yes, it was. |
Jeff,
I am repeating the questions ( with some modification for clarity) I asked in my second post :
1. How and where does the channel exits come into picture?
2. Does it require to set the channel parameters for the channel used by
processing application?
3. Do we need changes in the code of the processing application like we
do in sender application? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Sep 23, 2009 6:17 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| jeevan wrote: |
Jeff,
I am repeating the questions ( with some modification for clarity) I asked in my second post :
1. How and where does the channel exits come into picture?
2. Does it require to set the channel parameters for the channel used by
processing application?
3. Do we need changes in the code of the processing application like we
do in sender application? |
Ok so
- Channel exits are 'inserted' before compression on the outgoing and after decompression on the incoming channels
- Compression is transparent to the application. Once the channel compression settings match, compression/decompression is automatic.
- Why would you need to do that. Compression looks like a black box to the application.
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jeevan |
| Posted: Thu Sep 24, 2009 2:28 pm Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| fjb_saper wrote: |
| jeevan wrote: |
Jeff,
I am repeating the questions ( with some modification for clarity) I asked in my second post :
1. How and where does the channel exits come into picture?
2. Does it require to set the channel parameters for the channel used by
processing application?
3. Do we need changes in the code of the processing application like we
do in sender application? |
Ok so
- Channel exits are 'inserted' before compression on the outgoing and after decompression on the incoming channels
- Compression is transparent to the application. Once the channel compression settings match, compression/decompression is automatic.
- Why would you need to do that. Compression looks like a black box to the application.
|
We have just started the discussions about it and even have not have first meeting. But as far as I know, we are changing format of one fo the messages from binary to xml and with that the messages size is going to be very big. That is why they wanted to compression the origne of a message. That is what the app folks told me. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Sep 24, 2009 2:58 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
As pointed out, compression is a black box. There are a variety of data compression algorithms.
The decision to compress or not is not that simple. The effectiveness is, in great part, dependent on the type of data you intend to compress. Data compression is labor-intensive.
I'd suggest using Mr. Google to search for data compression algorithms+xml; then read all about it. Then develop some benchmark data to run against whichever vendor products you test.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Sep 25, 2009 12:18 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| jeevan wrote: |
| ...we are changing format of one fo the messages from binary to xml and with that the messages size is going to be very big. That is why they wanted to compression the origne of a message... |
So is the issue the size of the message on the queue, or the amount of bandwidth to transport them? If it's the first, have the application do the compression before passing the buffer, or if the latter, compress at channel level.
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| jeevan |
| Posted: Fri Sep 25, 2009 4:38 am Post subject: |
|
|
Grand Master
Joined: 12 Nov 2005
Posts: 1432
|
| exerk wrote: |
| jeevan wrote: |
| ...we are changing format of one fo the messages from binary to xml and with that the messages size is going to be very big. That is why they wanted to compression the origne of a message... |
So is the issue the size of the message on the queue, or the amount of bandwidth to transport them? If it's the first, have the application do the compression before passing the buffer, or if the latter, compress at channel level. |
I think, it is both. But the second is the major concern.
I am still researching how it works. If we use channel compression, do we need to create ccdt ? At the moment, the clients connect through server connection channel. Also, when the message are at the rest, will they still be compressed or not?
Thanks |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Sep 25, 2009 5:07 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| jeevan wrote: |
| ...If we use channel compression, do we need to create ccdt? |
I take it from that question you are using MQCONNX for the application's connection method?
| jeevan wrote: |
| ...when the message are at the rest, will they still be compressed or not? |
Think about where the compression is set and you'll realise you already know the answer to that question 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Sep 25, 2009 5:53 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| Quote: |
| So is the issue the size of the message on the queue, or the amount of bandwidth to transport them? |
Yes, message size and bandwidth are two considerations.
Data content is the third consideration. Some data compresses well (like redundant, adjacent characters), some does not (random adjacent characters). Research how xml compresses, which algorithm compresses xml well, then do testing to see if your data compresses well.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
