| Author |
Message
|
| RogerLacroix |
 Posted: Tue Mar 29, 2016 8:57 am Post subject: Message Properties |
 |
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
All,
I have a structure (about 25 fields) that is meta-data for a message. Since the meta-data structure is fixed length, I could put it in front of the message data as I have done over the years for other applications but I thought this would be a good use of message properties.
Questions (assume the messages per day will be over 1,000,000):
- Has anyone done any performance testing using message properties?
- Is creating 25 message properties per message too much? (when the application is doing million plus messages per day)
- Most of the fields of the meta-data are strings, so I could convert the non-strings to strings then put 1 message property which contains the entire mega-data structure. i.e. It would be about 400 bytes long. This means that the receiving application would need to parse the single message property to get the 25 fields.
I'm just curious how much of a performance hit it would be to use a large number (25) message properties per message when the sending application is putting a million plus messages per day.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
 |
| PaulClarke |
| Posted: Tue Mar 29, 2016 1:36 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Asking things like 'too much' is a little hard to answer. Wouldn't it be best to just try it and measure it and then you'll know just what the cost delta is?
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Mar 29, 2016 2:34 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
Hi Roger,
Most of the time penalty for a large number of properties would be CPU spent by MQ in marshalling all the property names and values into and out of the message.
If the fields are fixed length, it would be slightly more efficient to store them in one message property, and the app program directly accesses each fixed length field.
I would also be tempted to not use MQ message properties in this case, and just store the data structure in front of the message data. After all, fixed width string fields are the norm for MQ message formats, as it is easy coding and efficient for apps to process, and works well in cross-platform situations. Of course it lacks the versatility of XML, JSON, SOAP etc.
HTH, G.
_________________
Glenn |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Tue Mar 29, 2016 3:00 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Of course another consideration is forward compatibility. You say that there is a fixed format of 25 fields but is this true for all time ? A fixed format header would undoubtedly be the fastest but is also the least flexible and extensible. However, sometimes that is all you need....look at all the MQ headers  One of the downsides with headers though is data conversion - distributing a data conversion exit for all your data formats can be a bit of a pain.
Message Properties are a good way of allowing you to add extra properties if and when they are dreamt up. the other advantage of properties is that is lets you use selectors. Sometime it is useful to be able the get/browse only a subset of the messages efficiently.
Of course another way to have multiple elements to a message is to put everything in a PCF message. To be honest PCF messages are what I tend to do if I need to send a message containing lots of different pieces. They have lots of advantages. It is very easy to add/remove the individual pieces. PCF messages allow me to send lots of different formats of data and it is pretty quick to manipulate. Data conversion is handled automatically by MQ. There are also a number of tools which allow me to format out the contents fairly readily.
It all depends on the trade of betweenf Performance vs Usability vs Extensibility
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Wed Mar 30, 2016 3:46 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Both of you point out all of the reasons I asked the question. I'm leaning towards using 25 message properties but I thought I would ask before going too far down the path and then having a Homer moment - D'oh!
I know the MQSETMP call is a local call in the application's address space, so it should be very fast. It is just doing it 25 times that makes me wonder. Although, message properties allows for the greatest flexibility and I like that.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Mar 30, 2016 3:59 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
| RogerLacroix wrote: |
Hi,
Both of you point out all of the reasons I asked the question. I'm leaning towards using 25 message properties but I thought I would ask before going too far down the path and then having a Homer moment - D'oh!
I know the MQSETMP call is a local call in the application's address space, so it should be very fast. It is just doing it 25 times that makes me wonder. Although, message properties allows for the greatest flexibility and I like that.
Regards,
Roger Lacroix
Capitalware Inc. |
Will this be a client-bindings app? If so, there will be MQI network flows.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Wed Mar 30, 2016 4:46 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
No there won't. Message property verbs are all local to the application address space. The properties are only transported to the server when the usual MQPUT or MQGET call is made.
As an indication of this you can see that you can create a message handle, MQCRTMH, without actually giving it a valid connection handle at all but instead the value contact admin. M Q H C _ U N A S S O C I A T E D _ H C O N N
Cheers,
Paul
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Mar 30, 2016 8:25 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Ah. Got it. 
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Mar 31, 2016 4:51 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
... I haven't checked, and I'm sure Roger has. But.
Is there anything relevant in the performance reports?
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Thu Mar 31, 2016 3:50 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
| mqjeff wrote: |
| Is there anything relevant in the performance reports? |
I didn't see anything related to using a large number (25) of message properties.
Different subject: There are 10 types for a message property:
- MQTYPE_BOOLEAN
- MQTYPE_BYTE_STRING
- MQTYPE_INT8
- MQTYPE_INT16
- MQTYPE_INT32
- MQTYPE_INT64
- MQTYPE_FLOAT32
- MQTYPE_FLOAT64
- MQTYPE_STRING
- MQTYPE_NULL
For MQTYPE_STRING, MQ Knowledge Center says 'A character string. ValueLength must be zero or greater, or the special value MQVL_NULL_TERMINATED'.
General question: does anyone know what the type MQTYPE_BYTE_STRING is? All the MQ Knowledge Center says is 'A byte string. ValueLength must be zero or greater'. Not exactly useful.
Does it mean a hexadecimal string like: 5468697320697320612068657820737472696E67
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Thu Mar 31, 2016 5:02 pm Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
Roger,
A MQTYPE_BYTE_STRING is just that, a string of bytes. You say that the manual is ‘not exactly useful’ but I’m not sure what else there would be to say. You can put anything you like in this string of bytes and MQ will not do anything with them except transport them.
So, no it is not data like ‘54686973....’, those are just hexadecimal characters which describe bytes. Rather the actual bytes of 0x54686973. Needless to say sending bytes is twice as efficient, in storage, as sending the bytes as a string.
It is similar in principle to the MQCFBS PCF structure.
Cheers,
Paul
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
| tczielke |
| Posted: Fri Apr 01, 2016 6:57 am Post subject: |
|
|
Guardian
Joined: 08 Jul 2010
Posts: 941
Location: Illinois, USA
|
This manual doc provides a little more detail on what would constitue a MQTYPE_BYTE_STRING -> http://www.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.tro.doc/q044840_.htm?lang=en
"For MQBYTEn fields use MQTYPE_BYTE_STRING."
So examples would be:
MQBYTE24 MsgId; /* Message identifier */
MQBYTE24 CorrelId; /* Correlation identifier */
I agree with Roger that describing a MQTYPE_BYTE_STRING as "a byte string" is not exactly very helpful. Fundamentally, every data type (integers, character strings, etc.) is a string of bytes. However, I do understand it is kind of hard to succinctly describe "a string of bytes where MQ treats it as nothing more than raw bytes and does not assign any meaning to it like a string of characters, or an integer, etc.". Perhaps, providing some examples of what would be an MQTYPE_BYTE_STRING (e.g. MsgId) in the manual doc would be more helpful.
_________________
Working with MQ since 2010. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Apr 01, 2016 7:05 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
I think "byte string" is used consistently across the MQ knowledge center.
Granted, that might be hard to find if one is looking at a single page or set of pages.
A byte array might be a better term.
_________________
chmod -R ugo-wx / |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Fri Apr 01, 2016 7:39 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
From the APR:
| Quote: |
MQBYTEn - string of n bytes
MQCHARn - string of n characters |
I'd have preferred 'array,' as it implies an ordered arrangement.
Then, there's the Java 'string array.'
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| PaulClarke |
| Posted: Fri Apr 01, 2016 11:45 am Post subject: |
|
|
Grand Master
Joined: 17 Nov 2005
Posts: 1002
Location: New Zealand
|
An array doesn't imply an 'ordered arrangement' to me. An array to me implies that it is a series of the same object types which can be accessed directly by index. While you could of course do that with a byte string or character string or whatever it is not it's main purpose. So, for me personally, calling it an array would have been misleading. It is a string of bytes which should be treated as a string, not as lots of independently addressable units. It is a string that implies to me an 'ordered arrangement'.
As for MQ not naming it the same way as Java that would clearly not be possible since MQ is older than Java. Having said that, I'm no Java expert by any means, but doesn't Java also have the concept of ByteBuffers ? So Java doesn't always call them arrays either.
I guess it is a common problem in Computing generally that the nomenclature is not standard. Programming languages, command scripts, protocols and Development environments frequently use different names for the same things. That's why we are all paid the big bucks to sort it all out
Cheers,
Paul.
_________________
Paul Clarke
MQGem Software
www.mqgem.com |
|
| Back to top |
|
|
|
|
