| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
| Large message segmentation |
« View previous topic :: View next topic » |
| Author |
Message
|
| bruce2359 |
 Posted: Wed Jan 05, 2022 10:32 am Post subject: |
 |
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
| sl12345678 wrote: |
| bruce2359 wrote: |
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.
I don’t have a dev environment available to duplicate Morag’s testing. |
Yes, I am specifically trying to segment messages larger than 100MB. |
Break your >100MB app data into multiple <100MB segments, each with its own MQMD, then do individual MQPUTs for each segment.
The consuming app will need to do individual MQGETs to combine the segments into the original the app data.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
 |
| ashanks6 |
| Posted: Wed Jan 05, 2022 11:01 am Post subject: |
|
|
Newbie
Joined: 03 Jan 2022
Posts: 5
|
| bruce2359 wrote: |
| sl12345678 wrote: |
| bruce2359 wrote: |
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.
I don’t have a dev environment available to duplicate Morag’s testing. |
Yes, I am specifically trying to segment messages larger than 100MB. |
Break your >100MB app data into multiple <100MB segments, each with its own MQMD, then do individual MQPUTs for each segment.
The consuming app will need to do individual MQGETs to combine the segments into the original the app data. |
Hi Bruce,
Yes, that is a workaround we could use, but I'd rather have our folks use the functionality that MQ provides. I have opened a ticket with IBM support, and will post back the results of that submission. There is a lot of conflicting information on this board about this issue, and it is my hope to provide absolute clarity. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Wed Jan 05, 2022 11:34 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7719
|
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Jan 05, 2022 11:40 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| Yes, I have read the documentation and implemented the segmentation as IBM suggests, but I am still receiving the data length error. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jan 05, 2022 12:20 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
| sl12345678 wrote: |
| Yes, I have read the documentation and implemented the segmentation as IBM suggests, but I am still receiving the data length error. |
| sl12345678 wrote: |
| MessageSize = message.MessageLength; //124059223 |
Message length must be <100MB - the maximum size of an MQ physical message.
Reduce message length, and try again.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| ashanks6 |
| Posted: Wed Jan 05, 2022 12:27 pm Post subject: |
|
|
Newbie
Joined: 03 Jan 2022
Posts: 5
|
| PeterPotkay wrote: |
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=segmentation-reassembly-by-queue-manager
Yes, you can put and get messages > 100MB using this method without resorting to using multiple MQPUTs and MQGETs.
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=long-message-segmentation
| Quote: |
| Use this information to learn about segmenting messages. This feature is not supported on IBM® MQ for z/OS® or by applications using IBM MQ classes for JMS. |
|
Yes! Exactly. Our queue manager is on AIX btw.
And the OP is using .net. |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Jan 05, 2022 2:34 pm Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| bruce2359 wrote: |
| sl12345678 wrote: |
| Yes, I have read the documentation and implemented the segmentation as IBM suggests, but I am still receiving the data length error. |
| sl12345678 wrote: |
| MessageSize = message.MessageLength; //124059223 |
Message length must be <100MB - the maximum size of an MQ physical message.
Reduce message length, and try again. |
Thanks, Bruce - I am already able to send smaller messages successfully with the same code. It is my understanding that messages over 100mb are automatically segmented if MQMF_SEGMENTATION_ALLOWED flag is turned on.
We are trying to figure out what is missing that is causing the data length error instead of automatically segmenting the large message. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Jan 05, 2022 2:44 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20729
Location: LI,NY
|
Are you using a client channel to retrieve the message and what is the max msg length on your client channel / svrconn channel defined as?
If the queue manager assembles the message, does that mean it has to fit the max size of the client/svrconn channel? 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| hughson |
| Posted: Wed Jan 05, 2022 3:07 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1948
Location: Bay of Plenty, New Zealand
|
I did my original testing using the Q Utility so I can't easily post the source. So I have quickly created a small application that illustrates the same. I don't think it is necessary to post the whole source, as you can manage to do an MQCONN and an MQOPEN I'm sure, but here is the source of the MQPUT part of the simple C part.
| Code: |
MQMD MsgDesc = {MQMD_DEFAULT};
MQPMO pmo = {MQPMO_DEFAULT};
MQLONG MsgLength = 124059223;
:
:
/********************************************************************/
/* Put large message segmented */
/********************************************************************/
if (Verbose) printf("Putting %d bytes message to Queue '%s'\n",
MsgLength, pQName);
pBuffer = malloc(MsgLength);
if (!pBuffer)
{
printf("Unable to allocate %d bytes\n", MsgLength);
goto MOD_EXIT;
}
memset(pBuffer, '*', MsgLength);
MsgDesc.Version = MQMD_VERSION_2;
MsgDesc.MsgFlags = MQMF_SEGMENTATION_ALLOWED;
MQPUT(hConn,
hObj,
&MsgDesc,
&pmo,
MsgLength,
pBuffer,
&CompCode,
&Reason);
if (Reason)
{
printf("MQPUT to Queue '%s' failed with reason %d\n",
pQName, Reason);
goto MOD_EXIT;
} |
As I've said before, what you are seeing is not a restriction the queue manager puts on you, as demonstrated by this code, but might be a restriction the environment or language bindings you are making use of places on you.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software
Last edited by hughson on Wed Jan 05, 2022 3:33 pm; edited 1 time in total |
|
| Back to top |
|
|
| hughson |
| Posted: Wed Jan 05, 2022 3:23 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1948
Location: Bay of Plenty, New Zealand
|
I suspect that the problem is around the client channels. Try running your application locally bound to see if that works in the interim.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| hughson |
| Posted: Wed Jan 05, 2022 3:32 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1948
Location: Bay of Plenty, New Zealand
|
The limitation you are hitting is the client channel MAXMSGL limit. This limit is controlling the size of the buffer that the client can send to the queue manager. The client does not do the segmentation, the full >100MB buffer has to make it to the queue manager in order to for it to be segmented, and the client can only send a 100MB buffer.
So, you can do this segmentation of >100MB messages when locally bound but not when client bound.
When getting the reassembled message you will have the same problem. If the getter is a client, it will be restricted by the client channel MAXMSGL limit and even if the queue manager can reassemble the >100MB message, it can't be sent down the client channel.
Even though the queue manager allows it, it was, to be fair to all those saying the maximum is 100MB, never intended to be used as a way of bypassing the 100MB size. It was supposed to be about segmenting things up to 100MB into smaller sizes.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| ashanks6 |
| Posted: Wed Jan 05, 2022 3:38 pm Post subject: |
|
|
Newbie
Joined: 03 Jan 2022
Posts: 5
|
| hughson wrote: |
I did my original testing using the Q Utility so I can't easily post the source. So I have quickly created a small application that illustrates the same. I don't think it is necessary to post the whole source, as you can manage to do an MQCONN and an MQOPEN I'm sure, but here is the source of the MQPUT part of the simple C part.
| Code: |
MQMD MsgDesc = {MQMD_DEFAULT};
MQPMO pmo = {MQPMO_DEFAULT};
MQLONG MsgLength = 124059223;
:
:
/********************************************************************/
/* Put large message segmented */
/********************************************************************/
if (Verbose) printf("Putting %d bytes message to Queue '%s'\n",
MsgLength, pQName);
pBuffer = malloc(MsgLength);
if (!pBuffer)
{
printf("Unable to allocate %d bytes\n", MsgLength);
goto MOD_EXIT;
}
memset(pBuffer, '*', MsgLength);
MsgDesc.Version = MQMD_VERSION_2;
MsgDesc.MsgFlags = MQMF_SEGMENTATION_ALLOWED;
MQPUT(hConn,
hObj,
&MsgDesc,
&pmo,
MsgLength,
pBuffer,
&CompCode,
&Reason);
if (Reason)
{
printf("MQPUT to Queue '%s' failed with reason %d\n",
pQName, Reason);
goto MOD_EXIT;
} |
As I've said before, what you are seeing is not a restriction the queue manager puts on you, as demonstrated by this code, but might be a restriction the environment or language bindings you are making use of places on you.
I will reiterate for everyone who has said otherwise. You **CAN** do an MQPUT with a buffer longer than 100MB if you use segmentation. Stop telling the OP to reduce his message length!
Cheers,
Morag |
Interesting, our code is VERY similar (as you might expect). The only difference was in how we loaded our message (I used 'Z' instead of '*') and sizeof to load the messlen. I changed my code to match yours. I still fail with a 2010 return code. I'm afraid I'm now forced to conclude that the queue manager itself is configured differently than your queue manager.
We are able yo send 102457600 byte messages, so I can not for the life of me think of what could be different. |
|
| Back to top |
|
|
| ashanks6 |
| Posted: Wed Jan 05, 2022 3:46 pm Post subject: |
|
|
Newbie
Joined: 03 Jan 2022
Posts: 5
|
| hughson wrote: |
The limitation you are hitting is the client channel MAXMSGL limit. This limit is controlling the size of the buffer that the client can send to the queue manager. The client does not do the segmentation, the full >100MB buffer has to make it to the queue manager in order to for it to be segmented, and the client can only send a 100MB buffer.
So, you can do this segmentation of >100MB messages when locally bound but not when client bound.
When getting the reassembled message you will have the same problem. If the getter is a client, it will be restricted by the client channel MAXMSGL limit and even if the queue manager can reassemble the >100MB message, it can't be sent down the client channel.
Even though the queue manager allows it, it was, to be fair to all those saying the maximum is 100MB, never intended to be used as a way of bypassing the 100MB size. It was supposed to be about segmenting things up to 100MB into smaller sizes.
Cheers,
Morag |
Looks like we crossed streams (never cross streams).
This explains everything, and I thank you for leaning into this.
Seems the documentation for server auto segmentation should note this.
Perhaps some future version of the client will allow this.
In the meantime, it looks like we will have to split the message ourselves.
If you ever get to Phoenix, please send me a note and redeem a free drink on me  |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Jan 05, 2022 3:55 pm Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| hughson wrote: |
The limitation you are hitting is the client channel MAXMSGL limit. This limit is controlling the size of the buffer that the client can send to the queue manager. The client does not do the segmentation, the full >100MB buffer has to make it to the queue manager in order to for it to be segmented, and the client can only send a 100MB buffer.
So, you can do this segmentation of >100MB messages when locally bound but not when client bound.
When getting the reassembled message you will have the same problem. If the getter is a client, it will be restricted by the client channel MAXMSGL limit and even if the queue manager can reassemble the >100MB message, it can't be sent down the client channel.
Even though the queue manager allows it, it was, to be fair to all those saying the maximum is 100MB, never intended to be used as a way of bypassing the 100MB size. It was supposed to be about segmenting things up to 100MB into smaller sizes.
Cheers,
Morag |
Thank you for your responses! Is it possible to set the channel limit programmatically? I assumed it was being set on the server. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jan 05, 2022 4:42 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9442
Location: US: west coast, almost. Otherwise, enroute.
|
| sl12345678 wrote: |
| Is it possible to set the channel limit programmatically? I assumed it was being set on the server. |
Channel attributes here https://www.ibm.com/docs/en/ibm-mq/8.0?topic=attributes-channel-channel-types
SVRCONN channels answer-up to inbound client requests.
Programmatically, MQSET can set/alter a limited set of queue attributes only. What do you want to set the limit to - keeping in mind that 100MB is the maximum?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
