| Author |
Message
|
| sl12345678 |
 Posted: Tue Dec 28, 2021 1:50 pm Post subject: Large message segmentation |
 |
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
Hi, I am trying to test sending large messages over 100MB over MQ using message segmentation and keep receiving MQRC_DATA_LENGTH_ERROR.
Message size= 124059223
QMGR max = 104857600
Queue max = 104857600
Here's a snippet of the code:
message.MessageFlags += MQC.MQMF_SEGMENTATION_ALLOWED;
message.GroupId = MQC.MQGI_NONE;
message.Version = MQC.MQMD_VERSION_2;
var putMessageOptions = new MQPutMessageOptions();
putMessageOptions.Options = MQC.MQPMO_SET_IDENTITY_CONTEXT | MQC.MQPMO_LOGICAL_ORDER | MQC.MQPMO_SYNCPOINT;
queue.Put(message, putMessageOptions);
Are there other properties that need to be set in order to segment this message into smaller messages/a message group?
Any insight would be appreciated - thank you! |
|
| Back to top |
|
 |
| hughson |
| Posted: Tue Dec 28, 2021 8:07 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
Are you using a client connection? Have you set the MAXMSGL of the client and svrconn channels as well? What platform is your queue manager on and what version of MQ is it?
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 29, 2021 7:33 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Dec 29, 2021 7:49 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
The channels are set to the max length as well. I am using MQ 9.2 and .Net Full Framework 4.8.
I am trying to determine if the segmentation, or breaking the message into a smaller message/message group is done automatically by the program by turning on the segmentation flag in order to handle messages over 100MB.
I am able to put a 104MB message into the queue successfully. I am trying to use Segmentation and reassembly by queue manager as discussed here:
https://www.ibm.com/docs/en/ibm-mq/9.2?topic=segmentation-reassembly-by-queue-manager |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Dec 29, 2021 10:38 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The maximum physical message size MQPUT is 100MB. If your payload is bigger than 100MB, then your app must create (MQPUT) segments < 100MB. Reassembly of a segmented message is possible by qmgr, channel or your consuming app MQGET.
Search google for C samples of message groups and segmentation.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Dec 29, 2021 10:43 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| Ok, so the segmentation is not done automatically for messages larger than 100MB - I need to programmatically break them into smaller messages and then the QueueManager will assemble them with the MQGET? |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Dec 29, 2021 6:35 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| sl12345678 wrote: |
| Ok, so the segmentation is not done automatically for messages larger than 100MB - I need to programmatically break them into smaller messages and then the Queue Manager will assemble them with the MQGET? |
If you have to segment the message in your code, you will have to reassemble it the same way. 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Thu Dec 30, 2021 7:27 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| fjp_saper wrote: |
If you have to segment the message in your code, you will have to reassemble it the same way. |
Yes, I am trying to avoid manually segmenting the message, but I do not see another way to do so for messages over 100MB?
| bruce2359 wrote: |
The maximum physical message size MQPUT is 100MB. If your payload is bigger than 100MB, then your app must create (MQPUT) segments < 100MB. Reassembly of a segmented message is possible by qmgr, channel or your consuming app MQGET.
Search google for C samples of message groups and segmentation. |
|
|
| Back to top |
|
|
| hughson |
| Posted: Thu Dec 30, 2021 6:13 pm Post subject: |
|
|
Padawan
Joined: 09 May 2013
Posts: 1959
Location: Bay of Plenty, New Zealand
|
There seems to be some confusing, and inaccurate information in this thread.
I have just tried it, you can do an MQPUT of a message of size 124059223 bytes onto a queue (and qmgr) with MAXMSGL(104857600) using the MQMF_SEGMENTATION_ALLOWED flag and a V2 MQMD.
This results in two messages on the queue, one of 100Mb in size and the second of 18Mb.
There is no requirement to segment the messages yourself just because you have exceeded the maximum message size.
I did this with a program written in the 'C' language. It is possible that there is some limitation within the language you are using. But it is not a limitation of the queue manager.
Cheers,
Morag
_________________
Morag Hughson @MoragHughson
IBM MQ Technical Education Specialist
Get your IBM MQ training here!
MQGem Software |
|
| Back to top |
|
|
| ashanks6 |
| Posted: Mon Jan 03, 2022 3:24 pm Post subject: |
|
|
Newbie
Joined: 03 Jan 2022
Posts: 5
|
| hughson wrote: |
There seems to be some confusing, and inaccurate information in this thread.
I have just tried it, you can do an MQPUT of a message of size 124059223 bytes onto a queue (and qmgr) with MAXMSGL(104857600) using the MQMF_SEGMENTATION_ALLOWED flag and a V2 MQMD.
This results in two messages on the queue, one of 100Mb in size and the second of 18Mb.
There is no requirement to segment the messages yourself just because you have exceeded the maximum message size.
I did this with a program written in the 'C' language. It is possible that there is some limitation within the language you are using. But it is not a limitation of the queue manager.
Cheers,
Morag |
Hello Ms Hughson. I am an associate of the OP. I tried to replicate your results in 'C', but I seem to have failed. Any messlen setting above 104857600 returns an mqrc of 2010. Could you share what you set the messlen to?
Thank you. |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Jan 05, 2022 8:28 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| hughson wrote: |
There seems to be some confusing, and inaccurate information in this thread.
I have just tried it, you can do an MQPUT of a message of size 124059223 bytes onto a queue (and qmgr) with MAXMSGL(104857600) using the MQMF_SEGMENTATION_ALLOWED flag and a V2 MQMD.
This results in two messages on the queue, one of 100Mb in size and the second of 18Mb.
There is no requirement to segment the messages yourself just because you have exceeded the maximum message size.
I did this with a program written in the 'C' language. It is possible that there is some limitation within the language you are using. But it is not a limitation of the queue manager.
Cheers,
Morag |
Thank you for your reply. Can you please post the C code you used as well as the setup for the program so that we can compare it to ours? |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jan 05, 2022 8:37 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Rather than posting the snippet you offered, Please post your C code here, so we can take a closer look at it.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Jan 05, 2022 8:41 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
Here is the Put code:
// .Net Framework 4.8
// MQ Client 9.2.4.0
private void PutMessageInMQ(string requestMessage, string queueName, byte[] correlationId, string courtId)
{
MQMessage message;
_properties.Clear();
_properties.Add(MQC.TRANSPORT_PROPERTY, MQC.TRANSPORT_MQSERIES_CLIENT);
_properties.Add(MQC.HOST_NAME_PROPERTY, ConfigurationManager.AppSettings["PutHost"]);
_properties.Add(MQC.CHANNEL_PROPERTY, ConfigurationManager.AppSettings["PutChannel"]);
_properties.Add(MQC.PORT_PROPERTY, ConfigurationManager.AppSettings["PutPort"]);
using (var queueManager = new MQQueueManager(ConfigurationManager.AppSettings["PutQueueManager"], _properties))
{
try
{
using (var queue = queueManager.AccessQueue(ConfigurationManager.AppSettings["PutMQ"],
MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING | MQC.MQOO_SET_IDENTITY_CONTEXT | MQC.MQOO_INQUIRE))
{
int qmmml = queueManager.MaximumMessageLength; //104857600
int qmax = queue.MaximumMessageLength; //104857600
message = new MQMessage();
message.PutApplicationType = MQC.MQAT_WINDOWS;
message.Format = MQC.MQFMT_STRING;
message.MessageId = MQC.MQMI_NONE;
message.CorrelationId = MQC.MQCI_NONE;
message.Encoding = MQC.MQENC_NATIVE;
message.Persistence = MQC.MQPER_PERSISTENT;
message.MessageType = MQC.MQMT_DATAGRAM;
message.MessageFlags += MQC.MQMF_SEGMENTATION_ALLOWED;
message.GroupId = MQC.MQGI_NONE;
message.Version = MQC.MQMD_VERSION_2;
message.ApplicationIdData = string.Format("C{0}{1}{2}", courtId, ConfigurationManager.AppSettings["QInterfaceNumber"], ConfigurationManager.AppSettings["QOriginatorCode"]);
message.CorrelationId = correlationId;
message.Write(ASCIIEncoding.ASCII.GetBytes(requestMessage));
var putMessageOptions = new MQPutMessageOptions();
putMessageOptions.Options = MQC.MQPMO_SET_IDENTITY_CONTEXT | MQC.MQPMO_LOGICAL_ORDER | MQC.MQPMO_SYNCPOINT;
int MessageSize = 0;
MessageSize = message.MessageLength; //124059223
queue.Put(message, putMessageOptions); //Messages put on the queue and sent
message.ClearMessage();
queue.Close();
}
}
catch (Exception ex)
{
_logger.Error(ex);
}
finally
{
queueManager.Close();
queueManager.Disconnect();
}
}
} |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Jan 05, 2022 9:14 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.
I don’t have a dev environment available to duplicate Morag’s testing.
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| sl12345678 |
| Posted: Wed Jan 05, 2022 9:18 am Post subject: |
|
|
Novice
Joined: 28 Dec 2021
Posts: 10
|
| bruce2359 wrote: |
The maximum length of an MQ physical message is 100MB, including headers. Not a surprise that your app is getting data length error.
I don’t have a dev environment available to duplicate Morag’s testing. |
Yes, I am specifically trying to segment messages larger than 100MB. |
|
| Back to top |
|
|
|
|
