ASG
IBM
Zystems
Cressida
Icon
Netflexity
 
  MQSeries.net
Search  Search       Tech Exchange      Education      Certifications      Library      Info Center      SupportPacs      LinkedIn  Search  Search                                                                   FAQ  FAQ   Usergroups  Usergroups
 
Register  ::  Log in Log in to check your private messages
 
RSS Feed - WebSphere MQ Support RSS Feed - Message Broker Support

MQSeries.net Forum IndexGeneral IBM MQ SupportMessaging more than 100 MB

Post new topicReply to topic
Messaging more than 100 MB View previous topic :: View next topic
Author Message
AlexSTAL
PostPosted: Tue Jun 16, 2020 6:31 am Post subject: Messaging more than 100 MB Reply with quote

Newbie

Joined: 16 Jun 2020
Posts: 2

Good afternoon!
I apologize for the automatic translation of the message.

Please tell me, is it possible to send a message of more than 100 MB so that MQ performs segmentation (I use ActiveX)? Everywhere it is written that it is enough to use the value MQMF_SEGMENTATION_ALLOWED in the MsgFlag field, and MQ will do the rest.
I have a return code of 2010 MQRC_DATA_LENGTH_ERROR.

Or can you send a message of more than 100 MB using MQ only by breaking it into segments using the application?
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Jun 16, 2020 8:38 am Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8772
Location: US: west coast, almost. Otherwise, enroute.

The maximum size (length) of an MQ physical message is 100MB. An app can create a logical message that exceeds the 100MB limit by creating MessageGroup of physical messages. Read https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.1.0/com.ibm.mq.dev.doc/q023060_.htm
_________________
My life flows on in endless song;
How can I keep from singing?
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Tue Jun 16, 2020 3:12 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2154
Location: Melbourne, Australia

The 100MB physical limit was a decision made for MQ v1.0 and has remained unaltered since. These types of arbitrary limits are intentional, to make the designer think long and hard about any requirement to send more than 100MB in one MQ data entity. Large messages create extra demand on memory, CPU, disk and network.

In some situations, it is actually better to chunk the data into small messages based on smaller entities that often exist inside larger entities. Ideally, there should not be affinity between the chunks that will make the processing logic much more complicated (eg. needing to do marshalling or reassembly of data). IBM MFT is an example of doing this.

Also, the data should be analysed to identify wasted space, such as long runs of whitespace characters (eg. in fixed length fields), excessive use of delimeters or separator characters, or large numbers of empty XML elements. These should be minimised.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Tue Jun 16, 2020 3:34 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8772
Location: US: west coast, almost. Otherwise, enroute.

Also, what type of data needs 100MB plus? Not transactional data, would be my guess. Are you sending files? mp4's? binaries?
_________________
My life flows on in endless song;
How can I keep from singing?
Back to top
View user's profile Send private message
AlexSTAL
PostPosted: Wed Jun 17, 2020 9:29 pm Post subject: Reply with quote

Newbie

Joined: 16 Jun 2020
Posts: 2

I am sending XML generated by third-party software. 98% of files do not exceed 100 MB. But there are times when there is a lot of binary information in Base64 format inside.

I realized that I need to manually break it into pieces using segmentation mechanisms
Back to top
View user's profile Send private message
gbaddeley
PostPosted: Thu Jun 18, 2020 3:21 pm Post subject: Reply with quote

Jedi

Joined: 25 Mar 2003
Posts: 2154
Location: Melbourne, Australia

AlexSTAL wrote:
I am sending XML generated by third-party software. 98% of files do not exceed 100 MB. But there are times when there is a lot of binary information in Base64 format inside.

I realized that I need to manually break it into pieces using segmentation mechanisms


That's inefficient on 2 fronts, XML and Base64. Can you break the XML up into smaller elements, and send each? Can you decode the Base64, store the BLOB in a shared file system, or DB, or separate message, and just send the XML with a reference to it ? This may be easier and certainly more elegant than implementing MQ message segements / grouping.
_________________
Glenn
Back to top
View user's profile Send private message
bruce2359
PostPosted: Thu Jun 18, 2020 4:01 pm Post subject: Reply with quote

Poobah

Joined: 05 Jan 2008
Posts: 8772
Location: US: west coast, almost. Otherwise, enroute.

A thought or two:

Have you tried suppressing so-called "insignificant XML whitespace?" This might reduce enough of the data to fit within the 100MB physical message size limit. This should not require app changes at the receiving end.

And/or can the data be zipped (pax'd, tar'd, cpio'd, ...) before placed into the message payload?
_________________
My life flows on in endless song;
How can I keep from singing?
Back to top
View user's profile Send private message
Display posts from previous:
Post new topicReply to topic Page 1 of 1

MQSeries.net Forum IndexGeneral IBM MQ SupportMessaging more than 100 MB
Jump to:



You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Protected by Anti-Spam ACP


Theme by Dustin Baccetti
Powered by phpBB 2001, 2002 phpBB Group

Copyright MQSeries.net. All rights reserved.