| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| Messaging more than 100 MB |
« View previous topic :: View next topic » |
| Author |
Message
|
| AlexSTAL |
 Posted: Tue Jun 16, 2020 6:31 am Post subject: Messaging more than 100 MB |
 |
|
Newbie
Joined: 16 Jun 2020
Posts: 2
|
Good afternoon!
I apologize for the automatic translation of the message.
Please tell me, is it possible to send a message of more than 100 MB so that MQ performs segmentation (I use ActiveX)? Everywhere it is written that it is enough to use the value MQMF_SEGMENTATION_ALLOWED in the MsgFlag field, and MQ will do the rest.
I have a return code of 2010 MQRC_DATA_LENGTH_ERROR.
Or can you send a message of more than 100 MB using MQ only by breaking it into segments using the application? |
|
| Back to top |
|
 |
| bruce2359 |
| Posted: Tue Jun 16, 2020 8:38 am Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
|
| Back to top |
|
|
| gbaddeley |
| Posted: Tue Jun 16, 2020 3:12 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
The 100MB physical limit was a decision made for MQ v1.0 and has remained unaltered since. These types of arbitrary limits are intentional, to make the designer think long and hard about any requirement to send more than 100MB in one MQ data entity. Large messages create extra demand on memory, CPU, disk and network.
In some situations, it is actually better to chunk the data into small messages based on smaller entities that often exist inside larger entities. Ideally, there should not be affinity between the chunks that will make the processing logic much more complicated (eg. needing to do marshalling or reassembly of data). IBM MFT is an example of doing this.
Also, the data should be analysed to identify wasted space, such as long runs of whitespace characters (eg. in fixed length fields), excessive use of delimeters or separator characters, or large numbers of empty XML elements. These should be minimised.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Tue Jun 16, 2020 3:34 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
Also, what type of data needs 100MB plus? Not transactional data, would be my guess. Are you sending files? mp4's? binaries?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
| AlexSTAL |
| Posted: Wed Jun 17, 2020 9:29 pm Post subject: |
|
|
Newbie
Joined: 16 Jun 2020
Posts: 2
|
I am sending XML generated by third-party software. 98% of files do not exceed 100 MB. But there are times when there is a lot of binary information in Base64 format inside.
I realized that I need to manually break it into pieces using segmentation mechanisms |
|
| Back to top |
|
|
| gbaddeley |
| Posted: Thu Jun 18, 2020 3:21 pm Post subject: |
|
|
Jedi Knight
Joined: 25 Mar 2003
Posts: 2538
Location: Melbourne, Australia
|
| AlexSTAL wrote: |
I am sending XML generated by third-party software. 98% of files do not exceed 100 MB. But there are times when there is a lot of binary information in Base64 format inside.
I realized that I need to manually break it into pieces using segmentation mechanisms |
That's inefficient on 2 fronts, XML and Base64. Can you break the XML up into smaller elements, and send each? Can you decode the Base64, store the BLOB in a shared file system, or DB, or separate message, and just send the XML with a reference to it ? This may be easier and certainly more elegant than implementing MQ message segements / grouping.
_________________
Glenn |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Thu Jun 18, 2020 4:01 pm Post subject: |
|
|
Poobah
Joined: 05 Jan 2008
Posts: 9469
Location: US: west coast, almost. Otherwise, enroute.
|
A thought or two:
Have you tried suppressing so-called "insignificant XML whitespace?" This might reduce enough of the data to fit within the 100MB physical message size limit. This should not require app changes at the receiving end.
And/or can the data be zipped (pax'd, tar'd, cpio'd, ...) before placed into the message payload?
_________________
I like deadlines. I like to wave as they pass by.
ב''ה
Lex Orandi, Lex Credendi, Lex Vivendi. As we Worship, So we Believe, So we Live. |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
