| Author |
Message
|
| ruimadaleno |
 Posted: Tue May 13, 2014 4:01 am Post subject: Designing integration for big messages |
 |
|
Master
Joined: 08 May 2014
Posts: 274
|
Hi all,
my employer is promoting a major effort to re-design/re-implement a lot of information systems.
Today we have several .Net , VB programs that receive messages (from company suppliers) and perform intregration in several target information systems.
This messages are in the form of xml (from a web service) and file (shared folders on filesystem) and can reach sizes of 20MB (mainly because the message contains base64 elements, mainly photos that our suppliers need to exchange to the company)
This design has several problems, lot's of downtime, lack of support , aging of component.. etc etc 
We have the opportunity of redesign this programs !  we are considering the usage of IBM MQ + Broker to achieve it.
My first question/concert is regarding the message size. Can the combination of MQ + Broker deal with this messages ? I've done some searching in the MQ series forum, but found some old threads , pointing to some documents with broken links.
so i'm writing this thread to ask you to share your experience on MQ + Broker dealing with big messages. I'm in the design phase, so, what pitfall should i avoid from start ? any documentation on this subject ?
We are using Broker 8.0.0.3 on windows platform.
Best regards |
|
| Back to top |
|
 |
| kimbert |
| Posted: Tue May 13, 2014 4:33 am Post subject: |
|
|
Jedi Council
Joined: 29 Jul 2003
Posts: 5542
Location: Southampton
|
Other responders please note: this is not the usual 'large message' scenario. The messages do not consist of a large number of repeats of a small structure. It is a single, large, base64-encoded element.
_________________
Before you criticize someone, walk a mile in their shoes. That way you're a mile away, and you have their shoes too. |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue May 13, 2014 4:56 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
Our MQ + message broker infrastructure works as a "file transfer hub" and can cope with the large messages without any problem. Nowadays, size as 20 MB is not a big message - comparing for even 2 GB data transfers...
Interesting topics:
- FileInput / Output node environment related settings:
- - MQSI_FILENODES_MAXIMUM_RECORD_LENGTH
- - MQSI_FILENODES_ROOT_DIRECTORY
- MQ transaction log consideration:
- - circular vs. linear (laziness vs. anxiousness  )
- - bigger is better... |
|
| Back to top |
|
|
| zpat |
| Posted: Tue May 13, 2014 5:31 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Up to 100 mb messages are fine in MQ and broker (as long as the queues, transmit queues, channels, DLQ, all have this limit and there is sufficient queue space and log space).
However using groups of smaller messages can often be better and in effect has no size limit.
Groups of messages is a particular MQ feature (and also supported by broker).
Keep in mind any syncpoint implications whatever design you choose.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue May 13, 2014 5:33 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| kimbert wrote: |
| Other responders please note: this is not the usual 'large message' scenario. The messages do not consist of a large number of repeats of a small structure. It is a single, large, base64-encoded element. |
You got it absolutely right Kimbert.
Each message is a valid XML. Each one of this xml can have several elements of type base64. Each one of this elements represents a file that should be integrated into our enterprise content plataform (IBM Filenet) |
|
| Back to top |
|
|
| Esa |
| Posted: Tue May 13, 2014 5:39 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| ruimadaleno wrote: |
| Each one of this xml can have several elements of type base64. |
Which means that the usual 'large message' scenario in fact applies... |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue May 13, 2014 5:45 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| Tibor wrote: |
Our MQ + message broker infrastructure works as a "file transfer hub" and can cope with the large messages without any problem. Nowadays, size as 20 MB is not a big message - comparing for even 2 GB data transfers...
Interesting topics:
- FileInput / Output node environment related settings:
- - MQSI_FILENODES_MAXIMUM_RECORD_LENGTH
- - MQSI_FILENODES_ROOT_DIRECTORY
- MQ transaction log consideration:
- - circular vs. linear (laziness vs. anxiousness )
- - bigger is better... |
Tibor, thank you for your answer.
As far as i understand you are "exposed" to a system, built on MQ + Broker, wich has the ability to integrate 2GB data transfers.
This data transfer is a ratio or is it a maximum message size ? does your system has the ability to transfer a 2GB message size or does our sizes transfers several messages wich sums up a 2GB trafic ?
I'd like to know about sizing, parameters, queues configuration and other setting you've done to achieve this levels of message transfer. Would you share this knowledge with us ?
Best regards
Rui Madaleno |
|
| Back to top |
|
|
| Tibor |
| Posted: Tue May 13, 2014 7:07 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| ruimadaleno wrote: |
| This data transfer is a ratio or is it a maximum message size ? does your system has the ability to transfer a 2GB message size or does our sizes transfers several messages wich sums up a 2GB trafic ?" |
More precisely: this infrastructure was tested with files and messages up to 2GB.
| Quote: |
| I'd like to know about sizing, parameters, queues configuration and other setting you've done to achieve this levels of message transfer. Would you share this knowledge with us ? |
Most important this is the sizing of transaction log files. I personally prefer using circular logs, eg. for 20 GB log space (almost) you should configure logs:
| Code: |
LogPrimaryFiles=100
LogSecondaryFiles=100
LogFilePages=49152 |
When using SAN / contact admin, maximize the message throughput:
| Code: |
| LogWriteIntegrity=SingleWrite |
As zpat already mentioned, another useful idea is using MQ feature message groups, which helps your MQ and messages broker in message processing. Maximum message size is between 4 and 24 MB in every queue and works perfectly even with enormous files. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue May 13, 2014 7:13 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
To clarify the requirement i'm posting an example of an xml message:
Check the bold text , it's where the base64 is located
sorry for the long post 
<?xml version="1.0" encoding="iso-8859-1"?>
<TaskRequest xmlns="http://www.acornet.pt/AcoreanaAudaflow">
<Task>
<Accao />
<TaskID>703457</TaskID>
<ExternalRef />
<InsuranceCompanyID>0</InsuranceCompanyID>
<InsuranceCompany />
<ClaimId>9001559829/001</ClaimId>
<ClaimDate>2013-09-13T00:00:00.0000000+01:00</ClaimDate>
<TaskAdress>
<Street>PRACETA DA PRANCHINHA, 20</Street>
<City>PONTA DELGADA</City>
<PostCodeId>PONTA DELGADA</PostCodeId>
<PostCode />
</TaskAdress>
<CustomerName />
<CustomerType>SEGURADO</CustomerType>
<CustomerPhone />
<RepairShopName>AUTO AÇOREANA LDA diversa</RepairShopName>
<AssessmentType>2</AssessmentType>
<AssessmentTypeChangeMotive />
<AssessmentRequestDate>2014-05-08T00:00:00.0000000+01:00</AssessmentRequestDate>
<AssessmentDate>2014-05-09T00:00:00.0000000+01:00</AssessmentDate>
<AssessmentCloseDate>1900-01-01T00:00:00.0000000-00:00</AssessmentCloseDate>
<AssessmentReportDate>1900-01-01T00:00:00.0000000-00:00</AssessmentReportDate>
<RepairStartDate>2014-05-19T00:00:00.0000000+01:00</RepairStartDate>
<ClassificacaoSinistro>CIDS - Credor</ClassificacaoSinistro>
<IDS>false</IDS>
<TaskStatus>5</TaskStatus>
<TaskAction>1</TaskAction>
<IDPerito>519735</IDPerito>
<NomePerito>AUTO AÇOREANA LDA</NomePerito>
<ClaimLocal>VILA FRANCA DO CAMPO</ClaimLocal>
<ClaimDescription>o segurado estava estacionado Este processo substitui o processo 9001557301</ClaimDescription>
<ImpossibilidadeExecucaoServico />
<ImpossibilidadeFinalizarServico />
<ObsImpExecuctarServico />
<DataNovaVisita>1900-01-01T00:00:00.0000000-00:00</DataNovaVisita>
<DataConclusaoPeritagem>2014-05-13T14:04:59.0000000+01:00</DataConclusaoPeritagem>
<DataParticipacao>2014-01-22T00:00:00.0000000-00:00</DataParticipacao>
<DAAA_CorrectamentePreenchida>false</DAAA_CorrectamentePreenchida>
<DER>false</DER>
</Task>
<RepairShop>
<RepairShopName>AUTO AÇOREANA LDA diversa</RepairShopName>
<Contact />
<Adress>
<Street>PRACETA DA PRANCHINHA, 20</Street>
<City>PONTA DELGADA</City>
<PostCodeId />
<PostCode />
</Adress>
<Phone>296304900</Phone>
<HourValue1>28.50</HourValue1>
<HourValueDiscount1>6.00</HourValueDiscount1>
<HourValue2>28.50</HourValue2>
<HourValue3>0</HourValue3>
<HourValuePaint>28.50</HourValuePaint>
<HourValueDiscountPaint />
<PartDiscount>-1</PartDiscount>
<ReadyPayment>false</ReadyPayment>
<CanAlterValues>true</CanAlterValues>
<Fax />
<Email />
<Telemovel />
<NumeroContribuinte>512037353</NumeroContribuinte>
<Cesvi>true</Cesvi>
<RORServices />
</RepairShop>
<Vehicle>
<LicensePlate>21-52-QZ</LicensePlate>
<Brand>Toyota</Brand>
<Model>Hilux Pick-Up 08/97 -> 09/05 [LN/KDN145/165/170/190]</Model>
<Version>Hilux chassis cabina</Version>
<BuildingYear>2000</BuildingYear>
<BuildingMonth>12</BuildingMonth>
<Color />
<ChassisNumber>JTFVD526000005604</ChassisNumber>
<NumberDoors>2</NumberDoors>
<NumberSeats>3</NumberSeats>
<NumberKilometers>90900</NumberKilometers>
<GrossWheight>0</GrossWheight>
<VenalValue>0</VenalValue>
<ValueAsNew>0</ValueAsNew>
<SalvageValue>0</SalvageValue>
<CurrentValueAsNew>0</CurrentValueAsNew>
<KilometresType>1</KilometresType>
<Deal>false</Deal>
<DealValue>0</DealValue>
<EuroTaxValueSold>-1</EuroTaxValueSold>
<EuroTaxValueBuy>-1</EuroTaxValueBuy>
<TotalLoss>false</TotalLoss>
<Send2Coordenator>false</Send2Coordenator>
<Pistondisplacement>2446</Pistondisplacement>
<HorsePower>90</HorsePower>
<FuelType>O</FuelType>
<ManufacturerCodeAX>70</ManufacturerCodeAX>
<ModelCodeAX>70</ModelCodeAX>
<CategoriaViaturaAudatex>Ligeiro de Passageiros</CategoriaViaturaAudatex>
<ExistiuReboque>false</ExistiuReboque>
<Rebocadora />
<MotivoNaoCircula />
<HouveAveriguacao>false</HouveAveriguacao>
<MotivoAveriguacao />
<Observacoes />
</Vehicle>
<Inventory>
<contact admin>
<contact admin>
<contact admin>89</contact admin>
<Description>Jantes ferro</Description>
<Checked>false</Checked>
</contact admin>
</contact admin>
<SpareTyre>-1</SpareTyre>
<LeftFrontTyre1>-1</LeftFrontTyre1>
<RightFrontTyre1>-1</RightFrontTyre1>
<LeftRearTyre1>-1</LeftRearTyre1>
<RightRearTyre1>-1</RightRearTyre1>
<contact admin />
</Inventory>
<Policy>
<PolicyNumber>9001664461</PolicyNumber>
<InsuranceCompany />
<ProductDescription>20</ProductDescription>
<Make>TOYOTA HILUX</Make>
<Model />
<Year>2001</Year>
<CC>2446</CC>
<HorsePower>-1</HorsePower>
<GrossWeight>-1</GrossWeight>
<NO_OF_SEATS>0</NO_OF_SEATS>
<Fuel />
<Coverages>
<Coverage>
<CoverageId>17</CoverageId>
<Description>Quebra isolada de vidros </Description>
<InsuredValue>1000</InsuredValue>
<ExcessCodeDescription>50</ExcessCodeDescription>
<Excess>50</Excess>
<Affected>false</Affected>
<MinValue>0</MinValue>
<MaxValue>0</MaxValue>
</Coverage>
</Coverages>
<CoveredExtras />
<Verson />
<LicenceNumber>21-52-QZ</LicenceNumber>
</Policy>
<Claim>
<VehicleStatus>1</VehicleStatus>
<CauseOfAccident> Sem responsabilidade</CauseOfAccident>
<DamagedAreasObs />
<DamagedAreas />
</Claim>
<RorServiceList />
<BudgetList>
<BudgetData>
<BudgetId>1</BudgetId>
<Description />
<BudgetDate>2014-05-13T14:04:59.0000000+01:00</BudgetDate>
<BudgetKind />
<ManualBudgetType>-1</ManualBudgetType>
<AssessmentType>2</AssessmentType>
<AssessmentId>-1</AssessmentId>
<BudgetType>1</BudgetType>
<BudgetTypeAudaflow>Orçamento</BudgetTypeAudaflow>
<Conditional>false</Conditional>
<ManValue1>5.36</ManValue1>
<ManValueDiscount1>0.34</ManValueDiscount1>
<ManValue2>-1</ManValue2>
<ManValue3>-1</ManValue3>
<PaintValue>-1</PaintValue>
<PaintValueDiscount>-1</PaintValueDiscount>
<PartValue>110.16</PartValue>
<PartValueDiscount>-1</PartValueDiscount>
<NoTaxesValue>115.52</NoTaxesValue>
<NoTaxes>20.79</NoTaxes>
<WithTaxesValue>136.31</WithTaxesValue>
<Excess>0</Excess>
<ExecutionDays>1</ExecutionDays>
<Version />
<ExpirationDate>1900-01-01T00:00:00.0000000-00:00</ExpirationDate>
<PeritoID>-1</PeritoID>
<PeritoName />
<AproverID>-1</AproverID>
<AproveBy />
<Status />
<NaturezaOrcamento>Final</NaturezaOrcamento>
<StartRepairDate>2014-05-19T00:00:00.0000000+01:00</StartRepairDate>
<InputRecordIDX>1</InputRecordIDX>
<InputRecord>/OutputRecord>
<PaintMaterial>-1</PaintMaterial>
<TotalDiscount>-1</TotalDiscount>
<DiscountAfterTotal>0</DiscountAfterTotal>
<ExtraValue>-1</ExtraValue>
<ManHourValue>28.5</ManHourValue>
<Observacoes />
<DataPrevistaDesmontagem>2014-05-13T12:44:12.0000000+01:00</DataPrevistaDesmontagem>
</BudgetData>
</BudgetList>
<DocumentList />
<MessageList />
<Driver>
<DriverLicence />
<Name />
<PersonAddress>
<Street />
<City />
<PostCodeId />
<PostCode />
</PersonAddress>
<Identification />
<NumeroContribuinte />
<PhoneNumber />
<FaxNumber />
<Email />
<RegularDriver>false</RegularDriver>
<Observations />
</Driver>
<Segurado>
<DriverLicence />
<Name>JOSÉ LEONARDO PESTANA ROIAS</Name>
<PersonAddress>
<Street>AV 5 DE OUTUBRO, 13 A</Street>
<City>PENACOVA</City>
<PostCodeId>3360-181</PostCodeId>
<PostCode />
</PersonAddress>
<Identification>010391571</Identification>
<NumeroContribuinte>190181419</NumeroContribuinte>
<PhoneNumber />
<FaxNumber />
<Email />
<RegularDriver>false</RegularDriver>
<Observations> [Peritagem Definitiva até: 2500]</Observations>
</Segurado>
<Fotos>
<Foto>
<ImageName>CIMG4608.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124245765_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4608.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124302234_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4606.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124306234_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4607.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124308640_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4609.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124311078_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4611.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124313687_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4612.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124317468_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4613.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124322140_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4614.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124325046_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4615.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124327812_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4616.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124329500_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
<Foto>
<ImageName>CIMG4617.jpg</ImageName>
<InAutogere>false</InAutogere>
<IDImagem>001_AFSATA02_20140513124331437_703457</IDImagem>
<IDBudget>1</IDBudget>
<Image> Base64 goes here </Image>
</Foto>
</Fotos>
<Documents>
<Document>
<DocName>000_ACOREANA_201405081250048756353515020_703457.pdf</DocName>
<InAutogere>false</InAutogere>
<IDDocument>000_ACOREANA_201405081250048756353515020_703457</IDDocument>
<IDBudget>0</IDBudget>
<Doc> Base64 goes here </Doc>
</Document>
<Document>
<DocName>Output[IDX1].txt</DocName>
<InAutogere>false</InAutogere>
<IDDocument>1_20140513124231</IDDocument>
<IDBudget>1</IDBudget> <Doc> Base64 goes here </Doc>
</Document>
</Documents>
<FrameID>
<MessageID>ACR703457</MessageID>
<ReferringMessageID>5b0f2752_d07d_4806_afd4_8aeff2f68b62201405081253015996353515</ReferringMessageID>
<TransationID>MjAxNDA1MDgxMjUzMDE1OTk2MzUzNTE1</TransationID>
<ForeignKey />
</FrameID>
</TaskRequest> |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue May 13, 2014 7:28 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
We use FileNet for this kind of document storage and have had no problems with data traversing our v7.0.0.6 system that's 50Mb a pop and greater. One thing we do (which has more to do with the system we have fronting FileNet than anything else) is that for large data we split it out from the XML, drop the content (the base64 in your example) onto either a queue or a file system and send FileNet a URN to the location.
Data arrives either as file through ConnectDirect (or whatever IBM call it these days) or WMQ. We've taken a political decision not to allow more than 100Mb in a single data drop, which fits both the max size of a message and the default maximum file size for WMB. This not only allows us to distribute work across the server estate (i.e. chop up a 2Gb file) but also makes our valued customers think about what they're sending, where the transactional boundaries are, what the recovery point and SLAs are, etc, etc.
Having said that we still get 50Mb - 60Mb as a common single message. Interestingly, one of our largest XMLs (85Mbish) has not content but is a SQLServer database being sick - sorry, dumping the contents of it's tables by matching column name to tag name. Not sure what they plan to do when this hits our 100Mb limit - write a sensible SQL query perhaps?
I echo the comments of my associates that we size to carry this load. We use circular logging on the queue manager (as the recovery points are elsewhere) with log page size at the max, 65 primary logs (which is some convieient %age of our mounted disc) and the maximum number of secondary logs.
We also have resource monitoring the way stray dogs have fleas. So many statistics, so little value but we do know if things stop moving.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ruimadaleno |
| Posted: Tue May 13, 2014 8:10 am Post subject: |
|
|
Master
Joined: 08 May 2014
Posts: 274
|
| zpat wrote: |
Up to 100 mb messages are fine in MQ and broker (as long as the queues, transmit queues, channels, DLQ, all have this limit and there is sufficient queue space and log space).
However using groups of smaller messages can often be better and in effect has no size limit.
Groups of messages is a particular MQ feature (and also supported by broker).
Keep in mind any syncpoint implications whatever design you choose. |
Thank you for you reply zPat.
I'm not aware of the MQ feature Group of message. Can you clarify on how this feature can by valuable in this case ?
Best regards
´
Rui Madaleno |
|
| Back to top |
|
|
| zpat |
| Posted: Tue May 13, 2014 8:33 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Grouped messages and segmented messages are two different ways of asking MQ to carry related data over more than one physical message, but allow the receiving application to process it as one logical message.
You can read up on grouped and segmented messages in the infocenter.
If it's really one huge message, then segmented would be one to use.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Tue May 13, 2014 9:32 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
| Vitor wrote: |
| We also have resource monitoring the way stray dogs have fleas. So many statistics, so little value but we do know if things stop moving. |
Thank you, I love it. Punch out, you're done for the day. Go out on top like George Constanza.
_________________
Peter Potkay
Keep Calm and MQ On |
|
| Back to top |
|
|
| Tibor |
| Posted: Wed May 14, 2014 7:51 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| ruimadaleno wrote: |
| I'm not aware of the MQ feature Group of message. Can you clarify on how this feature can by valuable in this case ? |
Two useful things when using message groups:
(1) you can handle messages separately on broker - better throughput
(2) but these messages are consistent because of grouping properties |
|
| Back to top |
|
|
|
|
