| Author |
Message
|
| pvemuri |
 Posted: Mon Nov 01, 2004 1:28 pm Post subject: Large file sizes through WBIMB |
 |
|
Voyager
Joined: 05 Jan 2004
Posts: 83
Location: CA, USA
|
Hello,
If there is a large file of size 2 /3 GB coming in, how can we handle that through MB. Do we surely need to split it up before even being put into the input Queue? Please throw some light ...
Thanks,
pvemuri. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Mon Nov 01, 2004 1:52 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
A single Websphere MQ message can not be larger than 100 megabytes.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| kirani |
| Posted: Mon Nov 01, 2004 4:47 pm Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
Also, think practically. Do you really want to send one very very large message over the channel and have it processed by your application?
_________________
Kiran
IBM Cert. Solution Designer & System Administrator - WBIMB V5
IBM Cert. Solutions Expert - WMQI
IBM Cert. Specialist - WMQI, MQSeries
IBM Cert. Developer - MQSeries
|
|
| Back to top |
|
|
| RocknRambo |
| Posted: Mon Nov 01, 2004 5:10 pm Post subject: |
|
|
Partisan
Joined: 24 Sep 2003
Posts: 355
|
I'm not sure but...does Segmentation work for u.
-RR |
|
| Back to top |
|
|
| basha |
| Posted: Tue Nov 02, 2004 9:27 am Post subject: |
|
|
Apprentice
Joined: 25 Apr 2004
Posts: 31
|
Well,
The scenario is: A large file will come out from the source machine, we need to decide how to handle it.
1. As MQ cannot store that file in its Queue, there is no chance of handling the whole file.
2. We have to have an intermediate application handling that file and segmenting it and sending it to MB.
Correct me if I am wrong.
Thanks,
basha |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Nov 03, 2004 4:53 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Don't forget the option of file transfer between queue managers using reference messages. That way the actual file content does not have to become a message or series of messages.
This sounds like file transfer - not many applications would want to, or be able to consume a single message of that size. Maybe you should look at file transfer products (some of which operate over MQ). |
|
| Back to top |
|
|
| PGoodhart |
| Posted: Wed Nov 03, 2004 5:02 am Post subject: |
|
|
Master
Joined: 17 Jun 2004
Posts: 278
Location: Harrisburg PA
|
I recently spent some serious time looking into this and it became apparent that the best solution was to use a third party file transfer product (over MQ). All products I reviewed just dealt with so many issues that would otherwise have to be carefully implemented that it made sense (stuff like rollback, incomplete files, automatic reconnection, interaction with other technologies).
I do have a flow inhouse that consumes very large messages (300-700mb) (If you can send it over FTP in a resonable time it will go through) this was done with version 5 broker, 5.3 mq, 4 mb messages and segmentation. It seems to be working well under load.
Warmest Regards!
_________________
Patrick Goodhart
MQ Admin/Web Developer/Consultant
WebSphere Application Server Admin |
|
| Back to top |
|
|
| JohnMetcalfe |
| Posted: Wed Nov 03, 2004 7:17 am Post subject: |
|
|
Apprentice
Joined: 02 Apr 2004
Posts: 40
Location: Edinburgh, Scotland
|
For what it is worth, we looked hard at a product from Primeur called WMQI File Extender to handle large files through WMQI.
This product has a file repository that is a WMQI bolt on - you ftp the file into a 'virtual' Q in this repository, this 'Q' then acts as a input Q into WMQI via a Primeur plug-in node.
It looked very interesting (in the demos at least!), although we have not taken it on here (yet). |
|
| Back to top |
|
|
| kirani |
| Posted: Wed Nov 03, 2004 9:37 am Post subject: |
|
|
Jedi Knight
Joined: 05 Sep 2001
Posts: 3779
Location: Torrance, CA, USA
|
John, Do you have a link to their website.
_________________
Kiran
IBM Cert. Solution Designer & System Administrator - WBIMB V5
IBM Cert. Solutions Expert - WMQI
IBM Cert. Specialist - WMQI, MQSeries
IBM Cert. Developer - MQSeries
|
|
| Back to top |
|
|
| PGoodhart |
| Posted: Wed Nov 03, 2004 10:06 am Post subject: |
|
|
Master
Joined: 17 Jun 2004
Posts: 278
Location: Harrisburg PA
|
http://www.primeur.com/
_________________
Patrick Goodhart
MQ Admin/Web Developer/Consultant
WebSphere Application Server Admin |
|
| Back to top |
|
|
| kishoreraju |
| Posted: Wed Nov 03, 2004 10:26 am Post subject: |
|
|
Disciple
Joined: 30 Sep 2004
Posts: 156
|
| We developed a FileInputNode which directly reads the contents form the file and deletes that file after reading the content .I have doubt that Message Broker can Handle 2GB file. |
|
| Back to top |
|
|
| dkeister |
| Posted: Mon Nov 29, 2004 8:03 am Post subject: |
|
|
Disciple
Joined: 25 Mar 2002
Posts: 184
Location: Purchase, New York
|
This may be a little late but we had the same problem. A source file that clearly was larger than the message capacity and we had no control over the sender.
My first solution was to write a file input node that read the file from the landing zone ftp folder and break it into smaller messages and pass them to the remainder of the message flow to process the contents (insert into a DB then process the complete data when all records from the input file were received). The problem with this was the way the broker works, each message generated from the 'file input node' allocated additional memory to the broker process. There was (is) no way of freeing this memory up without stopping/starting the broker so eventually the flow would cause the broker to abend.
My second solution was to take the java code from the file input node and make it a stand alone process. This process checks the landing zone folder periodically and when a transfer is complete, it breaks the file into 'chunks' and puts them into an xml message with an attribute indicating first, middle, and last for the messages generated and put onto a queue. The message flow puts all records in a DB and processes when the last record is received. The first message of a set does initialization. Subsequent messages are put into the DB. The last message causes a compute node to actually process the data and build xml messages that contain transformed data.
It seems like a long way around, but I couldn't figure any other way. By the way, the output records also needed to be put into a number of output messages (i.e.an HR file might contain over 100,000 records and our XML designers LOVE VERBOSE XML even though a computer is reading the results.)
Any thoughts or comments? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Mon Nov 29, 2004 9:20 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
CommerceQuest's ProcessManager For Data is a good solution for this general problem.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
