| Author |
Message
|
| mdares |
 Posted: Wed Jun 07, 2006 8:49 am Post subject: File Transfer over MQ Series |
 |
|
Novice
Joined: 07 Jun 2006
Posts: 14
|
Hello,
I know this general topic has been covered here before, but I still haven't quite found the answer I am looking for. Here is my setup:
We are looking to send a file and some metadata describing the file (date created, some other application specific information) over MQ series to be handled by a generic listener. My original plan was to put all of this information into a seriazable java object, dump it onto the queue and reconstitute on the other side. The problem is that we may now have to allow for other products written in other languages to connect to this queue and drop data on as well.
Our next idea then is to use XML, but it seems like bad practice to dump a file between XML tags and expecting consistent behavior... (not to mention well formed XML).
So is there a best practice for this? I see that there are 3rd party products for file transfer, but we are not looking for anything complex (this is the simplest part of what we are trying to do here). I don't like implementing things without advice with only passing experience.
Sorry if this has been answered before,
Thanks!
Matthew Dares
P.S. - I'm not very far into the documentation yet, so if this is a RTFM I will take my lumps. |
|
| Back to top |
|
 |
| jefflowrey |
| Posted: Wed Jun 07, 2006 8:59 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Depending on the size of the metadata, and the capabilities of the programs that have to receive it - you can use an MQRFH2 header and put all the metadata into the usr folder or as NameValue strings.
IF some of your programs are in COBOL, though, processing an MQRFH2 is going to be ... hard...
In a previous life I have solved this problem by prepending an XML structured metadata document to the message data, with a known separator record. Then most programming languages can use position and substring functions to find the separator record and break the metadata from the real data. And it's easier to handle in COBOL than an MQRFH2.
This also doesn't require the message data to be valid XML.
Another approach is to base64 encode the message data, and insert that into a single field in the XML metadata. This is similar to the SOAP envelope concept.
SOAP itself is a third option.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| markt |
| Posted: Wed Jun 07, 2006 10:00 am Post subject: |
|
|
Knight
Joined: 14 May 2002
Posts: 508
|
While what Jeff says is fine, there are more serious problems to consider.
In particular: how do you deal with files that are larger than 100MB (or whatever you define to be the message max size on queues/channels).
Without a strategy for segmenting, reassembling, discarding "orphaned" segments (because half the messages have been rerouted to a DLQ?), knowing when all segments have arrived ... you are likely to be in trouble.
These are all things that have been designed into "real" file transfer products, and even the transfer component of MQ v6. And they're not simple.
(And that's before you get into other requirements like auditing, logging, security, scheduling ...) |
|
| Back to top |
|
|
| mdares |
| Posted: Wed Jun 07, 2006 10:49 am Post subject: |
|
|
Novice
Joined: 07 Jun 2006
Posts: 14
|
Thanks Jeff,
That helps get me going...
Markt,
I was under the impression that 4MB was the message size limit? Which on a good day I might have pushed. It would be 'impossible' for one of our files to exceed 100MB (or 4 for that matter) unless we have a bigger problem elsewhere in the application being designed. Auditing, logging and security are also handled application side on either end of the queue which we are using solely as a redundant delivery method for small reports.
Thank you both for your input!
Last edited by mdares on Fri Jul 21, 2006 6:09 am; edited 2 times in total |
|
| Back to top |
|
|
| wschutz |
| Posted: Wed Jun 07, 2006 11:01 am Post subject: |
|
|
Jedi Knight
Joined: 02 Jun 2005
Posts: 3316
Location: IBM (retired)
|
| Quote: |
| I was under the impression that 4MB was the message size limit? |
thats was a long time ago in a queue manager far away.... its 100 meg now
_________________
-wayne |
|
| Back to top |
|
|
| mdares |
| Posted: Wed Jun 07, 2006 12:00 pm Post subject: |
|
|
Novice
Joined: 07 Jun 2006
Posts: 14
|
Lol,
Probably like ... 5.3?
(Which happens to be what I am using...)
Thanks! |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jun 07, 2006 12:05 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
No, like MQ 2.x.
V5 qmgrs have always been able to support 100Mb messages.
The default maxmsgln on all objects, though, even in v6, is 4mb.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| mdares |
| Posted: Fri Jul 21, 2006 6:11 am Post subject: |
|
|
Novice
Joined: 07 Jun 2006
Posts: 14
|
| Offhand for reference, I base64 encoding our encrypted zip files and sending them as part of an XML document. This seemd to make the most sense from an interoperability point of view for us. |
|
| Back to top |
|
|
|
|
