| Author |
Message
|
| Neha |
 Posted: Thu Oct 19, 2006 12:56 pm Post subject: Sending 1 GB file from MQ client to MQ server...!!! |
 |
|
Novice
Joined: 19 Oct 2006
Posts: 13
|
Hello all,
I fairly new to MQ. I've been going thru some documentations and able to do some message transfer. i.e. MQ Server to server and MQ Client to server with in my own PC/Local Host.
I have good number of doubts as every new-comer should have 
I hope I am not the only one.
Well I could send and recieve msg thru utilities like
amqsput, amqsputc , amqsget and amqsgetc.
But most of the data is in files so how come MQ handles these files.
I have on my desktop(WIN XP MQ V 6.0) hard drive 1 GB file and I want to transfer from MQ Client to MQ Server(with in the same local host) and save it on another location.(And server to server trasfer also)
Could you help me in that. i tried to search for similar problem in this forum but didn't find it. If already posted by someone else, could you pls forward the link.
Thanks in advance. |
|
| Back to top |
|
 |
| fjb_saper |
| Posted: Thu Oct 19, 2006 2:01 pm Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
Well MQ has some limits. The default for a connection, channel, DLQ and such is 4MB?
The max used to be 100MB but it might have changed with 6.0.
Anyway for anything of that size (1 GB) I would advise you to chop it up into little chunks and reassemble at the other end...
Enjoy 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Thu Oct 19, 2006 4:05 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
MAXMSGLN is unchanged in v6 vs v5.3, the highest allowable value is 104857600.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Neha |
| Posted: Thu Oct 19, 2006 4:38 pm Post subject: |
|
|
Novice
Joined: 19 Oct 2006
Posts: 13
|
Thanks fjb_saper and Jeff.
So according to you I should break data and feed in the queue and re-assemble on recipient qmgr.
So that's the key.
B'cos I've seen various posts by people working in real time saying they have default value of queue and every thing. And since in corporation they might be dealing with lot of data with MQ, so do they break them and move data thru MQ and reassemble on recipient Qmgr?
If so how do they do that?? Also, how do they manage to break a file in say MPEG format?? |
|
| Back to top |
|
|
| kevinf2349 |
| Posted: Thu Oct 19, 2006 4:43 pm Post subject: |
|
|
Grand Master
Joined: 28 Feb 2003
Posts: 1311
Location: USA
|
The thing is there are a couple of options open to you.
1) Use reference messages to transfer the file.
2) Segment the large file into chunks and reconstruct at the other end.
....however, such large amounts of data (especially files) are probably better off using a recognised FTP product.
Search this site for FTP and you should find a wealth of disussion. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Oct 20, 2006 5:06 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
One thing that companies implementing MQ tend to do is convert from a batch/file oriented process to a message oriented process.
What that means is that where before a program would produce a whole batch of results, for example a file containing all of the orders that were entered today. Now the program will produce one message for each order.
So a lot of times, the need to move large files of data simply goes away.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Neha |
| Posted: Fri Oct 20, 2006 6:54 am Post subject: |
|
|
Novice
Joined: 19 Oct 2006
Posts: 13
|
Hi Kevin -
| Quote: |
| 1) Use reference messages to transfer the file. |
Could you pls shed little more light over this!!
I am learning MQ and sending a file isn't a need at this moment. I was just wondering how to do it leveraging MQ. I am also interested in your second comment:
| Quote: |
| 2) Segment the large file into chunks and reconstruct at the other end. |
How should I do that??
So Jeff, I now understand (I guess  ) going thru MQ documentation - sys admin guide, where no word "File" but a Message is mentioned. So I guess everything is msg.
| Quote: |
| What that means is that where before a program would produce a whole batch of results, for example a file containing all of the orders that were entered today. Now the program will produce one message for each order. |
Since I don't have real time experience in MQ, I'll try to simulate my imagination into practical situation ( I COULD BE TOTALLY WRONG)
Suppose I am getting messages into the server say NOT thru MQ, I don't know how,either since I never dealt with the program generating msg.
Now, I am either feeding those incoming msg in data base tables and then from there thru SQL queries I am putting those msg onto a MQ queue ( Assuming MQ is installed in this server) and moving it to remote qmgr.
Other scenario: I am getting Messages and feeding them onto a queue to transport it to remote qmgr.
Now, question is:
How would I tackle a message instead of a file!!! I've so far dealt FTP etc and so idea of Message is foreign to me.
A similar example that I could think of is: I type contents on an email and hit SEND. That's a msg since I didn't save it in a file and then send it. So is MQ like that!! If so, a program (which in my case is a person) should be reading incoming messages and copy those contents on the queue.
If so far my understanding is right, where I can get info about designing such program??
A suggestion is made: Break the data and then reassemble it.
So would I be breaking a file into messages; send it and on recieving end convert msg back to file or straight store messages into data base. If so, where can I get more info about desinging such program.
If incoming data is stored in database, (I have MS Access and db2 to experiment), fetching a data thru SQL in form of messages, needs a program. Where can I get more info about program??
Thanks guys in advance...
Last edited by Neha on Fri Oct 20, 2006 7:55 am; edited 1 time in total |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Oct 20, 2006 7:50 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Look, just send say 1MB per message, using GROUPid messages.
It's easy, see the Application Programming Guide.
The sending end indicates the start of a "group" and the last message in the group. The receiving end will be given each message in turn for that group until told that the last message has been retrieved.
Easy. What's all the fuss about?
If you look around you will even find getgroup.c and putgroup.c sample source code from IBM (there is also a Java version).
http://www-304.ibm.com/jct09002c/isv/tech/sampmq.html |
|
| Back to top |
|
|
| Neha |
| Posted: Fri Oct 20, 2006 11:58 am Post subject: |
|
|
Novice
Joined: 19 Oct 2006
Posts: 13
|
Thanks. I'll try to apply your suggestion. I pretty much think that I got the answer but if in doubt I'll bug you guys again.
Thanks guys for your suggestion.
Neha. |
|
| Back to top |
|
|
| ashoon |
| Posted: Sat Oct 21, 2006 9:44 am Post subject: why not use pm4data |
|
|
Master
Joined: 26 Oct 2004
Posts: 235
|
| as it does file transfer over MQ and can also do file-db as well via odbc. |
|
| Back to top |
|
|
| Neha |
| Posted: Wed Nov 29, 2006 7:45 am Post subject: |
|
|
Novice
Joined: 19 Oct 2006
Posts: 13
|
I could find a putgroup.c program in the above link. But couldn't find a similar program in java.
Is there a sample put program in java also! |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Nov 29, 2006 7:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
|
| Back to top |
|
|
| Neha |
| Posted: Wed Nov 29, 2006 8:07 am Post subject: |
|
|
Novice
Joined: 19 Oct 2006
Posts: 13
|
That was fast.
thanks - zpat.
I'll look into it. Thanks again. |
|
| Back to top |
|
|
| bruce2359 |
| Posted: Wed Nov 29, 2006 11:52 am Post subject: |
|
|
Guest
|
V6 includes an MQ File Transfer Application for doing the FTP-like thing.
Programaticaly: the RFH (Reference Message Header) method is still supported; breaking a file into 100meg (or less) pieces is yet another method. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Nov 29, 2006 12:15 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The v6 File Transfer Application is not production ready.
It's a fine tool for what it is, but it provides no security at all, and does not generally meet most of the requirements for enterprise file transfer.
Both markt and I have ranted here in various posts about the problem of enterprise file transfer using MQ. This is a frequently discussed topic.
And again, when designing for MQ, it is vastly better to design each message to hold a single unit of work, rather than many units of work. Most files are designed to hold many units of work, and so the change in design principle usually eliminates a lot of the need to do enterprise file transfer.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
|
|
