| |
|
RSS Feed - WebSphere MQ Support
|
RSS Feed - Message Broker Support
|
  |
|
| MQ file transfer |
« View previous topic :: View next topic » |
| Author |
Message
|
| rk400 |
 Posted: Fri Mar 21, 2003 8:39 am Post subject: MQ file transfer |
 |
|
Apprentice
Joined: 04 Mar 2003
Posts: 27
|
Hi,
I have MQ client on machine1 and MQ server(5.2) installed on Machine2.
I am asked to transfer files of size max 2MB using MQ. We are reading the entire file in ASCII mode in java and forming a MQ message and putting it in the queue manager of machine2. As the file size is 2MB, my application is running out of resources many times.
Is there a better way of tranferring the large files thru MQ???
thanx
RK |
|
| Back to top |
|
 |
| bbeardsley |
| Posted: Fri Mar 21, 2003 8:58 am Post subject: |
|
|
Acolyte
Joined: 17 Dec 2001
Posts: 52
Location: Dallas, TX, USA
|
RK,
It sounds like the problem is with the level of machine resources available, not necessarily MQ since I believe it supports msgs up to 4 MB in version 5.2. Is your maximum message size set >2MB?
If you could post the error you are getting, I'm sure someone here could help you with a solution. |
|
| Back to top |
|
|
| rk400 |
| Posted: Fri Mar 21, 2003 9:08 am Post subject: |
|
|
Apprentice
Joined: 04 Mar 2003
Posts: 27
|
thanx for the reply bbeardsley...
yes..u r right. I am not getting any problem with the MQ...
my question is
"Is there is a better way of tranferring the files from MQ client to server than forming a message of 2 MB" |
|
| Back to top |
|
|
| JohnMN |
| Posted: Fri Mar 21, 2003 9:42 am Post subject: |
|
|
Novice
Joined: 26 Feb 2003
Posts: 19
|
You can read the file in smaller pieces, and send each piece as a separate message. It's more work, because you need to keep track of how many pieces you are sending.
You also should check that every piece has arrived in the correct order. (In some strange cases, it is possible for a message to land on the DLQ, but for later messages to arrive on the target queue.)
But I don't think that will solve your problem.
In the first place, this will result in a higher total use of system resources. Each MQ message has an MQMD, MQPMO, etc. By sending multiple messages, you'll increase the number of bytes of data sent.
Instead, you need to find out why you are running out of resources.
Does your program send more than one file each time it executes? Does it fail on the first file sent, or only after several files have been sent? Are you relying on the Java garbage collector to release your buffers? You may need to reuse the buffer, or explicitly return it. (I'm not a Java guru, so I'm not sure of the exact terminology.) Are there a lot of other programs running at the same time?
If you get the error when you try to send the first message, then you should probably look into upgrading your machine.
There is feature of MQ called "Reference Messages". With these, you can send a file by telling MQ the name of the file. MQ will then handle all of the file operations.
But these are more difficult to write. You will need to write an exit program, which I think needs to be written in C or C++ (cannot be in Java). And it requires help (and approval) from your MQ Administrator.
John |
|
| Back to top |
|
|
| rk400 |
| Posted: Wed Mar 26, 2003 1:58 am Post subject: |
|
|
Apprentice
Joined: 04 Mar 2003
Posts: 27
|
Dear John,
The file I am talking about is of 2MB size and it has to travel thru atleast 5 mq servers before actually reaching the destination. We are encrypting the message with our algorithms,which makes it 2.5MB. On all the servers we have IIS,Tomcat,Oracle,MQserver installed. At each node I have to store the message in database before forwarding it to the next node.
Processing 2.5MB request at a time will definitely hamper the performance of the other applications during its processing time. If I break the message in to segmets and join them at the destination will take time but it will not affect the other applications performance.
Can we use MQ message segments for this???
RK |
|
| Back to top |
|
|
| JohnMN |
| Posted: Wed Mar 26, 2003 8:20 am Post subject: |
|
|
Novice
Joined: 26 Feb 2003
Posts: 19
|
Yes, you can use segmented messages, as long as none of the queue managers is z/OS (OS/390, MVS). This will provide some relief.
But I still think MQ is only a small part of the problem. There is a lot more than MQ involved in the process. Eventually, you will need to look at these other areas also.
John |
|
| Back to top |
|
|
| bduncan |
| Posted: Thu Mar 27, 2003 1:11 am Post subject: |
|
|
Padawan
Joined: 11 Apr 2001
Posts: 1554
Location: Silicon Valley
|
I had success in the past transferring large files via MQ by using C (Java is definitely not the optimal language for this application for several reasons) and rather than dealing with ASCII or loading the whole file into memory, etc., I would simply read the file byte by byte. After reading 16k, I would package it into a message (which was part of a logical group) and send it on its merry way. When I read the last chunk of the file, I would mark the message as the last in the group and then I'm done. On the receiving side, I would use a filestream to write out the file, again, byte by byte in 16k chunks. Because of this, my application never used more than 16k of memory for the file buffer.
_________________
Brandon Duncan
IBM Certified MQSeries Specialist
MQSeries.net forum moderator |
|
| Back to top |
|
|
|
|
|
| |
|
Page 1 of 1 |
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
|
|
