| Author |
Message
|
| edmondh |
 Posted: Wed Jan 10, 2007 7:54 am Post subject: Replacing FTP with MQ |
 |
|
Novice
Joined: 17 Feb 2006
Posts: 14
Location: London
|
Hi
I am trying to write a little application using MQ to replace FTP.
Currently I am making a MQ Client connection and putting my messages (&/OR groups of message) on a queue. Similarily, in parallel, my receiver application is reading messages and writing to destination file via MQ Client connection.
My problem is that I what to optimise my solution to work as well\fast as possible. I have enabled MQ Compression (I am using MQ6) to reduce message size over network, but is there anymore optimizations I could use?
Rgds
Ed
_________________
Thank you for your time
Ed |
|
| Back to top |
|
 |
| pathipati |
| Posted: Wed Jan 10, 2007 8:04 am Post subject: |
|
|
Master
Joined: 03 Mar 2006
Posts: 296
|
| message persistence, MQPUT1 call(if you have a single message to put on a queue). |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Jan 10, 2007 8:08 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
The biggest optimization you can make is to connect as infrequently as possible.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Jan 10, 2007 8:13 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| pathipati wrote: |
| MQPUT1 call(if you have a single message to put on a queue). |
If you're putting groups of messages, it's far more efficient to do MQOPEN / MQPUT / MQPUT / ... / MQPUT / MQCLOSE than MQPUT1 which does MQOPEN / MQPUT / MQCLOSE under the covers.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jsware |
| Posted: Fri Jan 12, 2007 1:55 am Post subject: |
|
|
Chevalier
Joined: 17 May 2001
Posts: 455
|
If you're sending a large file, send it as multiple messages. Every nth message write where you are to a "state" queue and then commit. Then read the state queue under synpoint and put the next n messages. When you get to 2n put a new state message and commit.
If your program crashes, when it restarts it can use the state queue to see how far it got and continue one. If it crashes mid transaction, the state queue will be rolled back to the beginning of the chunk and the part chunk gets rolled back too.
On the receiving side, do the reverse also with a state queue.
This means that MQ can start delivering the beginning of the file before you have put the last chunk. Your receiving program can start building the file before it has finished being produced. Your state queues will maintain progress for you and you'll know when the file has been delivered completely.
Have a look in the Application Programming Guide under "Putting and getting a group that spans units of work"
You can even get more cleverer (poor english intended  ) and have multiple files all streamed together across the same queue if you use something in the correlation ID of your messages/state queue to relate the messages to a file (a separate correl ID for each file).
_________________
Regards
John
The pain of low quaility far outlasts the joy of low price. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Jan 12, 2007 2:30 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| scottj2512 wrote: |
| You can even get more cleverer (poor english intended ) and have multiple files all streamed together across the same queue if you use something in the correlation ID of your messages/state queue to relate the messages to a file (a separate correl ID for each file). |
Please don't use transport headers to hold business information.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| ashoon |
| Posted: Fri Jan 12, 2007 6:10 am Post subject: buy pm4data |
|
|
Master
Joined: 26 Oct 2004
Posts: 235
|
| instead of building this why not just buy pm4data as it does everything you want to do?!?!? |
|
| Back to top |
|
|
| edmondh |
| Posted: Fri Jan 12, 2007 6:48 am Post subject: Replacing FTP with MQ |
|
|
Novice
Joined: 17 Feb 2006
Posts: 14
Location: London
|
Thanks guys for the feed back, the conversations is now reaching where I wanted it. Namely a high level discussion on potential solutions.
Thanks in particular to Scott. My own solution is not too different from yours. However I have another little requirement. I may want my data messages to be persistent or at least guarantee delievery.
So, I have sending and receiving applications. But before sender starts processing a file, sender sends simple 'handshake request'.
The idea of handshake request is essentially to ask receiver is it running and available to read data message from queue.
The receiver receives handshake request and replies with 'handshake reply'.
Once sender receives 'handshake reply', sender processes file.
When sender is finished processing file, a final series of handshake message are sent confirm complete file has been sent & receiver confirm it has processed all messages.
I use correlation id between sender and receiver to reconcile that correct messages are being processed.
I hope this isn't too much info. But does anyone see galring problems with this approach?
Am I mad to try and reinvent wheel?
I do have other proprietry requirment , like send thro broker & RFH2 headers etc etc, & that is why I am taking this approach.
Cheers
Ed
_________________
Thank you for your time
Ed |
|
| Back to top |
|
|
| exerk |
| Posted: Fri Jan 12, 2007 6:52 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| Quote: |
| But does anyone see glaring problems with this approach? |
Is that apart from using WMQ in a psuedo-synchronous mode as opposed to the asynchronous beastie it's designed to be? 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Jan 12, 2007 6:53 am Post subject: Re: Replacing FTP with MQ |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| edmondh wrote: |
So, I have sending and receiving applications. But before sender starts processing a file, sender sends simple 'handshake request'.
The idea of handshake request is essentially to ask receiver is it running and available to read data message from queue.
The receiver receives handshake request and replies with 'handshake reply'.
Once sender receives 'handshake reply', sender processes file.
When sender is finished processing file, a final series of handshake message are sent confirm complete file has been sent & receiver confirm it has processed all messages.
I use correlation id between sender and receiver to reconcile that correct messages are being processed.
I hope this isn't too much info. But does anyone see galring problems with this approach? |
Yes.
Don't do it this way.
Send the data when the sender is ready.
Receive the data when the receiver is ready.
Do not make the two dependant on each other.
You're not mad to reinvent the wheel.
It's just a much more complicated wheel than you think it is.
You probably don't even have a full set of requirements. You may, possibly, have a full set of functional requirements - but you're almost certainly missing a number of non-functional requirements like security, non-repudiation, replay, and etc that go into building a proper Enterprise File Transfer solution.
Talk to your IBM Sales representative about PM4Data.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| edmondh |
| Posted: Fri Jan 12, 2007 8:52 am Post subject: Re: Replacing FTP with MQ |
|
|
Novice
Joined: 17 Feb 2006
Posts: 14
Location: London
|
| jefflowrey wrote: |
Yes.
Don't do it this way.
Send the data when the sender is ready.
Receive the data when the receiver is ready.
Do not make the two dependant on each other.
|
Ok I agree, I am making process synchronous. Normally I will be sending data 'normally' in asynchrous manner but a special case I need to process synchronously. Imagine scenario where messages are too big for queue\queueManager, logs overload etc
Do u think it would work though. Where do u see complications?
Assuming both sender & receiver work reliably as expected?
| jefflowrey wrote: |
You probably don't even have a full set of requirements. You may, possibly, have a full set of functional requirements - but you're almost certainly missing a number of non-functional requirements like security, non-repudiation, replay, and etc that go into building a proper Enterprise File Transfer solution.
|
Yes, I am sure there are some aspects I have overlooked.....
| jefflowrey wrote: |
Talk to your IBM Sales representative about PM4Data. |
PM4Data not suitable, too expensive & too complicated.
I would need many licenses & too much maintainence.
_________________
Thank you for your time
Ed |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Jan 12, 2007 8:58 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
You don't know that PM4Data is too expensive and complicated until you know how much it costs to write and maintain yourself.
You don't know that until you know the requirements.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Fri Jan 12, 2007 6:31 pm Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| I find this amusing. Why is that everyone is so quick to recommend buying a product? I developed an MQ FTP replacement back in the mid-90's and have worked on improving since then. My client has been using it since the mid-90's and it certainly meets their needs. If you have the inclination, time and skill, why not work on it? |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Fri Jan 12, 2007 7:40 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
So you've been constantly improving your initial release for more than 10 years, and you still have things to make better...
And you wonder why people want to buy a product?
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Sun Jan 14, 2007 8:47 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| I certainly don't wonder why most users choose to purchase a product. But, in some (rare) circumstances, the user may choose to develop their own solution. In my particular case, the user has developed several of their own solutions, e.g. development environment/tools, automated job scheduler, etc. BTW, I choose to continually improve the MQ FTP replacement, not because the user wants a new feature, but because I see how some thing could be done better, and I have the time to incdulge myself. I don't see anything wrong with someone trying to learn something, even if there's a commercial alternative out there. If the person has the time and inclination, then why not? |
|
| Back to top |
|
|
|
|
