| Author |
Message
|
| sunny_30 |
 Posted: Mon Aug 20, 2007 5:34 pm Post subject: File transfer via the Hub. |
 |
|
Master
Joined: 03 Oct 2005
Posts: 258
|
Hi,
Need your expert opinions...
Here is the requirement:
Fast, secure & reliable transfer of files to single/multiple destinations.
Folder-to-folder(s) transfer between remote systems.
Eliminate the current point-to-point FTP/SCP implementation.
Implement a centralized hub in the middle.
File-size can vary from 0 to unlimited.
Will involve disparate environments (Windows/Unix).
Data transformation may/may-not be required.
The initial plan we are thinking to follow:
Surround servers will have MQ-Server/client installed.
Message-Broker is used to implement the Hub over MQ.
Route the files looking at the RFH2 header, say the source, file-name etc.
Database will be used to determine the destinations.
Is this a good idea to use MQ & Broker combination for the file-transfers?
If yes, what connector product is recommended on the end servers to communicate with MQ/Broker:
Message-Broker file-extender?
PM4data?
JText adapter?
or any other?
If not, what possible hurdles might be seen?
What are the other products/options to explore?
Thanks in advance,
Sunny. |
|
| Back to top |
|
 |
| Vitor |
| Posted: Mon Aug 20, 2007 11:09 pm Post subject: Re: File transfer via the Hub. |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sunny_30 wrote: |
If yes, what connector product is recommended on the end servers to communicate with MQ/Broker:
Message-Broker file-extender?
PM4data?
JText adapter?
or any other?
|
Yes. 
It's horses for courses, or software for requirements if you prefer. I like PM4Data but that's a personal preference and it does (like the other solutions) have weaknesses as well as strengths.
File transfer is a common topic in here and the search button will be your friend, but in the last analysis only your software evaluation & proof of concept will answer the question for you.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| manojsu |
| Posted: Tue Aug 21, 2007 1:12 am Post subject: |
|
|
Centurion
Joined: 19 Jul 2006
Posts: 147
Location: Bangalore
|
Sunny,
For transfer of data across disparate systems you can use simple java programs to move across of file or regular schedulers. Here in the encoding is not needed to be maintained as its going to be a ftp that is going to be done, for which you can use the WAS.
Regards
Manoj |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 21, 2007 1:20 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| manojsu wrote: |
For transfer of data across disparate systems you can use simple java programs to move across of file or regular schedulers. Here in the encoding is not needed to be maintained as its going to be a ftp that is going to be done, for which you can use the WAS.
|
This is of course true.
I think the poster's asking about file transfer leveraging the MQ infrastructure, which gives benefits over plain ftp. Which of course does still work.
Also writing java code to move files over MQ is a) reinventing the wheel and b) more complex than it first appears. It's this last point that makes selling solutions both viable & profitable.
But as I said above it all depends on the requirements, the complexity of your platform base, size of your files, etc, etc. There's no "right" answer.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Tue Aug 21, 2007 1:45 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
And let's not forget your audit requirements... 
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Aug 21, 2007 3:46 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| fjb_saper wrote: |
| And let's not forget your audit requirements... |
And security requirements.
And non-repudiation requirements.
And... and... and....
Start with the requirements. Then look at the budget.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 21, 2007 3:50 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jefflowrey wrote: |
| Then look at the budget. |
That most prized part of the software evaluation. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Aug 21, 2007 3:53 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| Vitor wrote: |
| jefflowrey wrote: |
| Then look at the budget. |
That most prized part of the software evaluation. |
That's why you look at the requirements FIRST.
If the proposed budget doesn't fit anything, then it's clear a larger budget is needed. Or the requirements weren't as "required" as was first thought.
But there's no sense spending any time on a POC or etc. for a $100,000 product when the budget is only $20,000 and the requirements are met by a $15,000 product.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Tue Aug 21, 2007 3:03 pm Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
Thanks for your replies.
Does any body know whats the max-size of the file that can be transferred using PM4data & also MB file-extender?
Just wondering why IBM hasnt built a robust 'file' transfer mechanism inside the Message-Broker to support the Folder-to-Folder transfer. It does the 'message' transfer from queue-to-queue so brilliantly, but not the 'files'. I think File transfers with data-transformation option using the MB will be a lot lot helpful for any client which results in speed & also secure transfers.
May be this strategy has something to do with making all the surround-apps to become more dependant on the MQ-Infrastructure by mq-puts & gets of messages. If its Folder-to-Folder, the applications can grow independent of the middleware leaving scope for the MQ replacement.
So as to achieve the 'Enterprise level file transfer', it is leaving the MB users with no option other than to buy an extra product that adapts with the Broker to do the Folder-to-Queue or vice-versa.
Am I thinking right..
--Sunny. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Aug 21, 2007 11:53 pm Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| sunny_30 wrote: |
| Just wondering why IBM hasnt built a robust 'file' transfer mechanism inside the Message-Broker to support the Folder-to-Folder transfer. |
First off - I am not now nor have I ever been associated with IBM's R&D, marketing, strategy or other areas. Other, more informed opinions on why they've done what they've done welcomed.
IBM bought PM4Data to fill the gap in MQ around file transfer over MQ missing from the base product. If all you're doing is folder to folder without transformation then you don't want the overhead of WMB.
As to max size, I don't know what the current limits are but they used to be fairly impressive. There's bound to be a web page some place.
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Aug 22, 2007 1:23 am Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
I don't think IBM does own PM4Data.
I believe they merely resell it.
I also don't think sunny_30 is "thinking right".
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| Vitor |
| Posted: Wed Aug 22, 2007 1:33 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| jefflowrey wrote: |
I don't think IBM does own PM4Data.
|
Thought MetaStorm had sold the rights to PM4Data. Shows what I know. 
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Aug 22, 2007 2:01 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Not forgetting reference messages that will move files for you between queue manager platforms using nothing more than base MQ.
Why not eliminate the files all together? Usually they are only used instead of a proper messaging design direct from the application.
You can also code a simple file to message program (and vice-versa) using REXX or COBOL or Java etc. Buying a product is expensive and only makes sense for a strategic implementation. |
|
| Back to top |
|
|
| fjb_saper |
| Posted: Wed Aug 22, 2007 2:16 am Post subject: |
|
|
Grand High Poobah
Joined: 18 Nov 2003
Posts: 20756
Location: LI,NY
|
| Quote: |
| May be this strategy has something to do with making all the surround-apps to become more dependant on the MQ-Infrastructure by mq-puts & gets of messages. If its Folder-to-Folder, the applications can grow independent of the middleware leaving scope for the MQ replacement. |
Maybe this strategy has to do with going towards a more responsive on line type of processing, (messaging system) while still catering for legacy and batch applications (file transfer).
Enjoy
_________________
MQ & Broker admin |
|
| Back to top |
|
|
| sunny_30 |
| Posted: Wed Aug 22, 2007 7:55 am Post subject: |
|
|
Master
Joined: 03 Oct 2005
Posts: 258
|
Certainly messaging-system has many advantages over the file-transfer system. My point was not to compare these both mechanisms to see which one is better. Im just trying to know why MB doesnt support 'files' transfer that well. Now I see its probably for all the great advantages 'messaging' provides...
Zpat - Yes. Its always better to eliminate the file transfers. But the organization I work at doesnt look to believe so. What their argument is, Im not sure & I cant make these decisions here. Their plan now is to eliminate the current Ftp/scp implementations, substitute that with MQ/MB to do the transfers. The implementation is estimated to take atleast 6 or 8 months, the system built will be running for 2 or 3 more years & ultimately SAP-XI is going to replace the whole MQ/MB file-transfering architecture that we build now.
Now, the idea is to integrate majority SAP systems around & move files upto 30 GB or more! Considering the requirements we have, still there is thinking going on whether to buy a product or build a own C++ connector to do the F2Q job.
Vitor, we already have MQ & MB being used for other implementations here & so Im free to use this product for the 'file-transfering' system too. I agree MB will only need to be used if transformation is required, instead just a sender & receiver PM4data pair can do all the work for us. We are planning to use the MB to determine the Routing logic looking at RFH2 & also coz we are unsure at this point that we wont be needing any transformation in future. MB can also be used if the Input file's 'mq-messages' will need to be routed to a queue rather than another 'file' destination.
Can I ask a different question. Do you think its a good plan to use SAP-XI for replacing the infrastructure that is built over MQ/MB. These decisions are already made & infact I will have to wait & see if I will also get replaced...
What does this product do better over MQ/MB? It probably does integrate SAP systems very well. But wdnt the job be done by a MB + SAP-adapter?
Thanks,
Sunny. |
|
| Back to top |
|
|
|
|
