| Author |
Message
|
| ymadden |
 Posted: Mon Jan 13, 2014 7:52 pm Post subject: FileInput - multiple FTP locations |
 |
|
Newbie
Joined: 13 Jan 2014
Posts: 5
|
We have a requirement to build a system that will poll multiple FTP sites, all having different login user names and passwords, source directories, polling intervals, file name patterns, etc. The files will also need to be transferred to different local directories. All of the configuration details will be read from a database that has a maintenance UI over top of it.
I'm not sure how to implement this without having to manually deploy a new instance of a flow pattern for each new entry or modification. Is it possible to somehow create multiple instances of a sub-flow or pattern at run-time, each with different configurable service settings?
We're running IIB 9 on Red Hat Linux. |
|
| Back to top |
|
 |
| zpat |
| Posted: Tue Jan 14, 2014 12:49 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
Sounds like a very old-fashioned and error prone design.
Tell them to use MQ messages and databases only!
If you must implement these horrible file based transfers despite all the danger of losing or duplicating data - then why not use FTE?
Bear in mind the true requirement is to integrate the applications - using files is just one (very bad) way to do it.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| ymadden |
| Posted: Tue Jan 14, 2014 4:32 am Post subject: |
|
|
Newbie
Joined: 13 Jan 2014
Posts: 5
|
| This system is for exchanging data with external customers and suppliers and these file-based transfers are our only option for some of them. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jan 14, 2014 4:47 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
How many times have I heard that!!?
Quite often there are options like MQ, or web services if you look beneath the surface. In any case coding a MQ application is very simple.
There were no MQ applications 20 years ago - people have written them for good reasons - why not join the modern world?
File transfer has no transactional integrity. You have to design it into the application to check all the data has arrived, with none missing, in the correct sequence and so on.
I've shown external parties how to use MQ client in the past - very easy and no cost for them to use it.
If you must use files, use a managed file transfer product. You need to ensure integrity, non-repudiation, authentication, authorisation and confidentiality.
SFTP is at least better than FTP (never use this).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| Vitor |
| Posted: Tue Jan 14, 2014 5:54 am Post subject: Re: FileInput - multiple FTP locations |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| ymadden wrote: |
| We have a requirement to build a system that will poll multiple FTP sites |
Why not have these people push the files to a central location, rather than polling multiple locations?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| ymadden |
| Posted: Tue Jan 14, 2014 7:25 am Post subject: Re: FileInput - multiple FTP locations |
|
|
Newbie
Joined: 13 Jan 2014
Posts: 5
|
| Vitor wrote: |
| Why not have these people push the files to a central location, rather than polling multiple locations? |
We have several trading partners that do push data files to us and I believe we have a good solution for this type of transfer. But we still have numerous companies and foreign-government agencies from which we must pull the data. I think we can use the built-in FtpServer configurable service, but I can't figure out how to get multiple FileInput nodes activated and responding to different configurable services.
I'm beginning to think that we might not be able to use the FileInput node for the FTP part. We may be able to implement some custom solution using a JavaCompute node, as shown here - http://www.ibm.com/developerworks/websphere/library/techarticles/1105_subrahmanyam/1105_subrahmanyam.html |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Jan 14, 2014 7:31 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
This is a managed file transfer solution.
Broker is not a tool for building or running managed file transfers.
The inbuilt broker File nodes are very poor at handling the complex solutions required for dealing with external business partners.
Your best solution is to build all of the communications involving file movement into a separate application - either one that you write yourself or one that you buy (like Connect:Direct or MQ MFT solution or etc. etc. etc. ) and then use that to deliver data in a sensible managed way for Broker to process it.
If you really want to host this solution in Broker, then as you say you have to put a bunch of complicated logic into a JavaCompute or .NETCompute or PHPCompute node, or write a custom input node that handles files better.
But again, it's much better to separate the file movement logic from the data transformation/integration logic. |
|
| Back to top |
|
|
| zpat |
| Posted: Tue Jan 14, 2014 8:05 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
SFTP should, at the very least, be mandated, and use SSH keys to authenticate the connection, not passwords.
In terms of how to code such a design in WMB (no matter how inadvisable) you can set the file node properties in ESQL environment variables these days (use the latest fix packs as I believe some environment options such as the FTP servername were added in PTFs).
You could drive the file input node multiple times using a propagate statement inside a ESQL loop in a compute node, having set the appropriate values in the environment variables.
Presumably if the servername set matched a configurable service name, it would pick up the actual details from that.
Seems easy enough to do, providing you can set the filenode properties that you need in ESQL. As I said there have been some enhancements in this respect so you may need a certain version or level of WMB to do this (and enable fixpack function with -f all).
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| hotshot |
| Posted: Thu Jan 16, 2014 3:15 pm Post subject: |
|
|
Novice
Joined: 04 Jan 2013
Posts: 19
|
Hi zpat,
Did I misunderstand your statement here:
| Quote: |
You could drive the file input node multiple times using a propagate statement inside a ESQL loop in a compute node, having set the appropriate values in the environment variables.
|
Do you actually mean
| Quote: |
| the file read node |
As far as I know, the FileRead node does not support reading from a remote location, unless this nice RFE: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=22098 has been implemented by IBM in version 9, I have only used it in v8.
As for the OP's question, I agree with most of the replies already given: if your project is about integrating new partners, you will probably find little you can not do these days with other technologies, for instance web services (even transferring considerable volumes).
If instead you need a solution for existing partners which are already hooked to file integration, you need to research a managed file transfer solution. |
|
| Back to top |
|
|
| Esa |
| Posted: Thu Jan 16, 2014 11:15 pm Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
| hotshot wrote: |
Hi zpat,
Did I misunderstand your statement here:
| Quote: |
You could drive the file input node multiple times using a propagate statement inside a ESQL loop in a compute node, having set the appropriate values in the environment variables.
|
Do you actually mean
| Quote: |
| the file read node |
|
Yes, he must mean that.
But making a flow programatically poll a remote location is bad design and may bring a few problems.
First of all, message flows are designed to run as short-lived transactional micro-flows.
Implementing a listener flow violates that principle.
Short-lived flow instances forgives most programming and design shortcomings so that the developers don't in general need to worry about memory handling, releasing parsers etc.
But even the tiniest mistake that causes no trouble with our everyday message transformation flows may mercilessly crash the execution group if the flow loops for a longer time.
You could -- and would have to, for cutting transactions shorter -- implement a listener flow that uses timer nodes to periodically offload itself.
But implementing that properly requires excellent Message Broker skills, too and will end up in quite a complex design in any case.
Who knows, maybe this is one of the reasons why File Read node doesn't support remote locations?
| ymadden wrote: |
I'm not sure how to implement this without having to manually deploy a new instance of a flow pattern for each new entry or modification. |
You can implement it by automating the deployment process. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jan 17, 2014 3:29 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
I must have meant File Read node. 
A more devious design could programmatically build and deploy a variable number of flows automatically and therefore use fileinput nodes.
There could be several ways to do this, from duplicating the CMF in XML format to using the new IIB (WMB v9) IBM Java Integration API.
An API to actually code applications?- seems a fairly strange thing for IBM to provide - but I guess this would be one use for it.
http://pic.dhe.ibm.com/infocenter/wmbhelp/v9r0m0/topic/com.ibm.etools.mft.doc/be43410_.htm
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| ymadden |
| Posted: Fri Jan 17, 2014 2:28 pm Post subject: |
|
|
Newbie
Joined: 13 Jan 2014
Posts: 5
|
Thanks for all of the input guys. I opened a PMR asking this same question and was told to contact IBM services through our rep, so I have now sent the same questions to our lab advocates (that I just discovered were assigned to us).
Currently, we are planning to use the FileRead node to monitor the directories for partners that push files to us.
| Esa wrote: |
| You can implement it by automating the deployment process |
I had considered this, even so far as to how we might issue mqsi* commands to the OS from a separate broker flow. There is a support pac that adds this functionality. That seems dangerous though but is a possible solution.
| Esa wrote: |
| implement a listener flow that uses timer nodes to periodically offload itself. |
This is sort of our current plan... To have a Timeout controlled flow periodically check our configuration table for new or modified records and offload the actual file transfer. This is the tricky part. We are pursuing having a second flow that takes messages from the first job and use a JavaCompute to do the actual file transfer. We are also looking at having a subflow with just an Input node and also a FileInput node (having the FileInput alone does not expose an input terminal to the main flow). I'm not sure how the FileInput node will behave in a subflow, or if it will even become activated. This too would require issuing mqsi*configurableservice commands to set the FTPService configurable service properties. Lots of options... none of them optimal... |
|
| Back to top |
|
|
| Esa |
| Posted: Sat Jan 18, 2014 12:38 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Keep it simple. That's the tricky part!
| ymadden wrote: |
| deploy a new instance of a flow pattern for each new entry or modification |
I assumed -- probably zpat and Vitor, too -- that your processing flow starts with an MQInput node and you plug little FileInput --> MQ Output flows in front of it as file adapters, one for each file input source.
But having read your posts I must ask you if you really deploy a new instance if the whole processing flow for every input source?
| ymadden wrote: |
| I had considered this, even so far as to how we might issue mqsi* commands to the OS from a separate broker flow. There is a support pac that adds this functionality. That seems dangerous though but is a possible solution. |
Are you planning to implement deployment automation with message flows that would run mqsi commands? I hope I misunderstood this...
A script that reads the configurations of all your ftp input sources from an XML file, overrides and deploys the bar file for each entry, ascript that you run every time you add or change a file source would be a much better alternative. If you cannot or are not allowed to run mqsi command on the production server, you can use CMP java API (called IBM Integration API in V9) to create a little java program that runs all this remotely. The same thing that zpat suggested a couple of replies back. The fact that you run IIB 9 makes the task a bit more complicated, you cannot just deploy more flows but you must deploy flows within applications. You may actually need generate the flows programmatically, as zpat said, because it seems to be impossible to just copy and paste flows within an application packaged in a bar file... |
|
| Back to top |
|
|
| zpat |
| Posted: Sat Jan 18, 2014 5:26 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5849
Location: UK
|
I think the IIB integration API has been enhanced to allow the dynamic/programmatic development/construction (as well as deployment) of message flows.
I certainly recall this from Hursley presentations as being a new feature.
_________________
Well, I don't think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Mon Jan 20, 2014 8:29 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
I think the IIB integration API has been enhanced to allow the dynamic/programmatic development/construction (as well as deployment) of message flows.
I certainly recall this from Hursley presentations as being a new feature. |
Yes, there is a set of Java APIS that let you do this kind of thing, in part packaged with the Pattern stuff.
But again, the file capabilities in broker are designed exclusively for working with files that are well-behaved and more or less "local".
Most of what the op is talking about here is NEITHER of those things. Again, a managed file transfer solution is the BETTER choice. |
|
| Back to top |
|
|
|
|
