| Author |
Message
|
| EternalLearner |
 Posted: Wed Nov 28, 2012 7:25 am Post subject: Runtime Design Considerations for Large no of FTP Interfaces |
 |
|
Novice
Joined: 20 Jul 2012
Posts: 16
|
Hi ,
I've got a requirement to deploy around 400 interfaces with FTP on both input and output side into message broker v8 running on Windows 2008.
The frequency of the files coming are very less like daily, weekly, monthly,querteerly or yearly.
Now question is how to design the runtime for deployment.A dedicated broker for the whole work is an option for me with grouping of more closely related interfaces in same execution groups.
Resourcewise I fear there might be problem specifically from CPU persspective because of huge no of remote scanning.
Can you please suggest some wise options in this case. Also how to arrive at the decision of whether 1 broker will be sufficient for this work or we need to split the interfaces across multiple brokers.
Using alternative options rather than broker for FTP work etc is ruled out as project management has decided to use WMB v8 for this.
Any suggestion/experience shared on this would be greatly appreciated.
Thanks. |
|
| Back to top |
|
 |
| McueMart |
| Posted: Wed Nov 28, 2012 7:39 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
| Quote: |
| Resourcewise I fear there might be problem specifically from CPU persspective because of huge no of remote scanning. |
You can configure the scan delay, so I dont think that should be your no. 1 concern.
Were you considering deploying these 400 interfaces in 400 separate flows? The memory and general slugishness of broker running 400 flows would be a bigger concern. |
|
| Back to top |
|
|
| lancelotlinc |
| Posted: Wed Nov 28, 2012 8:33 am Post subject: |
|
|
Jedi Knight
Joined: 22 Mar 2010
Posts: 4941
Location: Bloomington, IL USA
|
|
| Back to top |
|
|
| McueMart |
| Posted: Wed Nov 28, 2012 8:45 am Post subject: |
|
|
Chevalier
Joined: 29 Nov 2011
Posts: 490
Location: UK...somewhere
|
| Not the performance of the message processing; but the administration of the product certainly can be when there are very large number of resources (execution groups / message flows / whatever...). I appreciate the product team have improved this over time. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Nov 28, 2012 8:48 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
400 interfaces doesn't sound that big a workload for a single broker.
I'd avoid putting them all in the same EG....
I'd tend to design a set of EGs to handle all of the daily interfaces in reasonable time, and then spread the less frequent ones out so that if you happen to have 10 monthlys all at once, they aren't all in one EG.
For the quarterly and yearly, you might even consider just having them stopped most of the time and use scripts to start them a day or so before and stop them a day or so after. |
|
| Back to top |
|
|
| zpat |
| Posted: Wed Nov 28, 2012 8:53 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
Why don't you get the end points to push (i.e. send) files to the broker using FTP (or even better SFTP)?
Event driven designs are always better and more scaleable than timer polling driven ones.
Each Execution Group is a JVM. If the processing is identical I would suggest having about ten flows in each EG, with each flow having about 4 FileInput nodes, and about 10 E.G.s . But this is pure guesswork..
I seem to remember that WMB 8 has some file node enhancements - but I can't remember what they are offhand, what version were you thinking of using? |
|
| Back to top |
|
|
|
|
