| Author |
Message
|
| fundoo |
 Posted: Mon Aug 08, 2011 2:35 am Post subject: FTE Monitoring Question |
 |
|
Novice
Joined: 04 May 2005
Posts: 13
|
We have setup a FTE Monitor on a directory to transfer files without deleting from source directory for a month long. Every month a new directory will be created and new monitor as well.
1. Poll interval is set to every 10 sec and batch size 100.
2. Approx around 20000 files gets added to the directory in a 24 hour span.
In the first day, transfers are immediate as and when the file is added to the directory.
As days pass by, say on 8th day we noticed that MQFTE takes around an hour to trigger for the file after it was added to the directory and then does the transfers, resulting in a one hour delay.
And the time delay keeps increasing further in the following days.
What could be the reason?
Thanks. |
|
| Back to top |
|
 |
| shashikanth_in |
| Posted: Mon Aug 08, 2011 7:38 pm Post subject: |
|
|
Centurion
Joined: 26 Feb 2009
Posts: 123
|
| What sort of file matching pattern you have set? Is there a reason behind keeping the file at source after transfer is done? 20000 files per day mean a lot I believe. For example, on 8th day you have 160,000 files. FTE might be taking time to list all the files and then transfer only the required ones. |
|
| Back to top |
|
|
| fundoo |
| Posted: Mon Aug 08, 2011 11:42 pm Post subject: |
|
|
Novice
Joined: 04 May 2005
Posts: 13
|
Thanks for your response.
File matching pattern is *. We get only the files required to be stored/transferred in to this directory. It is a requirement that files be retained at source for some purpose.
Yes, i too thought FTE seems to be taking time to go through all the accumulated files. But have no option here, source files cant be removed.
Is there an option in MQFTE such that once a file is trigerred it will no longer be monitored again for changes. That way only the new files that are added will be triggered and transferred. That way the delay could be avoided in going over all the files in the directory to monitor for changes.
Any thoughts?
From a File Transfer product this kinda support should be there. We cant expect file should be deleted after the transfer everytime. |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Aug 09, 2011 2:16 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
How many file transfer programmes do you know of that hold a record of what files have been transferred, and can immediately differentiate those that have been transferred and those awaiting transfer when a global wildcard is specified? 
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| fundoo |
| Posted: Tue Aug 09, 2011 3:14 am Post subject: |
|
|
Novice
Joined: 04 May 2005
Posts: 13
|
I am not expecting MQFTE to remember all the files that were transferred. Files created after the time the previous poll occurred are the ones to be transferred in the next poll. There is no need to keep record what files had been transferred so far.
Anyone encountered similar situation before and is there a solution/workaround possible for this scenario? |
|
| Back to top |
|
|
| exerk |
| Posted: Tue Aug 09, 2011 3:37 am Post subject: |
|
|
Jedi Council
Joined: 02 Nov 2006
Posts: 6339
|
| fundoo wrote: |
| ...There is no need to keep record what files had been transferred so far... |
And if NO record was kept then all files would be transferred on each poll. It would be interesting to see the traffic pattern between your transfer node and the Coordination queue manager when there a 'few' files to transfer (beginning of the week) and when there are a lot of files to transfer (end of the week).
_________________
It's puzzling, I don't think I've ever seen anything quite like this before...and it's hard to soar like an eagle when you're surrounded by turkeys. |
|
| Back to top |
|
|
| shashikanth_in |
| Posted: Tue Aug 09, 2011 8:11 pm Post subject: |
|
|
Centurion
Joined: 26 Feb 2009
Posts: 123
|
AFIK FTE keeps track of the file contents after every poll interval. This is to make sure that it does not transfer the same file again if there are no changes to file. Not sure if the same applies to directory also.
As exerk mentioned, you will need to look at the traffic to see what's going on. What have you noticed in Current Transfer Log tab? |
|
| Back to top |
|
|
| camauz |
| Posted: Wed Aug 24, 2011 4:28 am Post subject: rsync |
|
|
Acolyte
Joined: 20 Aug 2007
Posts: 52
Location: Mojan, Italy
|
It's out of scope, but there's more than a chance "rsync" would perform better for this use case.
Regards.
Ch. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Wed Aug 24, 2011 4:39 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fundoo wrote: |
| source files cant be removed. |
This is the heart of the problem.
And I doubt rsync would easily handle a directory of 100,000 files very quickly, either.
Just because you can't *delete* the source files does not in any way mean that they have to stay in the same source directory. You can easily move them to a subfolder or another folder somewhere else.
What part of the business requirement in question does not allow you to move them to a subfolder or somewhere else after processing? |
|
| Back to top |
|
|
| fundoo |
| Posted: Thu Aug 25, 2011 4:34 am Post subject: |
|
|
Novice
Joined: 04 May 2005
Posts: 13
|
| Business requirement is Replication of files from source folder to the destination. Source folder files still continue to be used by the source side applications. |
|
| Back to top |
|
|
| Vitor |
| Posted: Thu Aug 25, 2011 4:54 am Post subject: |
|
|
Grand High Poobah
Joined: 11 Nov 2005
Posts: 26093
Location: Texas, USA
|
| fundoo wrote: |
| Business requirement is Replication of files from source folder to the destination. Source folder files still continue to be used by the source side applications. |
You quoted a figure of 20,000 new files per 24 hours; that's a significant number of directory entries very quickly. If you only create a new directory once a month that's 3.2m files (assuming these only get created Mon - Fri).
As others have said, after 8 days it will take any piece of software a while to go through 160,000 files especially with a new poll every 10 seconds.
How do the source side applications manage this collosal number of files? Is it by name & can you use that in the FTE? Why the transfer to another folder?
_________________
Honesty is the best policy.
Insanity is the best defence. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Thu Aug 25, 2011 4:57 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| fundoo wrote: |
| Business requirement is Replication of files from source folder to the destination. Source folder files still continue to be used by the source side applications. |
You can not guarantee that a file can be properly replicated until you can guarantee that the source application has completely finished with it.
This whole set of business requirements/application design strongly says that you need to abandon working with files at all, and use some other kind of container for the data. Like perhaps a DBMS.
Files are not a reliable or effective container for data that is being used in this manner. |
|
| Back to top |
|
|
| kimmyj |
| Posted: Tue Oct 18, 2011 5:26 pm Post subject: |
|
|
Novice
Joined: 26 Jun 2003
Posts: 19
Location: UK
|
Hi Fundoo
We have done this successfully at a client site using ANT scripts and having an archive directory to where the files get moved before they get transferred. We haven't had an impact in the performance of the monitor.
Cheers |
|
| Back to top |
|
|
| yeger |
| Posted: Mon Feb 13, 2012 1:12 pm Post subject: |
|
|
Newbie
Joined: 09 Mar 2005
Posts: 8
|
I recommand that you will move the transferred files to a differant storage so you will not need to list each transfer all the files over again.
it is best to move all files transfered so the source directory will be empty |
|
| Back to top |
|
|
|
|
