| Author |
Message
|
| Esa |
 Posted: Mon Feb 24, 2014 6:39 am Post subject: File Input node timestamps |
 |
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Timestamps from processing a small XML file (<2Kb):
| Code: |
<File>
...
<LastModified>2014-02-24T09:06:04.356</LastModified>
<TimeStamp>20140224_090543_266765</TimeStamp>
...
</File> |
From File Input.transaction Start event:
2014-02-24T09:06:08.552Z
Timestamp from File Input.transaction End event:
2014-02-24T09:06:12.502Z
According to the InfoCenter File.TimeStamp is the time when the node started processing the file.
I'm wondering what the node was doing for 25 seconds after starting to process the file and before starting the flow transaction?
Probably moving the file into mqsitransit folder (the LastModified timestamp)?
What else? There were no other files being processed at the same time. |
|
| Back to top |
|
 |
| Tibor |
| Posted: Mon Feb 24, 2014 7:08 am Post subject: |
|
|
Grand Master
Joined: 20 May 2001
Posts: 1033
Location: Hungary
|
| Hi Esa, don't you think it is only related to the polling interval? |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Feb 25, 2014 12:44 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
No, the polling interval is only 5 s.
My resolution is following:
This happens in system test and production environments.
The flow starts processing (TimeStamp) when the file appears in the polling directory. The file is transferred with sFTP by some external middleware.
LastModified timestamp happens 21 seconds after the processing starts. I guess the middleware holds a lock to the file for 21 seconds (regardless of file size, it appears). LastModified is the timestamp when the lock was released.
The four seconds from LastModified to File Input.transaction Start are used for copying the message to mqsitransit. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Tue Feb 25, 2014 4:07 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
It's entirely possible that the sftp server has the file locked for that time period.
Just because a file appears in a listing of the contents of a directory does not in any way mean that the file is complete or unlocked or ready to be read and processed.
Reading a file is a straightforward example of the halting problem. You only know the file is "readY" when you know that the program writing the file has halted.
So FileInput may be waiting for the file to stop showing some signs of being actively written to, or waiting for the sftp server to release any file system locks on it, before it tries to move it to mqsitransit.
The whole purpose of mqsitransit is to ensure that nobody else will mess with the file while Broker is processing it. |
|
| Back to top |
|
|
| Esa |
| Posted: Tue Feb 25, 2014 5:36 am Post subject: |
|
|
Grand Master
Joined: 22 May 2008
Posts: 1387
Location: Finland
|
Thanks, mqjeff.
The person responsible for the sftp system was unaware of any locking. It seems to me that the system is even incapable of applying any locking.
So it's quite likely that FileInput is waiting for a certain amount of time after the file size has ceased to grow. An undocumented feature?
Which raises the question how to disable this feature, if it really exists? It's unnecessary to delay processing when you know that the source system is already using rename or move strategy for preventing premature processing of files. Which doesn't apply to my current case, of course, but many others. |
|
| Back to top |
|
|
|
|
