| Author |
Message
|
| zpat |
 Posted: Thu Jun 11, 2009 11:36 pm Post subject: FileOutput Node and locks on files |
 |
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
I have a message flow that produces files in a directory using the FileOutput node. Because we can't use the FTP Append feature in WMB 6.1 (AIX) - I have to run another script to FTP the files, this runs at intervals.
Is there a risk that this script (Korn shell) would pick up a file as it was being moved from the transit directory to the target directory, or does WMB keep the file locked until it's completely ready and moved?
The documentation is not entirely clear on this point. Does the final file move (after the file is finished) actually involve data being copied or is it simply a directory update operation? |
|
| Back to top |
|
 |
| ovasquez |
| Posted: Fri Jun 12, 2009 3:55 am Post subject: |
|
|
Centurion
Joined: 09 Dec 2005
Posts: 141
Location: Lima, Peru
|
about your question, Yes is posibled
...BUT you can use long file(5MB) in FileOutput and test, is easy..
_________________
Oscar Vásquez Flores |
|
| Back to top |
|
|
| spazy |
| Posted: Fri Jun 12, 2009 6:59 am Post subject: |
|
|
Newbie
Joined: 02 Sep 2008
Posts: 8
|
in first part WMB store file to working directory and in second part move to target place.
I advice... after wmb stores big file in next step creates in another folder check file with zero size and script always pickup finshed file files. |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 12, 2009 7:19 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
The move from mqsitransmit is as atomic as the OS and disk subsystems and etc. allow it.
You should code your script, however, to assume the worst case.
This way you won't get surprised if something goes wrong.
Also, in general, file locks are not trustworthy. |
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jun 12, 2009 9:16 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
OK, I may make the script pause for 1 sec between reading the directory for the files to FTP and actually doing the FTPs - that should avoid any accidents.
I will submit an enhancement request to IBM to add the Append FTP option to WMB 6.1 |
|
| Back to top |
|
|
| mqjeff |
| Posted: Fri Jun 12, 2009 9:41 am Post subject: |
|
|
Grand Master
Joined: 25 Jun 2008
Posts: 17447
|
| zpat wrote: |
| OK, I may make the script pause for 1 sec between reading the directory for the files to FTP and actually doing the FTPs - that should avoid any accidents. |
... check that the modification time on the file hasn't changed as well.
 |
|
| Back to top |
|
|
| PeterPotkay |
| Posted: Fri Jun 12, 2009 10:55 am Post subject: |
|
|
Poobah
Joined: 15 May 2001
Posts: 7722
|
|
| Back to top |
|
|
| zpat |
| Posted: Fri Jun 12, 2009 12:30 pm Post subject: Re: FileOutput Node and locks on files |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
| zpat wrote: |
| Because we can't use the FTP Append feature in WMB 6.1 (AIX) - I have to run another script to FTP the files, this runs at intervals. |
Because I must append the data at the destination (which happens to be an AS/400) using FTP and WMB 6.1 does not support this (fairly common) FTP option.
I don't want to use NFS or other means of accessing the disks remotely since this has to replace a current FTP process with no changes. |
|
| Back to top |
|
|
| mqpaul |
| Posted: Mon Jun 29, 2009 3:13 am Post subject: Re: FileOutput Node and locks on files |
|
|
Acolyte
Joined: 14 Jan 2008
Posts: 66
Location: Hursley, UK
|
| zpat wrote: |
I have a message flow that produces files in a directory using the FileOutput node. Because we can't use the FTP Append feature in WMB 6.1 (AIX) - I have to run another script to FTP the files, this runs at intervals.
Is there a risk that this script (Korn shell) would pick up a file as it was being moved from the transit directory to the target directory, or does WMB keep the file locked until it's completely ready and moved?
The documentation is not entirely clear on this point. Does the final file move (after the file is finished) actually involve data being copied or is it simply a directory update operation? |
Once it's finished writing the file, WMB will use AIX calls to rename the file. So the file will remain in the transit directory before it's complete, and will appear in the output directory after it's complete. If your script is looking in the transit directory, it may well be trying to use the file when WMB wants to move it to the output directory.
The broker keeps file system locks on the file in the transit directory while it's writing to the file. However, if your flows stop writing for a significant time (currently 5-7 seconds), the broker will close the file and release the locks. Therefore you cannot rely on locks on files in the transit directory.
In summary, you should only be using the file in the output directory. That will be complete, and not locked.
_________________
Paul |
|
| Back to top |
|
|
| zpat |
| Posted: Mon Jun 29, 2009 3:30 am Post subject: |
|
|
Jedi Council
Joined: 19 May 2001
Posts: 5866
Location: UK
|
That what I do, I wouldn't want to mess about with files in the transit directory. Thanks.
Looks like the MQ user group will be at Hursley in time for the heatwave this week. No air-con in the old building I suppose? |
|
| Back to top |
|
|
|
|
