| Author |
Message
|
| guest |
 Posted: Tue Dec 19, 2006 1:34 pm Post subject: File Truncation issue |
 |
|
Acolyte
Joined: 11 Aug 2003
Posts: 52
|
Hello Everybody,
Platfor & Version: Windows 2003 Server, MQ 5.3, CSD12.
Problem Scenario: Some external application is copying the files in some directory. I have C program to put the files to MQ. The C program is used in a batch file with the required parameters. The batch filease are being invoked through WIndows Scheduler at the interval of 1 Mins. But when the file is written halfway by the external application and the Windows scheduler invokes the batch file, we are getting truncated files at the target end.
Requirement: As I came to know that in C it is possibe to determine the FILE Handle, more specifically it is possibe to know if the file is in use by other processes or not. If it is possible to determine the file is being used by other process or not, then I can implement some logic to ignore the file at that time. Is there nybody who did face issue something like this? Also is there anybody who implemented the FILE HANDLE in C? Please help.
Thanks. |
|
| Back to top |
|
 |
| JosephGramig |
| Posted: Tue Dec 19, 2006 1:38 pm Post subject: |
|
|
Grand Master
Joined: 09 Feb 2006
Posts: 1244
Location: Gold Coast of Florida, USA
|
Hmmm, how about writing the file as another name. When done rename the file to the name you are looking for and the rename should be atomic enough to keep you safe.
_________________
Joseph
Administrator - IBM WebSphere MQ (WMQ) V6.0, IBM WebSphere Message Broker (WMB) V6.1 & V6.0
Solution Designer - WMQ V6.0
Solution Developer - WMB V6.1 & V6.0, WMQ V5.3 |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 19, 2006 1:40 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Isn't it fun trying to implement transactionality with a non-transactional file system?
There isn't a good way on Windows that I know of to determine if a file is open for writing by another application - and even if there was that doesn't mean that the file isn't ready for processing or not.
The last time I had to do this, I wrote code that would look at the file size every interval, and wait until a particular file had stopped changing in size before trying to process it.
And then did a few more things to make sure that a) I didn't process it twice, b) nobody else would process it when I was working with it, c) that I deleted it properly when I was done.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| guest |
| Posted: Tue Dec 19, 2006 1:46 pm Post subject: |
|
|
Acolyte
Joined: 11 Aug 2003
Posts: 52
|
To Joseph: The requirement is to keep the same file name at target end as the source is sending.
To Jeff: Yes it is funny to me also the we are implementing some transaction with some non transactional system. But this is the requirement right now. Also I though abt the implementing delay in the C program. But the problem is the number of files and the volume of files are quite high. So I was thinking abt the Fike Handle as I have described.
Appreciate your suggessin or help.
Thanks. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 19, 2006 1:51 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
Again, I don't believe there is any good, reliable way to know if the file is not open by another application.
I remember that the Win32 APIs had some ways of checking these things - but I also remember that for example the IIS FTP server had ways of writing files that was ignored by the Win32 APIs.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Tue Dec 19, 2006 2:03 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Humm, why don't you just take a step back in the process?
Instead of copying the file from server 'X' to server 'Y' and then have a scheduled process 'S' on server 'Y' put it to a queue, do it all on server 'X'.
When process 'A' finishes writing the file on server 'X', have the process launch the 'File2Msg' program with the filename as a parameter.
Its not prefect but there will be fewer things to worry about. 
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 19, 2006 2:04 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
One reason not to do that, Roger, is if Server X is run by a business partner, and thus is not on your network at all...
Again, see my comments about the Joys of IIS FTP.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| guest |
| Posted: Tue Dec 19, 2006 2:12 pm Post subject: |
|
|
Acolyte
Joined: 11 Aug 2003
Posts: 52
|
Roger the thing is, the files are being copied from system A, B , C ... to my MQ server. Thats why they are not able to run the MQ put script to initiate the put.
Jeff is right on this. |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Tue Dec 19, 2006 2:30 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
That's not to say that Roger isn't right that you should really move the File->MQ translation to the sending servers.
In fact, you should work to migrate your file producing applications into MQ producing applications, and send more but smaller messages.
Unless they're at business partners.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| RogerLacroix |
| Posted: Tue Dec 19, 2006 2:45 pm Post subject: |
|
|
Jedi Knight
Joined: 15 May 2001
Posts: 3264
Location: London, ON Canada
|
Hi,
Remember, you can freely install MQ Client on any server, so that you could run your program on server 'A'. Secondly, a 'File2Msg' Java program means you don't need the MQ Client installed, just a couple of MQ jar files. Of course if you are using client mode connections, please make sure you use sync-point on your puts followed by a commit or rollback.
I can't find it right now, but many moons ago I remember having to use a backwards approach. You opened the file in exclusive mode for writing-only or reading & reading. If another process had it opened, then the open returned an error code. i.e. _open with _O_EXCL and _O_RDWR (or _O_EXCL and _O_WRONLY).
I've never had to deal with IIS but I think it was tested against ftp.
Regards,
Roger Lacroix
Capitalware Inc.
_________________
Capitalware: Transforming tomorrow into today.
Connected to MQ!
Twitter |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Wed Dec 20, 2006 8:46 am Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| If your application is written in C, you can get a lock/unlock on the file using the function _locking(). |
|
| Back to top |
|
|
| tleichen |
| Posted: Wed Dec 20, 2006 11:42 am Post subject: |
|
|
Yatiri
Joined: 11 Apr 2005
Posts: 663
Location: Center of the USA
|
MQ seemed like a sideline item in this question. What does the problem have to do with MQSeries? 
_________________
IBM Certified MQSeries Specialist
IBM Certified MQSeries Developer |
|
| Back to top |
|
|
| jefflowrey |
| Posted: Wed Dec 20, 2006 12:01 pm Post subject: |
|
|
Grand Poobah
Joined: 16 Oct 2002
Posts: 19981
|
| rtsujimoto wrote: |
| If your application is written in C, you can get a lock/unlock on the file using the function _locking(). |
In an ideal world where all applications honor that locking flag, including the OS.
_________________
I am *not* the model of the modern major general. |
|
| Back to top |
|
|
| rtsujimoto |
| Posted: Wed Dec 20, 2006 12:53 pm Post subject: |
|
|
Centurion
Joined: 16 Jun 2004
Posts: 119
Location: Lake Success, NY
|
| It takes (at least) two to tango. The file putter and getter have to honor the lock, otherwise what's the point. And, given the OP's scenario, there are only two participants, so it should work just fine. But, if the OP has no control on the getter, then this is not a viable alternative. |
|
| Back to top |
|
|
|
|
