EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Corrupt files after storage.close()

Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages
#4387
Posted: 11/23/2007 03:47:41
by Kenny Clement (Priority Standard support level)
Joined: 09/03/2007
Posts: 61

Internally, our application is using unison to synchronize files to a local storage.

We've noticed that unison occasionally reports corrupted archive files (these files are created and modified while synchronizing). This happens quite a lot.

We do not have a solid reproduction scheme for this yet, but we think this happens when we are writing files just before performing storage.close()

Does the close() function check for open/locked files? Can we let it 'wait' untill file operations are completed?
Can we check for this ourselves?

We've also noticed this happens when we kill our application (destroyOnProcessterminated = true), so the storage is killed too, which is exactly what we want, only, it seems to result in corrupted files.

Now that I think about it, I remember something about a transactions property.
We currently don't use transactions on the storage. Would these problems be resolved if we did? If so, what are downsides of using it? Is it slower? Otherwise, why wouldn't it be enabled by default.
#4391
Posted: 11/23/2007 05:04:04
by Volodymyr Zinin (EldoS Corp.)

Quote
Kenny Clement wrote:
We've also noticed this happens when we kill our application (destroyOnProcessterminat­ed = true), so the storage is killed too, which is exactly what we want, only, it seems to result in corrupted files.

Does it happen only in this case. Or does it happen also when you close the storages using the StorageClose routine? If so, does the ForceUnmount parameter is set in the latter case?
#4393
Posted: 11/23/2007 05:18:50
by Kenny Clement (Priority Standard support level)
Joined: 09/03/2007
Posts: 61

Thanks for the response.

ForceUnmount is indeed set to true.
We'll test if this resolves our issue.

However, the problem remains when a user resets the computer, or when our application crashes, (because destroyonprocessterminated is set to true).

Even though this is not standard behaviour, it shouldn't corrupt files. (NTFS doesn't do this either)
Would transactions prevent these issues?
#4394
Posted: 11/23/2007 06:04:40
by Volodymyr Zinin (EldoS Corp.)

NTFS uses transactions only for protect its internal integrity (boot sector, MFT, other meta-files). Other files data aren't protected by the NTFS transactions. Sure the files (as items) aren't lost on the NTFS volume, but its contents can be damaged.

[/QUOTE]
Quote
Kenny Clement wrote:
ForceUnmount is indeed set to true. We'll test if this resolves our issue.

If this parameter is not set then the unmounting can false (in case if there are some opened files still exist).

Quote
Kenny Clement wrote:
However, the problem remains when a user resets the computer, or when our application crashes, (because destroyonprocessterminate­d is set to true).

Resetting the computer cannot be correctly processed by SolFS (and by NTFS too).
Also when the destroyonprocessterminate­d parameter is set for a storage then the storage will be closed FORCIBLY if the process-initiator crashes. If you want do not use this feature. In this case if the application crashes you can restart it and correctly close the storage.


Quote
Kenny Clement wrote:
Now that I think about it, I remember something about a transactions property. We currently don't use transactions on the storage. Would these problems be resolved if we did? If so, what are downsides of using it? Is it slower? Otherwise, why wouldn't it be enabled by default.

You can try to use this feature. This fair prevents the files data integrity. But sure it is slower than the storages without transactions.

Also you can use the parameter EnableSystemCache. If you set it to true then SolFS does not use the system cache and the files data are directly written (without any delay) to a storage. It's also slower variant. But you can try it and may be it will be suitable for you.
This parameter currently is global, i.e. if it's set then it influences on all SolFS storages in the system. But in the next build we are going to correct it.


#4395
Posted: 11/23/2007 06:12:41
by Volodymyr Zinin (EldoS Corp.)

I forgot to add about the EnableSystemCache parameter. It doesn't help if a file is opened through the memory mapped mechanism.
#4396
Posted: 11/23/2007 06:34:19
by Kenny Clement (Priority Standard support level)
Joined: 09/03/2007
Posts: 61

Related to these problems:
Is there a way of checking whether there are files in use on the storage?

Next to SolFS, we're also using CrossCrypt.
The CrossCrypt driver has a property 'locked' to check whether there are files in use.
Does SolFS have something like this?

#4397
Posted: 11/23/2007 06:56:20
by Volodymyr Zinin (EldoS Corp.)

You can check it by means of the following code:

HANDLE hVolume = CreateFile("\\\\.\\c:", READ_CONTROL | FILE_READ_ATTRIBUTES, ...);
if (!DeviceIoControl(hVolume, FSCTL_LOCK_VOLUME, NULL, 0, NULL, 0, &BytesReturned, NULL)) {

//
// There are some files are opened on the volume
//
}
CloseHandle(hVolume);
#4401
Posted: 11/25/2007 13:32:54
by Nick Tucker (Basic support level)
Joined: 03/13/2007
Posts: 13

Is there a way to enumerate the process that are currently accessing the storage file? The above example also will include explorer if you have an explorer window open to the storage file. In this case it would be ok to force the closure.

By the way.. for the pascal users... :-)

//-------------------------------------------------------
function anyopenfiles(path:string):Boolean;
const FSCTL_LOCK_VOLUME = $00090018;
var BytesReturned : DWORD;
vHandle : thandle;
begin
Result := False;
path := trim(excludetrailingbackslash(path));

vHandle := CreateFile(PChar('\\.\' + path), GENERIC_READ or GENERIC_WRITE,
FILE_SHARE_READ or FILE_SHARE_WRITE, nil, OPEN_EXISTING,
FILE_FLAG_NO_BUFFERING, 0);

if vHandle <> INVALID_HANDLE_VALUE then
Result := not DeviceIoControl(vHandle, FSCTL_LOCK_VOLUME, nil, 0, nil, 0,
BytesReturned, nil);
CloseHandle(vHandle);
end;
//-------------------------------------------------------

Regards,

Nick
#4404
Posted: 11/26/2007 02:12:39
by Volodymyr Zinin (EldoS Corp.)

Quote
Nick Tucker wrote:
Is there a way to enumerate the process that are currently accessing the storage file?

No. Because this information is not logged. Also due to of fact that the operating system holds references to the files which were opened (in hope that this files will be opened again soon and their contents are already in the system cash), some files can be already closed by processes which opened this file but the system holds it. BTW this system's references can be forcibly closed by means of ReleaseUnusedFiles from CallbackFS api.
Also by EldoS: CallbackDisk
Create virtual disks backed by memory or custom location, expose disk images as disks and more.

Reply

Statistics

Topic viewed 4199 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!