EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Timeout while copying files with SolFS and CallbackFS

Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.
#3790
Posted: 09/18/2007 04:18:51
by Dan Cooper (Basic support level)
Joined: 06/27/2007
Posts: 125

I am using CallbackFS to mount a volume in Windows and using SolidFS to provide the underlying storage which has been working fine until we started testing with larger files. I have noticed that when a large file is being copied it starts to hang on the SolFS write operations and I am struggling to find a reason why. Here's the process for the file operation:

CbFsOpenFile is called which opens a SolFSStream and wraps this in a structure along with some other information and is passed back as an IntPtr in the FileContextHandle param.

CbFsWriteFile is called where the structure is extracted from the FileContextHandle and .Seek is called on the SolFSStream followed by .Write and .Flush.

CbFsCloseFile is called where the structure is extracted from the FileContextHandle and the .Close method is called on the SolFSStream and the FileContextHandle is freed.


This works until we start copying large files when the CbFsWriteFile event will hang on the .Write method of the SolFSStream. If I set the UseSystemCache to false and increase the Buffering I can write larger files without errors but we do still hit the hang when the file size increases and the system reports "Delayed write failed while copying file" and explorer then crashes.

Is there anything you can think of that may cause this?
#3791
Posted: 09/18/2007 04:30:59
by Dan Cooper (Basic support level)
Joined: 06/27/2007
Posts: 125

Update:

The operation also hangs when using a standard FileStream so I don't think it is related to SolFS.
#3793
Posted: 09/18/2007 05:03:01
by Volodymyr Zinin (EldoS Corp.)

Can you please give us more information about the problem: the operation system version, number of processor, size of the file which was copied.

Quote
Dan Cooper wrote:
The operation also hangs when using a standard FileStream so I don't think it is related to SolFS.

Whether the FileStream was opened with the system cache support enabled (i.e. FILE_FLAG_NO_BUFFERING flag was not set when the file was opened)? And whether the UseSystemCache was set?
#3794
Posted: 09/18/2007 05:11:20
by Dan Cooper (Basic support level)
Joined: 06/27/2007
Posts: 125

We are testing on WinXP SP2, Core2, filesize: 50MB.

I have created a test app and have discovered that the problem only seems to occur if the Stream is kept open and passed back in the FileContextHandle. If I open a new stream on each write it does not fail.
#3799
Posted: 09/18/2007 07:51:32
by Volodymyr Zinin (EldoS Corp.)

Whether the FileStream was opened with the system cache support enabled (i.e. FILE_FLAG_NO_BUFFERING flag was not set when the file was opened)? And whether the UseSystemCache was set?

Also please specify what build number of CallbackFS and what API (.NET 1.1, .NET 2.0, VCL, C++) you are using?
#3949
Posted: 10/04/2007 08:02:57
by Dan Cooper (Basic support level)
Joined: 06/27/2007
Posts: 125

Hi,

Sorry I haven't replies sooner, I have been working on other areas of the project.

I have made some changes to the way the system works to be more inline with the demo project supplied. I now open a new SolFSStream and Seek() on each OnWriteFile call which has improved the situation but we are still experiencing the 'Delayed write failed' error when copying large files.

The SolFS was opened with UseSystemCache set to true. CallbackFS version is 1.0.11.24856 and SolFS is version 3.1.0.80. Both are .NET 1.1 versions.

I have a feeling that the problem lies in one or more of the event handlers taking too long to return but I have been optimizing the code in the OnWrite handler as this is called most often without any improvement in the problem. Do you have any ideas what else could be causing this issue?

Thanks,
Dan
#3953
Posted: 10/04/2007 08:48:52
by Volodymyr Zinin (EldoS Corp.)

Please try to set the Timeout property to 0 (i.e. infinity) and check the problem again. Also check up, may be your Write callback routine returns any error. And try the new CallbackFS beta build.

If the problem when the Timeout set to 0 will also exist, then could you create a test application for us, so we can reproduce the problem ourselves?
#3954
Posted: 10/04/2007 09:12:43
by Dan Cooper (Basic support level)
Joined: 06/27/2007
Posts: 125

I set the Timeout property to 0 as suggested and I no longer get the popup saying "Delayed write failed..." but the copy operation does not progress and when I pause execution the line it always stops on the same line:

Code
using( Stream stream = Engine.Instance.StorageManager.OpenFile(fileContext.CachePath, false) )
{
   stream.Seek(Position, SeekOrigin.Begin);
   stream.Write(Buffer, 0, BytesToWrite);
   BytesWritten = (int)(stream.Position - Position);  <--- Always stuck here
   fileContext.Item.LocalSize = stream.Length;
   fileContext.Item.Size = stream.Length;
   _sync.QueueStorageItemUpdate(fileContext.Item);
   stream.Close();
}


It will be very difficult to produce a test application as it is tied in to a lot of our in-house code, I was hoping there may be something known that could cause this or an idea of where I should direct my attention to solve it.

Thanks,
Dan
#3955
Posted: 10/04/2007 09:54:05
by Volodymyr Zinin (EldoS Corp.)

Quote
Dan Cooper wrote:
stream.Write(Buffer, 0, BytesToWrite);
BytesWritten = (int)(stream.Position - Position);<--- Always stuck here

The program locked in the stream.Write call.

Quote
Dan Cooper wrote:
The SolFS was opened with UseSystemCache set to true.

Please re-check that the UseSystemCache is set to true (in CallbackFS, not in SolFS).

Quote
Dan Cooper wrote:
It will be very difficult to produce a test application as it is tied in to a lot of our in-house code, I was hoping there may be something known that could cause this or an idea of where I should direct my attention to solve it.

Only a compiled test program is necessary (without sources).
#3957
Posted: 10/04/2007 10:10:18
by Dan Cooper (Basic support level)
Joined: 06/27/2007
Posts: 125

UseSystemCache is set to true after construction of the CallbackFS object:

Code
<snip>
// Now we need a CallbackFileSystem object to handle the drive
_fs   = new CallbackFileSystem();

// We want to use the system cache
_fs.UseSystemCache = true;
</snip>


I will try to get a test app together but it will take some time as the application relies on connection to a server which is not available externally and various registry settings. It will not be easy to build a standalone application to demonstrate the issue.
Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.

Reply

Statistics

Topic viewed 13957 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!