EldoS | Feel safer!

Software components for data protection, secure storage and transfer

SolFS performance

Also by EldoS: CallbackProcess
A component to control process creation and termination in Windows and .NET applications.
#8011
Posted: 11/03/2008 02:12:47
by Jure Bogataj (Priority Standard support level)
Joined: 05/23/2006
Posts: 35

Hello!

I have a question about read performance on large SolFS storage files. Here are parameters for my specific storage file:

PageSize: 512
Encryption: No
Compression: No
CaseSensitive: No
StorageSize (on disk): 1,52 GB
Average file size inside storage: 13,80 KB (from 3,10 KB to 43,90 KB)
File count: 100.000
Directory count: 0 (all files are on the root of the storage file).

To open this storage (SolFSStorage.Open) my application needs 3-6 seconds. Here is code for opening storage file:

SolFS.SolFSStorage stg = new SolFSStorage();
stg.FileName = fn;
stg.ReadOnly = true;
stg.UseAccessTime = false;
stg.AutoCompact = 0;
stg.Open(StorageOpenMode.somOpenExisting);

Code for opening single file inside storage is following:

solFile = new SolFSStream(stg, fileName, FileMode.Open, FileAccess.Read);


Iteration with stg.FindFirst and stg.FindNext (closing with stg.FindClose) takes 35-40 seconds.

Are these timings normal behaviour of SolFS storage or can I do anything for SolFS to perform faster? Any parameters inside any of the function calls?

Best regards,
Jure
#8018
Posted: 11/03/2008 07:24:50
by Alexander Plas (EldoS Corp.)

Hello

First of all, you should increase page size at least to 4K. You can try to set 64K page size. It may significant increase read performance for your file sizes.

To speed up storage opening you should put your files into directories, approximately 1000 files per directory.

Also you can use FindFirstEx function instead of FindFirst. You can pass 0 in the last parameter of FindFirstEx function, if you don't need full name of the file (ffNeedFullName), file times (ffNeedFileTimes) and file size (ffNeedFileSize)
#8032
Posted: 11/04/2008 07:47:11
by Jure Bogataj (Priority Standard support level)
Joined: 05/23/2006
Posts: 35

Thank you for your response!

I can use 4K pages (and probably will), I just get more overhead with 4K pages (approx. 43%). That means that storage's disk size is approx 43% bigger than size of files it contains. With 512bytes PageSizes I get approx 16% overhead, so I thought I can speed up operations on 512B Pages also.

I will make use of directories then and use FindFirstEx for file browsing.

Best regards,
Jure
#8243
Posted: 11/23/2008 14:21:42
by Candin Jurca (Basic support level)
Joined: 11/23/2008
Posts: 1

I browsed the documentation and I found no clear definition of findfirst versus findfirstex functions. Which is the difference anyway ?

Best regards
Candin
#8244
Posted: 11/23/2008 22:27:52
by Eugene Mayevski (EldoS Corp.)

FindFirstEx has Flags parameter, which specifies, what information about the file should be returned. The less information is returned, the faster enumeration is. Speed different is not very significant, but it exists and it's worth paying attention to.


Sincerely yours
Eugene Mayevski
Also by EldoS: CallbackRegistry
A component to monitor and control Windows registry access and create virtual registry keys.

Reply

Statistics

Topic viewed 2889 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!