EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Opening Solid File store takes a long time

Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.
#11848
Posted: 11/30/2009 18:50:55
by Eric Dahlvang (Basic support level)
Joined: 09/11/2009
Posts: 29

store.Open(StorageOpenMode.somOpenExisting); is taking upwards of 20 minutes. We have around 85 gigs of files stored in it right now. Any ideas how we can speed up the start time?
#11850
Posted: 12/01/2009 00:58:14
by Alexander Plas (EldoS Corp.)

Hello.

Describe please properties and internal structure of your storage, i.e. page size, encryption, compression, structure of directories and files, average count of files in the directory, average size of file.
#11858
Posted: 12/01/2009 11:08:09
by Eric Dahlvang (Basic support level)
Joined: 09/11/2009
Posts: 29

This is the code we are currently using to open the store:
SolFSStorage.SetRegistrationKey(Resources.SolidFileSystemKey);
store = new SolFSStorage();
store.FileName = solidFileLocation;
store.UseTransactions = true;
store.UseAccessTime = false;
store.Open(StorageOpenMode.somOpenExisting);

We are currently not separating files into sub folders. We have tens of thousands of files in the root. The average file size is a few hundred K. We are not using encryption or compression. The page size is the default.

Our current system stores files with a Guid for the file name. We are considering using the first few characters of the file name as a folder structure. For instance, a file with the name {10a6d640-de9b-11de-8a39-0800200c9a66} would be stored in the \\10a\ directory. Would a configuration like this speed up the opening of the storage significantly?
#11861
Posted: 12/01/2009 11:43:52
by Alexander Plas (EldoS Corp.)

Two operations take a long time during the large storage opening. First is an enumeration of free pages. You can seed up this operation by increasing page size for the storage. Default page size is just 512 bytes. I think for your files 64K pages would be ideal. Second operation is the storage's root directory loading. To speed up this operation you can separate your files into different directories as you describe it.
#12403
Posted: 02/11/2010 14:24:17
by Eric Dahlvang (Basic support level)
Joined: 09/11/2009
Posts: 29

We have broken up the root directory into subdirectories as described above, and set the PageSize on the store to 64000. We also Compacted the store, and it continues to take upwards of half an hour to open. Do you have any other suggestions?

We are in development, so we are regularly doing internal releases, and starting/stopping our server. It is painful to have to wait so long between reboots. Any help at speeding up the time for opening the Solid File System would be greatly appreciated.
#12404
Posted: 02/12/2010 00:12:37
by Eugene Mayevski (EldoS Corp.)

Let's do the following:

if possible, please create a small program which will write to file the complete path to each file and it's size in "<filename>#9<size in bytes>" format and post this file to HelpDesk. We will try to recreate such storage locally. Also please specify there all SolFS properties that you set or the piece of code which allocates the storage.

We have certain ideas regarding how to improve loading time, but we need to investigate, what causes the slowdown in your particular case.


Sincerely yours
Eugene Mayevski
#12750
Posted: 03/09/2010 12:05:22
by Eric Dahlvang (Basic support level)
Joined: 09/11/2009
Posts: 29

Ticket # 16695 is resolved (I cannot see a way to post replies to the ticket anymore, so I am posting here.) Thank you for your assistance.

After recreating the store with 64k pages, and copying all of the files over, along with changing the names of the files to a maximum length of 22 characters, the store opens instantaneously. From 20 minutes, down to unmeasurable is a pretty good increase.

We appreciate your help with this.
Also by EldoS: CallbackDisk
Create virtual disks backed by memory or custom location, expose disk images as disks and more.

Reply

Statistics

Topic viewed 4528 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!