EldoS | Feel safer!

Software components for data protection, secure storage and transfer


Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages
Posted: 03/17/2011 18:22:56
by Bob Ferguson (Standard support level)
Joined: 01/24/2011
Posts: 6

I am finding my CBFS implementation which supplies data totally from memory arrays to be much slower than reading similar data from a hard drive.

The application reading the data reads 28 bytes at a time on each OnReadFile event.

I would have thought CBFS would be much quicker than a hard drive.

Am I correct in this assumption?
Posted: 03/17/2011 21:41:25
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

More information on this problem:

Our program is compiled in C# (DotNet 2.0)

The event that is "slow" is when a third party program (Not our own, but one that wants to use our virtual drive) browses to a folder and opens it.

When testing against a hard drive, using the same files as the virtual drive (we just copy the virtual drive to a disk folder), browsing to a folder takes less than a second

When testing against the virtual drive, loading the folder takes 6-7 seconds

When testing using the CBFS example app (Mapper I think?) which points to a disk copy of the Virtual Drive contents, it takes 8+ seconds. Noticeably slower than the virtual drive.

Looking at what the third part program is doing, it creates several temporary files, and reads/writes to these (and other files the virtual drive offers)
Specifically in the case of the temp files, it reads from them in 28 byte chunks, and reads from them a lot.

Over the 6 seconds it takes to open the folder, the app calls the CbfsRead 97,000 times and CbfsWrite 32,000 times.

We profiled the event, and the various callbacks and procedures don't seem to take 6 seconds. The profiler does come back with a "Waiting for synchronization" event that takes up almost all the missing time. Not sure what that means.

Transition to managed code takes less than 700ms

Using another program that does something very similar is almost instant, but rather than 97,000 read calls it was doing less than 6000.
Other than this, the virtual drive works well, and seems to look the same as the physical versions to the test apps.
Posted: 03/20/2011 04:04:32
by Eugene Mayevski (EldoS Corp.)

JFYI: we saw your message and our developers will check the problem during next week. I am sorry for the delay, yet we have several open issues to handle first.

Sincerely yours
Eugene Mayevski
Posted: 03/20/2011 17:29:30
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

More info:

We tried the c++ version of the example Mapper program (The one that maps a folder on your physical disk to a virtual drive) and pointed the same problem program to it.

The performance issue was still there
Posted: 03/21/2011 00:51:44
by Eugene Mayevski (EldoS Corp.)

Emm, now I am confused. Initially I thought the problem was that VDisk sample was slower than Mapper, and if the Mapper is also slow, then what are you comparing it to?

Sincerely yours
Eugene Mayevski
Posted: 03/21/2011 01:09:26
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

Sorry for the confusion, the example apps have odd names, and often EVERYTHING is called "mapper".

The Example app we are testing against is the one that you point to a folder, and a virtual drive is created so you can basically browse a disk folder as a virtual drive.

It is NOT the one with the 2 text files or the one with the 2 text files in folder structures.

The example app appears to be called "Mapper", at least that is what is on the main form titlebar

The visual difference is this is the only example app with text box below the MOUNT and UNMOUNT and above the ADD and DELETE buttons where you can enter a folder to be turned to a virtual drive.

We tested using this app because it (inherently) can do the same as our example, virtualize a tree list of files

We have tested the C# and C++ versions, both take 7+ seconds when a app browses to a folder.
Posted: 03/21/2011 01:21:27
by Eugene Mayevski (EldoS Corp.)

The samples are located in folders with different names, and the samples are named differently in documentation. So now I am confused even more. Since I didn't see those samples, I can't guess from your description, which ones you are referring to.

So... what are you comparing to what? 7+ seconds can be both slow and fast, depending on what you are comparing it to.

Sincerely yours
Eugene Mayevski
Posted: 03/21/2011 01:40:30
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

We have an app (not ours) that browses to a folder and loads a list of files

When you run against a real disk folder, it is instant. Far less than 1 second

If you browse against a Virtual folder, it takes 7+ seconds.

From the users point of view, they browse to a folder and see some files and some folders. Maybe 50-100.

7+ seconds to see a folder structure of 100 files is very slow.

I cleaned our source folder and reinstalled CBFS from scratch to confirm I have the exact name of the sample app.
It is in folder:
\Callback File System\Samples\dotNET\C#\Mapper\

The project file is called Mapper_VS2008.csproj

The Titlebar of the program is:

I have attached a screenshot of the Example app we are using to duplicate the performance results.

Posted: 03/21/2011 02:05:33
by Eugene Mayevski (EldoS Corp.)

Thank you, now it's more clear. You need to understand that calling back the user-mode code adds significant time to the total time needed to process the request. In case of Mapper sample, which redirects requests to another folder/disk, this time is even more increased because the request from your application goes to the kernel, then back to the callback handler in user mode, then again to the kernel (to read the data from the real disk), then this data is passed back to the user mode and back to the kernel. And only then it goes to your application. I.e. there are several extra kernel mode - user mode switches in this case. And if your application reads data in small chunks, then handling of such requests will be very slow (especially with unoptimized Mapper sample).

However, 7 seconds for browsing a directory with 100 files is not ok. Did you try reading the virtual disk's directory from Explorer? Does it give the same time?

Also please check VDisk or VMounter samples and see how different they are in time. This will give us some additional information.

Sincerely yours
Eugene Mayevski
Posted: 03/21/2011 02:14:20
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

The VDisk/VMounter samples aren't really going to give us anything, they are all very fast

Browsing the one of our Cbfs folders in Explorer is also fast as is using the Mapper example.

The issue is the temp files this third party app is writing when it browses to a folder.

As I mentioned in my first email above, it does 97,000 CbfsRead calls and about 32,000 CbsfWrite calls to a bunch of temp files it creates on the current folder every time a user browses to it.

We don't know why it does these calls, but the calls tend to be about 28 bytes in size for each one. Our guess is it reads 2 master index files, takes that information and writes a lot of temporary information, one "record" at a time to temp files it creates in the folder

That's fine when writing directly to a HDD folder, it's fast.
But when doing it via the Mapper example or our own Cbfs service, it's terribly slow.
Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages



Topic viewed 16548 times

Number of guests: 1, registered members: 0, in total hidden: 0


Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!