EldoS | Feel safer!

Software components for data protection, secure storage and transfer


Also by EldoS: SecureBlackbox
200+ components and classes for digital security, signing, encryption and secure networking.
Posted: 05/09/2011 03:51:16
by Bob Ferguson (Standard support level)
Joined: 01/24/2011
Posts: 6

Much appreciated Oleg.
Posted: 05/09/2011 06:54:23
by Eugene Mayevski (Team)

I really wish we could make it tomorrow as I don't like issues hanging in our ToDo list (this makes me nervous ;). However kernel-mode development is painfully slow (and file system drivers are probably one of the most complicated types of drivers, together with file system filter drivers). We can't give any estimates until the developers start to actually work on it. When we start and we get to know, what is involved (should we write complete caching ourselves or the OS offers some support for caching), then we'll be able to estimate time. I think 3-4 months is a good (more or less realistic) ballpark estimate. I will ask one of developers to start investigating, what the OS has to offer us in this aspect.

Also let me repeat - CBFS is the your best option even while caching is a problem. Reason #1 is that there are no comparable alternatives which could be used in a realistic time (less than those 3-4 months I mentioned), and reason #2 is that the issue is specific to only some usage scenarios.

If I were in your position, I would start to look not for alternatives, but for (a) optimization hints and (b) justification of such temporary problem to present it to the users (say "Our first version offers these and that unique features, yet this comes at a cost of lower speed. We plan to deliver speed improvements in 6 months").

You can implement some caching in user mode, as Oleg Savelos suggested, though in your case the main part of the problem is in context switches. But I think that user-mode cache should save about 30-40% of read operation speed.

Sincerely yours
Eugene Mayevski
Posted: 05/09/2011 15:11:51
by Bob Ferguson (Standard support level)
Joined: 01/24/2011
Posts: 6

Thanks Oleg and Eugene.
Can you explain a little more what you mean by caching in user mode?

We already store any data read from the physical DB in memory, so we only hit the HDD once for each set of data.
So the myriad reads and writes that are then requested by the application we are supplying data to are all done in a memory context.

Is that what you mean?

BTW the application we are supplying data to is a financial analysis program called MetaStock™.
Posted: 05/09/2011 23:57:31
by Eugene Mayevski (Team)

nisaus wrote:
We already store any data read from the physical DB in memory, so we only hit the HDD once for each set of data.

That's what I meant. So waiting for a cache would be the only option for now.

nisaus wrote:
BTW the application we are supplying data to is a financial analysis program called MetaStockā„¢.

Well, I can't blame the application makers, because most developers don't care about efficiency these days. Developers believe, that performing a data reading call is fast, and forget (or don't know) about the long path each request travels even in case of HDD.

Sincerely yours
Eugene Mayevski
Posted: 06/11/2011 20:11:54
by Ivan P (Priority Standard support level)
Joined: 04/11/2011
Posts: 70

Please ignore this message because I desided to create a [URL=https://eldos.com/forum/read.php?FID=13&TID=3026]separate thread[/URL]
Posted: 07/10/2011 11:42:23
by Eugene Mayevski (Team)

FYI: New beta build with file cache is available on pre-release downloads page. You are welcome to test it and provide feedback. Cache is enabled by default in this build and can be disabled if you experience problems with it.

Sincerely yours
Eugene Mayevski
Posted: 07/15/2011 20:00:29
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

Using the latest beta, here are our test results (using the same example app I wrote and sent before.

(Test basically writes a temp file then does a lot of small byte read/writes)

Tested against a physical disk Real Disk:
Time taken: 0:0.998

Tested against the old version we had before (Uncached)
Time taken: 0:5.811

Tested against the new beta (Cached)
Time taken: 0:2.418

Nice improvement, not as fast as a physical drive of course. I'll get into rebuilding the actual apps and putting through their paces again.

Thank you for the hard work!
Posted: 07/16/2011 00:42:00
by Eugene Mayevski (Team)

Callbacks are the cause of the major slowdown. The aim of the cache is to reduce the number of calls when the application writes or reads data in small chunks. The larger the block is, the less improvement will be achieved.

Sincerely yours
Eugene Mayevski
Also by EldoS: CallbackDisk
Create virtual disks backed by memory or custom location, expose disk images as disks and more.



Topic viewed 18843 times

Number of guests: 1, registered members: 0, in total hidden: 0


Back to top

As of July 15, 2016 EldoS business operates as a division of /n software, inc. For more information, please read the announcement.

Got it!