EldoS | Feel safer!

Software components for data protection, secure storage and transfer

OnReadFile and file size

Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.
#13122
Posted: 04/27/2010 01:06:09
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

I've got a simple virtual text database working at the moment, but am running into some issues with speed.

Assuming that by the time the OS is calling the OnReadFile events for data it needs to know the EXACT size of the data set to be returned, when do you TELL the OS what size that data is for a file?

Do you need to return the exact file size during the OnGetFileInfo event? (In the EndOfFile variable)

What I am seeing is if you browse to a folder with several files, you get one OnGetFileInfo event PER FILE visible in that folder (only those files visible in the explorer window)
The issue here is if there are 30 "files" each 100meg in size, we need to load 30x100 meg of data to see how big it is to tell the OS how big they are... and the user isn't even opening a file. In fact the file properties being displayed don't even use these values but rather use the values provided when OnEnumerateDirectory
#13125
Posted: 04/27/2010 02:05:19
by Eugene Mayevski (EldoS Corp.)

Yes, file size is obtained via OnEnumerateDirectory or via OnGetFileInfo event.

Unfortunately the only solution I can see in your case is either to cache actual sizes on the DB side so that they are read quickly, or have some fast data measurement mechanism ( or replace the DB with some better one ).

BTW did you check our Solid File System (Standard Edition)? It's a good companion to store large data blocks, optionally in DB-like manner (with metadata etc).


Sincerely yours
Eugene Mayevski
#13126
Posted: 04/27/2010 02:57:05
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

Quote
BTW did you check our Solid File System (Standard Edition)? It's a good companion to store large data blocks, optionally in DB-like manner (with metadata etc).


Yes I'd actually looked at SolidFS ages ago but it did not match our requirements

What we basically have is a database of records, the number of records and the size of each record is variable

The users can then view those records in a different format via a CBFS drive

Dates can be formatted differently, number values padded, with or without decimal places etc, so the size of every file is different for every user, and different depending on the time of day

The formatting can also be changed by the user on the fly so the size of a file changes constantly, which means we need to load them, format them and say "this is the size at the moment"

It's fine doing that once when the user loads the file, but doing it every time FileInfo is called badly hurts... especially since the values are NOT USED at that point (10 files in a folder, 10 FileInfos but the allocated/end of file sizes are not actually used to show the file size at that point)
#13127
Posted: 04/27/2010 04:03:12
by Gavin McKay (Standard support level)
Joined: 09/01/2009
Posts: 48

Are you using a local mount or a network mount? A local mount is extremely "chatty" and Windows Explorer will call enumerate events and file info events a lot. Network mounts are a bit more relaxed about enumeration and don't call these as often. That might cut down the number of calls.

If you have file sizes that are constantly changing, that is going to be an issue for sure... for my system the back-end database can change the size of files, but it is an infrequent event and I use notifychange to signal Explorer what is happening. If the file sizes change continuously, and for each user... well that's going to be tough :)
#13129
Posted: 04/27/2010 05:03:56
by Eugene Mayevski (EldoS Corp.)

Quote
Chris Schirlinger wrote:
What we basically have is a database of records, the number of records and the size of each record is variable

The users can then view those records in a different format via a CBFS drive

Dates can be formatted differently, number values padded, with or without decimal places etc, so the size of every file is different for every user, and different depending on the time of day

The formatting can also be changed by the user on the fly so the size of a file changes constantly, which means we need to load them, format them and say "this is the size at the moment"


This approach won't work, sorry. If the OS has obtained the size of the data via one of the mentioned events, it will expect the file to be of this size unless you use NotifyDirectoryChange to tell the OS to rescan the folder. Also, you can't give a different file size for each local user. The file system works for the OS, not for the user. So it doesn't matter, what user requests directory information - file size must be the same.


Sincerely yours
Eugene Mayevski
#13130
Posted: 04/27/2010 06:54:33
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

Quote
Gavin McKay wrote:
Are you using a local mount or a network mount?


A local mount and unfortunately I think we must use that method. Several of our staff feel (for some reason) that a network mount will confuse customers and refuse to allow me to use it.

However, I'll check the network mount and see how fast that is, thanks for that :)

Quote
Eugene Mayevski wrote:
This approach won't work, sorry. If the OS has obtained the size of the data via one of the mentioned events, it will expect the file to be of this size unless you use NotifyDirectoryChange


That's fine and and is what we are doing. The file size does change but not THAT fast

When GetFileInfo is called and then later the ReadFile the sizes are still the same unless an update has happened or the user changes the format at which point we NotifyDirectoryChange.

We imagine once the users have setup their formatting they will never change it, and otherwise the data is "updated" from a remote server once a day. My main point was that for every user the file size is different and changes daily, so we can't really store that in the DB

By the looks of it I will just have to be VERY clever about determining the file size... (Currently I am compiling unique conversion code for the users formatting at run time at the start to save time, just need more tricks like that!)
#13136
Posted: 04/27/2010 16:31:36
by Gavin McKay (Standard support level)
Joined: 09/01/2009
Posts: 48

Not quite sure how it would confuse customers - particularly since you can set a custom icon now on the drive in v3.x, as well as name the "share" anything you want. So you can customise that drive quite a lot to avoid confusion. But they aren't my staff :D

The main benefit I found when swapping to a network mount is that Explorer doesn't automatically try and enumerate sub-folders until you actually 'click' on them. This stopped a lot of calls to EnumDirectory for me and meant I didn't have to check my back-end database for changes as often.

Can file size calculations be done in a background thread? You could then pre-calculate for the user, and just check what has changed in the background and notifychange when necessary?
#13138
Posted: 04/27/2010 16:49:04
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

Quote
Gavin McKay wrote:
Not quite sure how it would confuse customers - particularly since you can set a custom icon now on the drive in v3.x, as well as name the "share" anything you want. So you can customise that drive quite a lot to avoid confusion. But they aren't my staff :D


Yeah, I was unsure how it could confuse anyone, but I've been told if we can't "disguise" it as a folder on the C: drive, we can't do it... I have a feeling they don't understand the virtual DB so I may breech the idea later on

Quote
Can file size calculations be done in a background thread? You could then pre-calculate for the user, and just check what has changed in the background and notifychange when necessary?


Hurm... I will have to think about that, there are a lot of files, and a lot of data (dozen GB if the user has bought everything) and they may only EVER look at... say 1 meg of that data... maybe I can do something threaded at update time for the files most often looked at

Thanks for the idea :)
#13139
Posted: 04/27/2010 19:52:39
by Gavin McKay (Standard support level)
Joined: 09/01/2009
Posts: 48

Perhaps if it "looks" like a folder/drive, it will do the job?

I note in the release notes:

"New major version of Callback File System includes ability to create virtual directory on an existing NTFS disk. Therefore, while the number of drives remains the same, an access to required data is provided through the virtual folder. For end-users or third-party applications it will look exactly the same as any conventional folder. Developers who prefer creation of virtual drives instead of virtual folders will be glad to hear that a custom icon can now be assigned to any virtual drive. This improves ability to visualize distinction between virtual and ordinary drives. Moreover, an installation of CBFS-based end product is now significantly simplified reducing the load on personnel installing end-products and making end-user installation easier. And even more, data stored on virtual disks or inside virtual directories can now be easily located by their ID in addition to regular searches by name. "

I haven't tried creating a folder on an existing NTFS drive but that might be enough for you to make it "look" like a folder :)

You could also use a picture of a "standard" drive (i.e. non-Network) as the icon for the virtual drive.

If it looks like a folder, and acts like a folder, surely it must be a folder lol
#13140
Posted: 04/27/2010 20:15:13
by Chris Schirlinger (Standard support level)
Joined: 02/18/2010
Posts: 38

Quote
Gavin McKay wrote:
New major version of Callback File System includes ability to create virtual directory on an existing NTFS disk

I'll have to check the CBFS terminology but I *think* that's what we are doing now

We have added a mounting point on c:\aaa\a (as an example) using:
Code
mCbFs.AddMountingPoint(@"c:\aaa\a", CallbackFileSystem.CBFS_SYMLINK_MOUNT_MANAGER, null);

I believe this only works under NTFS and acts like a Local Mount. It gets a bucket load of GetFileInfo requests when you click on a folder at least
Also by EldoS: CallbackRegistry
A component to monitor and control Windows registry access and create virtual registry keys.

Reply

Statistics

Topic viewed 1994 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!