EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Getting started with Callback File System

Introduction.

Callback File System consists of two components - a kernel-mode driver and a user-mode API. The driver works under the hood and you don't work with it directly. The API is linked to your user-mode application. You use the API to create, delete and manipulate the storage. The API also tells your application to perform certain actions which are required to actually implement the file systems.

Install and uninstall the drivers

Before the driver can be used it must be installed. This can be done using Install() API method or using the Installer DLL. API is used when the executable module of your application installs or uninstalls the driver. The Installer DLL was designed to be called from installation scripts, which are compiled into setup programs by special setup applications, such as Wise, InstallShield, InnoSetup etc..

After the driver is installed, it might be necessary to restart the system in order for the new driver to be installed and used. The API method or installer DLL will tell you if you need to restart the system.

The driver must be installed once. You don't have to install it each time you start your application. If the driver is not installed correctly, you will get error 2 ("file not found") which means that the driver could not be found or loaded.

When driver installation is complete, you can use the API to create and manage the storage with the virtual file system.

When you don't need the driver anymore, you can uninstall it using Uninstall() API method or using the installer DLL.

Create and delete the storage

Before CallbackFileSystem class can be used (and virtual disks can be created), you need to call Initialize() method to initialize the virtual filesystem.

The storage consists of the "drive" and the "media", similar to CD-ROM drive and the actual compact disk which is inserted to the drive.

So with Callback File System you first create a "drive" by calling CreateStorage() API method. Next, you assign a drive letter to the created "drive" using AddMountingPoint() API method. You can create a visible drive by assigning a drive letter or create an invisible drive by assigning a UNC path. The invisible drives (mounted via UNC paths) are not shown in Explorer or other file managers. They can, however, be accessed from the applications. Also the virtual file system can be mounted as a folder on the existing NTFS drive.

Finally, you insert the "disk" to the "drive" by using MountMedia() API method. But wait. How will the system know how to work with the given disk? You need to implement the actual file system. This is done by handling the callbacks (events) of CallbackFileSystem class as described below.

When you don't need a disk anymore, you call the opposite sequence of methods - first you call UnmountMedia() method to remove the "disk", then delete the "drive" using DeleteStorage() method.

Handle the callbacks (events)

Callbacks (event handlers) are the place where you implement the functionality of the file system. While we tried to hide the most complexities under the hood, there's still a lot of work for you to be done in order to implement the functionality of the file system.

All callback functions can be grouped into several groups by functionality. All groups of callbacks except named stream callbacks and security-related callbacks are mandatory. This means that your application must handle those callbacks.

Callback functions are called by the API when Windows operating system (the OS) gets to know about your "drive" and your "disk" and needs to work with them. Some requests are handled internally by CBFS, but most of them (those which perform actual work) are passed to your application by means of callbacks.

Tell the OS about file system capabilities

First of all the OS needs to know size and name of the disk. You need to handle OnGetVolumeId, OnGetVolumeLabel and OnGetVolumeSize events. Follow the links to get detailed information about these events. They are simple so there's nothing to discuss here about these events.

The OS can change the volume label. This type of request is passed to your application using OnSetVolumeLabel callback.

Let the OS get information about files and directories

The next important operation for any disk is to provide information about it's contents to the OS. When the OS needs to read contents of some directory (starting from the root directory), it sends the requests which are translated into OnIsDirectoryEmpty, OnEnumerateDirectory and OnCloseEnumeration callbacks.

OnIsDirectoryEmpty is a simple callback which is used to find out only the fact if some directory is empty. If the directory is empty, it is not enumerated.

OnEnumerateDirectory callback is called when the OS starts or continues enumeration of some folder. Unlike Win32 interface, which has FindFirstFile and FindNextFile functions, in Callback File System OnEnumerateDirectory callback is called for both first file and consequent entries in the directory. The API calls OnEnumerateDirectory callback in a loop until OnEnumerateDirectory returns false in FileFound parameter. The application must provide information about some file or directory, contained in the directory which is enumerated, via parameters of OnEnumerateDirectory callback. Each time different file or directory must be reported.

To store information about which file or directory was reported the last time (and so to know which file or directory must be reported next) the application can use EnumerationContext parameter. The following scenario can be used: if EnumerationContext is not allocated, this is the first call in a row to enumerate the given directory. The application can open access to the data, which represents the directory or perform other operation needed to get access to the list of the entries. Also the application allocates the context and passes it back to the API. If the context is already allocated, it is used for enumeration.

Once the application tells the API, that there are no more files or directories left to enumerate, the API calls OnCloseEnumeration callback and passes it the allocated enumeration context. You need to free the allocated resources and dispose of the enumeration context.

In addition to the above callbacks, two more callbacks are called during directory enumeration. They are OnOpenFile and OnCloseFile. Despite the names of these callbacks say about files, the callbacks are also called for the directory (Windows doesn't distinguish files and directories when it sends requests to the file system). These callbacks are described further.

Whle directory enumeration is used to get the list of entries in the directory, it's sometimes necessary for the OS to obtain information about some particular file. Such request is passed to your application via OnGetFileInfo callback. This callback is very similar to OnEnumerateDirectory and requests the same information, but gives you the name of the file or directory of interest and is only called once.

Create and open files

The next important operation is to create new files and open existing files. CallbackFileSystem object has two callbacks for this: OnCreateFile and OnOpenFile. As the names suggest, these callbacks are called when the OS needs to create new file or to open existing file.

When your application handles these callbacks, it usually opens access to some resource (be it a file, some remote resource or just a reference to a memory block). It is necessary to keep the handle to the opened resource somewhere in order to use this handle in consequent file operations. CallbackFileSystem offers FileHandleContext parameter. The application can store the above mentioned handle in the context. The context is passed to other callbacks, which are called when operations on open files (reading / writing / seeking / closing etc.) are performed. Your callback handlers will identify the opened file using this context.

To optimize multiple file opening operations which occur in parallel (for example, when you open two instances of some application which opens supplementary files or DLLs), Callback File System by default calls OnOpenFile callback only the first time the file is opened. If the file is opened successfully, the opened file is used for the next file open operations, which happen while the original file handle is opened. Usually this works correctly, but if the file is opened with different access mode, such scheme is not acceptable. To disable the described behaviour and have OnOpenFile callback be called every time, set CallAllOpenCloseCallbacks property to true.

As mentioned above, OnCreateFile and OnOpenFile are called for files and also for directories. The directories are created via OnCreateFile callback and they are opened for contents enumeration using OnOpenFile callback.

Close the files

After the file is read or written, it is closed by the OS. Closing doesn't necessarily happens as soon as the application, that used the file, calls CloseFile() Win32 function. In some cases the OS (it's cache manager) keeps the handle to the file open for some time.

Callback File System receives the request to close the file and calls OnCloseFile callback. Much like the OS, Callback File System driver delays closing of the files in order to save resources and time. This behaviour can be disabled by calling AllowDelayedClose() method and passing False as parameter.

OnCloseFile callback includes the file handle context as a parameter, and this context must be disposed of by your application.

If Callback File System called OnOpenFile callback only the first time the file was opened, OnCloseFile will also be called only when the last handle to the file is closed by the OS (see description of this behaviour above).

When directory contents enumeration is performed by the OS, OnCloseFile is called for the directory being enumerated when the enumeration is completed.

Read from files and and write to files

When the OS needs to read the data from the file or write the data to the file, Callback File System API uses OnReadFile and OnWriteFile respectively. Each of those callbacks includes parameter, which specifies the offset (position) in the file from which the application must read the data or to which the application must write the data. Also each of the callbacks includes FileHandleContext parameter. The file, on which the operation is performed, is identified using this context.

Handle file size change requests

Before writing the data the OS tells the file system, that certain amount of space must be allocated for the file to accept more data. This is done using OnSetAllocationSize callback. If size is not allocated, the OS won't be able to write the data. Allocation size should not be confused with file size. Allocation size is the space which is occupied by the file. File size is the actual size of the file contents. Allocation size is always equal or larger than file size. If the OS or some application needs to change the size of the files, it sends the request which is translated into OnSetEndOfFile callback. Each of the callbacks includes FileHandleContext parameter. The file, on which the operation is performed, is identified using this context.

Handle file renaming and deletion

File and directory renaming (including change of the file location in the directory tree) and file and directory deletion are atomic operations which must be handled via OnRenameOrMove and OnDeleteFile callbacks.

Create and delete directories

The directory creation must be performed in OnCreateFile callback (which is described above).

Deletion of the directories must be done in OnDeletFile callback.

Advanced operations

Advanced operations include manipulations with the named streams, security operations (retrieval and change of file and directory security attributes) and raw volume operations.

Work with named streams

Named (alternate) streams are the separate data chunks, stored in the file. They have their own names inside of the file. Support for named streams is optional.

Work with named streams is very similar to work with regular files. The named streams are opened via OnOpenFile callback and are read/written/closed in the same way as files are. When the named stream is opened, the file name contains a semicolon (":"), which delimits the file name and the stream name.

Enumeration of the named streams is done via OnEnumerateNamedStreams callback in a way similar to enumeration of directory entries. Once enumeration is complete, OnCloseEnumeration callback is called.

If you don't want to support named streams, don't assign a handler for OnEnumerateNamedStreams callback. In this case Callback File System will tell the OS that the named streams are not supported.

Handle security operations

Security operations include setting and retrieving ACL (access control lists) for the files and directories. Security operations are implemented via OnGetFileSecurity and OnSetFileSecurity callbacks. These callbacks are optional. If your application doesn't assign handlers for them, Callback File System will tell the OS that security operations are not supported.

What you may and may not do in callback handlers

The answer is short - the less calls (direct or indirect) to OS API is done the better. First of all, many system calls can lead to to a system-wide deadlock. Network operations are generally safe, but calls which involve the GUI can be unsafe. This is especially true when some third-party file system filter driver (such as antivirus or firewall application) comes into play.

Another reason is that the operation can take certain time and OS was not designed in the way that expects file system operations to be fast (milliseconds or a couple of seconds is an upper limit expected by the OS). So if your callback takes a minute to handle the callback, you can be sure that the whole system will be waiting for your callback (in some cases).

If allowed by file system implementation design, try to cache the file system operations in memory and perform all disk, GUI and network operations asynchronously, i.e. in a separate worker thread and not in the callback handler.

Asynchronous operations

The idea of asynchronous operations is to have a worker thread (or several threads) which will help the callbacks do their job. Callback function works with the local cache and the worker thread transfers the data to and from the actual data location. Implementation of the worker threads and interthread communication is outside of scope of this article.

Return to the list

|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!