Callback File System supports NTFS security attributes for files and directories. The OS sends security read and write requests to the file system and Callback File System passes those requests to your code via OnGetFileSecurity and OnSetFileSecurity callbacks. If you implement the handlers for these callbacks and store and retrieve the passed security information with your file system objects, Windows will show file security information to the user.
However when Windows sends requests to create or open the file or directory, it doesn't perform checking of access rights. It is the job of your callback handlers to retrieve security information from where you store it and use this information to validate the rights of the process which accesses the file system object. Then your code acts (i.e. provides or deny access to the file system object) according to results of such validation. Windows API offers an easy way to validate process access rights against some object's security attributes. This is AccessCheck() function of Windows API. You will find detailed description of this function in MSDN library.
Callback File System API offers very flexible security handling. It's supported using two security-related methods: GetOriginatorProcessName and GetOriginatorToken. First method lets you restrict access based on process name. Second method returns a system-defined security token of the calling process. You can use the security token to retrieve various security-related information using GetTokenInformation() function of Windows API.
To make use of the security checks you must first set CallAllOpenCloseCallbacks property to true.
GetOriginatorProcessName and GetOriginatorToken can be called from any callback. However not in all callbacks the calls would make sense. The file can't be read or written if it was not opened. This means that it makes sense to perform security checks, related to particular files, only in OnOpenFile and OnCreateFile callbacks, but not in OnWriteFile or OnReadFile callbacks. If you forbid creation or opening of the file, then the file won't be written or read.
Besides file opening/creation, it makes sense to check security in OnEnumerateDirectory, OnGetFileInfo and other callbacks, that retrieve information about the disk itself or it's directories and files.
Based on your checks you can accept or deny any operation. However, you may not alter the file based on the checks. I.e. if you reported file size to be 1 Kb, you should return exactly 1024 bytes when the file is read, and these 1024 bytes must be the same no matter which process performs reading. If you report some file to exist for process A, you must report the same file of the same size (unless file size has changed) to process B. You may not report the file to be a file for process A and a directory for process B.