CallbackRegistry API offers very flexible security handling. It's supported using two security-related methods: GetOriginatorProcessName and GetOriginatorToken. First method lets you restrict access based on process name. Second method returns a system-defined security token of the calling process. You can use the security token to retrieve various security-related information using GetTokenInformation() function of Windows API.
Based on your checks you can accept or deny any operation. However, you may not alter the file based on the checks. I.e. if you reported file size to be 1 Kb, you should return exactly 1024 bytes when the file is read, and these 1024 bytes must be the same no matter which process performs reading.