EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Filter notifications on SSD's

Also by EldoS: SecureBlackbox
200+ components and classes for digital security, signing, encryption and secure networking.
#32226
Posted: 02/17/2015 02:53:16
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

Hello,

Would you know whether there are any file-operation notifications which might be raised on an SSD drive and which the filter does not capture?

We have an application that uses the filter to listen to file notifications, and our tests indicate that all notifications are received and processed by the filter when running on a hard disk. Similar tests running on an SSD are showing 'gaps' in our results which makes us think that some writes are missed.

For e.g. Windows 8/2012 makes use of TRIM and UNMAP commands for SSD's as outlined here: https://msdn.microsoft.com/en-us/library/windows/desktop/hh848053(v=vs.85).aspx.
Is there a possibility that these commands are not captured by the filter?

This is just a hypothesis for now, as everything is possible including the possibility of hard-to-catch bugs in our app, but so far we do not have this indication, and we are getting in touch to start ruling out different hypothesis.

Thanks in advance for your feedback.

Regards,
Chris
#32227
Posted: 02/17/2015 03:47:22
by Vladimir Cherniga (EldoS Corp.)

Hello,

Quote
Chris Spiteri wrote:
Similar tests running on an SSD are showing 'gaps' in our results which makes us think that some writes are missed.

Callback Filter works on a file system level. Windows may use different cache policy for the SSD and hard drives. That may cause a less cached read/write requests. But in a file system level all requests must be filtered. As an addition you may check the file system activity with another tool like "Process Monitor" from Microsoft and compare your results.
Quote
Chris Spiteri wrote:
For e.g. Windows 8/2012 makes use of TRIM and UNMAP commands for SSD's as outlined here: https://msdn.microsoft.com/en-us/libra...s.85).aspx. Is there a possibility that these commands are not captured by the filter?


This is a disk level commands and they work directly with a storage level subsystem. File system filters works on another level, filtering file system requests. TRIM and UNMAP must work transparent for file system filter.
#32251
Posted: 02/17/2015 10:21:48
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

Thanks for your reply. We have set up ProcessMonitor to run for some time in conjunction with our tests so that we can compare results.

Best regards,
Chris
#32271
Posted: 02/19/2015 05:00:38
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

So we set up another test as follows:

1. Took md5 hashes of 128K blocks from a file on an SSD under test which we are monitoring; these hashes are used for comparison/change detections at the end of the test.

2. Set up Process Monitor to catch all activity on the file under test, while at the same time our test tool used the filter to capture offset-length pairs of write and eof notifications on the same file.

3. Performed random writes on the test file.

4. At the end of the test, took second set of Md5 hashes of 128K blocks from the test file, and compared with hashes in (1).

5. For each block whose md5 changed, we searched for the offset range in the ProcessMonitor and filter logs (for e.g. if block offset that changed was 104,857,600, range of possible writes was between 104,857,600 and 104,988,672, so we used regex's to search for entries that match 104 followed by 6 digits, which fall in that range.

6. No entries were found in both ProcessMonitor and filter logs for mismatched blocks.

So, even though the file actually changed in some areas, neither ProcessMonitor nor the filter were able to detect these changes.

We would appreciate it if you could assist us in troubleshooting this further.

Regards,
Chris
#32273
Posted: 02/19/2015 05:25:09
by Vladimir Cherniga (EldoS Corp.)

Did you control cached and non-cached writes ? Non-cached requests may be delayed in time, as they performed asynchronously based on cache policy of system.
#32336
Posted: 02/25/2015 11:32:19
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

We will try some tests with caching to see how it goes, but we are not very convinced that caching is influencing this behaviour.

We are noticing a particular operation in Process Monitor, FSCTL_FILE_LEVEL_TRIM, which seems to be zeroing some regions of the file.

According to MSDN, FSCTL commands can be processed by minifilter drivers (https://msdn.microsoft.com/en-us/library/windows/hardware/ff540367%28v=vs.85%29.aspx).

We would like to be able to detect the regions where the zeroing takes place using the offset/length values passed using the FSCTL_FILE_LEVEL_TRIM command and its FILE_LEVEL_TRIM_RANGE parameter as outlined here: https://msdn.microsoft.com/en-us/library/windows/desktop/hh447301%28v=vs.85%29.aspx

Is there a way how the CallbackFilter can detect these?

Thanks for your assistance so far.

Best regards,
Chris
#32337
Posted: 02/26/2015 02:51:50
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

The attached screenshot illustrates better what we mean.

We wrote a tool that uses DeviceIoControl functions to send FSCTL_FILE_LEVEL_TRIM and FSCTL_SET_ZERO_DATA commands to a file at particular offsets. Using ProcessMonitor to monitor the target file, we can see these commands being executed successfully.

We would like to be able to do the same thing using the CbFilter and would appreciate your assistance in this.
#32339
Posted: 02/26/2015 04:21:14
by Vladimir Cherniga (EldoS Corp.)

Quote
Chris Spiteri wrote:
We are noticing a particular operation in Process Monitor, FSCTL_FILE_LEVEL_TRIM, which seems to be zeroing some regions of the file.

Quote
The FSCTL_FILE_LEVEL_TRIM control code is a hint to the underlying storage system.

Such type or request may be processed in filter driver, but only the range of FILE_LEVEL_TRIM_RANGE structures may be processed. Actual data is trimmed in storage driver stack level, that is not available in file system filter driver stack, so you cannot control data flow directly. And the result of file level trim command is not determined without reading the file at specified offsets (according to MSDN). What do you plan to do with such information ?
#32340
Posted: 02/26/2015 04:29:42
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

We do not need the data itself, we just need the offset and length pairs of the FILE_LEVEL_TRIM_RANGE structure, which will indicate where the trimming changes have potentially taken place. Having these offset/length pairs, we can then examine the file ourselves to determine whether the trimming has actually occurred.
#32341
Posted: 02/26/2015 05:04:06
by Chris Spiteri (Standard support level)
Joined: 10/06/2014
Posts: 57

Will you be providing this feature as an update in a new build of the CbFilter?
Also by EldoS: CallbackProcess
A component to control process creation and termination in Windows and .NET applications.

Reply

Statistics

Topic viewed 11472 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!