EldoS | Feel safer!

Software components for data protection, secure storage and transfer

CBFS as local cloud cache

Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.
#28440
Posted: 02/18/2014 10:07:54
by Peter Williams (Standard support level)
Joined: 02/18/2014
Posts: 10

Hello, i'm using CBFS to point to a local directory on my server. This local directory acts as a cache for an AmazonS3 store. I am trying to write the part of my application that gets the file from the cloud when the it's not already cached locally.

Given a file path: [FileName = \\00\\00\\00\\file.pdf] that doesn't exist locally, i see that CbFsGetFileInfo(... String FileName, ...) is called for each directory in the path. My first approach is to create the directories as they are requested and hand them back to the SafeFileHandler. When i reach the end of the path however, because the file doesn't exist i don't know if the path in FileName is a directory or file.

Does CBFS have any way of telling me if a file or folder is being requested by CbFsGetFileInfo()?

Thank you.
#28441
Posted: 02/18/2014 10:11:13
by Eugene Mayevski (EldoS Corp.)

Quote
Peter Williams wrote:
Does CBFS have any way of telling me if a file or folder is being requested by CbFsGetFileInfo()?


You must know this yourself as it's you that provides the data. The OS can know the name of the file/directory to query from either the enumeration you've provided before (in which case you know what you have provided) OR if some application believes that the file or directory should exist on the disk (in which case you need to check existence of the file in the backend storage).


Sincerely yours
Eugene Mayevski
#28442
Posted: 02/18/2014 11:03:12
by Peter Williams (Standard support level)
Joined: 02/18/2014
Posts: 10

Hi Eugene, thank you for your quick response.

Quote
Eugene Mayevski wrote:
You must know this yourself as it's you that provides the data. The OS can know the name of the file/directory to query from either the enumeration you've provided before (in which case you know what you have provided)


I am using .net and calling
Code
cbfsPath = "S:\Test\00\00\00\test.pdf"
System.IO.File.Copy(cbfsPath, targetPath);


This results in CbFsGetFileInfo() being called for each directory, and i currently have it printing this:

Code
GetFileInfo: \Test, ----FileAttributes: 0
GetFileInfo: \Test\00, ----FileAttributes: 0
GetFileInfo: \Test\00\00, ----FileAttributes: 0
GetFileInfo: \Test\00\00\00, ----FileAttributes: 0
GetFileInfo: \Test\00\00\00\test.pdf, ----FileAttributes: 0
OpenFile: \, ----Properties: 16


So i am confused as to how GetFileInfo() would know if it's a directory or file being asked for, unless i am missing something or working with the wrong callback?

Quote
Eugene Mayevski wrote:
if some application believes that the file or directory should exist on the disk (in which case you need to check existence of the file in the backend storage).


Yes, i can check the backend storage to see if the file/folder exists there, but i cannot download it until i have the entire path to the file. I am trying to not make a lookup until i know the path being passed to CbFsGetFileInfo() is a file and not a directory.


Thanks,
Peter
#28443
Posted: 02/18/2014 11:07:47
by Eugene Mayevski (EldoS Corp.)

Quote
Peter Williams wrote:
So i am confused as to how GetFileInfo() would know if it's a directory or file being asked for


You should know this from the backend storage. The filesystem concept in Windows assumes that the filesystem is dumb and stores the data which it can read.

Quote
Peter Williams wrote:
Yes, i can check the backend storage to see if the file/folder exists there, but i cannot download it


You don't need to download it. You need to check if it exists and then get information about this object. The question of you creating some local caches is irrelevant here - you must get information from the storage and pass it to the OS upon request.


Sincerely yours
Eugene Mayevski
#28548
Posted: 02/25/2014 08:49:21
by Peter Williams (Standard support level)
Joined: 02/18/2014
Posts: 10

hi Eugene,

I've made some structural changes and now have things behaving the way i want when the requested file in the cloud is not already local:

GetFileInfo() callback returns cached file metadata in some datastore so the OpenFile() callback is hit.

OpenFile() callback is now downloading my file from the cloud asynchronously and then immediately returning, so as to avoid FS lock.

ReadFile() callback repeatedly sleeps until the download is complete, then it obtains a user context for itself and subsequent readfile() calls. The context is ever only obtained once.

CloseFile() callback closes the user context.

This is working 95% of the time correctly. The other 5% of the time the closefile() callback is hit before file reading is complete. I am noticing in my console logging that ref BytesRead resets to 0 every 4th callback, and the closefile() callback is always hit after one of these occurrences.

Here is a log of a file correctly read:
Code
Open File: mediumPdf_1.pdf
OpenFile: Download Cache File Async: \CloudTest2\00\00\00\multiple\mediumPdf_1.p
df
Read File: mediumPdf_1.pdf User Context: 0
ReadFile thread 51: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 51: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 51: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 51: Sleeping for file request: mediumPdf_1.pdf
Read File: mediumPdf_1.pdf: Position: 0 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf New User Context Set.
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 65536 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 131072 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 196608 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 262144 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 327680 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 393216 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 458752 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 524288 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 589824 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 655360 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 720896 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 786432 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 851968 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 917504 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 983040 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1048576 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1114112 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1179648 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1245184 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1310720 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1376256 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1441792 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1507328 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 727992
Read File: mediumPdf_1.pdf: Position: 1572864 Bytes to read: 62382, BytesRead: 0

Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 62382
Close File: mediumPdf_1.pdf
Close File: mediumPdf_1.pdf FileStream Closed.


And here is the same file calling close early and throwing an IOException:

Code
Open File: mediumPdf_1.pdf
OpenFile: Download Cache File Async: \CloudTest2\00\00\00\multiple\mediumPdf_1.pdf
Read File: mediumPdf_1.pdf User Context: 0
ReadFile thread 101: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 101: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 101: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 101: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 101: Sleeping for file request: mediumPdf_1.pdf
ReadFile thread 101: Sleeping for file request: mediumPdf_1.pdf
Read File: mediumPdf_1.pdf: Position: 0 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_1.pdf New User Context Set.
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 728040
Read File: mediumPdf_1.pdf: Position: 65536 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 728040
Read File: mediumPdf_1.pdf: Position: 131072 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Read File: mediumPdf_1.pdf User Context: 728040
Read File: mediumPdf_1.pdf: Position: 196608 Bytes to read: 65536, BytesRead: 65536
Read File: mediumPdf_1.pdf User Context existed, counter: 1
Read File: mediumPdf_1.pdf Bytes Flushed: 65536
Close File: mediumPdf_1.pdf
Close File: mediumPdf_1.pdf FileStream Closed.
Open File: mediumPdf_1.pdf
OpenFile: mediumPdf_1.pdf File Handle Obtained.
Close File: mediumPdf_1.pdf
Close File: mediumPdf_1.pdf FileStream Closed.



As you can see, close is called early, and then after the error open and close are both called again for some reason. Any insight?

Thanks in advance.
#28549
Posted: 02/25/2014 08:57:54
by Eugene Mayevski (EldoS Corp.)

I don't see any unexpected behavior in your description. Potentially the application (or several applications) may open, read and close the file whenever you want. You may not rely on any particular sequence of calls.

Check that you have CloseAllOpenCloseCallbacks property set to true and remember that the file can be opened and closed several times in parallel. OnOpenFile can be called several times in a raw, and so can OnCloseFile and OnCleanupFile (and the sequence will not be the opposite, i.e. the sequence

Open1
Open2
Open3
Close3
Close1
Close2

is possible.

.


Sincerely yours
Eugene Mayevski
#28552
Posted: 02/25/2014 09:11:23
by Peter Williams (Standard support level)
Joined: 02/18/2014
Posts: 10

Eugene,

closefile() only closes the handle if there is only 1 handle on the file, as shown above counter is 1 on each check of the handle, the application is making one request and then frozen, in test nothing else is requesting the file. In this very controlled scenario is this behavior still expected?


Also, in the above examples i have SerializeCallbacks set to false, and ParallelProcessingAllowed set to true. Should either of these be changed?

I have not set CloseAllOpenCloseCallbacks, i will try that next.

Thanks
#28553
Posted: 02/25/2014 09:14:38
by Eugene Mayevski (EldoS Corp.)

Now as I re-read your message and your log, I see the problem.

Please use GetOriginatorProcessName() method to log the name of the process which opens, reads and closes the file (i.e. add logging to all those callback handlers). Maybe we can find out which opens the file for the second time this way.


Sincerely yours
Eugene Mayevski
#28554
Posted: 02/25/2014 09:52:52
by Volodymyr Zinin (EldoS Corp.)

You can see what programs send requests by the use of Process Monitor from sysinternals.com. Run it and set there a filter to show only I/O operations for your CBFS virtual disk (by adding the filter "Path"->"begins with"->"X:").
#28556
Posted: 02/25/2014 10:13:19
by Peter Williams (Standard support level)
Joined: 02/18/2014
Posts: 10

Here is an error log with the process name retrieved at the start of open() read() and close().

Code
Open File: mediumPdf_9.pdf
Open File: Calling ProcessName: \Device\HarddiskVolume3\projects\async\IOTester\
bin\Debug\IOTester.vshost.exe
OpenFile: Download Cache File Async: \CloudTest2\00\00\00\multiple\mediumPdf_9.p
df
Read File: mediumPdf_9.pdf User Context: 0
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
ReadFile thread 201: Sleeping for file request: mediumPdf_9.pdf
Read File: mediumPdf_9.pdf: Position: 0 Bytes to read: 65536, BytesRead: 0
Read File: mediumPdf_9.pdf New User Context Set.
Read File: mediumPdf_9.pdf Bytes Flushed: 65536
Read File: mediumPdf_9.pdf User Context: 1053560
Read File: mediumPdf_9.pdf: Position: 65536 Bytes to read: 65536, BytesRead: 655
36
Read File: mediumPdf_9.pdf User Context existed, counter: 1
Read File: mediumPdf_9.pdf Bytes Flushed: 65536
Read File: mediumPdf_9.pdf User Context: 1053560
Read File: mediumPdf_9.pdf: Position: 131072 Bytes to read: 65536, BytesRead: 65
536
Read File: mediumPdf_9.pdf User Context existed, counter: 1
Read File: mediumPdf_9.pdf Bytes Flushed: 65536
Read File: mediumPdf_9.pdf User Context: 1053560
Read File: mediumPdf_9.pdf: Position: 196608 Bytes to read: 65536, BytesRead: 65
536
Read File: mediumPdf_9.pdf User Context existed, counter: 1
Read File: mediumPdf_9.pdf Bytes Flushed: 65536
Close File: mediumPdf_9.pdf
Close File: Calling ProcessName: \Device\HarddiskVolume3\projects\async\IOTester
\bin\Debug\IOTester.vshost.exe
Close File: mediumPdf_9.pdf FileStream Closed.
Open File: mediumPdf_9.pdf
Open File: Calling ProcessName: \Device\HarddiskVolume3\projects\async\IOTester\
bin\Debug\IOTester.vshost.exe
OpenFile: mediumPdf_9.pdf File Handle Obtained.
Close File: mediumPdf_9.pdf
Close File: Calling ProcessName: \Device\HarddiskVolume3\projects\async\IOTester
\bin\Debug\IOTester.vshost.exe
Close File: mediumPdf_9.pdf FileStream Closed.


My application is called IOTester.

As per Vladimir's suggestion, see this screenshot of procmon.

http://puu.sh/7a442.png

mediumpdf_8.pdf is successfully read, mediumpdf_9.pdf fails on the first read. Only my application is accessing the path.

Thanks
Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages

Reply

Statistics

Topic viewed 3815 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!