EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Customized file buffer size

Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.
#9405
Posted: 03/25/2009 06:31:18
by Amit Bhatia (Basic support level)
Joined: 03/25/2009
Posts: 13

Hello all,

I have been looking into the eldos CallBack SDK from last 2 weeks. I am not able to get one thing and I am not sure that is possible or not. I am explaining it as follows:

While implementing callback functions like Read and Write, we get Buffer allocated with the memory of size equal to size of file. I want that buffer to be big enough to accomodate a bigger file contents and hence the file which is demanded by the specific application, will be an altered/decoded file. I have the bigger file size written inside file only and hence can access that value in all those functions where we send the filecontext as per mapper sample source code. As these callback functions are OS generated, how can I help informing this to OS, some values which I need to alter or some more callback functions I need to implement.

I hope I am clear with my question.

void CbFsReadFile(
CallbackFileSystem* Sender,
CbFsFileInfo* FileInfo,
PVOID FileHandleContext,
__int64 Position,
PVOID Buffer, //This buffer I am pointing to
DWORD BytesToRead, // This contains the actual file size
PDWORD BytesRead
)

Thanks & Regards
Amit
#9407
Posted: 03/25/2009 07:18:24
by Volodymyr Zinin (EldoS Corp.)

Hello Amit,

CallbackFS just passes requests that the system initiated to the user mode callbacks. And the callback functions (CbFsReadFile, etc) are called with the same values as the system requests. For example if someone calls the win32 api ReadFile and specifies to read 0x4000 bytes then the system can requests to read either by 0x1000 bytes 4 times or 0x4000 bytes at once. Also it can request to read more than it was requested by the originator (a whole file for example).
So if you need a larger buffer during handling the CbFsReadFile request then allocate your own one. And after your specific processing copy data from it to the destination buffer.
#9417
Posted: 03/25/2009 10:57:11
by Amit Bhatia (Basic support level)
Joined: 03/25/2009
Posts: 13

I got your point, but my question is what if I need to decompress the data I read from file and want to return that expanded data to the application. As buffer size is BytesToRead, I can't put more data in buffer.

So I need some technique, by which I can tell OS that the file we are trying to open in an application is of expanedFileSize than originalFileSize. This way we can get buffer of size which will be of expanedFileSize than originalFileSize.

All this is assuming that application always requests for whole file.

Thanks & Regards
Amit
#9419
Posted: 03/25/2009 11:30:11
by Eugene Mayevski (EldoS Corp.)

You need to report decompressed file size in OnGetFileInfo and OnEnumerateDirectory callbacks. Your compressed file size doesn't make any sense to the OS. You need to tell the OS, how many data you will give it if needed. And the OS will perform read and seek operations based on uncompressed size. Your job is to
(a) store the original size of the compressed file somewhere;
(b) be able to perform seek on uncompressed data.


Sincerely yours
Eugene Mayevski
#9440
Posted: 03/30/2009 02:44:10
by Amit Bhatia (Basic support level)
Joined: 03/25/2009
Posts: 13

As per your information, I tried setting AllocationSize and EndOfFile variables in below 2 functions. But I am not able to get the desired buffer request in CbFsReadFile function as mention in the query above. Is there anythng else I need to set in functions below..?

void CbFsGetFileInfo(
CallbackFileSystem* Sender,
LPCTSTR FileName,
LPBOOL FileExists,
PFILETIME CreationTime,
PFILETIME LastAccessTime,
PFILETIME LastWriteTime,
__int64* EndOfFile,
__int64* AllocationSize,
__int64* FileId,
PDWORD FileAttributes
)


void CbFsEnumerateDirectory(
CallbackFileSystem* Sender,
CbFsFileInfo* DirectoryInfo,
PVOID* EnumerationContext,
LPCTSTR Mask,
INT Index,
BOOL Restart,
LPBOOL FileFound,
LPTSTR FileName,
PFILETIME CreationTime,
PFILETIME LastAccessTime,
PFILETIME LastWriteTime,
__int64* EndOfFile,
__int64* AllocationSize,
__int64* FileId,
PDWORD FileAttributes
)

Thanks
Amit
#9442
Posted: 03/30/2009 03:48:15
by Volodymyr Zinin (EldoS Corp.)

You can't obtain the desired buffer size. OnRead request can be called for any data size - from 1 byte to the whole file size (i.e. EndOfFile). But you can allocate an intermediate buffer with a necessary size and use it in the OnRead and OnWrite requests for compress/decompress necessary blocks of data. For example - if an OnRead request is called for one byte at the begin of a file, but you have the compressed data for the file and only 16-bytes block can be compressed/decompressed. In this case you use an internal buffer allocated by you, decompress the first 16-bytes block and return the first decompressed byte that is being requested in the OnRead function.
#9443
Posted: 03/30/2009 04:06:03
by Amit Bhatia (Basic support level)
Joined: 03/25/2009
Posts: 13

Hi Vladimir,

I understood that. But my question is how to tell OS that EndOfFile is at DecompressedFileSize than OriginalFileSize. What variable we have to set to make any application(through OS) able to request for data from 0 to DecompressedFileSize than 0 to OriginalFileSize.

For this I tried setting EndOfFile variable in the 2 functions metioned before, but that didn't helped.

Thanks
Amit
#9445
Posted: 03/30/2009 04:26:59
by Volodymyr Zinin (EldoS Corp.)

As Eugene said above - compressed file size didn't make any sense to the OS. It's up to you how your files are stored internally - either they are compressed or encrypted, or they are just records in some database, or something else. Specify only uncompressed file sizes in CbFsGetFileInfo and CbFsEnumerateDirectory requests, and return only uncompressed data form OnRead requests.
#9447
Posted: 03/30/2009 04:49:56
by Amit Bhatia (Basic support level)
Joined: 03/25/2009
Posts: 13

"Specify only uncompressed file sizes in CbFsGetFileInfo and CbFsEnumerateDirectory requests, and return only uncompressed data form OnRead requests."

Which variable in the function is to be set to specify that size?
I tried setting EndOfFile and AllocationSize variables, but in vain...

Thanks for your swift responses.

Regards
Amit
#9448
Posted: 03/30/2009 05:25:32
by Volodymyr Zinin (EldoS Corp.)

You do right by setting EndOfFile and AllocationSize variables. EndOfFile - is a real data size that a file contains and that can be read from the file. AllocationSize is a size of space that is allocated for a file in a storage where the file is located. AllocationSize can be >= than EndOfFile. For "usual" file systems that use block devices (disks, etc) AllocationSize is often multiple to sector size.

Quote
Amit Bhatia wrote:
I tried setting EndOfFile and AllocationSize variables, but in vain...

Please describe in more details what does it mean "in vain"? Thanks.
Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages

Reply

Statistics

Topic viewed 7055 times

Number of guests: 2, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!