EldoS | Feel safer!

Software components for data protection, secure storage and transfer

CallbackFileSystem: how to know the end of writing data

Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages
#5930
Posted: 04/18/2008 15:05:29
by Steffen Kux (Basic support level)
Joined: 04/11/2008
Posts: 5

Hello from germany,

I am just testing CbFs and wrote a test application (JAVA client with JNI connection to CbFs). The goal of the software is to represent a project (document) management system as a virtual file system.

Almost everything works but I have a problem with writing the data. I need the data in one block to send it to the database. In the event OnWriteFile I receive the file data sometimes in little parts (sometimes in pieces from start to end, sometime from back to end...). Often I know from the event OnSetAllocationSize how much data will be sent, but sometimes this doesn't work. Some software (e.g. MS Word) fire OnSetAllocationSize , send some data, fire again OnSetAllocationSize (with a bigger size) sends some more data and so on... :evil: so I don't know the end. How can I know, when I have all data received? But as I said before I need the whole data before sending it to the server.
:?: Thatswhy my question: How can I know, that all data is written?

Thank you very much for your advice!!!
Yours Steffen Kux.

Thank you very much
#5931
Posted: 04/18/2008 15:20:01
by Eugene Mayevski (EldoS Corp.)

Cache the data somewhere and submit them to the server when (or, better, after) OnFileClose is called.


Sincerely yours
Eugene Mayevski
#5932
Posted: 04/18/2008 15:48:36
by Steffen Kux (Basic support level)
Joined: 04/11/2008
Posts: 5

But this is also a problem. For example: If I create a new textfile (in explorer) open it with the notepad, enter some data and save it (and close the notepad), the data is written (from notepad always as one piece), but the file will not be closed. If someone would try to read the file again it remains still empty because I wouldn't have written it back to the server.
In this case the file is closed when I remove the mountpoint or unmount, and then the content is written, but this is to late, because I did send.

How can I know for sure, when writing is finished?
(Background: in the project management system saveing a document will create a new iteration which means it becomes checked in, thatswhy I need to know when I can perform the operation. When I write the file to memory or disk it doesn't matter, when I write it in pieces and even if the file will not be closed, but I need to decide when to send it as one piece and I need to dend it right after it seems to be written for the user to the virtual drive).

Is there any solution possible?

Thanks again.
Steffen Kux
#5935
Posted: 04/19/2008 03:30:47
by Eugene Mayevski (EldoS Corp.)

Quote
Steffen Kux wrote:
But this is also a problem. For example: If I create a new textfile (in explorer) open it with the notepad, enter some data and save it (and close the notepad), the data is written (from notepad always as one piece), but the file will not be closed. If someone would try to read the file again it remains still empty because I wouldn't have written it back to the server.


You must implement a full-featured cache, i.e. if the data is read, you read it from cache and not from remote location (where the old version is contained, or nothing at all if the cache is empty).

There's *absolutely* no problem in what you describe. You just need to think a little.


Sincerely yours
Eugene Mayevski
#5936
Posted: 04/19/2008 09:05:58
by Steffen Kux (Basic support level)
Joined: 04/11/2008
Posts: 5

Thank you for your answer. Be sure I thought a lot about these things. But it seems to me, we are talking about different things.

Of course I could solve the problem for one computer, but here I have a project communication system, where people almost around the world work together with the project documents. Thatswhy it is important, that a document is checked in immediatly after writing so that other persons somewhere else can use these documents. Only caching on one computer doesn't help, caching somewhere in the net is also not possible. (The project management system still exists and can be used by a web interface, we want to build a second interface for the direct use within the filesystem , but the web interface will be used also. We cannot change the this system, we can only use the interfaces (SOAP, RMI)) ).
And if someone puts in data via the webinterface, someone using the virtual file system must get this information and not some locally cached and by the other hand if I put in some data with the virtual file system, someone else must get this information (via the web interface or via the virtual file system on an other computer...

Thatswhy I need the information when writing is finished and I can snchronise the data. Is there any experience for doing something like that with CbFs?
Maybe I'm a little bit blind for a solution...

Thank you in advance.

Steffen Kux
#5937
Posted: 04/19/2008 12:03:51
by Eugene Mayevski (EldoS Corp.)

Ok, let me say it clear - you are taking the wrong approach. You can not track the abstraction. Never. Your design needs rethinking.


Sincerely yours
Eugene Mayevski
#5938
Posted: 04/19/2008 12:42:30
by Eugene Mayevski (EldoS Corp.)

Let's go into details... There's no such thing as "end of writing". When you write a sentence, the end of writing is marked by point mark. When the data is written to the file, end of writing is denoted by file close operation.

The only exception is databases, virtual file systems (like SolFS) and compound storages. They must be synchronized after some inactivity period, i.e. when no writing was done during 10 or 30 or 60 seconds or more. Maybe this approach will work for you, but to me it seems to be non-sense, as you are not actually synchronizing a database.


Sincerely yours
Eugene Mayevski
#5941
Posted: 04/21/2008 00:55:33
by Steffen Kux (Basic support level)
Joined: 04/11/2008
Posts: 5

I found the (simple) solution. (By the way, communication with the project data management system means communication with the underlying database!! Maybe I didn't describe it clear enough...)
Quote
Eugene Mayevski wrote: When the data is written to the file, end of writing is denoted by file close operation.

Now I found that CbFs has the option AllowDelayedClose an this option was set to true by default, meaning CbFs doesn't fire all close events just in time. Thats the reason I didn't receive close file events after finished writing. After setting AllowDelayedClose to false I receive the close file events right after writing the data and so I can synchronize (the cached data) with the database. :D

Thanks anyway.

Steffen Kux
#5942
Posted: 04/21/2008 01:13:36
by Eugene Mayevski (EldoS Corp.)

Quote
Steffen Kux wrote:
(By the way, communication with the project data management system means communication with the underlying database!! Maybe I didn't describe it clear enough...)


This is not the DB that I was talking about. It is tricky to keep a DB on virtual storage as it is never closed. But it doesn't matter where you store the files that you copy.





Sincerely yours
Eugene Mayevski
#5943
Posted: 04/21/2008 03:16:39
by Volodymyr Zinin (EldoS Corp.)

Hello,

Quote
Steffen Kux wrote:
Now I found that CbFs has the option AllowDelayedClose an this option was set to true by default, meaning CbFs doesn't fire all close events just in time. Thats the reason I didn't receive close file events after finished writing. After setting AllowDelayedClose to false I receive the close file events right after writing the data and so I can synchronize (the cached data) with the database.

Set also the property CallbackFileSystem.UseSystemCache to true. By setting this to true you tell the CallbackFS driver not to use the system caching for files. So the system cache manager won't hold references to previously opened files (in hope that the files will be opened again shortly and their data already in the cache) and the files will be closed at the time when applications close their handles.



Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.

Reply

Statistics

Topic viewed 10903 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!