EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Parallel callbacks

Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.
#16448
Posted: 05/13/2011 02:15:21
by Robert Baer (Basic support level)
Joined: 11/08/2010
Posts: 46

Hello,

is CBFS able to call callback functions in parallel? The description of SerializeCallbacks tells me that it is -- if they concern different files.

But I have the following behaviour:
When OpenFile is called for the file "A.txt" no other callbacks are called until OpenFile is finished. The enumeration of a different folder is waiting until OpenFile of "A.txt" is completed.

I can produce this behaviour with Mapper sample when adding a "Sleep(10000)" just before the call of CreateFile(): When opening a file and meanwhile enumerating a folder, the enumeration is waiting 10 seconds...

Can you reproduce this behaviour?

Regards,
Robert Baer
#16449
Posted: 05/13/2011 04:07:45
by Volodymyr Zinin (EldoS Corp.)

Hello Robert,

Quote
Robert Baer wrote:
s CBFS able to call callback functions in parallel? The description of SerializeCallbacks tells me that it is -- if they concern different files.

It's true.
But there are synchronization mechanisms in CallbackFS as well as in the system that don't allow to perform some operations in parallel. Usually it concerns such operations as Create, Open, Close which are performed not often and don't need much time to execute. In opposite the operations such as Read and Write can be called in parallel and can be time consuming (for example writing 1Gb of data). This is done so because an application usually opens a file once and then performs many I/O operations on it.
Of course we are trying to minimize global synchronization use and in the future releases it will be less.
BTW in order to have parallel call of the callbacks don't forget to set the CallbackFileSystem.ThreadPoolSize property to be more than one.
#16652
Posted: 06/14/2011 07:37:08
by Robert Baer (Basic support level)
Joined: 11/08/2010
Posts: 46

Dear Vladimir,

thanks for your answer. I just realized that there is a synchronization problem between ReadFile and EnumerateDirectory too. It takes a long time to enumerate a directory when another (big) file is read. You can reproduce this with Mapper sample by putting a Sleep() into ReadFile callback.

Will there be sychronization improvements in the next update? When is it planned? The performance is really bad like this.

Regards,
Robert Baer
#16653
Posted: 06/14/2011 08:04:13
by Volodymyr Zinin (EldoS Corp.)

Hello Robert,

Seems the problem is because there is only one worker thread which calls the callbacks (see the CallbackFileSystem.ThreadPoolSize property) or the callbacks are serialized (see the CallbackFileSystem.SerializeCallbacks property).
#16655
Posted: 06/14/2011 08:36:26
by Robert Baer (Basic support level)
Joined: 11/08/2010
Posts: 46

Dear Vladimir,

SerializeCallbacks property is set to false and ThreadPoolSize is set to 128.

This is how the callbacks are being called (first value is the time):

Code
15:27:26  >>> ReadFile
15:27:36  <<< ReadFile
15:27:36  >>> ReadFile
15:27:36  >>> EnumerateDirectory
15:27:36  <<< EnumerateDirectory


At 15:27:36 ReadFile is finished (with its 10 sec delay) and finally EnumerateDirectory is called for the first time. They are called in parallel afterwards but I have to wait 10 seconds to get the enumeration started.

Regards,
Robert Baer
#16698
Posted: 06/17/2011 09:59:18
by Volodymyr Zinin (EldoS Corp.)

I've tried to reproduce it but without success. I did it by setting the Sleep API call into the OnRead callback for the 1.txt file. Then opened this file which of course caused waiting, but navigating via the virtual disk structure as well as calling the CMD "dir" command worked without waiting. The OS on which I tried it was Win7 x64.
Perhaps your test scenario is different from it.
#16759
Posted: 06/21/2011 03:07:15
by Robert Baer (Basic support level)
Joined: 11/08/2010
Posts: 46

Dear Vladimir,

thank you for trying to reproduce this issue. I am able to reproduce it on Win7 x64 by performing the following steps:

1) Mount the virtual drive (Mapper sample)
2) Open 2 instances of Windows Explorer, both navigating to the virtual drive
3) Opening the 1.txt file in the first instance
-> Notepad pops up and waits...
4) Directly open a folder in the second instance
-> Windows Explorer waits for the OnReadFile operation to finish and shows the content finally when notepad is finished

I recognized that it is working fine when the file is opened for the second, third... time. Are you able to reproduce it like this?

Regards,
Robert Baer
#16813
Posted: 06/24/2011 10:36:20
by Volodymyr Zinin (EldoS Corp.)

I've reproduced it. The problem is really not in the enumeration callback. For some reason when from the other explorer window you are trying to enumerate a folder, explorer from one thread is performing reading a file in this folder (in my case it was an ".ico" file) and from another one is trying to open the same file. And this open request waits until the reading is finished.
#16840
Posted: 06/27/2011 05:34:13
by Robert Baer (Basic support level)
Joined: 11/08/2010
Posts: 46

Dear Vladimir,

thanks for reproducing this issue.

As a result, the problem is located in synchronisation of OnOpenFile callback?

I hope you are able to minimize the synchronisation locks for the next release. For us this is important because at the moment users are waiting for other users opening files in totally different folders.

Are you able to tell when you are going to release next version?

Regards,
Robert Baer
#16853
Posted: 06/28/2011 02:07:03
by Volodymyr Zinin (EldoS Corp.)

Hello Robert,

Yes. The problem is in synchronization of the OnOpenFile callback.
Of course we are going to upgrade it (to minimize maximally synchronization in CallbackFS in whole) in the nearest future. But I don't think it will be in the next release.
Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages

Reply

Statistics

Topic viewed 1645 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!