EldoS | Feel safer!

Software components for data protection, secure storage and transfer

The performance of version 4 and 5 of fusecbfs

Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages
#30219
Posted: 08/06/2014 03:46:16
by lingbo meng (Basic support level)
Joined: 09/16/2013
Posts: 24

The performance of writing small files(1024Bytes=1kBytes) ,the version 4 is better than version 5 .
Why ?

And in version 5,I have set
CBFS->SetFileCacheEnabled(1);
CBFS->SetNonexistentFilesCacheEnabled(1);
mknod=NULL;
setstrm=NULL
getstrm=NULL;
liststrm=NULL;
removestrm=NULL;
My using mode of the cbfs is liking fuse_mapper.

I test the performance using the tool is fastcopy. http://ipmsg.org/tools/fastcopy.html.en
#30220
Posted: 08/06/2014 04:12:46
by Volodymyr Zinin (EldoS Corp.)

Could you specify in details how you do testing and what a difference between the versions 4 and 5 you get?

Thanks.
#30221
Posted: 08/06/2014 04:38:08
by lingbo meng (Basic support level)
Joined: 09/16/2013
Posts: 24

Test tools is fastcopy ,copy 2000 files of 1kBytes to virtual disk of x.
the tools can display the rate of copy files.

The virtual disk of x is mounted by the mode of liking fuse_mapper.
#30226
Posted: 08/07/2014 01:02:55
by Volodymyr Zinin (EldoS Corp.)

CBFS v5 more actively uses cache than v4. Speed degradation can occur theoretically in this case because lots of small files are written through the cache but then the cached data has never been used.
Could you tell me the test result that you got (in Bytes/sec or so)? Also specify in more details what you do during the testing (i.e. parameters of fastcopy, whether the copied source files are located on the same disk or not, etc).

Thanks.
#30228
Posted: 08/07/2014 01:24:54
by lingbo meng (Basic support level)
Joined: 09/16/2013
Posts: 24

Today,I using fuse_mapper to test,I get the same result.version 4 is better than version 5.
一 The result is that,2000files ,1024Bytes per file.
version 4, copy 2000 files of 1kBytes ,the rate is 680 files/second or so.
version 5, copy 2000 files of 1kBytes ,the rate is 380 files/second or so.

二 The parameters of fastcopy is that,
1 the memmory size is 32MB
2 caculate the time of complete time

三 The copied source files are located on the same disk.
四 There is no different between enabled cache and disabled cache in version 5
#30237
Posted: 08/07/2014 21:25:53
by lingbo meng (Basic support level)
Joined: 09/16/2013
Posts: 24

Using default configure is ok,see the file of attachment
change the fastcopy.jpg to fastcopy.rar is ok.


#30253
Posted: 08/11/2014 05:09:01
by Volodymyr Zinin (EldoS Corp.)

Thank you for the information. We will check it and write the result later.
#30300
Posted: 08/13/2014 06:19:46
by Volodymyr Zinin (EldoS Corp.)

The problem is because of the CBFS file data cache. Lots of small files are copied sequentially and in CBFS4 the cache isn't used in this case, but for CBFS5 it's. In the next build we will correct the code for this case.
As a temporary solution for FastCopy you can disable the cache by setting the FileCacheEnabled property to false.
#30316
Posted: 08/13/2014 21:05:54
by lingbo meng (Basic support level)
Joined: 09/16/2013
Posts: 24

Is there some methods to solve this problem:
using the fastcopy to test of fuse_mapper,copy 2000files of 1kBytes from disk of d:\cbfs1k to z:\cbfs1k ,the rate is 320 files/second ;
but if destination is z:\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k\cbfs1k,the rate is 240files/second.
#30318
Posted: 08/14/2014 02:23:55
by Volodymyr Zinin (EldoS Corp.)

A file is copied in the following way:
1. The source file is opened.
2. The destination file is created.
3. One or several read/write requests are done.
4. The source and destination files are closed.

The difference in performance can be because during a file opening the file system parses full path in order to check whether the directories in the path exists. For CBFS FUSE it causes the getattr callback is called for each directory in the case it isn't cached yet in the CBFS metadata cache (by default this cache is enabled). In any case, even all the directories in the path are cached and the getattr callback isn't called for them each time, parsing such long path for 2000files requires time and the same is true for other file systems too (FATFS, NTFS).
Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.

Reply

Statistics

Topic viewed 3609 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!