EldoS | Feel safer!

Software components for data protection, secure storage and transfer

About metadata performance

Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.
#11075
Posted: 09/15/2009 21:56:39
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I checked the size of thread pool on debugging mode by calling myCbFs.GetThreadPoolSize(). It returned 65535.

To measure IOPS and MB/s together, I changed benchmark tool from Iozone(fileop) to Iometer.
Then, I compared our client FS on CBFS with samba client.
I set the size of test file was over 4GB to avoiding caching effect and request size to 4KB, 16KB, 32KB.

In conclusion, our client FS was about 10 times slower than samba.
Average I/O response time of ours was over 5ms, but samba under 1ms.

What makes the difference... it's a natural result or not?
Unfortunately, I can't find what to do on our client FS to reduce that.
#11078
Posted: 09/16/2009 21:30:10
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

When two threads open to read same file, their reads are performed synchronously one by one..?
When I run Iometer with single worker, our client FS shows better IOPS than samba for sequential read. But, according to increase worker#, our client FS went worse and samba went better.

For example,
thread A requests a mkdir /test/a;
thread B requestsa a mkdir /test/b.
In the above case, two mkdir events are requested in parallel?
#11080
Posted: 09/17/2009 04:02:13
by Volodymyr Zinin (EldoS Corp.)

Quote
Sangmin Lee wrote:
I checked the size of thread pool on debugging mode by calling myCbFs.GetThreadPoolSize(). It returned 65535.

It means - default value. In that case the number of worker threads is equal to the number of processors in the system.

Quote
Sangmin Lee wrote:
In conclusion, our client FS was about 10 times slower than samba. Average I/O response time of ours was over 5ms, but samba under 1ms.

What makes the difference... it's a natural result or not?

It's because CallbackFS calls the user mode callbacks and each such call causes thread context switching - the originator thread waits or does something else in case of asynchronous I/O, and a worker thread calls a callback. Also the callback code can add some deceleration.

Quote
Sangmin Lee wrote:
For example, thread A requests a mkdir /test/a; thread B requestsa a mkdir /test/b. In the above case, two mkdir events are requested in parallel?

The create and close operations currently can't be called in parallel. But usually this isn't a problem because for one file/directory opening a lot of other operations are made (for example a file is opened, several read/write is performed, and then it is closed).
The other callbacks (such as OnRead, OnWrite, OnSetFileSizes, OnGetFileSizes, OnEnumerateDirectory, etc) can be called simultaneously (but only if the calls are being made for different files).
#11090
Posted: 09/17/2009 21:41:46
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

Quote
In that case the number of worker threads is equal to the number of processors in the system.

You mean .. if my PC has a quad-core CPU, then simultaneously runnable work threads limit to 4?

I said our filesystem manages all file metadata in a remote, central metadata server. That is, file synchronization's also controlled by the server.
On filesystems like ours, to perform callbacks to same file synchronously is right?
I'm not sure ...
#11092
Posted: 09/18/2009 01:33:28
by Volodymyr Zinin (EldoS Corp.)

Quote
Sangmin Lee wrote:
You mean .. if my PC has a quad-core CPU, then simultaneously runnable work threads limit to 4?

Yes, but it's just the default value and you can set the number of worker threads in a value that is optimal for you.

Quote
Sangmin Lee wrote:
I said our filesystem manages all file metadata in a remote, central metadata server. That is, file synchronization's also controlled by the server. On filesystems like ours, to perform callbacks to same file synchronously is right? I'm not sure ...

As I wrote above CallbackFS calls the user callbacks for the same file synchronously (one by one). If it didn't follow this rule then there would be the following situation. For example some file region is being written by one process and at almost the same time (but a little later) another process starts to read the file data. So the first is write request and the second is read request. If CallbackFS processed them asynchronously then there would be a possibility when the OnRead and OnWrite callbacks for both requests are called at almost the same time by two worker threads, but the read callback is called a little earlier and your server processes it first, i.e. in the wrong order.
#11094
Posted: 09/18/2009 02:41:51
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

Quote
if my PC has a quad-core CPU, then simultaneously runnable work threads limit to 4?

I'm sorry it's a stupid question. It's natural that simultaneously runnable work threads are the same as CPU#.
My question was thread# being forked to perform events limit CPU#; in other words, only 4 worker threads are forked.

I know the situation like your explanation.
And I think FUSE on linux also might consider it.
I ported our FUSE-based client code to callback functions.
But, performance gap is over 10 times. The gap with CIFS, too.
I can't conclude what makes the gap.
I'm not sure I should admit as it's a gap between linux and windows.
That's my dilemma.
#11096
Posted: 09/18/2009 03:20:23
by Volodymyr Zinin (EldoS Corp.)

Quote
Sangmin Lee wrote:
My question was thread# being forked to perform events limit CPU#; in other words, only 4 worker threads are forked.

In the current version worker threads are not forked. There is a static pool (that is created during a CallbackFS virtual disk creation) with the number of worker thread equal to the ThreadPoolSize property.
But we are going to implement dynamic worker thread pool in future versions.

Quote
Sangmin Lee wrote:
I know the situation like your explanation. And I think FUSE on linux also might consider it. I ported our FUSE-based client code to callback functions. But, performance gap is over 10 times. The gap with CIFS, too. I can't conclude what makes the gap. I'm not sure I should admit as it's a gap between linux and windows. That's my dilemma.

If the gap exists with CIFS too then perhaps CallbackFS doesn't influence on it. Or is CIFS used in conjunction with CallbackFS code? Try to use the Windows profiler (for details google for "KrView", "Kernrate"). Maybe it will help to find the gap.
Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages

Reply

Statistics

Topic viewed 8512 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!