EldoS | Feel safer!

Software components for data protection, secure storage and transfer

About metadata performance

Also by EldoS: RawDisk
Access locked and protected files in Windows, read and write disks and partitions and more.
#10960
Posted: 09/04/2009 03:44:34
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I've tested FS metadata performance with window-based iozone fileop benchmark.

command: ./fileop -l 5 -u 5 -e -b -w -d m:

At first, I compared our FS with CIFS.
Our MDS and CIFS server are the same server.
The result is not good....at CIFS, rmdir recorded 236, but our FS, only 5.
Other operations(create, open, rmdir, ...) are not good, too.

I run it at our linux client. 'rmdir' recorded 243.

The difference between our linux and windows clients was CPU utilization of MDS daemon on our MDS.
When iozone run at our linux client, it showed above 20~30%.
But, in case of our windows client, it was under 2%.

I don't know why windows client requests slowly.
Can you consult me about that?
#10961
Posted: 09/04/2009 03:55:21
by Eugene Mayevski (EldoS Corp.)

I am sorry, but I didn't understand a single idea behind your post.

Please describe

0) what "MDS" is.
1) what architecture your file system has, i.e. whether you store the data locally or across network, whether you execute requests synchronously (right in callback / event handlers) or asynchronously (via worker threads) etc.
2) where the tool that you used is run from (the same system where CBFS works, other computer) and what exactly does (what operations are performed).


Sincerely yours
Eugene Mayevski
#10968
Posted: 09/06/2009 19:39:00
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I'm sorry I missed a explanation about our FS.

Quote
0)What "MDS" is?

==> 'MDS' is an abbreviation of MetaData Server, which is responsible for managing file metadata on our FS(GLORY-FS). It communicates with clients by TCP/IP.

Quote
what architecture your file system has, i.e. whether you store the data locally or across network, whether you execute requests synchronously (right in callback / event handlers) or asynchronously (via worker threads) etc.

==> Our FS is similar with google FS. Metadata and data path from clients are separated. Metadata is managed on an MDS and data on one more DS(data server)s.
We execute requests right in event handlers.

Quote
where the tool that you used is run from (the same system where CBFS works, other computer) and what exactly does (what operations are performed).

==> I used "fileop.exe within Iozone" on the same system where CBFS works.
I measured the time between send() and recv() in our CBFSOnCreateFile().

Code
GetSystemTime(&st);
   TRACE("CreateFile:BEFORE:mds_recv_res_head(): The system time is: %02d:%02d:%02d:%d\n", st.wHour, st.wMinute, st.wSecond, st.wMilliseconds);

recv_again:
   if ((sock_ret = recv(fd, ptr, len, 0)) <= 0) {
      if (sock_ret == 0)
      {
         res = -103;
      }
      else
      {
         res = sock_ret;
      }
      goto out;
   } else if (sock_ret < len) {
      ptr += sock_ret;
      len -= sock_ret;
      goto recv_again;
   }
   res = recv_size;

   GetSystemTime(&st);
   TRACE("CreateFile:AFTER:mds_recv_res_head(): The system time is: %02d:%02d:%02d:%d\n", st.wHour, st.wMinute, st.wSecond, st.wMilliseconds);


The time difference between BEFORE and AFTER was almost regularly 47ms.
I measured the same code on our LINUX client. It was under 10ms.
#10969
Posted: 09/07/2009 01:14:47
by Eugene Mayevski (EldoS Corp.)

You are actually measuring not performance of CBFS itself, but socket performance and performance of your metadata server, don't you? And you think that CBFS slows down network subsystem, is this correct?


Sincerely yours
Eugene Mayevski
#10972
Posted: 09/07/2009 02:11:29
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I don't think our poor metadata performance is due to CBFS.
Just, you're experts to Windows, so I hope you can consult about our problem.
#10973
Posted: 09/07/2009 02:29:54
by Eugene Mayevski (EldoS Corp.)

I see. First thing you need to do is copy the code you've shown above to outside of the callback handler, and check it's performance there. This will exclude possibility that CBFS internal locks affect performance. Please do this and let me know, and I will think what else we can suggest.


Sincerely yours
Eugene Mayevski
#10990
Posted: 09/09/2009 02:48:03
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I did it. Test environment was as the following.

- TCP server on LINUX
- TCP client on Windows XP

It run simply.
TCP client sends an mkdir request from to and receives a response from TCP server. Structures of request and response are the same as our FS.

I measured the ellapsed time between send() and recv() on TCP client.
It was from a few ms to tens of ms.
Accoring to the result, average processing time of mkdir is under 50ms.
Therefore, I think the result of Iozone fileop on our client may show tens of mkdirs.

At this point, I have a question.
After a directory by mkdir opeation is created, several open/close operations are performed.

In aspect of Iozone fileop, the completion time of an mkdir includes the time for all opens/closes after creation?
If it's right, I may understand it...
#10991
Posted: 09/09/2009 04:29:36
by Volodymyr Zinin (EldoS Corp.)

Quote
Sangmin Lee wrote:
In aspect of Iozone fileop, the completion time of an mkdir includes the time for all opens/closes after creation?

I don't think so. It seems programs, that wait for parent directory changes, re-enumerate this directory after the subdirectory has been created. And they do it asynchronously.
But of course, theoretically there can be other components that influence the directory creation process and perform some additional operations. For example, file system filter drivers can intercept file/directory creation and do some more work that can be done synchronously. So its time will be included to the time of the initial request.
#11026
Posted: 09/11/2009 09:27:28
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

Quote
whether you execute requests synchronously (right in callback / event handlers) or asynchronously (via worker threads) etc.


In our client FS, all events are performed synchronously.
It means all events must perform by a single thread. Right?
If then, poor performance might be a natural result...because all events may be serialized ..

If I want to process events asynchronously, where should I fork worker threads to execute them at? Like https://www.eldos.com/forum/read.php?PAGEN_2=3&FID=13&TID=1010&phrase_id=429650#nav_start_2?

Let me know how to do.
Thanks a lot for your help.
#11040
Posted: 09/14/2009 02:23:10
by Volodymyr Zinin (EldoS Corp.)

Events for same file/directory are performed synchronously one by one. It's necessary because several operations for a file can be made at the same time. For example a file can be read and written at the same time by different applications and CallbackFS must call the OnRead and OnWrite events in the same order as the original requests are made.
But operations for different files will be performed in parallel if you specify more worker thread during the CallbackFS initialization (for details see the CallbackFileSystem.ThreadPoolSize property).
Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.

Reply

Statistics

Topic viewed 8516 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!