EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Multi Threaded Speed

Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.
#5658
Posted: 03/28/2008 04:17:48
by Matt Redfearn Redfern (Standard support level)
Joined: 03/28/2008
Posts: 3

we have a few questions about the Standard Edition Solid File System that we would really appreciate some help with, I have

Is there any recommended methods to using multiple read/write thread to get the best thread performance out of the file system with multi core cpu's when working with multiple xml files?

We are using lots of 100Kb - 200Kb XML text files, could you recommend the block size and or the optimal read method to read these types of files with the fastest performance.

We are about to start testing the compression as our data compresses very well, does the compression occur on a single thread inside SFS? are there any best practices?

When compact is ran, is the file system locked?

We will be using multiple threads to work with multiple files concurrently, only one thread will open the file in write mode, can other threads open the same file in read only mode using SFS standard edition?

Thanks

Matt
#5660
Posted: 03/28/2008 07:44:12
by Alexander Plas (EldoS Corp.)

SolFS is completely thread-safe library. You can use any activity in different threads. SolFS doesn't uses additional internal threads for any operations such as compression or encryption.

If all your files are 100Kb or more it will be better to choose PageSize == 4Kb or more. To have better read/write performance you should use 16 * (PageSize - 32) buffer size, because SolFS by default compress data using block of 16 pages.

Compact operation is possible only if all storage files are closed and all other storage operations are completed. Compact locks whole storage.
#5664
Posted: 03/28/2008 13:23:48
by  NN NN
Hello,

Funny, I had a question about threading access to SolFS as well.

I am planning on opening multiple files in different threads. I will never be sharing the same file in different threads. I will also be reading multiple files in different threads.

Do these operations block each other? i.e. if different threads are writing to different files at the same time, will it go full speed ahead, or does writing to one thread block the other thread? Same question for reading?

Also, does writing to different files in the same thread lead to fragmentation?


#5669
Posted: 03/30/2008 07:27:37
by Eugene Mayevski (EldoS Corp.)

Operations on different streams/files won't affect each other.

Regaridng fragmentation: SolFS storages are usually located on the file system where they are fragmented anyway.


Sincerely yours
Eugene Mayevski
#5722
Posted: 04/03/2008 08:34:37
by Matt Redfearn Redfern (Standard support level)
Joined: 03/28/2008
Posts: 3

Thanks for the help, it is appreciated.

Matt
#33136
Posted: 04/25/2015 20:28:51
by William Egge (Standard support level)
Joined: 08/17/2011
Posts: 26

I don't believe the question has been answered, and I have the same question.

The response was that SolFS is thread safe, but the person was not exactly asking about thread safety. Something can be thread safe but not multi-threaded.

Take for example, if I were to place critical sections in my code - I can call it thread safe but it will still essentially be single threaded because of the blocking. It will not run "multi-threaded", to be truly multi-threaded I would need to allow multiple threads to be processing at the same time.

The question is if SolFS is "multi-threaded", not single threaded (in operation).

I did my own test and found that SolFS bogs down incredibly when there are multiple threads reading and writing streams. 6x slower than single threaded.


Answer?
Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.

Reply

Statistics

Topic viewed 5054 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!