EldoS | Feel safer!

Software components for data protection, secure storage and transfer

IO wait blocking issues

Also by EldoS: CallbackDisk
Create virtual disks backed by memory or custom location, expose disk images as disks and more.
#28464
Posted: 02/19/2014 13:41:56
by Brian Banet (Basic support level)
Joined: 02/19/2014
Posts: 3

Using CBFS.

We are managing files in a cloud data store with a local file cache. It's possible that files are not found in the cache and when this happens we need to go get them. Problem is the retrieval of the file in question might result in a delay, which we can manage, the CBFS processes are locked until control returns. We can't seem to find a way to allow CBFS to not block other requests when a long running request to get a file from cloud storage is running. Does this make sense? Any suggestions on how we can get things to run async?
#28465
Posted: 02/19/2014 13:52:31
by Eugene Mayevski (EldoS Corp.)

1. Set SerializeCallbacks property to false and MinWorkerThreadCount/MaxWorkerThreadCount - this will ensure parallel processing of requests to different files.
2. Set ParallelProcessingAllowed property to true to further optimize reading of data

But in any case, file open and close callbacks are always serialized - this is done to guard system resources. So you need to ensure that Create/Open and Close operations are as fast as possible.

And callbacks are always synchronous, so you can't postpone the request and tell Callback File System to call the callback later.


Sincerely yours
Eugene Mayevski
#28466
Posted: 02/19/2014 14:32:42
by Brian Banet (Basic support level)
Joined: 02/19/2014
Posts: 3

I understand the the callbacks need to be as fast as possible, but in the case of a file that is not cached (I.E. a file that is stored in S3), the speed at which that file is returned to me is controlled by both it's size and the connection speed.

Depending on the type of file request, it looks like ReadFile or GetFileInfo may get called depending on the situation. ReadFile appears to operate in parallel, but getFileInfo seems to be causing locks. Does this make sense?
#28467
Posted: 02/19/2014 14:57:47
by Brian Banet (Basic support level)
Joined: 02/19/2014
Posts: 3

Is there a way to override the readfile call and pump data from a file that is downloaded after the fact into it's BytesRead?
#28468
Posted: 02/19/2014 14:58:55
by Eugene Mayevski (EldoS Corp.)

Quote
bbanet@square-9.com wrote:
I understand the the callbacks need to be as fast as possible, but in the case of a file that is not cached (I.E. a file that is stored in S3), the speed at which that file is returned to me is controlled by both it's size and the connection speed.


That is, unfortunately, not the operating system's problem -- the OS was designed with dumb filesystems in mind, i.e. it expects the data to be available locally and quite fast. It's up to you how to arrange file caching.

In general you can set quite long timeouts (30 or even 60 seconds is rather sane timeout), but they can be not enough for slow networks.

Quote
bbanet@square-9.com wrote:
Depending on the type of file request, it looks like ReadFile or GetFileInfo may get called depending on the situation. ReadFile appears to operate in parallel, but getFileInfo seems to be causing locks. Does this make sense?


Depends on what you mean by locks. There always exists certain serialization because when the driver works with shared resources (eg. file or metadata cache) or with opened file or directory, it must lock access to this resource. Use of ParallelProcessingAllowed slightly reduces the number of locks. Still heavy caching if information by your application is recommended.


Sincerely yours
Eugene Mayevski
Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.

Reply

Statistics

Topic viewed 3435 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!