EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Write and delayed close

Also by EldoS: SecureBlackbox
200+ components and classes for digital security, signing, encryption and secure networking.
#16783
Posted: 06/22/2011 23:55:47
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I wrote a test program, which forks 2 worker threads.

Thread A:
{
create;
write;
close;
}

Thread B:
{
create;
write;
close;
}

Each thread runs as many as loop count. Each file can be the same or different.

The usage example: test.exe X:\test 10 10000.
X: is a mounted driver by CBFS;
10 is the number of worker threads being forked;
10000 is a loop count of each thread.

And, callback messages print out to system event viewer.

Before testing, I thought close is called immediately after write.
write;
close;
write;
close;
....

But, The result was different.

write;
write;
write;
write;
write;
write;
close;
close;
close;
write;
write;
close;
....

That's why? Windows OS kernel calls like them? or CBFS does?

Sangmin
#16784
Posted: 06/23/2011 01:54:20
by Volodymyr Zinin (EldoS Corp.)

CBFS does but Windows in some cases imposes restrictions on file systems to process the close request asynchronously (and in your case it occurs so).
Is it somehow important for you to have the calls in "right" order.
#16785
Posted: 06/23/2011 02:12:39
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

It's important on our filesystem(FS).
Our FS is a class of network FS.
File metadata and data all store to remote metadata nodes and storage nodes(DS) by TCP.
On data write, a TCP connection to DS is established on first WriteFile().
And, this connection is disconnected on CloseFile().

I should test a large of small writes(size is KB ~ hunreds of KB) for applications on our FS.

Because of delayed close, on our client, resource lack error happens and, on our DSs, connections are blocked until client does closesocket().
So, I hope when application calls close(), CloseFile() is immediately calls.

Sangmin
#16786
Posted: 06/23/2011 02:22:06
by Volodymyr Zinin (EldoS Corp.)

CBFS performs closing as soon as possible. Try to implement a counter for a file which is incremented in the open callback and decremented in the close callback. Then close the socket only when the counter is set to zero.
#16787
Posted: 06/23/2011 02:51:30
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

I implemented the counter, which increments 1 whenever OpenFile() to same file and decrements 1 on CloseFile().
Always, the count is 1.

I run a single thread, which performs create->write->close in order.
When a loop count is 10, do 10 couples of create and write and then, 10 close calls in a time.

Is there any special reason for delayed close?

I think it can cause significant serious performance/reliability problems on our FS. ㅠㅠ;;

Sangmin
#16788
Posted: 06/23/2011 03:10:03
by Volodymyr Zinin (EldoS Corp.)

Quote
Sangmin Lee wrote:
Always, the count is 1.

Alternatively the socket can be closed after some timeout in this case.
If the count stays non-zero quite a long time then it means that someone (a system component like the cache or memory managers as well as a third-party one like a FS filter belonged to an antivirus) holds a reference on the file and does not dereference it quite a long time.
#16789
Posted: 06/23/2011 03:17:49
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

Always, the count is 1.

I'm sorry the sentence causes misunderstanding.
It means just a single open to a file is called.
That is, the counter is 1 after OpenFile().
Therefore, if CloseFile() is called, the counter becomes 0 and the socket is closed.

Now, our problem is CloseFile() is not called immediately after write.

Sangmin
#16790
Posted: 06/23/2011 03:29:13
by Volodymyr Zinin (EldoS Corp.)

You specified:
Quote
write;
write;
write;
write;
write;
write;
close;
close;
close;
write;
write;
close;

As I see the count is not always 1. For some time it was 3.

Quote
Sangmin Lee wrote:
Now, our problem is CloseFile() is not called immediately after write.

Why is it necessary for you to have the close callback immediately?
#16791
Posted: 06/23/2011 03:49:23
by Sangmin Lee (Standard support level)
Joined: 06/03/2009
Posts: 57

Those 'write;' do to all different files.
Therefore, 'close;' calls do, too.

write A;
write B;
write C;
write D;
write E;
write F;
close A;
close B;
close C;
write G;
write H;
close D;
....
#16792
Posted: 06/23/2011 03:57:21
by Volodymyr Zinin (EldoS Corp.)

Close the socket almost immediately (on some timeout) after the write callback. Doesn't it solve the problem?
Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.

Reply

Statistics

Topic viewed 4281 times

Number of guests: 2, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!