EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Metadata caching speed problem

Also by EldoS: CallbackProcess
A component to control process creation and termination in Windows and .NET applications.
#24347
Posted: 03/27/2013 20:08:00
by Leo Chun (Standard support level)
Joined: 03/27/2013
Posts: 9

We have been testing Callbackfile system for our Media server. our media servers are based on IIS

When I activate the metadata caching. Sometimes it shows an error message "the attached device on this system is not working"(I just translated that error message it was written in Korean).
In most case it works well but occasionally that error message is displayed.

I found that there were no error messages when I just inactivated the Metadata Caching. But the problem was that inactivating the Metadata Caching affected the caching speed.

I am curious about that why it happened from time to time?
And the metadata caching should be activated in our system.
So please let me know how and what should we do

Please advise.
-Leo
#24350
Posted: 03/28/2013 05:34:01
by Volodymyr Zinin (EldoS Corp.)

Perhaps there is a bug inside your CallbackFS implementation or in CallbackFS itself. Is the problem reproducible with the Mapper sample?
Thanks.
#24412
Posted: 04/02/2013 04:45:27
by Leo Chun (Standard support level)
Joined: 03/27/2013
Posts: 9

Hello Vladimir,

I've just got code what my engineer wrote.
Please just ignore some unreadable Korean letters.
Cloud you take a look at below?
And please let me know what we did wrong.


// CbFsGetFileInfo()도 CbFsOpenFile()과 같은 방식으로 처리됨.
// 단, CbFsEnumerateDirectory() 에서는 해당 처리를 하지 않는다.




void CbFsOpenFile( CallbackFileSystem *lpCallbackFileSystem, LPCTSTR lpszFileName, ACCESS_MASK DesiredAccess,

DWORD FileAttributes, DWORD ShareMode, CbFsFileInfo *lpFileInfo, CbFsHandleInfo *lpHandleInfo)

{

...

// 가상 파일/디랙토리에 대한 접근인지 검사한다.

if (pCacheFS->GetExecuteSource( lpszFileName, &__cacheFile) == FALSE)


;

else {

...

}

}





void CbFsReadFile( CallbackFileSystem *lpCallbackFileSystem, CbFsFileInfo *lpFileInfo, __int64 Position, PVOID lpBuffer, DWORD BytesToRead, PDWORD lpBytesRead)

{

CCachedFS::LPFILE_HANDLE FileCtx = (CCachedFS::LPFILE_HANDLE)lpFileInfo->get_UserContext();





// 가상 파일인지 검사한다.

if (pCacheFS->IsExecuteSource( FileCtx->pFile) == FALSE)

;

else {

LPCTSTR lpszSourcePath = (LPCTSTR)(FileCtx->pFile + 1);

if (pCacheFS->FetchExecuteSource( FileCtx->pFile, lpszSourcePath, (LPBYTE)lpBuffer, Position, BytesToRead, lpBytesRead) == FALSE)

throw ECBFSError(ERROR_ACCESS_DENIED);




return;

}

}




// GetExecuteSource() 함수에 접근시 파일의 크기와 파일의 내용을 생성한다.

// 단, 1초 이내의 중복 처리는 무시한다.




// FetchExecuteSource() 함수의 경우 GetExecuteSource()가 생성한 파일 내용을 반환한다.
#24414
Posted: 04/02/2013 11:10:02
by Volodymyr Zinin (EldoS Corp.)

The code snippet you specified contains nothing concerning to the CallbackFS meta-data cache. I just see some calls of maybe your implementation of a cache and it says nothing to me.
Try to reproduce the problem with the Mapper sample (maybe modified somehow in order the problem occurs) and describe in details how to reproduce it.
Thanks.
#24473
Posted: 04/08/2013 04:01:17
by Leo Chun (Standard support level)
Joined: 03/27/2013
Posts: 9

Hello Vladimir,

This is Leo from Catenoid.
I've just made Mapper Sample.
please let me have your email account or the way to send you the mapper sample.
I made it as an zip file. the file size is around 9 Mbytes

best regards,
-Leo
#24474
Posted: 04/08/2013 04:12:42
by Leo Chun (Standard support level)
Joined: 03/27/2013
Posts: 9

I've made as an TXT. and attached it
Wish to know whether it is enough or not.
if it not enough then let me know.
I will send you full version.

thank you
Leo


[ Download ]
#24481
Posted: 04/08/2013 11:47:34
by Volodymyr Zinin (EldoS Corp.)

Hello Leo,

The file you attached is truncated. Please resend it to my email, which is zinin at eldos.com
But please describe in details what should I do in order to reproduce the problem.
Thanks.
#24482
Posted: 04/08/2013 14:29:50
by Volodymyr Zinin (EldoS Corp.)

I have received your sources. Thank you.
It seems the main problem is because your OnRead and OnWrite callbacks return that the quantity of read/written bytes is not the same as requested. This is an erroneous situation. Some programs in this situation just return error (like the "disk is corrupted" error), but other try to perform again the same read/write request several times in order it will be successful. For example a program opens a file as a memory mapped section, copy some data to the memory mapped section and maybe close the file. All is ok so far. But then the system memory manager starts flushing the data from the section memory to the file, but your OnWrite callback returns that not all the data has been written. In this case the memory manager tries to write the same block of memory several times and if it's still non-successful then it shows an error message to a user on the currently active desktop and throws away the flushing data.

If you want to restrict number of bytes that a file can contain then return an error from the OnSetFileSizes callback when the requested file size (the "end of file" size) is greater than allowed.
In the case to prohibit writing to a file set the CallAllOpenCloseCallbacks property and in the OnCreate/OnOpen callbacks check for the "write" access rights (for details see the DesiredAccess parameter for the CreateFile win32 API) and return the ERROR_ACCESS_DENIED.

About your implementation which dynamically changes file data (data which is located in the __resultBuffer buffer). It's possible to implement. But it's necessary to use NotifyDirectoryChange("\path_to_file\file_name", fanModified) in order to notify CallbackFS about the file changes. Otherwise CallbackFS (as well as programs which have the file opened) will consider (contain cached information) that the file still contain old data.
#24495
Posted: 04/09/2013 05:01:01
by Leo Chun (Standard support level)
Joined: 03/27/2013
Posts: 9

Hello Vladimir,

One more question.

As you understood, our system changes the information of the "z:\_status" at every time when other programs request it.

but the problem is when we are using NotifyDirectoryChange and someone edits a file then it seems like that all previous connections to _status are disconnected.

So I'd like to know how we can make the process to work like queue system.
I mean if there is anyone who is already connected then the next one should wait until the previous process is done.

Please advise.

Warm regards,
-Leo
#24496
Posted: 04/09/2013 05:02:59
by Eugene Mayevski (EldoS Corp.)

Quote
Leo Chun wrote:
So I'd like to know how we can make the process to work like queue system.


You can't. Windows is different from Unix in this aspect. Once you've declared that certain file contains certain data, you can only notify the OS that the data have been changed. But you can't feed each requester with different data (partially due to caching, partially due to other design specifics).

You need to re-think your design in regards to this.


Sincerely yours
Eugene Mayevski
Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.

Reply

Statistics

Topic viewed 1417 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!