EldoS | Feel safer!

Software components for data protection, secure storage and transfer

ParallelProcessingAllowed in CallbackFilter

Also by EldoS: CallbackProcess
A component to control process creation and termination in Windows and .NET applications.
#32641
Posted: 03/17/2015 09:14:08
by Jose Battig (Standard support level)
Joined: 09/24/2012
Posts: 10

Any plans to include ParallelProcessingAllowed feature in CallbackFilter?

We are trying to build a service that upon detection of an even on a particular root folder, it launches a separate process build on top of CallbackFileSystem to mount a virtual drive under this root folder.
Trying to do this causes a deadlock situation, because the mounting operation itself causes events to be sent to CallbackFilter application, and this events are serialized with the first event that initiated the need to launch this mounter application.
If ParallelProcessingAllowed were exposed to CallbackFilter we could simply ignore these events that we know are coming from the mounter application.

As it is we find no solution to this issue.
#32642
Posted: 03/17/2015 09:26:03
by Eugene Mayevski (EldoS Corp.)

Can't you just postpone the mounting procedure to after the callback is complete? This is done easily by eg. posting a Windows message to the main thread (or some worker thread) from the callback AND using a mutex both in the callback and the message handler code to ensure that while the callback handler is executed, the worker thread doesn't start the work.

ParallelProcessingAllowed doesn't have a direct equivalent in CallbackFilter and it would be hard (if at all possible) to do something like that in CallbackFilter.


Sincerely yours
Eugene Mayevski
#32647
Posted: 03/17/2015 17:59:01
by Jose Battig (Standard support level)
Joined: 09/24/2012
Posts: 10

Eugene, that would be an option that will make this work, but it's not an option for us given what we want to do to.

We are trying to present a virtual drive to Sterling Commerce Connect:Direct (also called NDM) product in an on-demand fashion, that why we make our filter call "block" until the mounter application confirms mounting has been successful. We perform this confirmation using a global windows event to avoid extra calls trying to check for the existence of the newly mounted junction.

So, we must block, we can't do the mounting asynchronously because NDM will try to enumerate the files when they are not ready yet. Unless somehow we can detect some subsequent even that will succeed only when mounter completes mounting, hopefully this event will get serialized AFTER all of the events generated by Mounter application.

Now, an idea we were going to prototype tonight:

What if we mount the virtual drive into another folder, and create a junction into the folder that NDM uses to access the files?
If an even happens in another location other than the target junction location, are these events posted to all junctions? Or only the target Junction gets the event?

Sebastian
#32652
Posted: 03/18/2015 03:26:24
by Vladimir Cherniga (EldoS Corp.)

Quote
Jose Battig wrote:
What if we mount the virtual drive into another folder, and create a junction into the folder that NDM uses to access the files? If an even happens in another location other than the target junction location, are these events posted to all junctions? Or only the target Junction gets the event?

If you set filter to junction point, you will get OnOpenFileC callbacks only for the junction point path. Junction point target will raise an events only if you set a filter to the junction point target path.
Quote
Jose Battig wrote:
Trying to do this causes a deadlock situation, because the mounting operation itself causes events to be sent to CallbackFilter ...

Do you need only a mounting callback to be work in parallel ?
#32660
Posted: 03/18/2015 16:32:02
by Jose Battig (Standard support level)
Joined: 09/24/2012
Posts: 10

We were thinking that by having the ability to process *any* callback to a given object in parallel, we could make the first access attempt block until the drive is mounted, and by identifying the process that issued the subsequent coming callbacks (from the mounter operation) we can safely ignore them knowing they are because of the mounting operation.

Then, when mounting is complete, we resume processing on the initial callback releasing the thread from the calling process (NDM application) that initiated the first callback.

Now, there's a question even of how threads operate callbackfilter. Do threads dequeue from a central events queue? Or there's a scheduler assigning events to be processed by a given thread?

I ask because if the second is true, we can even end up in deadlock situation bu using the Junction strategy, because the junction events may get assigned to be pre-processed by our blocked thread... unless filtering happens on a separate "system" thread that can't be blocked.

The developer working on this is still prototyping, but she could not reach final conclusion. She was even having issue getting the drive mounted on the junction. For some reason the mounter application seemed to be waiting for something to happen, and only completed mounting operation when callbackfilter application closed. She was using CreateProcess() API but I think she even tried with shellExecute with same results.

I will post here once we know better if Junction strategy works.
#32662
Posted: 03/18/2015 23:04:14
by Jose Battig (Standard support level)
Joined: 09/24/2012
Posts: 10

One more piece of information.

After a lot of testing, the developer found that apparently something that gets initialized when using CallbackFilter, makes this application that uses CallbackFileSystem block, until the app that uses CallbackFilter is shutdown.

She found that the mounter application blocks on this call:

Code
FCbFS.AddMountingPoint(MountAs, GetMountFlags(MountType, Hidden <> 0), nil);


calling this function:

Code
CbFsCreateSymLink()


Shutting down the filter application makes AddMountingPoint continue and succeed.

She also tried mounting in a completely different folder, not monitored by callbackfilter application.


Any ideas?
#32663
Posted: 03/19/2015 03:48:07
by Vladimir Cherniga (EldoS Corp.)

Quote
Jose Battig wrote:
Any ideas?

That could be some kind of recursion in callbacks. If you provide a modified sample code to reproduce the problem, I will create a new ticket to check the issue in helpdesk.

Quote
Jose Battig wrote:
Now, there's a question even of how threads operate callbackfilter. Do threads dequeue from a central events queue? Or there's a scheduler assigning events to be processed by a given thread?

There is a queue in kernel mode, processed in user mode by number of callback threads running in circle.
#32664
Posted: 03/19/2015 10:26:18
by Jose Battig (Standard support level)
Joined: 09/24/2012
Posts: 10

Vladimir, the question is if threads pull from queue or kernel mode "processor" pushes using a ring approach to threads.

If push with ring approach is used, then there chance of deadlock if number of threads is low and if operation on event produces new event(s) that eventually may be pushed into the blocking thread. Pull approach would not cause this deadlock, only idle threads will have the ability to pull from queue.

On the other side, the latest test that our developer did, she didn't even start filtering code, just linked most units from CallbackFilter and that caused CallbackFileSystem application to hang.

Some time today she will try to narrow down the issue and I'll post here the findings (hopefully with some sample code).
#32665
Posted: 03/19/2015 10:42:03
by Vladimir Cherniga (EldoS Corp.)

Quote
Vladimir, the question is if threads pull from queue or kernel mode "processor" pushes using a ring approach to threads.

User mode threads pull requests from kernel queue, using a custom form of request with DeviceIoControl() win32 api.

Quote
Jose Battig wrote:
...and if operation on event produces new event(s) that eventually may be pushed into the blocking thread...


One of the possible deadlock may be reproduced easy. In the case when user callback handler performs some actions that produce another callback (explicitly or implicitly ) - that will result in a 100% deadlock.
#32987
Posted: 04/07/2015 09:16:27
by Jose Battig (Standard support level)
Joined: 09/24/2012
Posts: 10

Final solution we implemented uses junctions. So we will monitor activity on a given folder, upon activity we mount on another folder that happens to be junctioned to the monitored folder, but the mounting operation won't trigger an even on the monitored folder.
We have a prototype working. So far is looking promising, but we have not tested yet in the real production scenario.
Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.

Reply

Statistics

Topic viewed 4857 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!