EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Min/MaxThreadCount thread pooled, or cycled?

Also by EldoS: CallbackDisk
Create virtual disks backed by memory or custom location, expose disk images as disks and more.
#36700
Posted: 05/10/2016 20:36:20
by Marius Storm-Olsen (Priority Standard support level)
Joined: 10/01/2015
Posts: 3

Hi,

I'm wondering if when you set a Min and MaxThreadCount for the callbackfilter, will the threads created to support the callback handling be static, or can they be terminated, created again, so you end up with a cycling set of "MaxThreadCount" number of varying threads?

I'm asking because I would like to use ZeroMQ for communication, but it requires a separate inproc "socket" per thread, so I would like to lazily create a new one only when I hit a new thread, up to MaxThreadCount. However, if the threads can be torn down, and new ones created, there will be a cycling effect, and I might end up spending more time creating new sockets than I would like.

Also, my internal engine for the callbackfilter usage has a static amount of threads, and it would be great if the threads didn't cycle, as I could then allocate each internal static worker thread to each thread from CallbackFilter.

So, any details on how threads are created/destroyed/increased/decreased in CallbackFilter would be greatly appreciated!

Thanks!
#36701
Posted: 05/11/2016 02:37:31
by Volodymyr Zinin (EldoS Corp.)

Hello,

Usually the worker thread pool contains the currently used threads (threads that are calling the callbacks) plus several more threads that are waiting for new requests. In the case a lot of waiting threads exists then some of them are terminated. If it is required more threads to process requests then new ones are created. But the number of threads are always in the span of Min and MaxThreadCount.
#36702
Posted: 05/11/2016 03:17:06
by Eugene Mayevski (EldoS Corp.)

To add to Volodymyr's answer, in your case it makes sense to employ a pool of sockets and/or threads and not bind them to CallbackFilter internal worker threads, but rather use some locks / monitors . All in all, I understand it so that you don't need strict binding of threads/sockets, but rather you need to guarantee that one socket is used by one thread.


Sincerely yours
Eugene Mayevski
#36719
Posted: 05/11/2016 14:06:44
by Marius Storm-Olsen (Priority Standard support level)
Joined: 10/01/2015
Posts: 3

Well, in the case of ZMQ (http://zguide.zeromq.org/page:all#Multithreading-with-ZeroMQ), the development paradigm is to avoid locking mechanisms completely, and just use the inproc messaging scheme between threads. However, this assumes that the ZMQ socket is created on the thread which passes the message.

If CBF cycles threads at random times, I won't be able to rely on that, and would require locking individual threads just to pass a message. And I wouldn't know when to destroy a previous socket created on the thread which has disappeared from the pool.

Is there a guarantee that there will be no thread cycling if I set Min/Max to the same? So, if I set Min=8, Max=8, I will always have the exact same 8 threads?
#36720
Posted: 05/11/2016 18:20:35
by Marius Storm-Olsen (Priority Standard support level)
Joined: 10/01/2015
Posts: 3

I guess most ideally would be to have CBF callbacks for the creation/deletion of CBF worker threads, allowing me to create/destroy application-specific TLS or assign a worker thread specific pointer (much like the PVOID* UserContext on I/O callbacks).

That way I could easily
MyStruct *threadData = (MyStruct*)cbFlt.GetWorkerThreadData();
on a callback function, to get thread-specific contexts.
#36722
Posted: 05/11/2016 23:21:47
by Eugene Mayevski (EldoS Corp.)

Yes, we have this idea in the wishlist for Callback File System, and we can implement it for all products. I have added the task to the ToDo list.


Sincerely yours
Eugene Mayevski
Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages

Reply

Statistics

Topic viewed 1960 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!