EldoS | Feel safer!

Software components for data protection, secure storage and transfer

SSLBlackbox - .Net issue with TElSSLArrayInfo not releasing

Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.
#35478
Posted: 01/09/2016 14:54:55
by Matt matt (Standard support level)
Joined: 12/22/2015
Posts: 11

Hi,

I've got an issue with memory not getting released in .Net after using the TElSSLClient/Server and Certificate PKI classes. I need to use a lot of these over time and get rid of them, and whilst the actual instances of these objects get removed from memory successfully (I use ANTS Memory Profiler to see what's around), what stays in memory and never goes away are instances of something called TElSSLArrayInfo.

*All* of these TElSSLArrayInfo classes that are left over have an FLength field of 65536, and a large byte[] allocated to them (65548 bytes?), and also have an FVacant field set to True.

Whilst I appreciate what seems to be some memory reuse... these things never seem to die!!!

Other smaller sizes of the TElSSLArrayInfo class do indeed disappear...

Any thoughts?

Thanks,
#35479
Posted: 01/09/2016 15:02:55
by Eugene Mayevski (EldoS Corp.)

Thank you for the report. Could you please clarify, what exactly version of SecureBlackbox, i.e. whether this is the latest release version or some previous build?


Sincerely yours
Eugene Mayevski
#35480
Posted: 01/09/2016 15:07:50
by Eugene Mayevski (EldoS Corp.)

I have reviewed the code and I believe you are talking about some global objects, which indeed stay in memory because there's no reliable place to release them (and so do some other global objects). Their number should be constant as you use the program, i.e. it should not grow. Do you see the growing number of these large objects over time? If not, then they are not worth bothering with.


Sincerely yours
Eugene Mayevski
#35482
Posted: 01/10/2016 04:47:18
by Matt matt (Standard support level)
Joined: 12/22/2015
Posts: 11

Hi,

I'm using version 14.0.286. The 'problem' seems to be with TElSSLClient....

From looking at the code briefly, each client seems to allocate 5 arrays using SSLMemoryManager of SSLBufferSize (which is 64K), all 5 arrays are held by this TElSSLCLient until it's destroyed. Then they go back as unallocated in the SSLMemoryManager pool. When I need a new a connection, it will try and get the memory from the pool, if not available, it will create a new one.... that's ok.


So for each connection, it's allocating about 320K... for the lifecycle of that connection.... if I burst my connections to say 100, for a few seconds, that's a fair bit of memory, but that then stays allocated forever and never, ever releases itself....

So my questions are I guess

1) Can you get away with smaller buffer sizes for those 5 internal buffers?
2) Can you release some (2?) of the buffers early once they've been used, e.g. the Handshake buffers, that way, they could get used by other SSLClients?
3) Could you allocate the memory pool in multiples of a blocksize so you could allocate say 8K, initially and then grow to 16K, 24K etc... as required.
4) Could you have a clean up routine that maybe we can call that would flush any buffers that have not been used for x amount of seconds?

Whilst option 3 may be the most work, how about 2 and 4????

Thanks,

Matt.
#35483
Posted: 01/10/2016 07:50:07
by Eugene Mayevski (EldoS Corp.)

I see your point. However, if you have a fairly constant number of connections (say 100 per second), then you have the constant number of buffers. Even if the number of connections drops, the chances are that it will be increased again. And While the buffers are not disposed immediately, they would be re-used later, when the number of connections grows.

To answer your questions:
1) no, 64K is the minimal size
2) there's a chance that renegotiation takes place later. Still your idea suggests that maybe we can make the pool more flexible.
3) I don't quite understand this. 64K is the needed size, but there's no need to have the buffers larger.
4) time-based expiration won't work well. The pool probably needs to be more intellectual, maybe with some adjustable upper limit of the number of waiting buffers. This needs to be discussed.


Sincerely yours
Eugene Mayevski
#35484
Posted: 01/10/2016 11:00:13
by Matt matt (Standard support level)
Joined: 12/22/2015
Posts: 11

Quote
1) no, 64K is the minimal size
3) I don't quite understand this. 64K is the needed size, but there's no need to have the buffers larger.


OK. if 64K is the smallest size for ALL 5 buffers then there's not much you can do - does seem rather large though! and any reduction would help...

Quote
2) there's a chance that renegotiation takes place later. Still your idea suggests that maybe we can make the pool more flexible.


Yes, if you could be more granular on memory allocation and return any buffers back to the pool (maybe the handshake ones when we're done handshaking) and then re-request later if need be, that would instantly save a great deal of memory.

Quote
4) time-based expiration won't work well. The pool probably needs to be more intellectual, maybe with some adjustable upper limit of the number of waiting buffers. This needs to be discussed.


Ok - will leave that to you guys!

Would be great to hear if you can do something for a future update.

Many thanks,

Matt.
#35485
Posted: 01/10/2016 11:48:57
by Matt matt (Standard support level)
Joined: 12/22/2015
Posts: 11

Also, could you confirm, which is best to call when wanting to shutdown the TElSSLClient or TElSSLServer classes, Destroy , or Dispose?
#35486
Posted: 01/10/2016 15:01:45
by Eugene Mayevski (EldoS Corp.)

Regarding your questions - I've sent my ideas to Ken, who implemented those pools and who can optimize them.

As for shutting down - you close the connection with a call to Close() method. Dispose should be used when you don't need an object anymore. Destroy is an internal method (it was made public for language- and platform-related reasons).


Sincerely yours
Eugene Mayevski
#35487
Posted: 01/11/2016 04:45:35
by Ken Ivanov (EldoS Corp.)

Hi Matt,

SecureBlackbox actually provides you with some level of control over the lifetime of the buffers.

First, you can switch off the memory manager entirely (and have the buffers allocated when they are needed and released when they are not needed anymore) by setting SBSSLCommon.Unit.SSLMemoryManager().Enabled to false. This will make your application only allocate the amounts of memory it needs right now, and release allocated buffers once it's over with them without storing them for future.

Next, if using the memory manager, you can still force it to release all cached buffers with SBSSLCommon.Unit.SSLMemoryManager().ReleaseAll() call. Note that you can call ReleaseAll() at any time; it won't affect any active SSL connections you might have up and running.

Please note that you might need to call GC.Collect() in addition to releasing the arrays to make the garbage collector actually remove them from memory.

Ken
#35503
Posted: 01/11/2016 11:52:45
by Matt matt (Standard support level)
Joined: 12/22/2015
Posts: 11

Hi,

Thanks for these hints, I'll take a look.
Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.

Reply

Statistics

Topic viewed 3210 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!