EldoS | Feel safer!

Software components for data protection, secure storage and transfer

SSLClient Memory Leak

Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.
#36186
Posted: 03/10/2016 18:07:58
by Charlie Jimenez (Standard support level)
Joined: 08/14/2012
Posts: 38

I have discovered a memory leak in SBB SSL Client (.Net Windows Forms). This memory leak is present in versions 13.0.276.0 and 14.0.288.0. I had to revert back to version 12.0.260.0 which does not have this leak.

My application creates and configures the SSLClient object on every connection, and disposes of it after closing the connection.

I performed a simple memory profile using ReSharper and the leak seems to be in SSLCommon.TElSSLMemoryManager->SBSSLCommon.TElSSLArrayInfo. The TElSSLArrayInfo object is using Byte[] arrays which are not being disposed when the SSLClient is disposed. The quantity of these objects just keeps increasing until memory is exhausted.

Thanks,

Charlie
#36187
Posted: 03/10/2016 18:15:15
by Eugene Mayevski (EldoS Corp.)

Thank you for the report.

It's hard to talk about "leaks" of managed objects in .NET. If managed objects are not referenced, they will be collected sooner or later. The memory manager, introduced in SSL, has been specifically designed not to release the objects but reuse them, if needed. This speeds up both TLS servers and TLS clients, especially during startup (when the new object is created).

It is possible, that in some particular usage scenario this mechanism doesn't work perfectly. However, it's not a "leak" but something that needs specific tune-up for your usage scenario.

If you could tell us a bit more about what your application does (how many clients are created per minute, how are those clients used etc.), maybe we could understand, why the number of objects grows.

Let me assure you, that we will find and remove the "leak" if we get more information from your about what's going on.

If you could run one of our test applications (in theory running HTTPGet in Samples\C#\HTTPBlackbox\Client directory should be enough) and see if the same leaks are reported there, doing this would let us know if we could easily reproduce the issue on our side, or the issue is really very project-specific and we need to keep asking you about the details.


Sincerely yours
Eugene Mayevski
#36188
Posted: 03/10/2016 18:25:57
by Eugene Mayevski (EldoS Corp.)

I have done a quick review of the SSL Client code and all arrays acquired from the memory manager are released back to it IF you call SSLClient.Dispose() properly. The arrays do stay in SSLMemoryManager (there's no mechanism to clear the memory manager or make it reduce its size automatically now). We did have one or two requests about such optimizations, so there's a chance that the optimizations will be added in SecureBlackbox 15 or 16. But the optimizations are not magic and they would need your code to adjust the behavior of SSLMemoryManager. Due to this it make sense for us to try to determine the root of your current problems, rather than wait for those optimizations.


Sincerely yours
Eugene Mayevski
#36189
Posted: 03/10/2016 18:34:52
by Charlie Jimenez (Standard support level)
Joined: 08/14/2012
Posts: 38

Thanks for your quick reply.

As I mentioned, I do indeed call SSLClient.Dispose after each connection closes. I have noticed that the component also exposes a .Destroy method and a .DestroyUnmanaged method. I have tried using those, but get exceptions because apparently Dispose calls those same methods internally.

My application can have from one to thousands of connections operating independently and simultaneously. Each connection lasts less than one second. A message is sent and a response is received. The connection is then closed.

When I discovered this problem I was running the application with 512 connections cycling at a rate of about 10 connections per second. Each connection object creates and disposes of its own SSLClient object. But it is naturally possible that more than one SSLClient may be active at any one time.

I hope this gives you a better picture of the environment.

Thanks,

Charlie
#36190
Posted: 03/10/2016 18:43:17
by Eugene Mayevski (EldoS Corp.)

Thank you very much for the detailed description. I was not trying to say that you didn't call Dispose, but rather described the overall scheme (also for those users, who might come read this thread later). Indeed we are having a better understanding of what is going on, now.

Each SSLClient allocates 5 buffers each being 64 Kb in size (in desktop version). If you have several thousands of connections, you can run out of memory no matter if the memory manager is used or not. This is a separate issue to deal with, if you encounter it.

With 512 active client connections memory fragmentation could lead to excessive memory use as well.

We will try to reproduce your scenario locally and see what this gives us.

One option would be to let you just turn off the memory manager completely - this will slow down operations but we won't be blamed for leaks :).

In any case, please let us reproduce the issue. It can take 3-4 working days, as we'll need to create a suitable test case (running so many active clients requires some servers to handle the load as well ... ).


Sincerely yours
Eugene Mayevski
#36191
Posted: 03/11/2016 03:28:20
by Ken Ivanov (EldoS Corp.)

Hi Charlie,

Thank you for very detailed information about the issue. Indeed the specifics of your task (particularly the one of hundreds of simultaneous connections) might cause excessive memory consumption with default configuration of SSL memory manager.

What you can do straight away is:

1) Try to turn the memory manager off completely. This will revert the behaviour of components to the one it used to be back in SBB 12. To switch the manager off use the following code:

SBSSLCommon.Unit.SSLMemoryManager().Enabled = false;

2) Change the ceiling value of the number of cached buffers. By default SSLMemoryManager keeps 100 buffers available for re-use by SSL components in future. You can try both decreasing and increasing this value (down to 25 and up to 250) and assess what outcome you can see:

SBSSLCommon.Unit.SSLMemoryManager().MaxCount = 25;

3) Try not only to Dispose() your SSL client objects but also null'ize them.

4) Finally, you can forcefully make the manager 'forget' all the cached buffers. Sometimes the Garbage Collector doesn't react to array releases properly and you need to push it to make it collect the otherwise unused arrays. Try to call the following code when your application's memory consumption becomes too high and check if it helps bring the consumption back to normal:

SBSSLCommon.Unit.SSLMemoryManager().ReleaseAll();
GC.Collect();

And some further comments following your description of the issue.

Quote
I performed a simple memory profile using ReSharper and the leak seems to be in SSLCommon.TElSSLMemoryManager->SBSSLCommon.TElSSLArrayInfo. The TElSSLArrayInfo object is using Byte[] arrays which are not being disposed when the SSLClient is disposed. The quantity of these objects just keeps increasing until memory is exhausted.

This is a bit weird and needs to be investigated further.

Could you please put a breakpoint somewhere in your code and make the execution flow stop on it when the application's overall memory consumption reaches a very high level (say extra 500MB over its normal behaviour). For example, you can add a temporary button on the form and put a breakpoint inside its Click event handler.

Once the breakpoint was triggered, please check the values of the following parameters:

- the number of SSL client objects currently alive,
- the value of SBSSLCommon.Unit.SSLMemoryManager().FAllocatedArrays.Count;

You can actually bundle this check with ReleaseAll()/GC.Collect() trick and check both at the same time.

Thanks in advance.

Ken
#36203
Posted: 03/11/2016 09:54:25
by Charlie Jimenez (Standard support level)
Joined: 08/14/2012
Posts: 38

Hi Ken,

Thanks for the detailed explanation. I will certainly try your suggestions and let you know how it turns out.

Charlie
#36206
Posted: 03/11/2016 12:15:15
by Charlie Jimenez (Standard support level)
Joined: 08/14/2012
Posts: 38

[FONT=Courier]Hi Ken,

Here are the results of your suggestions, using version 14.0.288.0. First as a bit of background, my application displays a lot of data and refreshes this display every second. Part of the display is the current CPU %, Working Set memory usage and Virtual Memory usage. When I start it up, the Working Set usage is around 130MB. This is the value that I was monitoring during these tests.

1) SBSSLCommon.Unit.SSLMemoryManager().Enabled = false;

This eliminates the memory consumption and behaves just like version 12 as you predicted.


2) SBSSLCommon.Unit.SSLMemoryManager().MaxCount = 25;

This did not seem to have any effect. I let it run until WS memory reached over 1 GB.


3) SBSSLCommon.Unit.SSLMemoryManager().ReleaseAll(); GC.Collect();

I inserted some code to execute the above commands when the WS memory usage reached 500 MB. Sure enough, the WS memory usage dropped back to 130MB after these commands executed.


4) The number of SSL client objects currently alive; The value of SBSSLCommon.Unit.SSLMemoryManager().FAllocatedArrays.Count;

What I did here was watch the WS value as it increased and stopped all connection activity when I reached certain values. So for each of these measurements, the number of SSL client objects was zero:

WS Memory (MB) / FAllocatedArrays.Count
249 / 5,145
530 / 16,630
1,023 / 58,648


Hope this helps.


Charlie
#36216
Posted: 03/14/2016 07:09:14
by Ken Ivanov (EldoS Corp.)

Hi Charlie,

Thank you for the detailed information.

Quote
What I did here was watch the WS value as it increased and stopped all connection activity when I reached certain values. So for each of these measurements, the number of SSL client objects was zero:

WS Memory (MB) / FAllocatedArrays.Count
249 / 5,145
530 / 16,630
1,023 / 58,648

The growth of the FAllocatedArrays.Count figure indicates that the buffers are not released properly by the consuming component (TElSSLClient). The most likely reason for that is missing Dispose() calls, which is supposed to release the buffers. I understand that you checked that already, but just to be sure, could you please re-check that Dispose() method is always called in your code? Could it be there is some logical branch that makes it possible for Dispose() not to be called in certain conditions?

We will also conduct some checks on our side. Which exactly version of desktop .NET framework are you using?

Ken
#36224
Posted: 03/14/2016 11:08:00
by Charlie Jimenez (Standard support level)
Joined: 08/14/2012
Posts: 38

Hi Ken,

My .Net framework version is 4.5.2

I placed some additional code in my application to monitor the dispose operation. At the time I disconnect I have the following

Code
          If mUseSSL AndAlso mSSLClient IsNot Nothing Then
            mSSLClient.CertStorage = Nothing
            Try
              mSSLClient.Dispose()
            Catch ex As Exception
              ProcessException(ex, Me.GetType.Name, MethodBase.GetCurrentMethod.Name, mName)
            Finally
              If Not mSSLClient.IsDisposed Then
                LogConnError(FailureEnums.ConnFailureTypes.SSLError, "SSL Client is STILL not disposed at LocalDisconnect!", EventSeverities.AppError, 0)
              End If
            End Try
            mSSLClient = Nothing
          End If


I also check again when I establish a new connection:

Code
        If mSSLClient IsNot Nothing Then
          LogConnError(FailureEnums.ConnFailureTypes.SSLError, "SSL Client is not nothing!", EventSeverities.AppError, 0)
          mSSLClient.CertStorage = Nothing
          Try
            mSSLClient.Dispose()
          Catch ex As Exception
           ProcessException(ex, Me.GetType.Name, MethodBase.GetCurrentMethod.Name, mName)Finally
            If Not mSSLClient.IsDisposed Then
              LogConnError(FailureEnums.ConnFailureTypes.SSLError, "SSL Client is STILL not disposed!", EventSeverities.AppError, 0)
            End If
          End Try
          mSSLClient = Nothing
        End If
        mSSLClient = New TElSSLClient


No exceptions or errors of any kind are logged.

I mentioned before about the Destroy and DestroyUnmanaged methods which TElSSLClient exposes? Are these methods useful in this or any other scenario?

Thanks,

Charlie
Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.

Reply

Statistics

Topic viewed 4671 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!