EldoS | Feel safer!

Software components for data protection, secure storage and transfer

ExecuteCommand throws error 110 under stress

Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.
#24315
Posted: 03/26/2013 18:08:13
by Michael Lovett (Standard support level)
Joined: 03/20/2013
Posts: 28

Hi there
I'm writing and running some code to stress the simple SSH and SFTP clients before I use them in my full app.

My stress code starts 45 threads, with each one executing ExecuteCommand to execute a remote script on a host machine in the cloud (Amazon EC2 instance). My code then goes into a loop whereby it watches for any of those ExecuteCommands to complete and when it does, it starts another one in its place, keeping the system as busy as possible with SSH activity. (I am still running with a "test license" so BB software might be inserting artificial delays)

What I'm finding is that after a few hundred iterations (typically around 400 or 500), one of the ExecuteCommands fails with error 110 ("connection lost"). When I look in the server log, the SSH server doesn't show anything out of normal. I don't see where a connection was attempted and then aborted or denied. So I don't have any real idea what the black box software is complaining about.

I also tried this same kind of test but only doing an Open() and Close() sequence on 45 threads, over and over. This code is not throwing the connection errors after hundreds of thousands of attempts, so it seems like the trouble is related to ExecuteCommand. (Tomorrow I will try SFTP downloads/uploads in the same kind of test)

Any ideas on things I might check or do to figure out why I'm getting the 110 error from ExecuteCommand?

Also, is there a way to turn on any diagnostics in the BB software in order to better understand the error?

Thanks!
Michael
#24318
Posted: 03/27/2013 02:03:09
by Vsevolod Ievgiienko (EldoS Corp.)

Thank you for contacting us.

Could you clarify:

1) What SBB edition do you use (VCL, .NET, Java).
2) Post here a call stack of the "connection lost" exception.

BTW, you can request an unlock key to remove delays temporary: https://www.eldos.com/sbb/trial-limitations.php Its reasonable to perform such tests with an unlock key.
#24326
Posted: 03/27/2013 08:04:07
by Michael Lovett (Standard support level)
Joined: 03/20/2013
Posts: 28

Thanks for the reply.

I did request an unlock key about a week ago but go no response :(

Yesterday I ended up buying the software because that's the only way I could apparently test it at "full speed", which I will do starting today.

The edition is VCL. An exception stack appears at the end of this email

Michael

-----------------------------------


See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.

************** Exception Text **************
SBSimpleSSH.EElSimpleSSHClientError: Connection lost (error code is 110)
at SBSimpleSSH.TElSimpleSSHClient.DoReceive(Object Sender, Byte[]& Buffer, Int32 MaxSize, Int32& Written)
at SBSSHCommon.TElSSHClass.DoReceive(Byte[]& Buffer, Int32 MaxSize, Int32& Written)
at SBSSHClient.TElSSHClient.DataAvailable()
at SBSimpleSSH.TElSimpleSSHClient.DataAvailable()
at SBSimpleSSH.TElSimpleSSHClient.IntMessageLoop(Boolean NoPeek)
at SBSimpleSSH.TElSimpleSSHClient.DoMessageLoop(Boolean NoPeek)
at SBSimpleSSH.TElSimpleSSHClient.Open()
at SBSimpleSSH.TElSimpleSSHClient.InternalExecuteCommand(String Cmd, Byte[]& StdErrData, Boolean RedirectStdErr)
at SBSimpleSSH.TElSimpleSSHClient.ExecuteCommand(String Cmd, Byte[]& StdErrData)
at BlackBox_Asynch_Testing_1.Form1.RunRemoteScript(Int32 id) in c:\Users\michaell\Documents\Visual Studio 2012\Projects\BlackBox_SSH_SFTP_SimpleExamples\BlackBox_Asynch_Testing_1\BlackBox_Asynch_Testing_1\Form1.cs:line 93
at BlackBox_Asynch_Testing_1.Form1.<>c__DisplayClass5.<AsyncRunScript>b__4() in c:\Users\michaell\Documents\Visual Studio 2012\Projects\BlackBox_SSH_SFTP_SimpleExamples\BlackBox_Asynch_Testing_1\BlackBox_Asynch_Testing_1\Form1.cs:line 122
at System.Threading.Tasks.Task`1.InnerInvoke()
at System.Threading.Tasks.Task.Execute()


************** Loaded Assemblies **************
mscorlib
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18033 built by: FX45RTMGDR
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
----------------------------------------
BlackBox_Asynch_Testing_1
Assembly Version: 1.0.0.0
Win32 Version: 1.0.0.0
CodeBase: file:///C:/Users/michaell/Documents/Visual%20Studio%202012/Projects/BlackBox_SSH_SFTP_SimpleExamples/BlackBox_Asynch_Testing_1/BlackBox_Asynch_Testing_1/bin/Debug/BlackBox_Asynch_Testing_1.exe
----------------------------------------
System.Windows.Forms
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18037 built by: FX45RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
----------------------------------------
System.Drawing
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18022 built by: FX45RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
----------------------------------------
System
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18033 built by: FX45RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
----------------------------------------
System.Configuration
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.17929 built by: FX45RTMREL
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
----------------------------------------
System.Xml
Assembly Version: 4.0.0.0
Win32 Version: 4.0.30319.18033 built by: FX45RTMGDR
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
----------------------------------------
SecureBlackbox
Assembly Version: 10.0.233.0
Win32 Version: 10.0.233.0
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/SecureBlackbox/v4.0_10.0.233.0__5a62fa96d0ac431a/SecureBlackbox.dll
----------------------------------------
SecureBlackbox.SSHClient
Assembly Version: 10.0.233.0
Win32 Version: 10.0.233.0
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/SecureBlackbox.SSHClient/v4.0_10.0.233.0__5a62fa96d0ac431a/SecureBlackbox.SSHClient.dll
----------------------------------------
SecureBlackbox.SSHCommon
Assembly Version: 10.0.233.0
Win32 Version: 10.0.233.0
CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/SecureBlackbox.SSHCommon/v4.0_10.0.233.0__5a62fa96d0ac431a/SecureBlackbox.SSHCommon.dll
----------------------------------------

************** JIT Debugging **************
To enable just-in-time (JIT) debugging, the .config file for this
application or computer (machine.config) must have the
jitDebugging value set in the system.windows.forms section.
The application must also be compiled with debugging
enabled.

For example:

<configuration>
<system.windows.forms jitDebugging="true" />
</configuration>

When JIT debugging is enabled, any unhandled exception
will be sent to the JIT debugger registered on the computer
rather than be handled by this dialog box.
#24330
Posted: 03/27/2013 08:45:30
by Vsevolod Ievgiienko (EldoS Corp.)

Quote
The edition is VCL. An exception stack appears at the end of this email

The stack looks like a .NET one, but not VCL.

What kind of commands do you try to execute and what amount of data do they return? There are two possible reasons for such error:

1) The connection is closed by timeout.
2) The firewall or router limits network throughput.

Please try to increase TElSimpleSSHClient.SocketTimeout, for an instance, twice and check if this solves the problem.
#24334
Posted: 03/27/2013 09:43:48
by Michael Lovett (Standard support level)
Joined: 03/20/2013
Posts: 28

Sorry, it is .NET. Not sure why I typed VCL?

Anyway, the script that's being executed is a tiny python script that runs very fast, and simply returns a numeric string such as "100".

I will try increasing the SocketTimeout.
#24335
Posted: 03/27/2013 09:52:40
by Ken Ivanov (EldoS Corp.)

Michael,

Quote
I did request an unlock key about a week ago but go no response :(


We checked our records and found out that your time-limited unlock key was sent to you on March 21 at 00:50 GMT. If you didn't receive it, this could mean a problem with your junk mail filter. If you haven't received your SecureBlackbox license as well, please check your junk mail folders or contact us via Helpdesk so that we could provide you the license via other means.
#24344
Posted: 03/27/2013 13:26:26
by Michael Lovett (Standard support level)
Joined: 03/20/2013
Posts: 28

Thanks for the reply. I did get my license key. Not sure why the other un-lock key did not make it through.

Back to the problem:

Trying the SocketTimeout at higher and higher values did not help.

A few more pieces of info have come up:

- the 110 errors start around the 60-70 second mark.
- once one ssh client throws the 100 error, many others do as well


- I don't see this same behavior when I do multi-threaded testing of downloading files (using simple sftp) nor do I see this same problem when doing multi-threaded testing of Open()/Close() calls only (with no ExecuteCommand)

Still not understanding the fundamental problem...
#24345
Posted: 03/27/2013 14:16:41
by Ken Ivanov (EldoS Corp.)

Michael,

We inspected the relevant piece of TElSimpleSSHClient's code, and I guess we have pinpointed a suspicious place in code that can lead to the problem you described. To confirm or reject this assumption, we will need some assistance from your side. I'll create a Helpdesk ticket for you now, let's continue the discussion there.
#24346
Posted: 03/27/2013 14:23:59
by Michael Lovett (Standard support level)
Joined: 03/20/2013
Posts: 28

Got the email, tx. Will post any new findings to the ticket; also, if you need me to test anything, just let me know

M
#24353
Posted: 03/28/2013 08:32:46
by Michael Lovett (Standard support level)
Joined: 03/20/2013
Posts: 28

Quote
Ken Ivanov wrote:
We checked our records and found out that your time-limited unlock key was sent to you on March 21 at 00:50 GMT.


I actually would still like to do a bit of testing with the unlock key (even though I now have my license key). Can you tell me what email address the unlock key was sent to? I checked the spam folders for both my personal and work accounts for 3/21 around midnight and see nothing in either of them.

Also, who would be the sender of the email?
Thanks
Michael
Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.

Reply

Statistics

Topic viewed 1836 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!