EldoS | Feel safer!

Software components for data protection, secure storage and transfer

SendData sync method hanging

Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.
#28043
Posted: 01/18/2014 21:42:17
by steve cook (Standard support level)
Joined: 11/15/2013
Posts: 11

I occasionally have problems where calling methods like TElSSHTunnelConnection.SendData() and TElSSHTunnelConnection.Close() from my SSH Server hang (most likely because the other end point is closed/closing).

I would like to use an async SendData() method, but I can't find one. Is there any? And what files do I need to reference? And is there any example code?
#28044
Posted: 01/19/2014 02:57:11
by Eugene Mayevski (EldoS Corp.)

The SSH server is a pure "data converter" -- its SendData method converts the data (encrypts them etc) and sends new data to your code, where you send it to the client using socket or in some cases other channel.

I don't think it's data conversion that hangs - instead it's socket being blocked or returning error.

The SSH server doesn't have built-in sockets. Your code deals with sockets. Consequently it's up to you to check if the socket is ready before writing the data to it.

There exist several approaches - you can use asynchronous socket functions or you can set the socket to non-blocking mode and check the result, or you can use select() method to check if the socket can be written to.

In general, socket operations are a bit beyond the scope of our support and there exists plenty of information in Internet about how to deal with sockets right.


Sincerely yours
Eugene Mayevski
#28048
Posted: 01/19/2014 09:04:55
by steve cook (Standard support level)
Joined: 11/15/2013
Posts: 11

The strange thing is that when this locks up my TElSSHServer.OnSend() implementation never seems to be called.

This is roughly the code that I'm using:

Code
        srv = new TElSSHServer();
        srv.KeyStorage = Keys;
        srv.OnReceive += srv_OnReceive;
        srv.OnSend += srv_OnSend;
        srv.OnAuthAttempt += srv_OnAuthAttempt;
        srv.OnAuthPassword += srv_OnAuthPassword;
        srv.OnOpenShell += srv_OnOpenShell;
        srv.OnCloseConnection += srv_OnCloseConnection;
        srv.OnError += srv_OnError;
        srv.Open();

...

    private void srv_OnOpenShell(object Sender, SBSSHCommon.TElSSHTunnelConnection connection)
    {
        Connection = connection;
        Connection.OnData += Connection_OnData;
    }

...

    //call this function from various threads to write messages
    //behaviour seems the same both with and without synchronisation
    public void Write(byte[] buffer, int offset, int size)
    {
        Connection.SendData(buffer, offset, size);
     }

    //send data to client (SendData callback)
    private void srv_OnSend(object Sender, byte[] Buffer)
    {
        try
        {
            int toSend = Buffer.Length;
            int sent;
            int ptr = 0;
            Logger.Debug("Sending...");    // <- never see this debug trace when SendData() locks
            while (toSend > 0)
            {
                if (m_Socket == null || !m_Socket.Connected)
                {
                    Logger.Debug("Send socket not connected");
                    Close();
                    return;
                }

                sent = m_Socket.Send(Buffer, ptr, toSend, SocketFlags.None);
                if (sent < 0)
                {
                    Logger.Debug("Send socket failed " + sent);
                    Close();
                    return;
                }
                ptr += sent;
                toSend -= sent;
            }
        }
        catch (ObjectDisposedException ex)
        {
            //if object is disposed, then its just that the client has disconnected - ignore
            Close();
        }
        catch (SocketException ex)
        {
            //if socketexception is thrown, then its just that the client has disconnected - ignore
            Close();
        }
        catch (Exception ex)
        {
            Logger.Error("SSH service socket send operation failed (" + Service.Port + ") to " + RemoteIP, ex);
            Close();
        }
    }


Mostly the code above works fine, but in some circumstances (I think when the network client is rapidly started/stopped), SendData() is called, but I never see the debugging trace in srv_OnSend().

I did notice that SendData() seems to acquire a write lock with an infinite timeout. In case the lock can't be acquired, is there any way to adjust the timeout period?
#28050
Posted: 01/19/2014 09:38:21
by Eugene Mayevski (EldoS Corp.)

Indeed there exists a lock in the SendData method, however if it's blocked, it means that some other thread has acquired a lock and never released it.

If the application (your server) locks itself under debugger, can you please inspect the call stack of each thread of the server and capture it if this thread is somehow related to SSH?

Maybe we will be able to find the problem this way.

Also please inspect your code to see that there are not exceptions thrown and not handled in it -- if the exception is thrown by the event handler, I can imagine that some mistake in our code doesn't expect this exception and doesn't properly release the global lock.


Sincerely yours
Eugene Mayevski
#28051
Posted: 01/19/2014 20:57:15
by steve cook (Standard support level)
Joined: 11/15/2013
Posts: 11

Not sure if related, but I did find this uncaught exception on the server. Not sure if the fact that it was uncaught meant that the resources were not released/closed properly. In any case, looks like the exception is coming from inside the Close() method. :

Code
An unhandled exception occurred and the process was terminated.

Application ID: /LM/W3SVC/1/ROOT

Process ID: 11984

Exception: System.ArgumentOutOfRangeException

Message: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index

StackTrace:    at System.Collections.ArrayList.get_Item(Int32 index)
   at SBSSHServer.TElSSHServer.Close(Boolean Forced)
   at SSHSession.reader()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
#28052
Posted: 01/20/2014 02:31:45
by Vsevolod Ievgiienko (EldoS Corp.)

Hello.

Even if an exception happens inside TElSSHServer.Close method the global lock is unlocked properly. But the fact that exception happens means that something is definitely done wrong. Could you post your full code to Helpdesk with some step-by-step description of how we can reproduce the problem.
#28053
Posted: 01/20/2014 02:40:38
by Eugene Mayevski (EldoS Corp.)

Looks like you close and/or remove the tunnel in response to OnClose. That causes the exception. Please review your code.


Sincerely yours
Eugene Mayevski
#28054
Posted: 01/20/2014 02:54:12
by steve cook (Standard support level)
Joined: 11/15/2013
Posts: 11

Quote
Looks like you close and/or remove the tunnel in response to OnClose. That causes the exception


Where is the correct place to put the socket teardown code?
#28055
Posted: 01/20/2014 03:02:20
by Eugene Mayevski (EldoS Corp.)

Quote
steve cook wrote:


Where is the correct place to put the socket teardown code?


I am not talking about the socket but about the tunnel object. It's ok to close and dispose of the socket in OnClose handler, but somehow your code destroys the tunnel object in this case.


Sincerely yours
Eugene Mayevski
#28432
Posted: 02/18/2014 02:52:11
by steve cook (Standard support level)
Joined: 11/15/2013
Posts: 11

I removed the spurious tunnel close(), but I still have the problem.

I've put together the following simple test case that can easily reproduce the problem. Just requires C# and the SSH libs. Can you take a look and let me know what I'd need to change to fix the issue?

Code
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Sockets;
using System.Text;
using System.Threading.Tasks;
using System.Net;
using System.Threading;

using SBSSHServer;
using SBSSHCommon;
using SBSSHKeyStorage;

namespace SSHChat
{
    //example app that echoes data between SSH sessions
    //run the app, then open up two SSH terminals (e.g. puTTY) to connect to the specified port.
    //anything typed into the terminal will be echoed to all other connected terminals.
    //
    //Bug: sometimes deadlocks in Connection.SendData().
    //     to reproduce, just select a large amount of text and paste it into two terminals
    //     at the same time (enough text that it takes a few seconds to echo through).
    //     This will cause a deadlock.

    class Program
    {
        static void Main(string[] args)
        {
            SSHChat chat = new SSHChat();
            chat.Start();

            while (true)
                Thread.Sleep(10000);
        }
    }

    class SSHChat
    {
        private static string ELD_10_EVALKEY = //put licence key here;
        private static string Interface = "127.0.0.1";
        private static int Port = 6666;

        private TcpListener serverSocket;

        private List<SSHSession> sessions = new List<SSHSession>();

        public SSHChat()
        {
            //set licence key
            SBUtils.Unit.SetLicenseKey(ELD_10_EVALKEY);
        }

        internal void Start()
        {
            //open a listening socket on specified IP/port
            serverSocket = new TcpListener(IPAddress.Parse(Interface), Port);
            serverSocket.Start();

            Thread thread = new Thread(new ThreadStart(Handler));
            thread.Start();
        }

        internal void Handler()
        {
            bool running = true;
            while (running)
            {
                try
                {
                    //get incoming connections
                    Socket clientSock = serverSocket.AcceptSocket();

                    //create a new SSH protocol handler for this connection
                    SSHSession session = new SSHSession(this, clientSock);
                    session.Start();
                    sessions.Add(session);
                }
                catch (Exception e)
                {
                    //otherwise some internal error
                    Console.Out.WriteLine("Error in SSH service: " + e.Message);
                }
            }
        }

        internal void Broadcast(byte[] buffer)
        {
            foreach (SSHSession session in sessions)
            {
                session.Broadcast(buffer);
            }
        }
    }

    //sesssion that echoes data to the other session
    class SSHSession
    {
        private SSHChat sshChat;
        private Socket clientSock;
        private TElSSHServer server;
        public TElSSHTunnelConnection Connection;

        public SSHSession(SSHChat sshChat, Socket clientSock)
        {
            this.sshChat = sshChat;
            this.clientSock = clientSock;
        }

        internal void Start()
        {
            TElSSHKey key = new TElSSHKey();
            key.Generate(SBSSHKeyStorage.Unit.ALGORITHM_RSA, 1024);

            server = new TElSSHServer();
            server.KeyStorage = new TElSSHMemoryKeyStorage();
            server.KeyStorage.Add(key);
            server.OnReceive += server_OnReceive;
            server.OnSend += server_OnSend;
            server.OnAuthAttempt += server_OnAuthAttempt;
            server.OnAuthPassword += server_OnAuthPassword;
            server.OnOpenShell += server_OnOpenShell;
            server.OnTerminalRequest += server_OnTerminalRequest;
            server.OnCloseConnection += server_OnCloseConnection;
            server.OnError += server_OnError;
            server.Open();

            Thread readerThread = new Thread(new ThreadStart(reader));
            readerThread.Start();
        }

        void server_OnError(object Sender, int ErrorCode)
        {
            Console.Out.WriteLine("Terminal request");
            server.Close(true);
        }

        void server_OnCloseConnection(object Sender)
        {
            Console.Out.WriteLine("Closing connection");
        }

        void server_OnTerminalRequest(object Sender, TElSSHTunnelConnection Connection, SBSSHTerm.TElTerminalInfo Info, ref bool Accept)
        {
            Console.Out.WriteLine("Terminal request");
            Accept = true;
        }

        void server_OnAuthPassword(object Sender, string Username, string Password, ref bool Accept, ref bool ForceChangePassword)
        {
            Console.Out.WriteLine("Authenticate password");
            Accept = true;
        }

        void server_OnAuthAttempt(object Sender, string Username, int AuthType, ref bool Accept)
        {
            Console.Out.WriteLine("Authenticate");
            Accept = true;
        }

        void server_OnOpenShell(object Sender, TElSSHTunnelConnection connection)
        {
            Console.Out.WriteLine("Open Shell");
            Connection = connection;
            Connection.OnError += Connection_OnError;
            Connection.OnData += Connection_OnData;
        }

        void Connection_OnError(object Sender, int ErrorCode)
        {
            Console.Out.WriteLine("Connection error:" + ErrorCode);
        }


        public void reader()
        {
            try
            {
                bool connected = clientSock.Connected;
                while (connected)
                {
                    //TODO check poll/available/connected is best way to wait on socket read
                    if (clientSock.Poll(1000000, SelectMode.SelectRead))
                    {
                        if (clientSock.Available > 0)
                            server.DataAvailable();
                        else
                            connected = false;
                    }

                    if (connected)
                    {
                        if (clientSock == null)
                            connected = false;
                        else
                            connected = clientSock.Connected;
                    }
                }
            }
            catch (Exception e)
            {
                Console.Out.WriteLine("Exception in reader " + e.Message);
            }

            try
            {
                //close session on exit
                Console.Out.WriteLine("Shell closing");
                server.Close(true);
            }
            catch (Exception e)
            {
                Console.Out.WriteLine("Error closing connection " + e.Message);
            }
        }

        //send data to socket
        object sendLock = new object();
        void server_OnSend(object Sender, byte[] Buffer)
        {
            try
            {
                lock (sendLock)
                {
                    Console.Out.WriteLine("OnSend");
                    int toSend = Buffer.Length;
                    int sent;
                    int ptr = 0;
                    while (toSend > 0)
                    {
                        if (clientSock == null || !clientSock.Connected)
                        {
                            return;
                        }

                        sent = clientSock.Send(Buffer, ptr, toSend, SocketFlags.None);
                        if (sent < 0)
                        {
                            return;
                        }
                        ptr += sent;
                        toSend -= sent;
                    }
                }
            }
            catch (Exception e)
            {
                Console.Out.WriteLine("Exception in send:" + e.Message);
            }
        }

        void server_OnReceive(object Sender, ref byte[] Buffer, int MaxSize, out int Written)
        {
            Written = 0;
            try
            {
                Console.Out.WriteLine("OnRec");
                if (clientSock.Poll(100000, SelectMode.SelectRead))
                {
                    Written = clientSock.Receive(Buffer, MaxSize, SocketFlags.None);
                }
            }
            catch (Exception e)
            {
                Console.Out.WriteLine("Exception in rec:" + e.Message);
            }
        }

        //handle incoming data and echo to other client
        void Connection_OnData(object Sender, byte[] Buffer)
        {
            Console.Out.WriteLine("OnData");
            sshChat.Broadcast(Buffer);
        }

        internal void Broadcast(byte[] buffer)
        {
            Connection.SendData(buffer);
        }
    }
}
Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.

Reply

Statistics

Topic viewed 3636 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!