EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Help with timeout during long FTPS upload

Also by EldoS: MsgConnect
Cross-platform protocol-independent communication framework for building peer-to-peer and client-server applications and middleware components.
#4864
Posted: 02/06/2008 13:49:00
by Gary Mui (Basic support level)
Joined: 02/06/2008
Posts: 5

Hi,
I'm trying to diagnose a problem a client is having while using the SBSimpleFTPS.TElSimpleFTPSClient class to send files via FTPS. Normally, everything works fine. However, for long file transfers (up to 2 hours or more), when the FTP Server tries to send the "226, data transfer okay" message back to the FTPS client, it detects that the socket has been broken.

I suspect that because of the long transfer time, the network and/or firewall are simply closing the connection due to the lack of data being sent over the control socket connection. Does this sound like a reasonable explanation?

To avoid this problem, I've seen other FTP clients (e.g. CuteFTP) have a "Keep Alive" functionality where the client will periodically send benign FTP commands like PWD, LIST, etc. in order to make sure the control socket connection doesn't get closed. Is this capability available in the SBSimpleFTPS.TElSimpleFTPSClient client? If not, is it possible to spawn a separate thread that will periodically issue FTP commands?

Any help or guidance on this problem would be much appreciated!

Thanks,
Gary Mui
#4865
Posted: 02/06/2008 14:22:55
by Eugene Mayevski (EldoS Corp.)

Quote
Gary Mui wrote:
I suspect that because of the long transfer time, the network and/or firewall are simply closing the connection due to the lack of data being sent over the control socket connection. Does this sound like a reasonable explanation?


Yes, I would think about this too. I even think I know the time - 2 hr 38 min. That's standard TCP stack behaviour.

The problem with such "keep alives" is that FTP protocol is a state machine. When in download or upload state, the server expects to receive no data. And the client expects to receive only 226 reply after successful transfer. There's no room (in the standard) for any other command during the transfer. I have reviewed the RFC 959 now and I see no clue about how one can implement sending anything over the command channel while the data is being transferred.

Those other applications, do they send the commands while in transfer state?


Sincerely yours
Eugene Mayevski
#4866
Posted: 02/06/2008 14:23:40
by Eugene Mayevski (EldoS Corp.)

Quote
Eugene Mayevski wrote:
the server expects to receive no data.


By this I meant "receive nothing over control channel".


Sincerely yours
Eugene Mayevski
#4867
Posted: 02/06/2008 14:30:12
by Gary Mui (Basic support level)
Joined: 02/06/2008
Posts: 5

Thanks for the (very fast) reply.

I was thinking about that also, i.e. whether those client implemented keep alives happen during an ongoing file transfer. I will try it out to see if it does.

But does this mean that almost no ftp transfers can take longer than 2 hours and 38 minutes?

Thanks,
Gary
#4868
Posted: 02/06/2008 14:53:09
by Gary Mui (Basic support level)
Joined: 02/06/2008
Posts: 5

I think you were right on those keep alives. I tried using Core FTP (lite) and configured it to send NOOP keep alives every 5 seconds. I saw that it was sending them, but after I initiated a 500 mb transfer, the keep alives stopped. As soon as the file transfer was complete, they resumed. I'm not sure if that's a problem inherent in the protocol or if it's because the lite version of CoreFTP is limited to one simultaneous connection. I'll have to find a client that can do multiple simultaneous transfers and see if they operate on the same control connection (though I doubt it).

Of course, my trial license for CuteFTP which I referred to in my original post expired today.

Thanks,
Gary
#4869
Posted: 02/06/2008 15:24:25
by Gary Mui (Basic support level)
Joined: 02/06/2008
Posts: 5

It looks like the underlying TCP connections have an optional SO_KEEPALIVE option that is meant to send data like heartbeats to determine if the server it has connected to is still up. Can you confirm whether the socket connection created by the client FTPS library sets this socket option? I know it can be enabled by Java, but am not familiar enough with .NET to know if this can similarly be done.

Also, is the underlying socket connection exposed via an API so that any socket options can be separate outside of the API?

Thanks!
Gary
#4874
Posted: 02/07/2008 02:32:03
by Eugene Mayevski (EldoS Corp.)

Well, keep-alive flag is designed for slightly different purpose - testing the connection. I am not sure if this flag will help you. We can turn it off for the next beta version of SBB 6 in order for you to check how it behaves.

The sockets are buried deep inside and can't be accessed directly, sorry.


Sincerely yours
Eugene Mayevski
#4917
Posted: 02/08/2008 09:21:48
by Gary Mui (Basic support level)
Joined: 02/06/2008
Posts: 5

I understand that Keep-alive is used at TCP layer to determine if the server end of a connection has disconnected and if it does, to close the connection. Otherwise, the Keep-alive would "help" keep the connection open as long as the server side replied to the keep-alive request.

Are you saying that the keep-alive option for that client socket is already turned on? I wouldn't expect that turning it off would help.

Has anyone else had this problem before where long-running ftp transfers were having problems because of the control connection getting closed? We've had other FTP clients successfully transfer files over several days (4 gig files) but the one client that we know of that uses Blackbox to transfer to our FTP server fails after 2 hours (or so).

Thanks,
Gary
#4918
Posted: 02/08/2008 09:38:38
by Eugene Mayevski (EldoS Corp.)

I was thinking about something else when typing - the option is OFF now and we will make it ON in the next build.


Sincerely yours
Eugene Mayevski
Also by EldoS: Callback File System
Create virtual file systems and disks, expose and manage remote data as if they were files on the local disk.

Reply

Statistics

Topic viewed 2608 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!