EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Looking for tips on how to improve SFTP transfer speed

Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.
#6120
Posted: 05/05/2008 22:11:17
by Charles DeWeese (Standard support level)
Joined: 04/17/2008
Posts: 52

I'm looking for tips on how to improve SFTP transfer speed.

I know there's an entry in the manual that suggests adjusting the DownloadBlockSize and PipeLineLength but it seems like these settings need to be server specific as what works great on one server may not work at all on another server. No matter how I tweak them I can't seem to get ideal results. Oh and I do have AutoAdjustTransferBlock set to false.
#6122
Posted: 05/06/2008 00:31:13
by Charles DeWeese (Standard support level)
Joined: 04/17/2008
Posts: 52

To let me give you a good example, why is the SophisticatedClient VCL demo so much slower than the SimpleSFTPClient VCL demo?

I understand that they both use SBB in different ways to provide the same functionality but when comparing the throughput of each demo there's not even close to being similar.

SophisticatedClient = 256 KB/s download
SimpleSFTPClient = 2.85 MB/s download

It just so happens that I designed my program to work similar to SophisticatedClient and it too suffers the same poor transfer speeds.
#6124
Posted: 05/06/2008 02:08:19
by Eugene Mayevski (EldoS Corp.)

SophisticatedClient is based on ElSftpClient component. It doesn't have pipelining at all, i.e. the second block is sent after confirmation for the first block delivery has been sent. Of course, this is much slower, than with pipelining, when many blocks are sent, then confirmation for at least one is received and the next one is sent and so on. Implementing pipelining right is a big job.


Sincerely yours
Eugene Mayevski
#6135
Posted: 05/06/2008 08:36:48
by Charles DeWeese (Standard support level)
Joined: 04/17/2008
Posts: 52

Thank you for your explanation.

Is there an example of how to use pipelining with SynchronousMode = false.
#6138
Posted: 05/06/2008 10:15:21
by Charles DeWeese (Standard support level)
Joined: 04/17/2008
Posts: 52

I believe I have worked it out for download file transfers but I'm running into some difficulty with uploads, the problem as I see it is that there isn't an event triggered once a data packet has been transmitted via the transfer manager with a value of the size of the packet being transmitted.


ElSftpClient.OnTransferCompleted is triggered for downloading and I can keep track of when the transfer manager has completed and needs to be feed more data, but for uploads this event is called with a count=0 so I can't keep track of the data leaving the transfer manager and being sent.

If I assume that each time OnTransferCompleted triggers for an upload the amount of data sent is equal to UploadBlockSize then I start seeing data corruption in the uploaded file.
#6140
Posted: 05/06/2008 10:46:10
by Ken Ivanov (EldoS Corp.)

Would you be so kind to specify the exact task you need to resolve? TElSFTPClient reports the progress of the transfer via the OnProgress event.
#6141
Posted: 05/06/2008 10:49:41
by Charles DeWeese (Standard support level)
Joined: 04/17/2008
Posts: 52

I'm trying to implement pipelining as such to speed up transfers when using TElSFTPClient directly.
#6143
Posted: 05/06/2008 11:02:13
by Ken Ivanov (EldoS Corp.)

TElSFTPClient does already contain built-in support for pipelining. Please use PipelineLength property.
#6144
Posted: 05/06/2008 11:13:02
by Charles DeWeese (Standard support level)
Joined: 04/17/2008
Posts: 52

I'm sorry but I am now rather confused, If you look a few posts up and read the reply by Eugene Mayevski he says "SophisticatedClient is based on ElSftpClient component. It doesn't have pipelining at all"

What I'm trying to do is use ElSftpClient in asyncronized mode similar to the SophisticatedClient demo. I pointed out that this demo is very slow and I'm trying to understand how to improve the throughput while keeping ElSftpClient asyncronized mode.
#6145
Posted: 05/06/2008 11:24:22
by Ken Ivanov (EldoS Corp.)

I suppose, Eugene mentioned the SophisticatedClient demo, not the TElSFTPClient component.

Please try to
* set AutoAdjustTransferBlock property to true,
* increase FILE_BLOCK_SIZE constant to $20000

and check if the speed is comparable to the speed of SimpleSFTPDemo.

Please also note, that SophisticatedClient demo uses Delphi sockets that are quite slow.
Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.

Reply

Statistics

Topic viewed 2516 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!