EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Quite Happy

Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.
#553
Posted: 06/28/2006 09:19:42
by Santiago Castaño (Standard support level)
Joined: 04/16/2006
Posts: 155

Hi,

I'm writing only to congratulate you for your work with SFTP speed and Version 4.4.0.90.

I started using SFTP-SSH components a year+ ago, and i realized of the slow speed i was having against WinSCP. After messing up with TransferBlockSize, i was able to speed up from 300Kb/sec to 600Kb/sec on upload for example.

But with UploadBufferSize, DownloadBufferSize (and PipeLineLength) i am able now to upload up to 2400Kb/sec!! a 4x improvement! and WinSCP stays in 2100Kb/sec. In download speed i get exactly the same results as WinSCP now; so i'm very satisfied.

Keep up the good work
#556
Posted: 06/28/2006 10:21:50
by Antti Karttunen (Basic support level)
Joined: 05/30/2006
Posts: 5

Hello,

I am also very pleased with the improvements in the new 4.4.90 version of the SFTP component, although the transfer speed is not so crucial to me as reliability is.

Santiago, would you mind sharing any tips on how you have been able to get the best speedups? I have planned to do some speed-optimizations in the near future, but it would be nice to hear what kind of results others have obtained. What have you found to be good combinations of the UploadBufferSize, DownloadBufferSize and PipeLineLength? Have you tested only with OpenSSH or with several different server types?
#558
Posted: 06/28/2006 10:43:00
by Santiago Castaño (Standard support level)
Joined: 04/16/2006
Posts: 155

Hi!,

I'm still testing all, but i am satisfied with this results...

I set pipelinelenght to 32 (default i think). UploadBufferSize to $40000 and DownloadBufferSize to $200000. I know that the upload buffer should be 32768, but openssh (on CENTOS that we use) support that buffer size. We don't have to connect to other kind of server.

Then, i upload/download files using "adaptative buffers"; I start asking to transfer $2000 bytes, and get the time in ms of the transfer, if it isn't greater than a timeout (2 seconds), i increase this buffer by *2; and always this until Maximum_buffer (size of the buffer*10). If it exceeds this timeout of transfer, i divide by 2 the data to transfer.

This way, i get most of the speed on local connections; and on busy networks or over the net i get a more "response" application (if i want to cancel i don't have to wait for 128000 bytes to be transfered).

Anyway, i should investigate more on pipelinelength, because i hace 32, but i think that i'm not getting more than 10 (and i think that i don't have to... so maybe i'll set pipelinelength to 5-10) and adjust the maximum buffer size (to decrease memory requirements).

Anyway, the best top speeds are when i upload $40000 bytes or download $200000 bytes. Openssh seems to run with downloads of more bytes, but no improvement is seen; also the same with uploading.

As you say that you need reliability instead of transfer speed i recomend you to ask for less bytes at the beginning, and make them go as the speed of the connection (maybe i'm wrong, so don't take all my words :p).

#562
Posted: 06/28/2006 11:02:41
by Santiago Castaño (Standard support level)
Joined: 04/16/2006
Posts: 155

Also, note that i'm not using DownloadFile nor UploadFile methods. Because i need to control all the process of upload/download.

But i tested them, and setting all the UploadBufferSize=0=DownloadBufferSize and pipelinelength withouth touching, i gout more or less 1900Kb/sec on download and upload. So those parameters are quite well auto-calculated. If you don't mind too much the speed; maybe that is a very good solution that Eldos bring to us easily.

I wanted to get at least the speed of WinSCP, because our clients used it and they always were saying that our application was slower on transfers of big files :(.
#564
Posted: 06/28/2006 11:09:45
by Ken Ivanov (EldoS Corp.)

Quote
Also, note that i'm not using DownloadFile nor UploadFile methods. Because i need to control all the process of upload/download.

Would you be so kind to specify, what kind of control do you need? You can use the ElSimpleSFTPClient.OnProgress event along with DownloadFile/UploadFile methods to track the progress of the current operation (and to interrupt it if needed). Do you need some more transfer-related information?
#567
Posted: 06/28/2006 13:05:42
by Santiago Castaño (Standard support level)
Joined: 04/16/2006
Posts: 155

Yes of course!.

First of all, i needed to feed the data with streams, and not with path-strings <- Important

Next, sometimes, i need to set SBSftpCommon.fmAppend in the OpenFile.

So i can't use your useful routines, anyway, i want to know a thing from your algorithm. I see the PrepareTransfer routine... I haven't analized it deeply, but my guess is... does your routines adapt to a congested network or a slow internet and a fast local network?. Because when yesterday i tested this routines to see if i could move a little bit of mine to this; i found that if i upload/download to a server in internet, when i hit my "cancel" button, it didn't responds as fast as mine; because i think that it was trying always to send/receive 128000 bytes; so in the onprogress where i call a routine similar to application.processmessages, this line was never called and the button seemed most of the time to not respond.

Are the routines speed-adaptative in any way?. An interesting thing anyway is to be able to feed streams, and specify creation attributes at least in uploading.

Anyway, it's a great great job what you did. I'm impatiently waiting to see PKI working with chunk on bytes instead of a whole file; and i'll be very very happy :p
#568
Posted: 06/28/2006 13:23:46
by Ken Ivanov (EldoS Corp.)

Thank you very much for the feedback.

Quote
First of all, i needed to feed the data with streams, and not with path-strings

We will consider implementing streams support in one of the future build updates.

Quote
Next, sometimes, i need to set SBSftpCommon.fmAppend in the OpenFile.

Hmm, the ability to resume downloads/uploads also seems to be quite useful. Added to the to do list.

Quote
I haven't analized it deeply, but my guess is... does your routines adapt to a congested network or a slow internet and a fast local network?

No. PrepareTransfer just breaks the file to be read/written into a number of chunks according to values of UploadBlockSize/DownloadBlockSize and PipelineLength values. If AutoAdjustTransferBlock property is set to true, it uses optimal block size and pipeline length values (obtained experimentally).

Quote
Are the routines speed-adaptative in any way?

Both 'yes' and 'no'. If AutoAdjustTransferBlock is set, ElSFTPClient uses different optimal block sizes for different servers, but it does not take into account the actual network-layer bandwidth.

Quote
I'm impatiently waiting to see PKI working with chunk on bytes instead of a whole file; and i'll be very very happy :p

It is almost ready -- hope to publish it in a week or two.
#569
Posted: 06/28/2006 13:38:08
by Santiago Castaño (Standard support level)
Joined: 04/16/2006
Posts: 155

Quote

Both 'yes' and 'no'. If AutoAdjustTransferBlock is set, ElSFTPClient uses different optimal block sizes for different servers, but it does not take into account the actual network-layer bandwidth.


It could be and excellent feature to use gettickcount/diff (as you do in many routines), and adjust dinamically the transfer block based on a new property i.e. TransferBlockTimeOut; because i don't know how all the rest of the people do their programs, but if my client wants to connect to his server from his home, the button cancel (and all the rest of the application, seems frozen and not responsive if he runs for example emule in the middle of the transfer ;)). Anyway, the solution is to give the users the ability to transfer in a thread; but if they transfer synchronally (they never choose to transfer in a thread); it seems quite unresponsive.

Great to see your answers, i hope to wipe my routines and use yours; i'm sure they'll be 200% more perfect than mines.
#570
Posted: 06/28/2006 13:42:20
by Antti Karttunen (Basic support level)
Joined: 05/30/2006
Posts: 5

Santiago,

thank you very much for your detailed analysis and good tips. I like the idea of sending smaller chunks of data first. It is also nice to know that the default parameters work quite well.

Stream support would be a nice addition in the future.
#571
Posted: 06/28/2006 14:09:22
by Eugene Mayevski (EldoS Corp.)

Quote
Santiago Castaño wrote:
if my client wants to connect to his server from his home, the button cancel (and all the rest of the application, seems frozen and not responsive if he runs for example emule in the middle of the transfer ). Anyway, the solution is to give the users the ability to transfer in a thread; but if they transfer synchronally (they never choose to transfer in a thread); it seems quite unresponsive.


There's OnProgess event, in whose handler you can (a) call ProcessMessages and (b) abort operation.


Sincerely yours
Eugene Mayevski
Also by EldoS: CallbackRegistry
A component to monitor and control Windows registry access and create virtual registry keys.

Reply

Statistics

Topic viewed 11137 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!