EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Question about memory usage

Also by EldoS: BizCrypto
Components for BizTalk® and SQL Server® Integration Services that let you securely store and transfer information in your business automation solutions.
Posted: 04/14/2008 02:32:23
by WaZZu (Basic support level)
Joined: 05/05/2006
Posts: 5

Hello All,

I would like to know what's wrong in memory usage by SolFS.
Please take a look at the code:
SolFSStream *inStream= NULL;
    inStream = new SolFSStream(st, fileName, true,
                           true, true, false, false,       
                           st->get_FilesPassword(), st->get_FilesEncryption(), 0,
                           st->get_Compression(), st->get_CompressionLevel(), 16);
catch(ESolFSError *err)
    // Handle exception
char *buf = new char[4096];
size_t len = outStream.Length(); // file stream we would like to save into SolFS
while(len > 4096)
    size_t szr = outStream.Read(buf, 4096);
    inStream->Write(buf, szr);
    len -= szr;
if(len > 0)
    size_t szr = outStream.Read(buf, len);
    inStream->Write(buf, szr);
delete inStream;
delete [] buf;

When you loop this code for more than 10,000 files the memory usage becomes very high, and never goes down, even after all procedures are finished.
Then, I tried to import that many files into SolFS container using SolFS Explorer. I noticed the same thing, memory never gets cleaned up and stays very high.

Thank you.
Posted: 04/14/2008 03:02:23
by Volodymyr Zinin (EldoS Corp.)


What version of the product are you using?
Posted: 04/14/2008 03:03:10
by Volodymyr Zinin (EldoS Corp.)

Vladimir Zinin wrote:
What version of the product are you using?

Sorry, I mean standard or driver edition.
Posted: 04/14/2008 03:25:59
by WaZZu (Basic support level)
Joined: 05/05/2006
Posts: 5

Hello Vladimir,

This is standard SolFS edition.
Posted: 04/14/2008 09:40:37
by Alexander Plas (EldoS Corp.)

Unfortunately I not able to reproduce your problem.
Can you specify details of your storage (page size, encryption, compression method and compression level) and your files (minimal, maximal and average file size, type of files in context of the compression) ?
Posted: 04/14/2008 10:21:03
by WaZZu (Basic support level)
Joined: 05/05/2006
Posts: 5

page size: 4096
encryption: ecAES256_HMAC256
compression: not used
min file size: 500bytes
max file size: 8-10MB
average file size: 400-800KB
# of files: ~15,000
SolFS version: 3.2.85
Most of the files are text files and XDR encoded files. Some of them Base64 encoded files.

I wonder why it's happening to me. I tried to comment out opening and writing to SolFS streams. No changes in the memory difference in this case. Right after I start creating and writing to SolFS streams, memory usage just keeps increasing as new streams are created. I made sure streams are closed after I used them. As I said, I tried to import these files using SolFS Explorer. Memory usage by SolFS Explorer boosted by 40-50MB. It would be OK, if after completion of import operation, SolFS would release that memory back, but it stays increased as it was. Then, if you try to import again, memory starts to grow again, and never goes down. You have to re-open SolFS container in order to release the memory.
So, I think there are some problems in memory management.
Thank you, any help is highly appreciated.

Posted: 04/15/2008 08:11:44
by Eugene Mayevski (EldoS Corp.)

Still no luck.

Let's try to do the following:

Create a small sample project that will setup the parameters you mentioned and then create 10 000 files in SolFS storage, but (!) instead of putting the actual file, fill the 4096-byte buffer with pattern <0x00><0x01><0x02>..<0x0FFF> and save this buffer to SolFS stream.

Finally, please send us the compiled version and the source code.

Sincerely yours
Eugene Mayevski
Posted: 04/16/2008 23:53:29
by WaZZu (Basic support level)
Joined: 05/05/2006
Posts: 5

Hello Eugene,

I've posted a sample project to the Helpdesk system a couple of days ago, since I couldn't attach it here. Please let me know what the current state is for this issue.

Thank you, appreciate your help.
Also by EldoS: CallbackDisk
Create virtual disks backed by memory or custom location, expose disk images as disks and more.



Topic viewed 4743 times

Number of guests: 1, registered members: 0, in total hidden: 0


Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!