EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Can SolFS replace a ZIP archive ?

Also by EldoS: Rethync
The cross-platform framework that simplifies synchronizing data between mobile and desktop applications and servers and cloud storages
#33337
Posted: 05/20/2015 04:22:04
by Jens Olesen (Basic support level)
Joined: 11/20/2009
Posts: 7

My customer has a lot of files placed in a hierarchy of folders on a Windows file server.
Every folder contains files received from / send to a specific partner within a month.

All files are pretty small (a few kilobytes) and the content is text (csv / xml)

Many partners and a long history means many folders:

Code
Partner1\
  Send\
    Archive\
      2015_04\
      2015_03\
      ...
      2010_03\
      ...
  Received\
    Archive\
      2015_04\
      2015_03\
      ...
      2010_03\
      ...
Partner2\
   ...


Due to diskspace and backup this setup is becoming a problem, and we have tried to zip the content of each month into one file (one zip file pr. month).

A .zip archive for each month is an ideal solution because of:
1) Disk space
2) Backup of a large file is much faster than backup of many small files
3) Users with access to the zip-archives can open the archives using std. tools (like Explorer) and read the content.

However zip of a large number of files (worst case 1.500.000) is a lengthy and errorprone process - and I don't think it is a realistic alternative.


I have read about SolFS and but I am in doubt about these points:
1) How many FS instances should I have (1 for everything, one for each partner, 2 for each partner, one for each month, ...)
2) Will the backup process see every archive (FS instance) as one big file ?
3) Will human users be able to open every FS instance using std. tools (like Explorer) and read the content ?
4) What version of SolFS will we need ?
5) Do we need to install low level software at the customer site ?

Thanks
#33338
Posted: 05/20/2015 04:43:25
by Eugene Mayevski (EldoS Corp.)

Quote
Jens Olesen wrote:
1) How many FS instances should I have (1 for everything, one for each partner, 2 for each partner, one for each month, ...)


At least you need a separate container file for each partner. Maybe it also makes sense to separate them by month (i.e. have a new file for each month).

Quote
Jens Olesen wrote:
Will the backup process see every archive (FS instance) as one big file ?


Yes.

Quote
Jens Olesen wrote:
Will human users be able to open every FS instance using std. tools (like Explorer) and read the content ?


With Application Edition one can use our tools (either console one or the GUI sample which you can modify for your needs) to export/extract files.

With OS edition one can mount the container file as a virtual drive with a disk letter, then work with it as with the regular disk.

Note, that Explorer is not a "standard tool" -- it has built-in support for ZIP archives, but ZIP archives are not a part of the filesystem hierarchy and in order to open the file contained in the archive one would need to extract the file anyway (Explorer does this for you when you double-click the file in archive in order to open it).

Quote
Jens Olesen wrote:
4) What version of SolFS will we need ?
5) Do we need to install low level software at the customer site ?


Depends on your needs. SolFS Application edition is a direct replacement for ZIP archives, and OS edition lets you mount the disk to access them as a part of the system (so that third-party applications could read and optionally modify files easily).

NOTE: with your particular files (1.5 Kb per file) SolFS won't be able to compress them (this is due to how filesystems compress files, NTFS does the same). You can either employ your own compression when writing these small files, or put them uncompressed.

Also I'd like to recommend you to test speed of file addition with SolFS. It can turn out that the narrow place of your scheme is not ZIP components or SolFS, but the procedure of reading the millions of files off the filesystem. In this situation you might find it better to avoid creation of millions of files on the real disk in the first place (this can be done by employing SolFS OS edition).


Sincerely yours
Eugene Mayevski

Reply

Statistics

Topic viewed 2896 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!