EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Implement block mode for objects in Azure

Also by EldoS: Solid File System
A virtual file system that offers a feature-rich storage for application documents and data with built-in compression and encryption.
#33339
Posted: 05/20/2015 08:40:22
by VoxPopuli Robot  (EldoS Corp.)

Currently objects are written to Azure in page mode, which gives slightly larger size and format incompatible with other software. Block mode would let other software use the original data, written with CloudBlackbox.

If you like the idea, vote for it on https://www.eldos.com/sbb/wishlist.php
#33560
Posted: 06/04/2015 08:47:07
by Paolo Furini (Basic support level)
Joined: 06/04/2015
Posts: 3

Can you please elaborate on this?

Do you mean that, with the current version of CloudBlackBox, it isn't possible to use the data saved to Azure cloud storage directly in other apps or consumed by http clients (like a browser)? What "other software" are you referring to?

My usage scenario could be: a client uploads a file using FTP(S) on a virtual FS using your components, and CloudBlackBox as the backend storage connected to azure storage. The same client should be able to eventually download that files using resource uri syntax (https://myaccount.blob.core.windows.net/mycontainer/virtualpath/vp2/myblob) or attaching a CDN endpoint to it.
Is it currently possible? What are the real benefits in using Block blobs vs page blobs.

Thank u for the clarifications..
#33562
Posted: 06/04/2015 11:50:39
by Eugene Mayevski (EldoS Corp.)

Quote
Paolo Furini wrote:
Do you mean that, with the current version of CloudBlackBox, it isn't possible to use the data saved to Azure cloud storage directly in other apps or consumed by http clients (like a browser)? What "other software" are you referring to?


Yes, this is correct. Even when you don't use encryption and compression, in Azure CloudBlackbox adds its own information, making files a bit larger (not importand) and not "compatible" with any other applications.
Quote
Paolo Furini wrote:
The same client should be able to eventually download that files using resource uri syntax (https://myaccount.blob.core.windows.ne...p2/myblob) or attaching a CDN endpoint to it. Is it currently possible?


At the moment this is not possible.You can vote for a feature and we will possibly implement it in future updates.


Sincerely yours
Eugene Mayevski
#33575
Posted: 06/05/2015 10:18:45
by Paolo Furini (Basic support level)
Joined: 06/04/2015
Posts: 3

Thanks Eugene for the clarification, but in the original feature request you say:
Quote
Block mode would let other software use the original data, written with CloudBlackbox


I'm wrong in assuming that implementing the block write mode would let other software use the objects directly, or is there more work to do?
Maybe a switch needs to be added, to let the user decide if saving or not additional metadata along with the files?

Another question regarding the FTPSBlackBox component used in combination with azure cloud storage: is it mandatory to use CloudBlackBox for such an integration, or is there another way to read/save block blobs directly in azure, and to present them like a virtual FS to the FTP client?

Regards,
Paolo
#33576
Posted: 06/05/2015 10:32:30
by Eugene Mayevski (EldoS Corp.)

Quote
Paolo Furini wrote:
I'm wrong in assuming that implementing the block write mode would let other software use the objects directly, or is there more work to do? Maybe a switch needs to be added, to let the user decide if saving or not additional metadata along with the files?


What seems to be a small addition from functionality point of view is a much larger action for us.

In order to maintain our products as integrated solutions and not chaotic collections of losely linked functions we need to consider, how the requested feature fits into the products and into our development plans for the product.

If we decide that the feature makes sense, we need to determine, how it can be fit from design point of view - will it be some option exposed via a property, or a method to be called, or a new class. Next we need to define the best way to integrate the feature into the existing code. It is possible, that we will need to develop much larger scope of functionality to ensure that the feature in question doesn't stand out of the feature croud, but fits nicely into the product.

When it comes to development, each feature must be implemented for a number of target editions and supported platforms. Then it must be tested, so we need to create a set of tests (again for the extensive number of supported editions and platforms). And that's not all. As we offer components and classes, grouped into various packages, in many cases new feature needs to be added to several dozens of package source files, installation scripts and alike. Finally we need to document each feature and maybe even write a separate how-to.

This doesn't mean that we decline all feature requests. In opposite all of them are evaluated and many are included into the development plans. The timeframe for implementing them is not fixed, however. In some cases the feature can be implemented relatively quickly, or it can be offered as a custom development service (for a fee). Other features can request hundreds of man-hours for implementing. So the time really depends on complexity and demand for the functionality.

Quote
Paolo Furini wrote:
Another question regarding the FTPSBlackBox component used in combination with azure cloud storage: is it mandatory to use CloudBlackBox for such an integration, or is there another way to read/save block blobs directly in azure, and to present them like a virtual FS to the FTP client?


I am not sure that I understand what information you want to present in which way and how FTPS is at all related to CloudBlackbox. CloudBlackbox uses TElHTTPSClient as a transport, as HTTP/HTTPS is a standard way to access cloud storages.

So the question is - do you want to use a cloud storage instead of a local file location (i.e. take files from the cloud and upload to FTP server and vice versa) OR you want to access the cloud using FTP protocol?


Sincerely yours
Eugene Mayevski
#33578
Posted: 06/05/2015 11:40:57
by Paolo Furini (Basic support level)
Joined: 06/04/2015
Posts: 3

Quote
Eugene Mayevski wrote:
What seems to be a small addition from functionality point of view is a much larger action for us.
...


Sorry, but I'm confused now.. The original feature request didn't originate from you or your team (there's your name on it) ? And in that 2-lines message you wrote: "Block mode would let other software use the original data", but in your first reply to my post you wrote something contradictory to your original post: "in Azure CloudBlackbox adds its own information, making files a bit larger (not importand) and not "compatible" with any other applications". So maybe there's something wrong or missing in your original feature request..
Regarding the complexity I agree with you and I was not trying to diminishing it.

Quote
So the question is - do you want to use a cloud storage instead of a local file location (i.e. take files from the cloud and upload to FTP server and vice versa) OR you want to access the cloud using FTP protocol?


I want to use Azure cloud storage as the backend of my FTP(S) server, presenting the client a virtual file system. But ideally the files should be saved as plain blobs accessible by other azure code, or, with the user consent, publicly available.
This FTP server is a workaround for some "legacy" corporate apps that are able to publish documents only by using FTP.

Regards,
Paolo
#33580
Posted: 06/05/2015 12:38:14
by Eugene Mayevski (EldoS Corp.)

Quote
Paolo Furini wrote:
The original feature request didn't originate from you or your team (there's your name on it) ?


That's complicated :). I've copied the request of one of our users, and its my name in the wishlist specified. The forum posts are made from under special VoxPopuli Robot account (VoxPopuli is our software which powers the wishlist).

Our developers will explain you more about these modes.


Sincerely yours
Eugene Mayevski
#33582
Posted: 06/05/2015 12:51:51
by Ken Ivanov (EldoS Corp.)

Hi Paolo,

It seems there is some misunderstanding about functionality offered by both Azure and SecureBlackbox.

In page mode objects can only be written as a set of 512 byte pages (this is a requirement of Azure), therefore SecureBlackbox uses padding to support objects with sizes which are not multiple of 512. As there is no general standard for such padding, any third party software (e.g. Azure SDK) will not be able to remove padding added by SecureBlackbox correctly. This is what Eugene meant by 'incompatibility'.

Objects uploaded in block mode can be of any size (there are different restrictions in place though), so it would be possible to make SecureBlackbox compatible to other Azure clients on the market by getting use of the block mode.

Ken
Also by EldoS: CallbackFilter
A component to monitor and control disk activity, track file and directory operations (create, read, write, rename etc.), alter file data, encrypt files, create virtual files.

Reply

Statistics

Topic viewed 1187 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!