EldoS | Feel safer!

Software components for data protection, secure storage and transfer

Error on Amazon S3 Buckets in Frankfurt

Also by EldoS: Callback File System
Create virtual file systems and disks, expose and manage remote data as if they were files on the local disk.
#33027
Posted: 04/13/2015 02:57:26
by ntr1 (Standard support level)
Joined: 02/05/2014
Posts: 73

Hi,

it seems when you try to sent files to an Amazon S3 Bucket created in Frankfurt region from the AWS console, it occurs a "400 bad request error".

The component always creates buckets in the US region.

It seems a known issue:

https://github.com/boto/boto/issues/2741
https://trac.cyberduck.io/ticket/8302

Is there an option/property to solve this ?
#33028
Posted: 04/13/2015 05:34:20
by Ken Ivanov (EldoS Corp.)

Thank you for the detailed guidance on the issue. It looks as the newly introduced geographical buckets only support AWS signatures of version 4. I've just added implementing support for that type of signatures as a top priority task to our to do list, so you may expect it available in one of early version 13 builds.

Ken
#33597
Posted: 06/08/2015 02:28:07
by ntr1 (Standard support level)
Joined: 02/05/2014
Posts: 73

Hello,

thank you for adding to the Cloudblackbox component the new signature and geographic support for Amazon S3.

+ [All] (Cloud) Added support for version 4 signatures in AWS S3
* [All] (Cloud) Geographic locations support in AWS S3 improved


Can you give me a sample code on how to write into a bucket located in Frankfurt?
#33599
Posted: 06/08/2015 05:33:52
by Ken Ivanov (EldoS Corp.)

Basically, what you need to do to access buckets located in Frankfurt is:

- force the AWS data storage component to use version 4 signatures by setting the TElAWSS3DataStorage.UseVersion4Signatures property to true;

- set TElAWSS3DataStorage.Region property to awsrEUFrankfurt.

After adding the above modifications your application should access EU buckets just fine.

Ken
#33601
Posted: 06/08/2015 09:00:32
by ntr1 (Standard support level)
Joined: 02/05/2014
Posts: 73

Hello,

thank you for your quick answer. It works.

However, I still have a problem.

I've tried to list all buckets and then to get the region or each bucket using this method:


GetBucketLocation(sBucketName);


Unfortunately, when the bucket name is that one in Frankfurt, an exception is raised:


'HTTP request failed with code 400, message is Bad Request AuthorizationHeaderMalformed'


If instead I set the region property to Frankfurt before calling GetBucketLocation, it works. But I must assume I don't know the region of the bucket...
#33604
Posted: 06/09/2015 05:41:39
by Ken Ivanov (EldoS Corp.)

Hi,

That was a nice catch on your side, thank you. The problem seems to be caused by a bug in the Amazon S3 service. Apparently, you can't request a bucket location from the default S3 service endpoint (s3.amazonaws.com) when the bucket is located at a different geographic location (the endpoint always returns error 400 for such buckets). However, if you use a different, 'dedicated', geographic endpoint, it will work out just fine:

Code
  // Works fine
  FStorage.Region := awsrUSWestNCalifornia;
  Location := FStorage.GetBucketLocation('frankfurtbucket');

  // Works fine
  FStorage.Region := awsrEUFrankfurt;
  Location := FStorage.GetBucketLocation('frankfurtbucket');

  // Won't work
  FStorage.Region := awsrDefault;
  Location := FStorage.GetBucketLocation('frankfurtbucket');


So it seems to be a good idea to explicitly use a dedicated endpoint when requesting a bucket location.

Ken
#34832
Posted: 10/20/2015 12:54:57
by Zoë Peterson (Priority Standard support level)
Joined: 05/24/2007
Posts: 22

We've had customers run into this too, and I'm not exactly clear on the fix.

Are we supposed to set UseVersion4Signatures ourselves? It sounds like you're suggesting that we have to query for the bucket region explicitly before any interaction, then set UseVersion4Signatures manually by somehow knowing whether a particular region requires them or not. If that's the updated signature type, why isn't doesn't SBB use it as the default behavior everywhere? And if it does need to be set by region, why isn't SBB encapsulating that knowledge for us?
#34836
Posted: 10/20/2015 16:37:20
by Ken Ivanov (EldoS Corp.)

Hi Zoë,

Generally, Amazon's current aims, as far as we understand them, is to transition all its authorization processes to version 4 signatures. Therefore a good strategy for you and other AWS users would be to always use version 4 signatures in new applications (as we apparently cannot guarantee that Amazon will support version 3 signatures in all new features it rolls out, as they showed with Frankfurt buckets). This will require you some bucket location handling though, as you would always need to set the region explicitly.

We will probably make version 4 the default signature type in SBB 14. We didn't do that straightaway, as this is a serious breaking change and we didn't want to break the user code - at the same time expecting version 4 signatures to gain larger share of Amazon services until the next major SBB version is released. For the same reasons, we didn't encapsulate the location knowledge into the components.

Anyway, thank you very much for your suggestions. They make perfect sense and make a great contribution towards our understanding of SecureBlackbox users' needs.

Ken
Also by EldoS: Callback File System
Create virtual file systems and disks, expose and manage remote data as if they were files on the local disk.

Reply

Statistics

Topic viewed 2309 times

Number of guests: 1, registered members: 0, in total hidden: 0




|

Back to top

As of July 15, 2016 EldoS Corporation will operate as a division of /n software inc. For more information, please read the announcement.

Got it!