Amazon S3 question

[Sadler]

Hello,

I have an application to build which is going to involve upload of files. The application has to run under SSL and I have specific requirements that:

1. All files are encrypted whilst stored (i.e. at rest)
2. Data is held data centres which are either in the UK or EU

Item number 2 is not going to be an issue as it looks like I can specify locations.

I can see that S3 can be set up to encrypt/decrypt files on the way in and out of storage but it is not something I am familiar with, since the encryption/decryption is handled at the S3 end it looks like it won't impact on what I am doing in Fabrik?

Anyone have any experience or guidance?

Burnsy

 

[cheesegrits]

Question 2 ... Yes, you can specify the region for your bucket. But ... first issue, don't choose "EU - Frankfurt" as your region, as it only supports the new v4 auth, and the library we use for S3 only supports v3. This is an issue we are going to have to resolve at some point, because as of last year all new regions added to ECS will only support v4, so we'll need to rewrite our S3 upload adapter using a different library, or hope the existing library gets updated for v4.

EDIT - just found a PR on the lib we use, where someone has added v4 auth support! Yay. Hasn't been merged yet, but looks solid, and I can start testing it.

https://github.com/tpyo/amazon-s3-php-class/pull/128

So ... my statement is still correct, don't choose Frankfurt at the moment. But it looks like the v4 auth issue will be resolved soon (I just offered to use Fabrik users as guinea pigs for the v4 branch, lol!).


Question 1 ... encrypted at rest. That might be an issue, and we may need to add that as a feature to the settings and code. I've never actually tried, but I think in order to encrypt, the REST API has to specifically request encryption during the PUT operation. I may be wrong, but I don't think you can just designate a bucket as "encrypted" and everything uploaded to it is automatically encrypted. It has to be specifically requested for each upload.

I'll look in to it, but I'm currently about 97.365% sure I'm right. So we'd need to add "encrypt uploads" as an option in the S3 settings, and add the underlying code to the upload processing to enable encryption.

If I am right, would you be prepared to share the cost of implementing this feature? What we usually do when there is a feature which would be of interest / use to other users, but is something only one person is currently asking for, we offer to split the hours 50/50. I can take a quick look and give you an initial estimate, if you are interested.


Oh, and related to 1, we do have an SSL option on the s3 settings, which will ensure that we specify https for all interaction with S3, including any download links to the files shown to your users.

-- hugh

 

[cheesegrits]

Note that there would be one point where the file would not be encrypted "at rest", which is during upload. Just because of the way S3 and http work, we have to act as "piggy in the middle" when you submit a form and upload a file. It gets uploaded (by the web service) to the temp directory on your server, then we push it up to S3, and delete the temporary file, during our form submission processing.

There is no way to proxy the upload directly from the browser to s3. It has to go through your server. And offhand I can't think of any way we (Fabrik) can encrypt that temporary file, as it's written by the web server.

If this is a show stopper, you could probably work round it by using an encrypted file system for the temporary partition on your server (whatever partition your web server is configured to use as temp).

-- hugh

 

[Sadler]

Hello Hugh,
thanks for the replies.

I think it would be worthwhile getting some initial costs as I think this is something that we would use quite a lot in future and getting it done makes sense.

In terms of the temporary file I think this is something that at our level we could live with, if push came to shove I think we would honestly be able to say we had taken all "reasonable" steps to secure the data and that is pretty much us covered.

I will send you my contact details off list for the pricing.

Burnsy

 

[cheesegrits]

OK. I'm actually in the process of completely rewriting our S3 adapter to use the official Amazon SDK. It's the only way we're going to be able to work with the new v4 authentication. I was hoping that change for the library we currently use would suffice, but it's proving somewhat buggy.

I'll get back to you in PM soon.

-- hugh