S3 — A Modern Approach to Hiding Secrets!

What you gonna learn in this story:

• What are AWS and S3 exactly?
• How to set up AWS CLI 2?
• How to create a bucket and store objects?
• How to change access policies?
• How to hide secrets and share them securely?

What is AWS? What is S3?

Amazon S3 Logo

AWS stands for “Amazon Web Services” and “S3” for “Simple Storage Service”. It is a cloud storage solution provided by Amazon as a part of AWS. AWS along with GCP by Google and Azure by Microsoft, which are the top 3 cloud computing platforms that offer web services on the go.

AWS S3 offers high reliability and scalability to host content that needs near-zero downtime and has seamless integration with other services.

Now, why do you need S3 as a beginner?

You can host your own static website on an S3 bucket and later make it dynamic with an AWS EC2 instance which is easily integrable. You can add as many services as you like as later. Or you can host your web-app somewhere and use it as a fast and secure cloud storage solution.

Our own Hello IITK uses S3 as a storage solution!

There are two important terms in S3: “bucket” and “object”. Buckets are essentially containers or a big “folder” to store files known as “objects”. Note that buckets don’t have other “buckets” or subfolders inside them, they just have objects.

Of course, you can name objects in such a way that some hierarchy becomes apparent. For example:

users/sam.txt, users/john.txt, users/tom/profile.json, images/jpeg/aw2.jpg, images/data/aw4.png

But, all the files above lie together in some bucket. This is known as a flat structure with no “depth” or hierarchy.

To get started, first, you need to sign up for AWS, which, unfortunately, requires a valid debit or credit card. But fret not, you can log out and log in again, you will be able to access vital user information such as access-key-ID and region-name and generate secret-access-key to presign URLs and stuff. However, you won’t be able to use any service such as EC2. Also, note that AWS offers generous resources in its free tier for small projects, so you won’t be charged unless you are doing some serious stuff and engaging a lot of people.

After logging in, this is how your console should look (https://aws.amazon.com/console/),

AWS Management Console

Do you see “Ohio” at the top-right? Click on it, to see a list of “regions” which you can select from. The default is “US East(Ohio) us-east-2”. The region-code that you will require later is “us-east-2”. If you choose Mumbai in the Asia Pacific region, then your region-code will be “ap-south-1”.

Actually, AWS has various data centres all over the world to reduce latency and increase data reliability. Though this is an endpoint to make requests here, later when you create servers, you would like it to be near your target audience, so that they have a better user experience.

To generate keys, click on “My Security Credentials” -> “Access Keys” to go to

Access keys

Click on “Create New Access Key”. Your keys will be generated, and you should download the keys and keep them safe otherwise, anyone can impersonate you.

Let us set up AWS CLI now. It enables you to do everything by the command-line interface. By the way, the latest version is AWS CLI 2, which is recommended for you.

For Linux users, you can visit the official page: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html .

The commands are:

$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 
unzip awscliv2.zip 
sudo ./aws/install

For Windows users, you can visit the official page: https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-windows.html . The download link for the latest installer:

https://awscli.amazonaws.com/AWSCLIV2.msi

Check whether you have successfully installed by:

D:\>aws --version
aws-cli/2.0.48 Python/3.7.7 Windows/10 exe/AMD64

Now, let us configure the AWS CLI!

You can choose any of the four output formats in below from “JSON”, “YAML” (“YAML-stream” too), “text”, “table”. If you are confused, enter “json” below.

For more information regarding output format, you can visit https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html .

D:\>aws configure
AWS Access Key ID [None]: #Your AWS Access Key that you created                                                    AWS Secret Access Key [None]:  #Your AWS Secret Access Key that you created along with the above Access Key                                                        Default region name [None]: #Your region code Ex: us-east-2                                                                  Default output format [None]: # Your preferred format. Ex: json

Yay! You have successfully configured AWS CLI! Now, let us create a bucket.

I have chosen the name of my bucket: “my-medium-test-bucket” and the region as “ap-south-1”. Your bucket will be created in a data centre in that region.

D:\>aws s3api create-bucket --bucket my-medium-test-bucket --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1                                                                      {                                                                                                           "Location": "http://my-medium-test-bucket.s3.amazonaws.com/"                                        }

Note that all regions except us-east-1 have to add this snippet at the end as shown:

--create-bucket-configuration LocationConstraint=ap-east-1 #If your region in the command is ap-east-1

Note that the “arn” for this bucket would be: “arn:aws:s3:::my-medium-test-bucket”. It identifies the bucket uniquely. Let us upload a file to it :

D:\>aws s3 cp blog1.png s3://my-medium-test-bucket                                                      upload: .\blog1.png to s3://my-medium-test-bucket/blog1.png

If you check it by going to the URL: http://your_bucket_name/your_file_name, you will not be able to access it, because S3 blocks public access by default. You can allow public access to some objects by selecting them and choosing “Make Public” in “Actions” tab in the S3 console online or else, you can take help of AWS Policy Generator: http://awspolicygen.s3.amazonaws.com/policygen.html to generate the appropriate policy.

To make all the objects public, select Policy Type as “S3 Bucket Policy”, Principal (anything that will use your object in future) as “*” and select “GetObject” in “Actions” and put your arn (appended with a /* Ex: arn:aws:s3:::my-medium-test-bucket/*). Add the statement and click on “Generate Policy”.

Voila! You have generated your policy that you can put in “Buckets Policy” section in “Permissions” tab online, or you can save it somewhere say policy.json and use CLI:

D:\>aws s3api put-bucket-policy --bucket my-medium-test-bucket --policy file://policy.json

The policy generated here was:

{
  "Id": "Policy1604387043942",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1604387039687",
      "Action": [
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::my-medium-test-bucket/*",
      "Principal": "*"
    }
  ]
}

Now, you should be able to access all the objects that you have uploaded. But, keep in mind that everyone from anywhere can do so anonymously. You can learn more about bucket policy on how to restrict them to certain groups, and only allow a certain level of access to some group of objects only.

Now, let me upload a secret on my bucket that I don’t want anyone to access anonymously.

So, I uploaded a file called ‘secret.txt’ on my bucket. Then, I changed my bucket policy on allowed “Resources” key to “arn:aws:s3:::my-medium-test-bucket/blog*”. This allows are above uploads to be accessed simply (anonymously) except my secret.

That’s better! No peeping tom can see my secret!

Now, I want my secret should only be shared with some people possessing a unique URL.

This URL will be presigned by me, that is, I let AWS know that I have authorised the principal to do a particular action with the resource (GET or PUT etc).

So, how do we create a presigned URL?

First of all, configure your AWS CLI to that of your endpoint of the bucket. In my case, my bucket was created in ap-south-1 region, hence, I configured AWS accordingly.

Configuring my AWS CLI 2

Then, you can simply :

aws s3 presign s3://my-medium-test-bucket/secret.txt --expires-in 604800
https://my-medium-test-bucket.s3.ap-south-1.amazonaws.com/secret.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJKSSB6VHGHQT7DUA%2F20201108%2Fap-south-1%2Fs3%2Faws4_request&X-Amz-Date=20201108T175708Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=2a98874927469d3bfc901a676e87b8473996894cfccde802786d04c70d6be621

Note that, 604800 corresponds to 1 week in seconds and that is the maximum time, that you can use the presigned URL for in this case and it is also the maximum expiration time that you can set!

My Secret!

References: https://docs.aws.amazon.com/AmazonS3

All images used here are my own images or images available online publicly.

This blogpost is a part of the Programming Club, IIT Kanpur blog series.