Amazon S3

This article covers all the necessary steps to access files on Amazon S3, AWS's simple storage solution, from inside Workspace.


Create an Amazon S3 Bucket

You need to have an existing Amazon S3 bucket (which is the S3 equivalent of a file folder). If you don't have a bucket yet, follow the instructions in this AWS documentation article to create one.

Locate your access key credentials

To programmatically access resources on AWS you need to create an access key that consists of an Access key ID and a Secret key that has the right permissions for what you intend to do with the S3 bucket from inside Workspace.
If you don't yet have such an access key, follow the instructions in this AWS documentation article to create a new access key.

Create a new workspace

Create a new, empty workspace, or click this link to create a workspace in your own account that contains all the Python code you need to connect to Amazon S3.

Store account key credentials in Workspace

To use the account key credentials in this workspace, you need to store them in Workspace. To do so securely, you can use Environment variables.
In your new workspace, in the left sidebar, click on "Environment", and click on "+" next to "Environment variables". You need to create 2 environment variables:
  • AWS_ACCESS_KEY_ID: Set this to the access key ID you got in the previous step.
  • AWS_SECRET_ACCESS_KEY: Set this to the secret key you got in the previous step.
  • Set a meaningful "Environment Variable Set Name", e.g. "AWS Access Key".
Set up AWS environment variables in Workspace.
After filling in all fields, click "Create", "Next" and finally, "Connect". Your workspace session will restart, and AWS_ACCESS_KEY_ID and AWS_SECRET_KEY will now be available as environment variables in this workspace.

Connect to Amazon S3 with Python

You can now switch to Python to access the files. We'll use boto3 for this, which is the official Python package to create, configure, and manage AWS services, among which Amazon S3. It's already installed by default, so we only have to import it.
import boto3
To verify that everything was set up okay, let's list all the objects (files) in a specific S3 bucket. Make sure to update the AWS_BUCKET_NAME to a bucket that is available in your AWS account.
AWS_BUCKET_NAME = "datacamp-workspacedemo-workspacedemos3-prod" # change this
# Create the s3 resource
s3 = boto3.resource('s3')
# Load the bucket with specified name.
bucket = s3.Bucket(AWS_BUCKET_NAME)
# List all objects (files or folders) in the bucket
[ obj.key for obj in bucket.objects.all() ]
This should've worked! Note that you don't need to explicitly fetch and provide the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables; boto3 expects these environment variables to be there with those names, and loads them behind the scenes.

Going deeper

Other than listing files, boto3 allows you to download and upload files, manage buckets, etc. Consult the boto3 documentation to learn more.