Uncategorized

AWS S3 Bash Unit Testing Using localstack

Localstack is an awesome idea. It’s a locally run AWS mocking environment. It’s a cool thing to have if you’re developing an AWS based application.

However, I stumbled across localstack because I was looking for ways to unit test my infrastructure. Even if cloud infrastructure is super fast compared to the “old school” way of infrastructure, for developing, it’s still pretty slow (try turning on and off a Redshift DB a couple of times just because you forgot to configure the security group properly).

This is where localstack comes in for me.

Let’s do some test-driven infrastructure development. A simple S3 bucket, deployed via a bash script.

Sidenote: I’ll be using BATS to unit test the bash script if you don’t know how that works, check out this post. And spin up localstack in a docker container, that way you can access the dashboard at http://localhost:8080

I assume you already have localstack running. For docker run

docker run -it -p 4567-4581:4567-4581 -p 8080:8080 atlassianlabs/localstack:latest

Step 1: Write the test

First, we create the test in a separate test file test/s3-cp-test.bats. Note, all AWS commands have the option —endpoint-url=http://localhost:*  set to send them to our localstack.

load 'libs/bats-support/load'
load 'libs/bats-assert/load'

@test "s3-cp-tests.bats:s3-cp-tests.bats: Should create a S3 bucket called mytestbucket." {
# Execute our script to test
sh "/bash-test-proj/s3-cp.sh" "--endpoint-url=http://localhost:4572";

# unit test
compare_to=$(aws --endpoint-url=http://localhost:4572 s3 ls | awk '{print $3}')
assert_equal $compare_to "mytestbucket"

# tear down
aws --endpoint-url=http://localhost:4572s3 rb s3://mytestbucket --force
}

The load commands load bats as submodules as detailed in the medium post linked above. Then we execute our script “s3-cp.sh” which is supposed to create an S3 bucket with the name mytestbucket.

Note that we pass the endpoint as an argument. The script is supposed to use some regular AWS profile if no argument is passed, and only go to localstack for unit tests.

To assert if the bucket has been created, we simply need to use the aws s3 ls command and parse the output for the name. We use the assert-equal function from bats-assert to compare the result of aws s3 ls to the string “mytestbucket”.

Finally, we tear down the bucket with aws s3 rb.

Step 2: Run the test

If you run this, using bats you’ll have a failing test. Depending on how you installed bats you can either run

bats s3-cp.sh

– or ./test/libs/bats/bin/bats test/s3-cp-test.bats

Now let’s write the script to get this test green. And yes, I just noticed I mixed the inputs of assert_equal up.

Step 3: Write the script

We create a script called s3-cp.sh and put the following code in:

#!/bin/bash

# we'll pass $1=--endpoint-url=http://localhost:4572 
aws $1 s3 mb s3://mytestbucket

 

We make it callable with an argument as only the unit test is supposed to go to the localstack. Otherwise, this will create a bucket with whatever default profile is loaded in aws CLI.

Step 4: Run the test again

Alright, let’s go back to our test and see whether it turns green.

Done! Within a sec. You can run it on commit (spinning up a localstack somewhere) and use it for basic developing.

Additionally

Unit testing infrastructure is an extremely underserved area it seems to me. Still, localstack is one option to have tests which are “better than nothing”. And localstack is in rapid development. Nevertheless, it’s still pretty early. The cloudformation module isn’t working correctly in my tests, and I’m missing RDS support. More notes:

  • You can always check the resources created at the dashboard http://localhost:8080/#/infra if you’ve launched localstack via Docker.
  • While localstack doesn’t check your credentials (that’s ok, it shouldn’t in my opinion), you’ll still have to create a .aws/credentials file with the corresponding profile.
  • There’s a good stackoverflow answer showing how to use terraform together with localstack. Ironically I couldn’t get it to work.

If you’re looking for more ways of testing AWS/cloud infrastructure, check out the blog post at continio, and of course the very promising recently open sourced gruntwork tool called terratest.

1 thought on “AWS S3 Bash Unit Testing Using localstack”

Leave a Reply

Your email address will not be published. Required fields are marked *