Spring and DynamoDB Integration Testing

Deploying cloud native Spring applications on cloud providers allows you to leverage great managed tools such as databases, object storage systems and queues. In my current project many of our microservices, which are deployed on Amazon ECS, use DynamoDB for storage. This is great of course, but how do you integration test these services? In this post I’ll show you how you can use TestContainers to do integration tests in a Spring microservice backed by DynamoDB.

Introduction

So let’s start with a relatively standard Spring Microservice. Our 'counter' service follows the fairly typical architecture where we have a CounterController that handles the HTTP requests. The controller calls the CounterService which is autowired into the controller. The service handles the (simple) business logic and handles mapping from database entities to data transfer objects (DTO’s). The service calls the autowired CounterRepository. This repository is in charge of all the CRUD operations on our data.

Note	You might notice that there are no @Autowired annotations on the controllers. This is not needed anymore since Spring Boot 1.4 or so!

Then there is a single configuration class. It defines the AmazonDynamoDB bean used in the repository and integration tests that is used to communicate with the DynamoDB instance. The AWS region and DynamoDB endpoint are defined in the application.yml.

Note	The endpoint is currently set to 'foobar'. Keep in mind that if you use an actual DynamoDB endpoint here and start the application a 'counters' table will be created!

TestContainers

So for this neat counter service we want to start building some integration tests. There are multiple approaches to create these tests. One is to just use the real DynamoDB service and have a separate user to do tests under. The downside of this approach is that it’s easy to end up in a situation where there are two tests running in parallel that are interfering with each other. You could solve this problem by creating random prefixes for tables, but that is harder to implement.

Another option is to install DynamoDB locally. Fortunately AWS has created a 'simulation' DynamoDB distribution that can be used specifically for testing. It is not meant as a production service, it’s just made for testing. The downside is that you now have to depend on something installed and running on a local machine. It is rather complex to get this to work in your build; you need to integrate the installation in for example your build.gradle or Maven POM. And if something else is already running on that port you’re out of luck.

Fortunately there’s a third option: TestContainers. The solution awesome in it’s simplicity; you just start a docker container running whatever you want to test against. Most databases have docker images available that are ready to use. TestContainers is a simple library that you plug into your tests that can start the container for you.

Keep in mind though: running a TestContainers integration test requires Docker to be installed and running!

Integration Test

So what does the test look like? If you check out the supplied examples they went for a reusable abstract 'base' test that does the setup which the concrete test classes can then extend. I went for the same approach in both this test as well as previous implementations for other customers.

Dependencies

The only dependencies we need for testing are the TestContainers dependencies and the spring-boot-starter-test dependencies:

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.9.1</version>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <version>${spring.boot.version}</version>
    <scope>test</scope>
</dependency>

Spring-boot-starter-test contains libraries like JUnit, Mockito and AssertJ already and also manages versions for us.

AbstractIntegrationTest

So let’s look at our AbstractIntegrationTest first. There are two important parts to this base class:

private static final int DYNAMO_PORT = 8000;
@ClassRule
public static GenericContainer dynamoDb =
    new GenericContainer("amazon/dynamodb-local:1.11.119")
        .withExposedPorts(DYNAMO_PORT);

This bit of code creates a GenericContainer. A GenericContainer can be used to run pretty much any Docker container. But it does not know or understand what is actually running inside. This is where the .withExposedPorts(DYNAMO_PORT) part comes in. DynamoDB listens on port 8000 internally, but that is not what we will be exposing to the outside world. If there would be something else listening on port 8000 already it would fail to start. What .withExposedPorts(DYNAMO_PORT) does is expose the container’s internal port 8000 to a random available port.

The second important bit is overriding the configuration of the system so that it does not try to connect to the endpoint configured in the application.yaml, but connects to the container we just started instead. That’s where our ApplicationContextInitializer instance comes in:

public static class Initializer implements
    ApplicationContextInitializer<ConfigurableApplicationContext> {
    @Override
    public void initialize(ConfigurableApplicationContext configurableApplicationContext) {
        String endpoint = String.format("aws.dynamo.endpoint=http://%s:%s",
            dynamoDb.getContainerIpAddress(),
            dynamoDb.getMappedPort(DYNAMO_PORT));

        TestPropertyValues.of(endpoint).applyTo(configurableApplicationContext);
    }
}

This bit of code does two things. It formats the container address and container mapped port into a string like aws.dynamo.endpoint=http://localhost:32456. Then it builds and applies these TestPropertyValues to the current context. This happens before the AmazonDynamoDB bean we defined in the config is created.

Now any integration test that wants to use DynamoDB can simply inherit from this class and it will have a DynamoDB instance started for it! If you create a simple library out of it, it can even be reused in different microservices.

CounterIntegrationTest

Now we can easily implement our CounterIntegrationTest by extending AbstractIntegrationTest. I autowire in a TestRestTemplate (for calling the controllers), the AmazonDynamoDB client (to set up data) and the CounterRepository (so I can reset the data in the @Before method:

@Before
public void clear() {
    //Clear all items from the table before every test
    repository.findAll().forEach(c -> repository.delete(c.getCounter()));
}

It’s important to keep in mind that the DynamoDB instance is not recreated for every single test case: this would be way too slow. So it is your responsibility to reset the data to a known state. In this case this is done by just deleting every item in the counters table.

So now we can easily build our tests and assertions. I’ll show you just one example because they all work more or less the same:

@Test
public void findAll() {
    putCounter("findAll", 42);
    var response = restTemplate.getForObject("/counter", Counters.class);

    assertThat(response.getCounters()).containsExactly(new Counter("findAll", 42));
}

This tests first inserts a single counter with value 42 using the putCounter utility method. It then calls the controller via the TestRestTemplate and asserts that the returned collection contains only the exact counter we created.

Note	In case you haven’t seen these Fluent assertions before; check out AssertJ. It’s awesome.

Check out the rest of the CounterIntegrationTest for more inspiration!

Conclusion

The TestContainers library makes it very easy for us to write integration tests in situations where in-process databases like H2 can’t be used. I have applied this technique not only for DynamoDB, but also on other projects where we used Cassandra and Kafka. The abstract tests were published as a separate library (in 3 flavours; Cassandra, Kafka and Cassandra + Kafka) that were used by most of the microservices.

Also make sure you check out which specialised containers are available in TestContainers. Currently there is no specialised container for DynamoDB but there is one for for example Kafka!

If you have feedback or questions please contact me on Twitter or open an issue!