Setup Spring DataSource from values stored in AWS Secret Manager

I was tasked migrating an application to AWS recently. The company wanted the application to store the database credentials in AWS’s Secret Manager. The issue that I had is that if I setup a configuration with a custom DataSource, I lose some of the auto configuration. It’s possible to define my own DataSource but that added code I didn’t need. This also became apparent with trying to get Flyway to work too. I then found a workaround that worked for the use case I wanted.

To start, we need to setup a secret inside AWS. For this post I have a username and password item defined using the same secret name. We could store everything we need in here, including the URL, the driver, connection pool info, flyway connection details etc. Here is an example screenshot of a a secret defined for the database:

AWS ECR Task Configuration

Now that the secret is defined, we need a way to send this information in. For our purposes, I’ll be using a ECR container and define this in the secret section. The AWS task definition would look something like this:

[
  {
      "name": "my_app",
      "image": "someUserID.dkr.ecr.us-east-1.amazonaws.com/env/app:1.2.3",
      "memoryReservation": 512,
      "essential": true,
      "portMappings": [
          {
              "containerPort": 8080,
              "protocol": "tcp"
          }
      ],
      "secrets": [
          {
            "name": "DB_FLYWAY_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/flyway_db_connection_info_blah"
          },
          {
            "name": "DB_APP_CREDS_SECRET",
            "valueFrom": "arn:aws:secretsmanager:us-east-1:1:secret:/env/db_connection_info_blah"
          }
      ],
      "environment": [
          {
            "name": "DB_HOST",
            "value": "somehost"
          },
          {
            "name": "DB_PORT",
            "value": "5432"
          }
      ]
  }
]

The values we’re sending in the secret section is the ARN of actual secret. What gets sent in is actually a JSON string with the key/value pairs of the secret. Just make sure your task is given the secretsmanager:GetSecretValue permission for the ARN of the key(s) you’re using. Make sure this is defined in the secret section so they aren’t visible in the AWS UI. Now onto the Java code.

Java Code

First of all I need to define my own properties. I want this app to work in AWS but also for local test and therefore I don’t want to rewrite the world. If the secret is set, great we’ll use that. If not, I just want it to work as-is. To get around this, I extend the properties that Spring come with and add a special awsDbSecret property which will be the JSON object that AWS sends in:

@Getter
@Setter
@ConfigurationProperties(prefix = "spring.datasource.my-app-custom")
public class CustomDataSourceProperties extends DataSourceProperties {
    /**
     * The AWS secret is JSON format containing the database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}

@Getter
@Setter
@ConfigurationProperties(prefix = "spring.flyway.my-app-custom")
public class CustomFlywayProperties extends FlywayProperties {
    /**
     * The AWS secret is JSON format containing the flyway database connection (username/password)
     * information.
     */
    private String awsDbSecret;
}

Now that we have custom properties defines. Lets set up the application.yml file:

spring:
  datasource:
    driver-class-name: org.postgresql.Driver
    // Here I'm injecting DB_HOST and DB_PORT from AWS. This can be hardcoded or can be stored in the AWS secret as well.
    url: jdbc:postgresql://${DB_HOST}:${DB_PORT}/db-schema
    my-app-custom:
      aws-db-secret: ${DB_APP_CREDS_SECRET}
  flyway:
    my-app-custom:
      aws-db-secret: ${DB_FLYWAY_CREDS_SECRET}

Now we have properties we want. Let’s setup a custom @Configuration which does the magic for us.

import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.autoconfigure.flyway.FlywayProperties;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.configurationprocessor.json.JSONException;
import org.springframework.boot.configurationprocessor.json.JSONObject;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.PropertiesPropertySource;
import org.springframework.util.StringUtils;

import java.util.Properties;


@RequiredArgsConstructor
@Slf4j
@Configuration
@EnableConfigurationProperties({CustomDataSourceProperties.class, CustomFlywayProperties.class})
@Order(Ordered.HIGHEST_PRECEDENCE)
public class DatabaseConfig2 {

    private final ConfigurableEnvironment environment;

    @Primary
    @Bean
    public DataSourceProperties customDataSourceProperties(CustomDataSourceProperties customDataSourceProperties, DataSourceProperties dataSourceProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customDataSourceProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customDataSourceProperties.getAwsDbSecret());
                dataSourceProperties.setUsername(dbCredentials.getString("username"));
                dataSourceProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.datasource.username", dataSourceProperties.getUsername());
                properties.setProperty("spring.datasource.password", dataSourceProperties.getPassword());
                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-datasource-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Weather datasource AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS credentials secret was configured. Falling back to properties set for username/password.");
        }

        return dataSourceProperties;
    }

    @Primary
    @Bean
    public FlywayProperties customFlywayProperties(CustomFlywayProperties customFlywayProperties, FlywayProperties flywayProperties) {
        boolean usedAwsSecret = false;
        try {
            if (StringUtils.hasText(customFlywayProperties.getAwsDbSecret())) {
                JSONObject dbCredentials = new JSONObject(customFlywayProperties.getAwsDbSecret());
                flywayProperties.setUser(dbCredentials.getString("username"));
                flywayProperties.setPassword(dbCredentials.getString("password"));

                // Needed since Hikari reads properties directly
                Properties properties = new Properties();
                properties.setProperty("spring.flyway.user", flywayProperties.getUser());
                properties.setProperty("spring.flyway.password", flywayProperties.getPassword());

                environment.getPropertySources().addFirst(new PropertiesPropertySource("aws-custom-flyway-properties", properties));

                usedAwsSecret = true;
            }
        } catch (JSONException e) {
            log.error("Weather flyway AWS secret property was set but an error occurred parsing the value");
        }

        if (!usedAwsSecret) {
            log.info("No AWS flyway credentials secret was configured. Falling back to properties set for username/password.");
        }

        return flywayProperties;
    }

}

Final Notes

And that’s it! This works because we are defining our own custom DataSourceProperties file and making it the primary bean. Now Spring will use the properties that have the username and password set. By extending the class, we get all of Spring’s current and future configuration items plus our extra decoder.

You may ask: Why is the @Primary annotation needed if our class extends DataSourceProperties? This is because Springs expect one and only one DataSourceProperties to be defined. It will fail to start if it detects two or more beans (ConfigurationProperties) defined. This gets around the framework by adding a little extra processing to Spring’s properties before it’s used anywhere else.

• Auto configuration can’t occur until it injects the DataSource
• DataSource can’t be created until it has the DataSourceProperties
• DataSourceProperties can’t be injected until the bean marked as @Primary is defined first

The @ConfigurationProperties prefix can be whatever you like. It does not need to start with spring.datasource if you want to isolate the property namespace to be something more custom.

One final note. You can also use Spring’s spring-cloud-starter-aws-secrets-manager-config as well. In my use case I could not since I needed access to multiple secrets due to the company terraform modules I needed to use. spring-cloud-starter-aws-secrets-manager-config only allows access to one secret and thus wasn’t suitable for my case.