Toggle Light / Dark / Auto color theme

Toggle table of contents sidebar

Centralized Config Management

POC-style projects often have numerous hardcoded values, with some constant values being used multiple times. This pattern make projects difficult to maintain and prone to errors. In contrast, a production-ready project requires a centralized location to store all configurations. Once configurations are defined, we no longer allow hard-coded values and only reference configurations.

In this project, I implemented a light-weight config management system with the following folder structure:

/config
/config/define # config schema definition
/config/define/main.py # centralized config object
/config/define/app.py # app related configs, e.g. app name, app artifacts S3 bucket
/config/define/cloudformation.py # CloudFormation related configs
/config/define/deploy.py # deployment related configs
/config/define/lbd_deploy.py # Lambda function deployment related configs
/config/define/lbd_func.py # per Lambda function name, memory size, timeout configs
/config/define/name.py # AWS Resource name related configs
/config/init.py # config value initialization

• The define module defines the configuration data schema (field and value pairs).

  • To improve maintainability, we break down the long list of configuration fields into sub-modules.

  • There are two types of configuration values: constant values and derived values. Constant values are static values that are hardcoded in the config.json file, typically a string or an integer. Derived values are calculated dynamically based on one or more constant values.

• The init module defines how to read the configuration data from external storage.

  • On a developer’s local laptop, the data is read from a config.json file.

  • During CI build runtime and AWS Lambda function runtime, the data is read from the AWS Parameter Store.

Below is the implementation of the init module:

# -*- coding: utf-8 -*-

import os
import json
from config_patterns.jsonutils import json_loads

from ..logger import logger
from ..paths import path_config_json, path_config_secret_json
from ..runtime import IS_LOCAL, IS_CI, IS_LAMBDA
from ..boto_ses import bsm

from .define import EnvEnum, Env, Config

if IS_LOCAL:
    # ensure that the config-secret.json file exists
    # it should be at the ${HOME}/.projects/aws_lambda_python_example/config-secret.json
    # this code block is only used to onboard first time user of this
    # project template. Once you know about how to handle the config-secret.json file,
    # you can delete this code block.
    if not path_config_secret_json.exists():  # pragma: no cover
        path_config_secret_json.parent.mkdir(parents=True, exist_ok=True)
        path_config_secret_json.write_text(
            json.dumps(
                {
                    "shared": {},
                    "envs": {
                        "dev": {"password": "dev.password"},
                        "int": {"password": "int.password"},
                        "prod": {"password": "prod.password"},
                    },
                },
                indent=4,
            )
        )

    # read non-sensitive config and sensitive config from local file system
    config = Config.read(
        env_class=Env,
        env_enum_class=EnvEnum,
        path_config=path_config_json.abspath,
        path_secret_config=path_config_secret_json.abspath,
    )
elif IS_CI:
    # read non-sensitive config from local file system
    # and then figure out what is the parameter name
    config = Config(
        data=json_loads(path_config_json.read_text()),
        secret_data=dict(),
        Env=Env,
        EnvEnum=EnvEnum,
    )
    # read config from parameter store
    # we consider the value in parameter store is the ground truth for production
    config = Config.read(
        env_class=Env,
        env_enum_class=EnvEnum,
        bsm=bsm,
        parameter_name=config.parameter_name,
        parameter_with_encryption=True,
    )
elif IS_LAMBDA:
    # read the parameter name from environment variable
    parameter_name = os.environ["PARAMETER_NAME"]
    # read config from parameter store
    config = Config.read(
        env_class=Env,
        env_enum_class=EnvEnum,
        bsm=bsm,
        parameter_name=parameter_name,
        parameter_with_encryption=True,
    )
else:
    raise NotImplementedError