Automate Project Setup with DevOps Best Practices

1. Summary

Setting up automation scripts and a folder structure for a production-ready project can be a tedious and error-prone process. This project is a template project that has all of DevOps practices that can be reused across similar projects. By doing this, we can minimize overhead efforts and devote more time to developing business logic.

cookiecutter is a tool that can generate a codebase folder structure from a template project. The tool leverages the Jinja template language to perform advanced string replacements in text content, as well as in file and folder names. Developers can use this tool to convert their projects into templates for future use. Below is the automation process to generate a code skeleton for AWS + Lambda + Python styled project:

package_name [my_package]: aws_lambda_python_project
author_name [Firstname Lastname]: Firstname Lastname
author_email [firstname.lastname@email.com]: firstlast@email.com
semantic_version [0.1.1]: 0.1.1
aws_profile [my_aws_profile]: my_aws_profile
aws_account_id [111122223333]: 123456789012
aws_region [us-east-1]: us-east-1

Converting a concrete project into a template project can also be a tedious and error-prone process. To simplify this process, I created a tool cookiecutter_maker, a Python tool that is the inverse of cookiecutter and can speed up the process of creating templates.

2. Concept

  • System Python: the Python used by your OS. a lot of OS internal software depends on this, don’t touch it anytime!

  • User Python: additional Python installed by user that can be used as the base Python interpreter for virtuale environment. I usually use pyenv to install and manage multiple Python versions.

  • Virtualenv Python: the virtual environment that purposely created for your project, it is isolated from the User Python and System Python.

3. Generate Project Folder Structure

We have to install the tool cookiecutter to generate the folder structure from the template project. You should install this tool to your “User Python”.

pip3.8 install cookiecutter

Then follow the instruction to create a new project from the template. You will be prompted to enter a bunch of project config values. (These are defined in the project’s cookiecutter.json.) In this tutorial, we created a git repo called dataops_example-project, and the python library name is dataops_example.

# use specific version
cookiecutter https://github.com/MacHu-GWU/cookiecutter-aws-lambda-python-project --checkout tags/${version}

# for example (v5 is the latest as of 2023-02-18)
cookiecutter https://github.com/MacHu-GWU/cookiecutter-aws-lambda-python-project --checkout tags/v5
../_images/generate-a-new-project-from-template.png

Now you have a folder structure on your local laptop.

4. Initialize your Git Repository

Now we want to check in the code skeleton to your Git repository. You can use any Git vendor you like (GitHub, GitLab, BitBucket, AWS CodeCommit, etc.). In this tutorial, I will use the AWS CodeCommit.

  1. Go to AWS CodeCommit Console to create a repo called dataops_example-project.

  2. Follow this AWS Document to clone the repo using git-remote-codecommit (GRC). If you have GRC CLI installed and the default aws CLI is the one with the AWS Account you are working on, then you can just run the following command to clone the repo:

git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/dataops_example-project
../_images/clone-repo.png

  1. Copy the code skeleton to the repo folder and commit the code skeleton to the repo. You can use either the git CLI or any Git Client or GUI. I personally use the GitHub desktop.

../_images/github-desktop-add-local-repo.png ../_images/github-desktop-check-in-code-skeleton.png

  1. Refresh the AWS CodeCommit Console, you should see the code skeleton in the repo.

../_images/after-check-in-code-skeleton.png