CI/CD: Automating our build and deploy process

September 27, 2021 · 12 minute read

Johnson K

Engineering

TLDR

Having a continuous integration / continuous delivery (CI/CD) pipeline set up for each of our git repositories is an essential part of Mage’s software development life cycle. It allows us to greatly speed up the process of testing, building, and deploying our application without sacrificing quality assurance.

Outline

  • Introduction to CI/CD

  • Importance of CI/CD in software development

  • CircleCI and its alternatives

  • How Mage uses CircleCI for our CI/CD pipeline

  • CircleCI configuration

Introduction to CI/CD

The “CI” stands for “continuous integration”, and it’s the practice of regularly building and testing code after each code change is merged to the main branch. Different tests can be run depending if code is committed to a secondary branch or the main branch.

The “CD” stands for either “continuous delivery” or “continuous deployment”. Continuous delivery and continuous deployment both build on top of continuous integration and enable automatic deployments of code changes to testing/staging and production environments

Delivering your app to the interwebs. (Source: Giphy/TikTok)

However, continuous delivery requires a manual step, such as an approval, before deploying to production, whereas continuous deployment automates the production deployment to completion without an approval step, assuming all tests and builds have successfully completed.

Importance of CI/CD in software development

Using a CI/CD pipeline automatically tests new code commits at frequent intervals so bugs or breaking changes can be caught quickly and rectified before being deployed to production. The release process is much quicker, easier, and safer once the CI/CD pipeline is already set up, so it makes your team’s software development life cycle much more efficient.

CircleCI and its alternatives

CircleCI flow (Source: https://circleci.com/docs/2.0/about-circleci/)

CircleCI

is currently one of the most popular CI/CD tools on the market, and it supports several widely used programming languages, including Python, JavaScript, Java, Go, and Ruby on Rails. It also supports integration with both

GitHub

and

Bitbucket

.

Though CircleCI is the tool that Mage uses and that we will go into greater detail in the rest of this article, there are several other suitable CI/CD tools available, such as

Jenkins

,

TeamCity

, or

Travis CI

.

As stated in our

Startup tech stack

article, we previously used Travis CI but migrated our CI/CD pipeline to CircleCI because it allowed better control over configuring our pipeline (e.g. setting up manual approvals for certain jobs before the next job can run), was less restrictive in terms of number of concurrent jobs and users, and was a much better value for us (ended up costing five times less than Travis CI).

How Mage uses CircleCI for our CI/CD pipeline

Our CircleCI backend workflow

We have different CircleCI

workflows

for each of our services, but we’ll be discussing the different parts of our backend workflow below. A workflow defines a collection of

jobs

and the order in which they are run, and a job is a collection of steps. All steps in a job are executed in a single

Docker container

or

virtual machine

(VM).

Github integration

We host all of our code repositories in GitHub, so we

integrate CircleCI

with all of our services except the front-end service, which we use

Vercel

for. We discuss how we use Vercel in the “Infrastructure” section of our

Tech stack

article.

When working on secondary branches, code commits pushed to the remote branch on GitHub will only build a development environment in a Docker container and run tests, but not make any deployments.

Testing/Debugging

In addition to the automated testing that CircleCI provides on every code commit pushed to the remote branch, CircleCI also allows us to SSH into test instances to check our code or test environment for debugging purposes.

Eliminating bugs in your pipeline. (Source: Giphy)

This SSH feature can be found by clicking on the “Rerun” button in the top right corner of the CircleCI job page and selecting “Rerun Job with SSH”:

Database migrations

On the main branch after all tests have successfully passed, CircleCI runs our database migrations for our

MySQL

database hosted on

Amazon Relational Database Service

(

RDS

). This synchronizes the production database state with any updated models from our

Django

backend.

Staging environment

After successfully running the database migrations, CircleCI deploys our

backend

application to a staging environment in Amazon

Elastic Beanstalk

and also runs some

smoke tests

automatically. With the staging environment deployed, we can run additional manual tests and confirm that any new or updated APIs are working properly.

Notifications

CircleCI can send

emails

to members of a CircleCI project’s team whenever a

workflow

succeeds or fails to complete. There is also a

Slack integration

that can send Slack messages at any step during the workflow. We have a message sent to a specific Mage Slack channel whenever any manual approvals are required to remind us to go back into the CircleCI UI.

Manual approvals

We require a

manual approval

in CircleCI for initiating any deployments to production. This allows us to do any additional custom testing in the staging environment first, and it also gives us control over the timing of our deployments.

Manual approvals actually require a team member to go into the CircleCI UI and click an “Approve” button, like in the screenshot below:

Building Docker images

As part of our CircleCI

workflow

, we can also create custom Docker images using our own

Dockerfiles

.

We use

AWS Lambda

to execute and quickly complete complex data processing tasks asynchronously. In order to configure our

AWS Lambda

functions, we need to build custom Docker images and push them to Amazon Elastic Container Registry (

ECR

). This is done automatically in CircleCI after approving the manual approval hold.

Production deployments

In addition to updating the AWS Lambda functions, simultaneous CircleCI

jobs

will be run for deploying our application and backend workers to production in

Amazon

Elastic Beanstalk.

Deployment rollbacks

There may be situations where we want to rollback a deployment. For instance, if we already deployed our application to production but want to revert back to a previous version because we discovered an uncaught bug, we may be able to do that using CircleCI.

In order to rerun an individual job in a workflow after it has already successfully completed, CircleCI doesn’t have an explicit feature that allows us to do this from the UI. However, as a workaround, we can use the “rerun the job with SSH” feature like in the “Testing/Debugging” section above. We simply go back to the workflow for that particular code commit, and rerun the individual deployment job.

Note that the rerun deployment job should be able to reference the relevant Docker image with a unique version tag (e.g. a commit

SHA hash

value) if the job is deploying an application using Docker containers, which is the case for our

API service

.

CircleCI configuration

A

config.yml

file using

YAML syntax

is required for defining the steps, jobs, and workflows in CircleCI. We’ll go over some of the different parts of a

config.yml

file, but the best place for details on the different configuration keys would be the

CircleCI documentation

.

Version

A minimum CircleCI version of 2.1 is required for several different features, so it’s not recommended to use a version lower than that.

1
version: 2.1

Orbs

Orbs

are convenient packages of configuration elements (i.e.

jobs

,

commands

,

executors

) provided by CircleCI. There are many orbs available in CircleCI’s

orbs registry

.

Some of the orbs used by Mage include the Slack, AWS Elastic Beanstalk, AWS CLI, and AWS ECR orbs. The Slack orb allows CircleCI to send messages to one of our Slack channels, while the AWS orbs allows us to easily run CLI or other commands specific to those tools.

1
2
3
4
5
orbs:  
   slack: circleci/slack@4.3.1  
   aws-elastic-beanstalk: circleci/aws-elastic-beanstalk@1.0.2  
   aws-cli: circleci/aws-cli@1.4.0  
   aws-ecr: circleci/aws-ecr@6.15.3

Jobs

Most of the sections under “How Mage uses CircleCI for our CI/CD pipeline” above represent a single job in Mage’s backend workflow. For example, there is a job for running tests, a job for deploying to a staging environment, a job for building the AWS Lambda Docker image and deploying it to AWS ECR, etc.

Each job has a number of steps that get executed in a single Docker container or VM. In our CircleCI jobs, we generally use Docker containers whenever possible because they can spin up quicker than booting up a VM, and our jobs aren’t so resource-intensive that they require being run in a VM.

If your jobs only require a minimal amount of CPU and memory, a small Docker container uses less resources than a medium VM (the smallest size available in CircleCI), resulting in less

CircleCI credits

being used per minute.

The

slack/on-hold job

provided by a CircleCI orb has the following steps:

The CircleCI

docs

have several examples of how the configuration should look like inside the

config.yml

file for jobs.

Workflows

There is a lot of customization regarding how a CircleCI

workflow

can run. Besides defining the order that jobs should run in, we can specify which branches we want certain jobs to run on, fan-out to run multiple jobs concurrently to save time, or require several jobs to successfully complete before a specific job can proceed.

We showed a diagram of Mage’s backend workflow earlier in the article at the very beginning of the “How Mage uses CircleCI for our CI/CD pipeline” section. This is just one of many ways to utilize the power of CircleCI workflows.

Closing thoughts

No matter which CI/CD tool you decide to use, the important thing is just having a CI/CD pipeline in place to begin with and then building off of it. Migrating to another platform can always happen later if absolutely necessary.

Relax while your pipeline takes care of the work for you. (Source:

Hang out with us

Join our community and chat about startups, AI/ML, and product development.

Like what you see? Join the team.

Mage is making AI and ML accessible to product developers. Join us and build beautiful and intuitive devtools.

Want to give us feedback or ask questions?

Please chat with us live by joining our Discord channel or send us an email.