FEA506 - Implement automated build and deployment pipeline

Feature ID	FEA506
Subsystem the feature is part of	Backend/Infrastructure
Responsible person	Matias Korpela
Status	Implemented

Description

This feature integrates CI/CD pipelines for the frontend and backend repositories. It automates the process of building Docker images and pushing them to the GitLab container registry. After this, the images are automatically deployed to either the staging or production environments.

Restrictions, requirements and use cases related to this feature

All relevant issues related to or contributing to the definition of the feature are gathered here

ID	Description
Use Case 1	Development and production pipelines
FUNC-REQ-C0017	Automated build and deployment pipeline
FUNC-REQ-C0021	Account for staging and production environments in deployment pipeline

Preliminary user stories

• US021 As a developer, I want to set up a CI/CD pipeline that automatically builds, tests, and deploys the web app to different environments, such as staging and production. #79
• US029 As a platform engineer, I want to implement an automated build and deployment pipeline using tools like GitLab CI/CD to streamline the release process and ensure consistent deployments. #80

User interface mock-up

Implementation

Gitlab CI/CD Pipelines were created for frontend and backend repositories. A Gitlab runner was installed on the development server to run the pipeline jobs. The pipeline first builds a Docker image from the source and pushes it to GitLab container registry. After this, the images are automatically deployed to either the staging or production environments.

Gitlab container registry

Frontend

# .gitlab-ci.yml

# Build a Docker image with CI/CD and push to the GitLab registry.
# Deploy to staging or production environment.

# Based on docker template:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml

variables:
  DOCKER_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

# All branches but main are tagged with $DOCKER_IMAGE_NAME (defaults to commit ref slug)
# Main branch is tagged with `latest`

workflow:
  rules:
    - if: $CI_COMMIT_REF_NAME == "main"
      variables:
        DOCKER_IMAGE_NAME: $CI_REGISTRY_IMAGE:latest
    - if: $CI_COMMIT_BRANCH
      when: always

docker-build:
  # Use the official docker image.
  image: docker
  stage: build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - |
      if [[ "$CI_COMMIT_BRANCH" == "main" ]]; then
        docker build --pull -t "$DOCKER_IMAGE_NAME" .
      else
        docker build --pull --build-arg BUILD_MODE=staging -t "$DOCKER_IMAGE_NAME" .
      fi
    - docker push "$DOCKER_IMAGE_NAME"
  tags:
    - ff-2024-t7-runner
  # Run this job in a branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH
      exists:
        - Dockerfile

deploy-staging:
  image: ubuntu
  stage: deploy
  script:
    - |
      if docker ps -a --format {{.Names}} | grep -q "tukko-frontend"; then
        docker container stop tukko-frontend
        docker rm tukko-frontend
      fi
    - docker run -d -p 80:80 --name tukko-frontend $DOCKER_IMAGE_NAME
    - docker image prune --force
  tags:
    - ff-2024-t7-runner
  # Run this job in a development branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH == "dev"
      exists:
        - Dockerfile

deploy-production:
  image: ubuntu
  stage: deploy
  # Set up SSH credentials
  before_script:
    - install -m 600 -D /dev/null ~/.ssh/id_rsa
    - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
    - ssh-keyscan "$SSH_PROD_HOST"  > ~/.ssh/known_hosts
  # Connect to the server and start docker
  script:
    - ssh $SSH_PROD_USER@$SSH_PROD_HOST << ENDSSH
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - |
      if docker ps -a --format {{.Names}} | grep -q "tukko-frontend"; then
        docker container stop tukko-frontend
        docker rm tukko-frontend
      fi
    - docker run -d -p 80:80 --pull always --name tukko-frontend $DOCKER_IMAGE_NAME || err=1
    - docker image prune --force || err=1
    - exit \$err && ENDSSH
  # Delete SSH credentials
  after_script:
    - rm -rf ~/.ssh
  tags:
    - ff-2024-t7-runner
  # Run this job in main branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      exists:
        - Dockerfile

Backend

# .gitlab-ci.yml

# Build a Docker image with CI/CD and push to the GitLab registry.
# Deploy to staging or production environment.

# Based on docker template:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Docker.gitlab-ci.yml

variables:
  DOCKER_IMAGE_NAME: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG
  # mkdir /opt/gdcdeployment
  # chown gitlab-runner /opt/gdcdeployment/ | chown gitlab-runner /opt/gdcdeployment/traffic-visualizer-backend/
  DEPLOYMENT_PATH: /opt/gdcdeployment/traffic-visualizer-backend

# All branches but main are tagged with $DOCKER_IMAGE_NAME (defaults to commit ref slug)
# Main branch is tagged with `latest`

workflow:
  rules:
    - if: $CI_COMMIT_REF_NAME == "main"
      variables:
        DOCKER_IMAGE_NAME: $CI_REGISTRY_IMAGE:latest
    - if: $CI_COMMIT_BRANCH
      when: always

docker-build:
  # Use the official docker image.
  image: docker
  stage: build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$DOCKER_IMAGE_NAME" .
    - docker push "$DOCKER_IMAGE_NAME"
  tags:
    - ff-2024-t7-runner
  # Run this job in a branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH
      exists:
        - Dockerfile

deploy-staging:
  image: ubuntu
  stage: deploy
  script:
    - pwd
    - rm -rf $DEPLOYMENT_PATH
    - mkdir $DEPLOYMENT_PATH
    - cp docker-compose.yml redis-stack.conf .env $DEPLOYMENT_PATH/
    - echo "DOCKER_IMAGE_NAME=$DOCKER_IMAGE_NAME" >> $DEPLOYMENT_PATH/.env
    - ls -la $DEPLOYMENT_PATH
    - docker compose -f $DEPLOYMENT_PATH/docker-compose.yml up -d
    - docker image prune --force
  tags:
    - ff-2024-t7-runner
  # Run this job in a development branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH == "dev"
      exists:
        - Dockerfile

deploy-production:
  image: ubuntu
  stage: deploy
  # Set up SSH credentials
  before_script:
    - install -m 600 -D /dev/null ~/.ssh/id_rsa
    - echo "$SSH_PROD_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
    - ssh-keyscan "$SSH_PROD_HOST"  > ~/.ssh/known_hosts
  # Connect to the server and start docker
  script:
    - scp docker-compose.yml redis-stack.conf .env $SSH_PROD_USER@$SSH_PROD_HOST:$DEPLOYMENT_PATH
    - ssh $SSH_PROD_USER@$SSH_PROD_HOST << ENDSSH
    - find $DEPLOYMENT_PATH -type f -not \( -name "docker-compose.yml" -or -name "redis-stack.conf" -or -name ".env" \) -delete
    - echo "DOCKER_IMAGE_NAME=$DOCKER_IMAGE_NAME" >> $DEPLOYMENT_PATH/.env
    - ls -la $DEPLOYMENT_PATH
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
    - docker compose -f $DEPLOYMENT_PATH/docker-compose.yml up --pull always -d ||err=1
    - docker image prune --force || err=1
    - exit \$err && ENDSSH
  # Delete SSH credentials
  after_script:
    - rm -rf ~/.ssh
  tags:
    - ff-2024-t7-runner
  # Run this job in main branch where a Dockerfile exists
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      exists:
        - Dockerfile

Testing / possible acceptance criteria

Testcase	Test source	Responsible
Testcase 1	FUNC-REQ-C0017	Ops