Power your GitLab Auto DevOps pipelines with Crossplane

Introduction

Most of the times the CI pipeline’s automated test phase requires a database to be provisioned so that the business application can be tested in its entirety. In this guide we will provision a managed Postgres database on the Google cloud platform using Crossplane and GitLab’s Auto DevOps pipeline as part of CI. A little bit about GitLab’s Auto DevOps, GitLab Managed Apps and Crossplane in the subsequent sections.

GitLab Auto DevOps

GitLab provides a feature called Auto DevOps which automatically creates CI/CD pipelines for your source code repository. It is an opinionated pipeline with reasonable defaults. The pipeline can be customized by way of a yaml configuration. Auto DevOps facilitates standardisation and consistency of the CI/CD pipelines.

GitLab Managed Apps

GitLab provides a feature of a One-Click Install of an application onto a Kubernetes cluster connected to a repository. Applications such as Prometheus, Cert-Manager or the newly added Crossplane control plane can be installed via the Managed Apps dashboard.

Crossplane

Crossplane is an open source multi cloud control plane. It allows you to provision cloud services the kubectl way by exposing them as Kubernetes resources by the use of CRDs and controllers. Crossplane connects to the cloud provider using service account credentials and orchestrates creation of services via the cloud provider APIs. This helps developers and administrators alike in provisioning and configuration of these services using Kubernetes resources hence, allowing a uniform interface for those working on the platform. It follows the model similar to a StorageClass and a PersistentVolumeClaim where an administrator could define the class of services available to be consumed and the developers request for a claim.

Reference: Crossplane Architecture

This guide will demonstrate the following.

• Create a sourcecode repository for a sample application in GitLab.
• Connect a Kubernetes cluster to that respository.
• Install Crossplane via the GitLab Managed Apps interface.
• Run the Auto DevOps pipeline for the repository which will provision a managed Postgres instance of the cloud provider (CloudSQL in this case)
• Verify the application is able to connect to the Postgres instance.

Create a repository and a GitLab Managed cluster.

Login to gitlab.com and create a new project. We will use the following repository as a sample application in this guide.

https://github.com/mahendrabagul/go-postgres-example

Once the project is created, you can use the instructions given on landing page to push your application source code to the newly created repository.

The next step is to setup a cluster for the project. In order to do this, Navigate to the “Operations/Kubernetes” in the left hand menu. GitLab allows creation of a cluster or adding an existing cluster to the repository. In this guide we will create a new GKE cluster via the GitLab interface.

Click on the “Add Kubernetes Cluster”. You will be landed on “Add a Kubernetes cluster integration”. Select “Google GKE” from “Create new Cluster tab”. You will then be asked to sign to you google account. Select your account/Log in to your account. (In my case, I already have a project in Google Cloud account. You can refer this link to create a new project on Google Cloud.

You will then be presented with a form with a bunch of fields to be filled.

Enter the details for the cluster configuration details. Select the GitLab-managed cluster option since this will basically provision a service account with the right RBAC permissions and which is used by the GitLab managed apps

Once the cluster is provisioned on GCP the GitLab UI redirects to the configuration of the connected cluster.

GitLab uses Helm (2.x) to install the Managed Applications on the connected Kubernetes cluster. Hence, the pre-requisite of installing Crossplane is to first install Helm as the managed app.

Install the Ingress controller and Crossplane with Google cloud Platform as the choice of Stack via the Managed Apps interface.

Crossplane needs to be configured with the cloud provider to allow provisioning of managed services. Follow this guide to configure Crossplane with the cloud provider.

Important Note:

You should have the following resources created on your Kubernetes cluster after configuring Crossplane.

• A Connection and a GlobalAddress kinds in order to allow connection from the workload on the GKE cluster with CloudSQL.
• Resource Classes. These are classes of service for CloudSQL that can be defined by an administrator.
• Necessary RBAC permissions if the GKE cluster is an existing cluster added to the repo.

Auto DevOps Configuration Options

The Auto DevOps pipeline can be configured in multiple ways to utilize Crossplane for provisioning Postgres. The above steps to configure Crossplane will create 2 CloudSQLInstance resources, one of them will be referenced as the class of service via the Auto DevOps configuration.

• The Environment variables, AUTO_DEVOPS_POSTGRES_MANAGED and AUTO_DEVOPS_POSTGRES_MANAGED_CLASS_SELECTOR provision PostgreSQL using Crossplane.
• If the Auto DevOps chart is bundled as part of the source code repository, the values.yaml needs the following changes.
  • postgres.managed set to true which will select a default resource class.
  • postgres.managed set to true with postgres.managedClassSelector providing the resource class to choose based on labels. For eg: postgres.managedClassSelector.matchLabels.gitlab-ad-demo="true" will select the CloudSQLInstance class which satisfies the claim based on the labels.

The Auto DevOps pipeline would get triggered upon a commit or executed manually. The pipeline provisions a postgresqlinstance which is a resource defined by the Crossplane CRDs. This essentially is a claim for creating a Postgres Instance referencing a CloudSQLInstanceClass based on labels or a default class.

Verify the creation of a postgresqlinstance

A quick look on the google cloud SQL dashboard should show the newly provisioned postgres instance.

The sample application creates a table and inserts a few records in the database. Logs from the example application

Looking for help with building your CI/CD pipelines? learn why so many startups and enterprises consider as their CI/CD consulting & services provider.

Further References

• https://crossplane.io/docs/master/
• https://docs.gitlab.com/ee/topics/autodevops/
• https://www.youtube.com/watch?v=Zv6llSEUDYk\&list=PLFGfElNsQthbiSy32TsYZyVgMk8Rv54kN\&index=5\&t=0s