Skaffold - Building and Deploying Kubernetes Apps Simplified!

Harshal Jarikote

22^nd July 2020

11 mins

In this blog post, we will look at Skaffold & how it can be used to improve the developer workflow for Kubernetes applications.

What is Skaffold?

Skaffold is a Google container tool, which handles the workflow for building, pushing and deploying the Kubernetes applications.

How to install Skaffold:

Skaffold comes as a client-side only single binary cli tool. Its installation is as simple as downloading the binary & copying it to /usr/local/bin. Detailed Skaffold Installation steps.

We will look at how we can use Skaffold to build & deploy Botkube.

We can generate simple skaffold.yaml configuration file with skaffold init command in the project’s root directory. skaffold init command will take us through a wizard to generate a simple skaffold.yaml file, as below :

$ skaffold init
apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: botkube
build:
  artifacts:
  - image: infracloudio/botkube
    context: build
deploy:
  kubectl:
    manifests:
    - deploy-all-in-one-tls.yaml
    - deploy-all-in-one.yaml

Do you want to write this configuration to skaffold.yaml? [y/n]: y
Configuration skaffold.yaml was written
You can now run [skaffold build] to build the artifacts
or [skaffold run] to build and deploy
or [skaffold dev] to enter development mode, with auto-redeploy

But, we need to make some more changes to the generated skaffold.yaml file to make the build & deploy work with Botkube’s file structure. The changes we need to make for build the step work are:

>>>
  artifacts:
  - image: infracloudio/botkube
  context: build
>>>
<<<
  artifacts:
  - image: infracloudio/botkube
  context: .
  docker:
    dockerfile: build/Dockerfile
<<<

Image tagging policies in Skaffold:

We can set a Docker image tagging policy with below configuration in build section:

<<<
  build:
    local:
      push: false     # Set true to a push built images to remote Docker repository
    tagPolicy:
      gitCommit: {}
<<<

Skaffold supports below tag policies:

Tag by git commit
Tag by current date & time
Tag by environment variables based template
Tag by digest of the Docker image

The updated skaffold.yaml file will look like this:

apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: botkube
build:
  local:
    push: false      # Setting false will not push the image to remote repository
  tagPolicy:
    gitCommit: {}
  artifacts:
  - image: botkubetest
    context: .
    docker:
      dockerfile: build/Dockerfile
deploy:
  kubectl:
    manifests:
    - deploy-all-in-one.yaml

Skaffold workflow pipeline can be run in below modes:

skaffold run – to build & deploy once
skaffold dev – to trigger the watch loop build & deploy workflow with cleanup on exit
skaffold debug – to run a pipeline in debug mode

Now, we can run skaffold dev command to build the Docker image & deploy to the Kubernetes cluster (uses current Kubernetes context). We can also add kubeContext parameter in deploy section of skaffold.yaml to explicitly specify Kubernetes context.

NOTE : For Minikube, we need to run eval $(minikube docker-env) & set imagePullPolicy to Never, so that Minikube can get access to the locally built Docker images, else we will get ImagePullBackOff error.

The skaffold dev output looks like below & Botkube will be deployed to Minikube:

$ skaffold dev
Listing files to watch...
- botkubetest
Generating tags...
- botkubetest -> botkubetest:v0.10.0-6-gf90361d-dirty
Checking cache...
- botkubetest: Not found. Building
Found [minikube] context, using local docker daemon.
Building [botkubetest]...
Sending build context to Docker daemon 9.087MB
Step 1/17 : FROM golang:1.12-alpine3.10 AS BUILD-ENV
---> 6d6865cf688e
...
...
...
...
Step 17/17 : ENTRYPOINT /go/bin/botkube
---> Using cache
---> a610c9924625
Successfully built a610c9924625
Successfully tagged botkubetest:v0.10.0-6-gf90361d-dirty
Tags used in deployment:
- botkubetest -> botkubetest:a610c9924625c825bc7775ab959921adcf075f83c64474ef1899174d884821e2
local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
- namespace/botkube created
- configmap/botkube-configmap created
- secret/botkube-communication-secret created
- serviceaccount/botkube-sa created
- clusterrole.rbac.authorization.k8s.io/botkube-clusterrole created
- clusterrolebinding.rbac.authorization.k8s.io/botkube-clusterrolebinding created
- deployment.apps/botkube created
Watching for changes...

Supported Builders & Deployers in Skaffold:

For deploying the applications to a remote Kubernetes cluster, we need to either push locally built images to remote Docker repository & pull image in remote cluster or use in-cluster builders like kaniko.

We can to set push: true in skaffold.yaml file in order to push the built image to remote Docker repository. Skaffold uses local Docker login credentials to push to the remote Docker repository. Also, we need to create a custom image pull secret in the cluster & provide it in the Kubenetes deployment specification to be able to pull the images in the cluster.

Skaffold currently supports following Builders:

- Dockerfile
  - locally with Docker
  - in-cluster with Kaniko
  - on cloud with Google Cloud Build
- Jib Maven and Gradle
  - locally
  - on cloud with Google Cloud Build
- Bazel locally
- Cloud Native Buildpacks
  - locally with Docker
  - on cloud with Google Cloud Build
- Custom script
  - locally
  - in-cluster

source https://skaffold.dev/docs/

Skaffold currently supports following Deployers:

- Kubernetes Command-Line Interface (kubectl)
- Helm
- Kustomize

In order to use Helm as Deployer, we need to update the skaffold.yaml file as below:

apiVersion: skaffold/v2alpha3
kind: Config
metadata:
  name: botkube
build:
  local:
    push: false                         # Setting false will not push the image to remote repository
  tagPolicy:
    gitCommit: {}
  artifacts:
  - image: botkubetest
    context: .
    docker:
      dockerfile: build/Dockerfile
deploy:
  statusCheckDeadlineSeconds: 600       # Period Skaffold waits for deployment to stabilise
  kubeContext: minikube                 # Context to deploy the application to
  helm:
    flags:
      upgrade:
        [--timeout=600]                 # Helm upgrade timeout
      install:
        [--timeout=600]                 # Helm install timeout
    releases:
    - name: botkube
      chartPath: helm/botkube/
      skipBuildDependencies: true       # Skip Helm dependencies build
      namespace: botkube
      version: v0.10.0
      setValues:
        communications.webhook.enabled: true
        communications.webhook.url: <WEBHOOK_URL>
        config.settings.clustername: <CLUSTER_NAME>
        #image.repository: infracloudio/botkube
        #image.tag: v0.10.0
      imageStrategy:
        helm: {}

As per Skaffold docs, when we run skaffold dev, following steps are carried out:

1) Collects and watches your source code for changes
2) Syncs files directly to pods if user marks them as syncable
3) Builds artifacts from the source code
4) Tests the built artifacts using container-structure-tests
5) Tags the artifacts
6) Pushes the artifacts
7) Deploys the artifacts
8) Monitors the deployed artifacts
9) Cleans up deployed artifacts on exit (Ctrl+C)

source: https://skaffold.dev/docs/

skaffold_pipeline_stages

Source

Profiles in Skaffold:

Skaffold supports multiple profiles to allow deployment to different environment/clusters. Profiles can be specified in skaffold.yaml file as follows:

apiVersion: skaffold/v2alpha3
kind: Config
profiles:
  - name: minikube
    activation:
      - kubeContext: minikube
      - env: ENV=local_dev
    build:
      ...
      # Build configuration for minikube profile
      ...
    deploy:
      ...
      # Deploy configuration for minikube profile
      ...
  - name: eks-cluster
    activation:
      - kubeContext: <EKS_CLUSTER_CONTEXT_NAME>
      - env: ENV=integration
    build:
      ...
      # Build configuration for eks-cluster profile
      ...
    deploy:
      ...
      # Deploy configuration for eks-cluster profile
      ...

Below command can be used to run Skaffold with different profiles:

skaffold [run|dev] -p <profile_name> -f <skaffold_config_yaml_file>

We can use Skaffold in CI/CD pipeline for building, deploying separately or both as below:

Build:

We can just build the project with below command.

skaffold build [-p <profile>] > build_result.json

build_result.json file mainly contains image tags that are generated, among other things.

Deploy:

This is useful for deploying pre-built images with Skaffold.

skaffold deploy [-p <profile>] --build-artifacts=build_result.json

build_result.json can also consist of just the image tags like below.

{
  "builds": [
    {
      "imageName": "REPO_NAME/IMAGE_NAME",
      "tag": "REPO_NAME/IMAGE_NAME:IMAGE_TAG"
    },
    {
      "imageName": "REPO_NAME/IMAGE_NAME",
      "tag": "REPO_NAME/IMAGE_NAME:IMAGE_TAG"
    }
  ]
}

Delete Resources:

This will delete all the cluster resources created by Skaffold.

skaffold delete [options]

Logging and Port-forwarding :

Logging:

Log Tailing is enabled by default for dev and debug mode & disabled by default for run mode. It can be enabled with the --tail flag.

skaffold run --tail

Details about logging.

Port-forwarding:

Port forwarding is disabled by default, can be enabled by using --port-forward. Port forwarding is supported only in dev & debug modes.

There are two types of port forwarding:

Automatic Port Forwarding
User Defined Port Forwarding

Details about port forwarding.

Highlighting Features of Skaffold:

Skaffold comes as a client-side only single binary cli tool. As there is no on-cluster component, hence no overhead or maintenance work on cluster.

Skaffold has a pluggable architecture, so we have a choice of using suitable tools at each stage in pipeline.

skaffold-architecture

Source: https://skaffold.dev/docs/design/

Skaffold provides built-in support for the following tools, at each stage in pipeline.

skaffold_tools

Source: https://skaffold.dev/docs/design/

It is very easy to share projects (code & configuration) with just two steps git clone and skaffold run.

Skaffold automatically manages logging and port-forwarding for the deployments in dev & debug mode.

Single skaffold.yaml file can be used to deploy to multiple environments using below mechanisms:

Skaffold profile
User level configuration
Environment variables
Cli flags

Skaffold is very easy to use tool to automate the end-to-end developer workflow for Kubernetes applications. The pipelines created with skaffold are fast & efficient. It has potential to be used as the CI/CD tool as well. It supports multiple tools at each stage of the workflow, It has a lot of features compared to other such tools, making it even better choice.

If you would like to hear more about Improving Kubernetes Developer Workflow with Kind & Skaffold, take a look at this talk from one of my InfraCloud colleagues, Vishal Biyani, from PDC Tech Carnival 2020.

Looking for help with building your DevOps strategy or want to outsource DevOps to the experts? learn why so many startups & enterprises consider us as one of the best DevOps consulting & services companies.

References:

Skaffold documentation
Skaffold github repository
Checkout all the configuration options available for skaffold.yaml.
All Skaffold CLI command options.
Sample examples for all types of configurations.
For a more complex Skaffold implementation checkout Fission: Serverless Functions for Kubernetes.