How to Develop a Custom Provider in Terraform

Saravanan Gnanaguru

22^nd October 2021

15 mins

Terraform Introduction and Overview

Terraform is an Infrastructure as Code technology and it is used to create immutable infrastructure. It allows infrastructure to be expressed as code in a simple, human readable language called HCL (HashiCorp Configuration Language). It supports managing resources across all the major cloud providers. Terraform is used to create, manage, and update infrastructure resources such as physical machines, VMs, network switches, containers, Kubernetes and more. Almost any infrastructure type can be represented as a resource in Terraform.

Deliver infrastructure as code with Terraform using declarative .tf configuration files.
Plan and predict changes: Terraform provides an elegant user experience for operators to safely and predictably make changes to infrastructure. It reads configuration files and provides an execution plan of changes, which can be reviewed for safety and then applied and provisioned.
Extensible providers allow Terraform to manage a broad range of resources, including IaaS, PaaS, SaaS, and hardware services.
Create reproducible infrastructure: Terraform makes it easy to re-use configurations for similar infrastructure, helping to avoid mistakes and save time.

This post is intended for Terraform users who are having a basic understanding of Terraform and its usage and are likely willing to develop custom Terraform provider. Let’s get started!

What are Terraform Providers?

Creating and maintaining resources using Terraform rely on plugins called providers. Each provider plug-in is responsible to interact with cloud providers, SaaS providers, and other APIs. Most providers configure a specific infrastructure platform (either cloud or self-hosted). Providers can also offer local utilities for tasks like generating random numbers for unique resource names.

Each provider adds a set of resource types and/or data sources that Terraform can manage. Every resource type is implemented by a provider; without providers, Terraform can’t manage any kind of infrastructure. Terraform providers enables extensibility not only for cloud infrastructure, but it allows managing objects which can be created through exposed API calls as well.

Terraform Custom Provider

Below are some of the possible scenarios for authoring a custom Terraform provider, such as:

An internal private cloud whose functionality is either proprietary or would not benefit the open source community.
A “work in progress” provider being tested locally before contributing to the registry.
Extensions of an existing provider.

How Terraform and Provider plugin works?

As per the Terraform documentation:

Terraform core

Terraform Core is a statically-compiled binary written in the Go programming language. The compiled binary is the command line tool (CLI) terraform, and this is the entrypoint for anyone using Terraform.

The primary responsibilities of Terraform Core are:

Infrastructure as code: reading and interpolating configuration files and modules

Resource state management

Construction of the Resource Graph

Plan execution

Communication with plugins over RPC

Terraform plugins

Terraform Plugins are written in Go and are executable binaries invoked by Terraform Core over RPC.

Terraform Provider Plug-in Design Source: Terraform documentation

Each plugin exposes an implementation for a specific service, for example: AWS, or provisioner, such as bash. All Providers and Provisioners used in Terraform configurations are called as plugins. Terraform Core provides a high-level framework that abstracts away the details of plugin discovery and RPC communication so developers do not need to manage them.

The primary responsibilities of Provider Plugins are:

Initialization of any included libraries used to make API calls.

Authentication with the Infrastructure Provider.

Define Resources that map to specific Services

The primary responsibilities of Provisioner Plugins are:

Executing commands or scripts on the designated Resource after creation, or on destruction.

Please note our post focuses on how to develop Provider Plugins

Terraform CLI file and provider plugin installation

Terraform 0.13+ uses .terraformrc CLI config file to handle the provider installation behavior. So, we need to create the config file under the path $HOME/.terraformrc and add below content:

plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"
disable_checkpoint = true

There are two methods available to do the provider installation (from Terraform 0.13+).

Explicit Installation Method A provider_installation block in the CLI configuration allows overriding Terraform’s default installation behaviors, so you can force Terraform to use a local mirror for some or all of the providers you intend to use. In the explicit installation method, we will need to have a provider_installation block.

Implicit Local Mirror Method If the CLI configuration file does not have a provider_installation block, then Terraform produces an implied configuration.

We will be using Implicit local mirror method to install our custom provider.

Default behavior of terraform init, is usually to attempt to download the provider from the Terraform registry from the internet. Since we are mimicking the custom provider scenario, we can override this behaviour by implicit method. Using the implicit method, Terraform will implicitly attempt to find the providers locally in the plugins directory ~/.terraform.d/plugins for Linux systems and %APPDATA%\terraform.d\plugins in Windows systems.

What is required to develop custom Provider?

Just a basic Go development knowledge is enough to get started.
Exposed API details from the service provider for managing resources.

How to Install and configure Terraform

Refer here for installing Terraform

Windows:

Download and extract the Terraform executable
Add Terraform executable path to ENV PATH variable

In Linux flavours, extract and copy the Terraform executable in /usr/bin path to execute it from any directory.

Install Go and setup development environment

Follow the Go installation steps mentioned in official Go website and getting started with Go.

Custom Provider source code details

Go to $HOME/go/src path and create code.

cd $HOME/go/src
mkdir tf_custom_provider

Required source files for custom provider are:

main.go
provider.go
resource_server.go

The code layout looks like this:

.
├── main.go
├── provider.go
├── resource_server.go

Functionality of the Provider

We will be creating a provider with the below functionality. Since this is going to be an example, we will be mocking the Terraform Resource create and delete functionalities. We will also be utilizing the random UUID generator API and it will be added as part of the Create functionality, to show the ability to invoke the API call. The API can be modified later with actual resource creation API for cloud provider, on prem service provider or any As a Service provider API.

main.go

Go entry point function is main.go.

// main.go
package main

import (
        "github.com/hashicorp/terraform-plugin-sdk/plugin"
        "github.com/hashicorp/terraform-plugin-sdk/terraform"
)

func main() {
        plugin.Serve(&plugin.ServeOpts{
                ProviderFunc: func() terraform.ResourceProvider {
                        return Provider()
                },
        })
}

provider.go

provider.go will have the resource server function calls.

// provider.go
package main

import (
        "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
)

func Provider() *schema.Provider {
        return &schema.Provider{
                ResourcesMap: map[string]*schema.Resource{
                     "example_server": resourceServer(),
                },
        }
}

resource_server.go

All the resource creation has to be coded in resource_server.go. This file will have the resource function declaration and definition like create, delete etc, it also gets the input params required to create resources.

As part of this example provider, Resource server has the following functionalities:

Create
Delete

// resource_server.go
package main

import (
        "net/http"
        "log"
        "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
)

func resourceServer() *schema.Resource {
        return &schema.Resource{
                Create: resourceServerCreate,
                Read:   resourceServerRead,
                Update: resourceServerUpdate,
                Delete: resourceServerDelete,

                Schema: map[string]*schema.Schema{
                        "uuid_count": &schema.Schema{
                                Type:     schema.TypeString,
                                Required: true,
                        },
                },
        }
}

func resourceServerCreate(d *schema.ResourceData, m interface{}) error {
        uuid_count := d.Get("uuid_count").(string)

        d.SetId(uuid_count)

        // https://www.uuidtools.com/api/generate/v1/count/uuid_count
        resp, err := http.Get("https://www.uuidtools.com/api/generate/v1/count/" + uuid_count)
        if err != nil {
                log.Fatal(err)
        }
        defer resp.Body.Close()

        return resourceServerRead(d, m)
}

func resourceServerRead(d *schema.ResourceData, m interface{}) error {
        return nil
}

func resourceServerUpdate(d *schema.ResourceData, m interface{}) error {
        return resourceServerRead(d, m)
}

func resourceServerDelete(d *schema.ResourceData, m interface{}) error {
        d.SetId("")
        return nil
}

Our example code implements mock resource creation for the provider called ‘exampleprovider’. In an actual implementation, it has to be changed for the provider name of the respective cloud or on-premises server. Most providers have API calls to be consumed for resource operation like create/update/delete etc.. So, we need to define the logic of resource operations like create and delete using the custom provider API calls, to apply the Terraform template.

After adding the logic for resource operations in resource_server.go, our custom provider is ready to get tested.

Build the Custom Provider code

go mod init
go fmt
go mod tidy
go build -o terraform-provider-example

Steps to copy provider executable to plugins directory

In order to copy and use the custom provider we have created, we need to create the below directory structure inside the plugins directory:

Linux based system - ~/.terraform.d/plugins/${host_name}/${namespace}/${type}/${version}/${target}
Windows based system %APPDATA%\terraform.d\plugins\${host_name}/${namespace}/${type}/${version}/${target}

Where:

host_name-> somehostname.com
namespace-> Custom provider name space
type-> Custom provider type
version-> semantic versioning of the provider (ex: 1.0.0)
target-> target operating system

Our custom provider should placed in the directory as below:

~/.terraform.d/plugins/terraform-example.com/exampleprovider/example/1.0.0/linux_amd64/terraform-provider-example

So, as a first step, we need to the create the directory as part of our provider installation:

mkdir -p ~/.terraform.d/plugins/terraform-example.com/exampleprovider/example/1.0.0/linux_amd64

Then, copy the terraform-provider-example binary into that location:

cp terraform-provider-example ~/.terraform.d/plugins/terraform-example.com/exampleprovider/example/1.0.0/linux_amd64

Create Terraform `.tf` files

Let’s test the provider by creating main.tf, by providing the resource inputs. We have added the number of server count (uuid_count) as an input parameter for the demo purpose.

main.tf

Create main.tf file with code to create custom provider resource:

resource "example_server" "my-server-name" {
	uuid_count = "1"
}

version.tf

Create a file called versions.tf and add the path to custom provider name and version:

terraform {
  required_providers {
    example = {
      version = "~> 1.0.0"
      source  = "terraform-example.com/exampleprovider/example"
    }
  }
}

Test the Provider and output values

Execute the following Terraform commands to verify the custom provider functionalities we have added.

Terraform initialization

When we run terraform init command, the Terraform core fetches the provider plugin from the local path, since we have configured the provider in the versions.tf file. During the Terraform initialization, the custom provider has been cached into ~/.terraform.d/plugin-cache directory, to re-use the provider during next run.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding terraform-example.com/exampleprovider/example versions matching "~> 1.0.0"...
- Using terraform-example.com/exampleprovider/example v1.0.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform plan

The terraform plan command, uses the server definition defined in the main.tf file.

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # example_server.my-server-name will be created
  + resource "example_server" "my-server-name" {
      + id         = (known after apply)
      + uuid_count = "1"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Terraform apply

The terraform apply command invokes the resourceServerCreate function, we have defined in resource_server.go file.

$ terraform apply -auto-approve=true

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # example_server.my-server-name will be created
  + resource "example_server" "my-server-name" {
      + id         = (known after apply)
      + uuid_count = "1"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
example_server.my-server-name: Creating...
example_server.my-server-name: Creation complete after 0s [id=1]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Cleaning up

Terraform destroy

The terraform destroy command invokes the resourceServerDelete function, we have defined in resource_server.go file.

$ terraform destroy -auto-approve=true
example_server.my-server-name: Refreshing state... [id=1]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # example_server.my-server-name will be destroyed
  - resource "example_server" "my-server-name" {
      - id         = "1" -> null
      - uuid_count = "1" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.
example_server.my-server-name: Destroying... [id=1]
example_server.my-server-name: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.

Conclusion

In this technical blog post, we covered the topics below:

How Terraform provider works.
What is Terraform custom provider.
Steps to create and build an example Terraform provider.
Steps to use the custom provider.
What happens while invoking the Terraform CLI commands.

Readers of this article can use the sample code given above and modify the API call with their own API for managing their resources. Also here is GitHub repo link to the source code. Hope you enjoyed the article, if you have any queries or feedback, let’s connect and start a conversation on LinkedIn.

Looking for help with building your DevOps strategy or want to outsource DevOps to the experts? learn why so many startups & enterprises consider us as one of the best DevOps consulting & services companies.

Bibliography and Reference