# How to Develop a Custom Provider in Terraform

## **Terraform Introduction and Overview**

Terraform is an Infrastructure as Code technology and it is used to create immutable infrastructure. It allows infrastructure to be expressed as code in a simple, human readable language called HCL (HashiCorp Configuration Language). It supports managing resources across all the major cloud providers. Terraform is used to create, manage, and update infrastructure resources such as physical machines, VMs, network switches, containers, Kubernetes and more. Almost any infrastructure type can be represented as a resource in Terraform.

* Deliver infrastructure as code with Terraform using declarative `.tf` configuration files.
    
* Plan and predict changes: [Terraform provides](https://www.infracloud.io/terraform-consulting/) an elegant user experience for operators to safely and predictably make changes to infrastructure. It reads configuration files and provides an execution plan of changes, which can be reviewed for safety and then applied and provisioned.
    
* Extensible `providers` allow Terraform to manage a broad range of resources, including IaaS, PaaS, SaaS, and hardware services.
    
* Create reproducible infrastructure: Terraform makes it easy to re-use configurations for similar infrastructure, helping to avoid mistakes and save time.
    

This post is intended for Terraform users who are having a basic understanding of Terraform and its usage and are likely willing to develop custom Terraform provider. Let’s get started!

## **What are Terraform Providers?**

Creating and maintaining resources using Terraform rely on plugins called providers. Each provider plug-in is responsible to interact with cloud providers, SaaS providers, and other APIs. Most providers configure a specific infrastructure platform (either cloud or self-hosted). Providers can also offer local utilities for tasks like generating random numbers for unique resource names.

Each provider adds a set of resource types and/or data sources that Terraform can manage. Every resource type is implemented by a provider; without providers, Terraform can’t manage any kind of infrastructure. [Terraform providers](https://www.infracloud.io/terraform-consulting/) enables extensibility not only for cloud infrastructure, but it allows managing objects which can be created through exposed API calls as well.

## **Terraform Custom Provider**

Below are some of the possible scenarios for authoring a custom Terraform provider, such as:

* An internal private cloud whose functionality is either proprietary or would not benefit the open source community.
    
* A “work in progress” provider being tested locally before contributing to the registry.
    
* Extensions of an existing provider.
    

## **How Terraform and Provider plugin works?**

As per the [Terraform documentation](https://www.terraform.io/plugin/how-terraform-works):

### **Terraform core**

> Terraform Core is a statically-compiled binary written in the Go programming language. The compiled binary is the command line tool (CLI) `terraform`, and this is the entrypoint for anyone using Terraform.
> 
> The primary responsibilities of Terraform Core are:
> 
> * Infrastructure as code: reading and interpolating configuration files and modules
>     
> * Resource state management
>     
> * Construction of the Resource Graph
>     
> * Plan execution
>     
> * Communication with plugins over RPC
>     

### **Terraform plugins**

> Terraform Plugins are written in Go and are executable binaries invoked by Terraform Core over RPC.

![Terraform Provider Plug-in Design](https://d33wubrfki0l68.cloudfront.net/d42962f7b2484e6a85f74f66589a9b09a909ff4e/f826d/assets/img/blog/developing-custom-provider-terraform/terraform-provider.png align="left")

*Source:* [*Terraform documentation*](https://www.terraform.io/plugin)

> Each plugin exposes an implementation for a specific service, for example: AWS, or provisioner, such as bash. ***All Providers and Provisioners used in Terraform configurations are called as plugins.*** Terraform Core provides a high-level framework that abstracts away the details of plugin discovery and RPC communication so developers do not need to manage them.
> 
> The primary responsibilities of Provider Plugins are:
> 
> * Initialization of any included libraries used to make API calls.
>     
> * Authentication with the Infrastructure Provider.
>     
> * Define Resources that map to specific Services
>     
> 
> The primary responsibilities of Provisioner Plugins are:
> 
> * Executing commands or scripts on the designated Resource after creation, or on destruction.
>     

Please note our post focuses on how to develop Provider Plugins

### **Terraform CLI file and provider plugin installation**

Terraform 0.13+ uses `.terraformrc` CLI config file to handle the provider installation behavior. So, we need to create the config file under the path `$HOME/.terraformrc` and add below content:

```markdown
plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"
disable_checkpoint = true
```

There are two methods available to do the provider installation (from Terraform 0.13+).

**Explicit Installation Method**  
A `provider_installation` block in the CLI configuration allows overriding Terraform’s default installation behaviors, so you can force Terraform to use a local mirror for some or all of the providers you intend to use. In the explicit installation method, we will need to have a `provider_installation` block.

**Implicit Local Mirror Method**  
If the CLI configuration file does not have a `provider_installation` block, then Terraform produces an implied configuration.

We will be using Implicit local mirror method to install our custom provider.

Default behavior of `terraform init`, is usually to attempt to download the provider from the Terraform registry from the internet. Since we are mimicking the custom provider scenario, we can override this behaviour by implicit method. Using the implicit method, Terraform will implicitly attempt to find the providers locally in the plugins directory `~/.terraform.d/plugins` for Linux systems and `%APPDATA%\terraform.d\plugins` in Windows systems.

## **What is required to develop custom Provider?**

1. Just a basic Go development knowledge is enough to get started.
    
2. Exposed API details from the service provider for managing resources.
    

### **How to Install and configure Terraform**

Refer here [for installing Terraform](https://www.terraform.io/downloads)

Windows:

* Download and extract the Terraform executable
    
* Add Terraform executable path to ENV PATH variable
    

In Linux flavours, extract and copy the Terraform executable in /usr/bin path to execute it from any directory.

### **Install Go and setup development environment**

Follow the Go installation steps mentioned in [official Go website](https://go.dev/doc/install) and getting started with Go.

## **Custom Provider source code details**

Go to `$HOME/go/src` path and create code.

```markdown
cd $HOME/go/src
mkdir tf_custom_provider 
```

Required source files for custom provider are:

* `main.go`
    
* `provider.go`
    
* `resource_server.go`
    

The code layout looks like this:

```markdown
.
├── main.go
├── provider.go
├── resource_server.go
```

### **Functionality of the Provider**

We will be creating a provider with the below functionality. Since this is going to be an example, we will be mocking the Terraform Resource create and delete functionalities. We will also be utilizing the random UUID generator API and it will be added as part of the Create functionality, to show the ability to invoke the API call. The API can be modified later with actual resource creation API for cloud provider, on prem service provider or any `As a Service` provider API.

#### **main.go**

Go entry point function is `main.go`.

```markdown
// main.go
package main

import (
        "github.com/hashicorp/terraform-plugin-sdk/plugin"
        "github.com/hashicorp/terraform-plugin-sdk/terraform"
)

func main() {
        plugin.Serve(&plugin.ServeOpts{
                ProviderFunc: func() terraform.ResourceProvider {
                        return Provider()
                },
        })
}
```

#### **provider.go**

`provider.go` will have the resource server function calls.

```markdown
// provider.go
package main

import (
        "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
)

func Provider() *schema.Provider {
        return &schema.Provider{
                ResourcesMap: map[string]*schema.Resource{
                     "example_server": resourceServer(),
                },
        }
}
```

#### **resource\_server.go**

All the resource creation has to be coded in `resource_server.go`. This file will have the resource function declaration and definition like create, delete etc, it also gets the input params required to create resources.

As part of this example provider, Resource server has the following functionalities:

* Create
    
* Delete
    

```markdown
// resource_server.go
package main

import (
        "net/http"
        "log"
        "github.com/hashicorp/terraform-plugin-sdk/helper/schema"
)

func resourceServer() *schema.Resource {
        return &schema.Resource{
                Create: resourceServerCreate,
                Read:   resourceServerRead,
                Update: resourceServerUpdate,
                Delete: resourceServerDelete,

                Schema: map[string]*schema.Schema{
                        "uuid_count": &schema.Schema{
                                Type:     schema.TypeString,
                                Required: true,
                        },
                },
        }
}

func resourceServerCreate(d *schema.ResourceData, m interface{}) error {
        uuid_count := d.Get("uuid_count").(string)

        d.SetId(uuid_count)

        // https://www.uuidtools.com/api/generate/v1/count/uuid_count
        resp, err := http.Get("https://www.uuidtools.com/api/generate/v1/count/" + uuid_count)
        if err != nil {
                log.Fatal(err)
        }
        defer resp.Body.Close()

        return resourceServerRead(d, m)
}

func resourceServerRead(d *schema.ResourceData, m interface{}) error {
        return nil
}

func resourceServerUpdate(d *schema.ResourceData, m interface{}) error {
        return resourceServerRead(d, m)
}

func resourceServerDelete(d *schema.ResourceData, m interface{}) error {
        d.SetId("")
        return nil
}
```

Our example code implements mock resource creation for the provider called ‘exampleprovider’. In an actual implementation, it has to be changed for the provider name of the respective cloud or on-premises server. Most providers have API calls to be consumed for resource operation like create/update/delete etc.. So, we need to define the logic of resource operations like create and delete using the custom provider API calls, to apply the Terraform template.

After adding the logic for resource operations in `resource_server.go`, our custom provider is ready to get tested.

### **Build the Custom Provider code**

```markdown
go mod init
go fmt
go mod tidy
go build -o terraform-provider-example
```

## **Steps to copy provider executable to plugins directory**

In order to copy and use the custom provider we have created, we need to create the below directory structure inside the plugins directory:

* Linux based system - `~/.terraform.d/plugins/${host_name}/${namespace}/${type}/${version}/${target}`
    
* Windows based system `%APPDATA%\terraform.d\plugins\${host_name}/${namespace}/${type}/${version}/${target}`
    

Where:

* host\_name-> [somehostname.com](http://somehostname.com)
    
* namespace-> Custom provider name space
    
* type-> Custom provider type
    
* version-> semantic versioning of the provider (ex: 1.0.0)
    
* target-> target operating system
    

Our custom provider should placed in the directory as below:

```markdown
~/.terraform.d/plugins/terraform-example.com/exampleprovider/example/1.0.0/linux_amd64/terraform-provider-example
```

So, as a first step, we need to the create the directory as part of our provider installation:

```markdown
mkdir -p ~/.terraform.d/plugins/terraform-example.com/exampleprovider/example/1.0.0/linux_amd64
```

Then, copy the `terraform-provider-example` binary into that location:

```markdown
cp terraform-provider-example ~/.terraform.d/plugins/terraform-example.com/exampleprovider/example/1.0.0/linux_amd64
```

## **Create Terraform** `.tf` files

Let’s test the provider by creating [`main.tf`](http://main.tf), by providing the resource inputs. We have added the number of server count (`uuid_count`) as an input parameter for the demo purpose.

### [**main.tf**](http://main.tf)

Create [`main.tf`](http://main.tf) file with code to create custom provider resource:

```markdown
resource "example_server" "my-server-name" {
	uuid_count = "1"
}
```

### [**version.tf**](http://version.tf)

Create a file called [`versions.tf`](http://versions.tf) and add the path to custom provider name and version:

```markdown
terraform {
  required_providers {
    example = {
      version = "~> 1.0.0"
      source  = "terraform-example.com/exampleprovider/example"
    }
  }
}
```

## **Test the Provider and output values**

Execute the following Terraform commands to verify the custom provider functionalities we have added.

### **Terraform initialization**

When we run `terraform init` command, the Terraform core fetches the provider plugin from the local path, since we have configured the provider in the [`versions.tf`](http://versions.tf) file. During the Terraform initialization, the custom provider has been cached into `~/.terraform.d/plugin-cache` directory, to re-use the provider during next run.

```markdown
$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding terraform-example.com/exampleprovider/example versions matching "~> 1.0.0"...
- Using terraform-example.com/exampleprovider/example v1.0.0 from the shared cache directory

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```

### **Terraform plan**

The `terraform plan` command, uses the server definition defined in the [`main.tf`](http://main.tf) file.

```markdown
$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # example_server.my-server-name will be created
  + resource "example_server" "my-server-name" {
      + id         = (known after apply)
      + uuid_count = "1"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
```

### **Terraform apply**

The `terraform apply` command invokes the `resourceServerCreate` function, we have defined in `resource_server.go` file.

```markdown
$ terraform apply -auto-approve=true

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # example_server.my-server-name will be created
  + resource "example_server" "my-server-name" {
      + id         = (known after apply)
      + uuid_count = "1"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
example_server.my-server-name: Creating...
example_server.my-server-name: Creation complete after 0s [id=1]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
```

## **Cleaning up**

### **Terraform destroy**

The `terraform destroy` command invokes the `resourceServerDelete` function, we have defined in `resource_server.go` file.

```markdown
$ terraform destroy -auto-approve=true
example_server.my-server-name: Refreshing state... [id=1]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # example_server.my-server-name will be destroyed
  - resource "example_server" "my-server-name" {
      - id         = "1" -> null
      - uuid_count = "1" -> null
    }

Plan: 0 to add, 0 to change, 1 to destroy.
example_server.my-server-name: Destroying... [id=1]
example_server.my-server-name: Destruction complete after 0s

Destroy complete! Resources: 1 destroyed.
```

## **Conclusion**

In this technical blog post, we covered the topics below:

* How Terraform provider works.
    
* What is Terraform custom provider.
    
* Steps to create and build an example Terraform provider.
    
* Steps to use the custom provider.
    
* What happens while invoking the Terraform CLI commands.
    

Readers of this article can use the sample code given above and modify the API call with their own API for managing their resources. Also [here is GitHub repo link](https://github.com/chefgs/terraform_repo/tree/master/tf_custom_provider_new) to the source code. Hope you enjoyed the article, if you have any queries or feedback, let’s connect and start a conversation on [LinkedIn](https://www.linkedin.com/in/saravanan-gnanaguru-1941a919/).

Looking for help with building your DevOps strategy or want to outsource DevOps to the experts? learn why so many startups & enterprises consider us as one of the [best DevOps consulting & services companies](https://www.infracloud.io/devops-consulting-services/).

### **Bibliography and Reference**

* [Terraform Provider Installation](https://www.terraform.io/cli/config/config-file#provider-installation)
    
* [How Terraform Works With Plugins](https://www.terraform.io/plugin/how-terraform-works)
    
* [Conformity Provider](https://registry.terraform.io/providers/trendmicro/conformity/latest/docs)
    
* [Explicit Installation Method Configuration](https://www.terraform.io/cli/config/config-file)
    
* [Explore InfraCloud’s Terraform Consulting capabilities](https://www.infracloud.io/terraform-consulting/)