Posted on

Managing your Kubernetes cluster using Helm and Terraform

In a previous post, I explained how to manage a Kubernetes cluster with FluxCD. This showed a way to implement a GitOps workflow, which means using a Git repository as the source of truth for the state of your cluster.

Flux introduces multiple new objects in your Kubernetes clusters, and requires using custom software. This makes it harder to adopt by smaller teams without dedicated platform engineers. On the other hand, most teams are already using Terraform. In this article, I will show how to make use of Terraform to manage your kubernetes clusters. This way could be considered GitOps-lite because while changes are kept track of in a Git repository, they are neither enforced nor automatically pulled from the repository.

I will use resources from Google Cloud Platform in the following examples, but everything in this article should be doable in any major cloud platform.

A nice earthly forest
Photo by Geran de Klerk

Setting up a cluster with Terraform

First, make sure you have the Terraform CLI installed or download it here. Then, create a new Git repository

mkdir cluster
cd cluster
git init
git remote add origin <your-github-repo>

We will then define a basic Google Kubernetes Engine cluster (with two nodes) and indicate Terraform to store the state in a remote Google Cloud Storage bucket, which you will need to create manually. The following HCL should go in the main.tf file:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "3.5.0"
    }
  }
  backend "gcs" {
    bucket = "your-unique-tf-state-bucket"
    prefix = "terraform/state"
  }
}

variable "project_id" {
  description = "project id"
}

variable "region" {
  description = "region"
}

provider "google" {
  project = var.project_id
  region  = var.region
}

resource "google_container_cluster" "primary" {
  name               = "${var.project_id}-gke"
  location           = var.region
  initial_node_count = 2
}

This defines the variables “region” and “project_id”, which we pass to the google provider, as those will contain the data relevant to our GCP project. The values of these variables can either be set interactively when running terraform commands or kept in the terraform.tfvars file.

You can then proceed to run terraform init to initialize your project and state. After that, run terraform plan, which will display the changes terraform is about to do, and run terraform apply to apply those changes and create your cluster.

Since GKE resources have a state that is a lot more complex than the HCL file above, I suggest using the output of terraform state show google_container_cluster.primary to refactor the HCL into a more complete overview of the state. This can in the future avoid unwanted changes to appear in the plans.

Authenticating to your cluster

You can now authenticate to your cluster using the Google Cloud CLI. First install it by following the instructions here. Then log into Google Cloud using gcloud auth login, which should open your browser for a login prompt. After that, use gcloud config set project to indicate which project you are working on. Finally, run gcloud container clusters get-credentials [YOUR_PROJECT_ID]-gke --zone=[YOUR_REGION]

You should now see your cluster appear when you run kubectl config get-contexts and two nodes should be visible by running kubectl get nodes.

A helm
Photo by Frank Eiffert

Installing Helm charts

We will now use the helm terraform provider to install podinfo and set the value replicaCount to 2. To do this, create a new HCL file, with the following content:

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
  }
}

resource "helm_release" "podinfo" {
  name       = "podinfo"

  repository = "https://stefanprodan.github.io/podinfo"
  chart      = "podinfo"

  set {
    name = "replicaCount"
    value = 2
  }
}

You’ll need to run terraform init -upgrade to install this new provider. Then you can run terraform plan and terraform apply, just like in the previous section.

This will act just like helm upgrade -install and the podinfo release should now appear when running helm list. Using kubectl get pods should show you two podinfo pods running. You can access the service by running kubeclt port-forward service/podinfo 9898:9898 and then curl localhost:9898.

Going further

Directly applying changes from your local environment doesn’t scale well when multiple people are committing changes to the infrastructure. It also causes safety issues, as changes could be applied locally without having been through a PR review. One way to solve that problem would be to use GitHub Actions to automate planning the changes when a PR is applied and applying when a PR is merged. The SaaS Atlantis can also help solve that problem, as it will act as a GitHub bot that will plan/apply in response to comments.