OpenTofu for Azure Databricks: IaC for Small Teams and Projects

In the previous post, I drew the blueprint to an application landing zone for Azure Databricks, with hub-spoke network architecture and a modular approach. Building on that, we focus on Azure Databricks and do a breakdown of OpenTofu as well. It is a perfect infrastructure-as-code tool for smaller teams and proof-of-concept projects.

This post covers how to approach IaC without trying to build an enterprise data platform on day one.

Why OpenTofu?

If you are not sure which provisioning setup is right for you, there are two things you need to keep in mind.

• First, what engine do you need? The engine is the command-line tool that reads your configs, and calls the provider APIs. In this case the choice might be between Terraform CLI and OpenTofu, both of which are running the same HCL and same providers so the difference is in licensing, governance and engine features—about which you can read more here.
• Second, where will the provisioning run? These workloads can run on a CI runner, which is already available inside your source control platform or they can run on a managed platform like Terraform Enterprise, HCP Terraform, Scalr etc.

These two decisions are linked; as if you would want HCP Terraform to run your plans, the choice is made, you need Terraform. If CI runners are sufficient, especially at the start of a project, licensing costs can be avoided by using OpenTofu.

OpenTofu is the community-governed open-source fork of Terraform, uses the same provider binaries, including the Databricks provider managed by Databricks. For a proof of concept, an MVP or a small team, I recommend starting infrastructure-as-code with OpenTofu and the CI runner already available inside the source control platform. 

Provisioning layers

The provisioning setup, as well as the recommended IaC project structure can best be grasped by three layers.

First of all, Azure, managed with the `azurerm` provider, covering the infrastructure topics already included in Part 1 of this post. Main elements here are networking, storage, Key Vaults and the Databricks workspace.

Second, the Databricks account layer, managed with the Databricks provider, pointed at the account API. This holds all things which workspaces share in a region, such as Unity Catalog metastore, workspace assignments, account groups or network connectivity setup for serverless compute.

And finally, the Databricks workspace layer, managed with the same provider pointed at the workspace URL. This includes everything inside one single workspace: storage credentials, external locations, catalogs and schemas, grants, SQL warehouses and policies.

The three layers and the structure of a kickstarter IaC solution is illustrated on the next figure.

The project is built from four stacks, each with its own state file: 

• `bootstrap/` creates the state storage and the CI identities running once before anything else.
• `platform/` creates the regional Unity Catalog metastore and its root storage. This is optional though: when the region already has a metastore, the landing zone takes its ID instead.
• `envs/landing-zone/` is an environment definition, which is applied once per environment.
• `envs/monitoring/` is an optional add-on, which attaches to Azure Log Analytics adding diagnostics and alerts to an existing environment.

The other parts of the figure are not stacks. Variable files (`.tfvars` inputs) or simply configurations are collected in `envs/config/`. Here, one variable file per environment contains everything that makes the given environment different from others, such as name, network range, hub and DNS zones, role groups etc.

The module library in `modules/` holds building blocks of the solution.

• Infrastructure blocks provision through Azure; they create the resource group, the network, the private DNS zones, storage accounts etc. Generally, you can go with some optional add-ons, which can be used for service principals, network security for serverless compute and monitoring.
• Account blocks provision through the Databricks account API objects, which a workspace receives from outside of itself, such as the metastore assignment, role groups, or the network connectivity configuration for serverless compute.
• Workspace blocks provision through one workspace’s API: `unity-catalog` creates the storage credential, external location and catalog binding; `databricks-compute` creates the SQL warehouse and policies, while `databricks-access` gives the role groups access to the workspace and privileges to use catalogs, schemas, locations, and computes.

Deploying with OpenTofu

Let’s walk through how the solution is deployed: how OpenTofu runs for every stack, the CI/CD setup principles, and some common provisioning scenarios.

This is where we can understand why OpenTofu is a good choice for starting your Infrastructure-as-Code approach. The engine needs no platform of its own, so the pipelines and wrappers around the command can run locally or optimally inside CI/CD. The managed platform functionality, such as plan reviews and approvals are handled through custom CI/CD pipelines.

The DevOps approach

One of the most important principles of a successful IaC solution is to automate what can be automated. The repository should be the only place where the infrastructure is defined, and the CI/CD pipelines are the only identities which can change it.

Choosing OpenTofu steers the whole setup towards the source control platform: 

• CI/CD pipelines designed properly to ensure quality of code and the protection of the deployed environment,
• CI runners prepared to execute the commands,
• a federated identity so no secret is stored, and
• properly designed pull request form and process to enable reviews.

The starting logic and complexity can be small but must follow GitFlow at all times to ensure a robust and quality-assured process. The following table illustrates a simple template solution which ticks all the most important boxes.
 

The key ideas behind quality assurance through this process are

• commits must only be done on `feature` branches, which are short-lived, and merge into `development` at all times,
• `release` branches for every release, cherry-picked if necessary, pull request into `main`,
• pull requests should have a fixed body and an automatic plan every time, result commented into the PR.

Working with smaller MVPs or proof-of-concepts, we do experience the need to simplify the above. But it is crucial to know what we can trim and what we cannot: environments can be trimmed, the process cannot.

Dropping the development or UAT environment removes some of the assurance, as changes can only be deployed to production. There will be increased risk, but this can still be managed as long as GitFlow quality gates remain intact.

Without the development environment, the merge into the `development` branch still runs, but it plans only. Without the UAT environment, the manual deployment to the UAT environment is not needed, but the release branch still serves the purpose of freezing a release and planning it against `main` and the production environment.

Deployment scenarios

Another challenge we face regularly is when IaC is not yet or just partially used. Projects rarely start in an empty subscription, some assets usually exist, and in most cases, realistically there are several assets which are clicked together by hand. Migration to IaC approach is a whole other story, so the key question in these cases: how can we apply the illustrated modular solution?

The good news is that the stack order and general deployment remain the same both while deploying to a fresh state or working with already existing assets. The design principle that makes this work is that anything the solution depends on but does not own is an input, and never hardcoded. Take the hub network as an example: the landing zone needs it to peer, but it should not create it, therefore, the hub’s resource ID is a variable. 

Deployment with a fresh state

A fresh state means that there is an empty subscription, but never an empty tenant—some things must exist before, and done by hand: the CI identity and its roles, the Databricks account console claimed once, and the tenant’s own rules.

The first run presents us with a few problems:

• The state backend cannot store its own creation, so the `bootstrap` run applies with local state and migrates to the storage account it created. Only during the first run, the state file is kept on disk until the storage is ready.
• Account-level Databricks resources need an account admin, thus the metastore and everything else on the account level can only be created by an account admin. There is a human step involved: someone must grant the account admin privileges to the pipeline’s identity.
• The workspace-level provider needs the workspace URL, which only exists after the workspace is created during the same run. We usually approach this with ordered applies, where we create the storage and then the workspaces and the rest of the infrastructure.

Most important considerations when working with a fresh stack are to have a naming prefix and convention, have address ranges figured out, create or reference a metastore, and address networking questions thoroughly.

Deployment with existing assets

Most of the time, there will be several assets already existing in the cloud platform. Here, the first and most important step is to make an inventory of the existing assets and methods in the Customer’s cloud platform team and environment.

This inventory generally turns out to be some collection of the following.

• On Azure, there is a hub network and private DNS zones owned by cloud engineering, storage accounts and Key Vaults are usually created on the portal. Sometimes there is a whole Databricks workspace already with Azure Policy assignments which limit what the IaC can create.
• On Databricks, there is the regional metastore and groups, which are synced from EntraID, catalogs and grants made on the UI, warehouses and compute policies created by hand.
• Identities are usually created by IT in the form of service principals, as project teams are generally not permitted to create app registrations.
• As for the tooling, there might be some IaC somewhere in the organization. The hub network might be managed by the cloud engineering in Terraform code, the policies by some central repository.

The last point is crucial, as an object can only have one owner, thus if something is already managed by another team, it must not be imported into the state of the Databricks IaC solution. These assets are references for us, and the practical question is how do we get their IDs—it is highly important to bridge the gap between data teams and cloud engineering at this point, and coordinate how we use the existing infrastructure. Every object goes into one of the following groups:

1. Consume: someone else owns the object and our code only references it, it takes the ID as a variable and never creates or changes it. Examples are typically the hub network, the metastore, and service principals.
2. Adopt: the object exists, but from now on our code owns it; it is imported with an `import` block, which OpenTofu plans and modifies as needed. In case of larger inventories or takeovers, there are several tools to generate the configuration and state of objects, e.g., `aztfexport` for Azure resources or the Databricks exporter for account and workspace objects, though it is still in early development.
3. Ignore: when the object is not in the scope of the project.

The design of the IaC solution should reflect the above groups: every dependency that may exist should be addressed as a variable, so the code creates it when it is empty. This way, the same configuration can work whether the object exists or not—the metastore is the cleanest example.
 

# simplified: the same shape serves every "existing or not" dependency variable "metastore_id" { default = "" }   # set it to reference an existing metastore

locals {
 metastore_id = var.metastore_id != "" ? var.metastore_id : data.terraform_remote_state.platform[0].outputs.metastore_id
}

data "terraform_remote_state" "platform" {
 count = var.metastore_id == "" ? 1 : 0
 # … coordinates of the platform stack's state …
}

Lessons learned

We already applied the illustrated approach with small projects and teams, and found that OpenTofu, as an IaC tool, always behaved as a drop-in, working reliably every time regardless of complexity, existing assets or CI platform. I can summarize the most important lessons learned in three major points.

The first deployment of infrastructure and day-to-day operations are two different scopes so you should manage your expectations.

1. The first run is allowed to contain hands-on work, do not be afraid to get your hands dirty—afterwards you will be able to automate everything and streamline the process.
2. A self-hosted runner is a must if the environment is meant to be private, otherwise something has to be opened: public access to the storage account, public front-end of the workspace or allow-lists for ever-changing runner address ranges. Make sure to include the setup of self-hosted runners in your plans if they are not available already.
3. The location of the backend state is a compliance question—some source control platforms offer a solution to keep the state next to the code, GitLab for example can store it inside the project without locking or encryption. This saves the bootstrap stack, but moves infrastructure state outside of the cloud tenant and infrastructure.