Terraform security best practices (2022)

This article provides a breakdown of the most important Terraform security best practices to consider when implementing an Infrastructure as Code (IaC) environment. Terraform is a highly popular IaC tool offering multi-cloud support. IaC means that infrastructure is deployed automatically and configured at scale, which has immediate benefits for efficiency and consistency. In addition, the code can be stored in solutions like GitHub, which provide further benefits such as version control, documentation, and pipeline functionality. However, like many new technologies there are common security pitfalls to avoid during adoption.

The following sections discuss our most important Terraform security best practices:

The importance of Terraform State

Terraform must keep track of the resources created. When Terraform environments are created and deployed, Terraform tracks the resources that it created, and stores this mapping in a Terraform State File. When deploying changes to the configuration, Terraform consults it’s State File to see what resources must be created or modified.

Problems associated with State Files

By default, Terraform creates a single local file called ‘terraform.tfstate’. This may be sufficient for independent development, but development often involves large teams which gives rise to the following problems:

• State Files must be shared. Every member of the team must have access to the State File with the latest configuration.
• Locking State Files. State Files must be locked to prevent concurrent changes and prevent race conditions that would corrupt the file.
• State Files are not encrypted. Anybody with access to the State File and a text editor can read your configuration and any sensitive values.
• Maintaining State isolation between different environments. Using the same configuration between environments (like test, staging and production) is preferred. Maintaining total isolation between these environments is also required.

How not to share Terraform State Files (in Git)

Historically the Terraform documentation recommended that .tfstate files are committed to version control (e.g. Git) in order to share them. This is no longer considered Terraform security best practice, thanks to the introduction of backends. State Files (and their backups) should not be committed to Git and should instead be added to gitignore to prevent accidental commits. Many organisations have accidentally exposed State Files that leaked sensitive secrets (API keys, AWS access keys etc).

Terraform Remote Backends

To tackle the issues with sharing State Files, Terraform introduced the concept of Remote Backends. By default Terraform uses a Local Backend, but by choosing to use a dedicated Remote Backend, this configures Terraform to store it’s state in a remote location. These Backends, once correctly configured, can solve the problems identified with sharing State Files and there are a number of options to choose from depending on your use case. Some commonly used options include Amazon S3, AzureRm, Google Cloud Storage, and Hashicorp’s own Terraform Cloud.

Preventing misconfigurations in Terraform Remote Backends

The configuration of Remote Backends is of particular importance. For example, storing your State File in an Amazon S3 Bucket will not automatically encrypt it unless you configure your bucket to be encrypted. Similarly, Amazon S3 does not support file locks and is typically configured in conjunction with DynamoDB to lock the State File. If using S3 as a Remote Backend, remember to ensure that you enable S3 Bucket Versioning to maintain a versioned history of all changes. This will allow you to rollback changes and guard against corruption of the State File.

Ensure that you have detailed knowledge of the Remote Backend, what features it supports and how it needs to be configured before committing to a solution. For assistance with this, our Cloud Configuration Review service provides an in-depth assessment of services used to deploy Terraform Backends.

Using Terraform Cloud to store and manage State

In 2022, the best approach to remote State Files is likely to be a dedicated premium service like Terraform Cloud (previously Terraform Enterprise). This service offers encrypted remote state management, shared variables, remote operations, and has a free tier with a limited number of users (5). Paid users benefit from additional features such enforcing policies to add guardrails to deployments.

Creating separation between environments with Terraform Workspaces

The code used to manage IaC environments is highly sensitive, therefore care should be taken to separate code environments such as development, testing, and production. Sensitive variables within each environment should also be unique, to avoid shared values (such as database passwords) being used both in development and production environments.

Terraform Workspaces can help alleviate some of the common problems with environment separation. Every Terraform configuration has a Backend associated with it, where persistent data and the important State File is stored. Persistent data is stored in the Terraform Workspace, and initially there is only one “default” Workspace.

Some Terraform Backends support multiple Workspaces, allowing multiple States to be associated with a single configuration. Where multiple Workspaces are used there is still only one Backend, but you could have multiple instances deployed without needing to configure a new Backend or change authentication credentials e.g., AWS keys and database passwords.

A common use case for multiple Workspaces is to create a parallel distinct copy of infrastructure to test changes before deploying to production. They are often associated with feature branches, which are torn down once changes are merged. The common problem with copying infrastructure in this manner is that they tend to be used as the development, testing and production environments. This means they share a common Backend, which is a security concern as there is a lack of isolation between environments.

Many organisations will have a requirement to separate multiple deployments of the same infrastructure, either for supporting development stages (development and production) or isolating customer deployments of an application or environment. In these scenarios, Workspaces are not the solution as each deployment should have its own Backend. The security best practice here is to produce modular code that can be implemented with an independent Backend configuration and variables. Terraform provides a registry of modules, both community sourced and modules created by verified partners such as AWS, Azure and Google. We recommend using verified modules from trusted partners unless you are comfortable with reviewing the security of code imported from untrusted sources.

How to manage secrets in Terraform

When building infrastructure using IaC, it is common to need to include secrets. For example, the following code creates a new AWS RDS database instance and includes username and password variables:

 

In this example the username and password values have been hardcoded into the configuration. Hardcoded credentials should be avoided, so developers often replace sensitive values with a variable (usually stored in a .tfvars file):

 

This is an improvement provided the accompanying variable files are not committed to source control alongside the main configuration. However, the values of these variables may still be exposed if they are included in any output by Terraform. As of Terraform 0.14, Terraform supports the creation of variables marked as “sensitive.” This can be done by declaring the variable as sensitive in the Terraform configuration:

 

This prevents the password from being displayed in output (such as log files). Nevertheless, the sensitive values will still be present in the .tfstate file. This file keeps track of resources created by the configuration and maps them to real-world resources. As a result, .tfstate files should not be committed to version control.

Using this approach, sensitive values will always be stored somewhere in the source code environment, even if using a separate .tfvars file or environment variables. How do these secrets get shared with other users needing access and where should we store them if not in version control?

Secure storage of secrets in Terraform

It is possible to use tools to encrypt secrets. git-crypt allows for the encryption of files committed to a Git repository which are then decrypted when they are checked out. Unfortunately, it does not scale well for large teams and requires users to create individual GPG public keys, or worse, a shared symmetric key. Similarly, the terrahelp tool allows for encrypting the .tfstate files as well as values displayed in output. However, neither of these options are a replacement for a dedicated Secrets Manager.

Secrets Managers – The best way to store Secrets

All major cloud platforms provide functionality for secrets storage, which is considered Terraform security best practice. AWS has several to choose from; KMS for storing encryption keys, the AWS Secrets Manager, and the AWS Parameter Store. Microsoft Azure Key Vault and Google Cloud Secret Manager offers very similar functionality. All of these products can be used to ensure that secrets are stored externally to IaC repositories.

Alternatively, there are many 3rd party solutions, such as Hashicorp Vault, which is an identity-based secrets and encryption management system. Vault (and similar solutions) store key/value data encrypted at rest but can also dynamically create secrets for use with Cloud services.

Adding Guardrails

Compliance as Code is the term used for codification of your compliance controls. Whilst compliance is a complex matter, there are tools and techniques that can be incorporated at an early stage to enhance your Terraform security best practices.

Even if you have added your .tfvars, .tfstate and .tfstate.backup files to .gitignore to prevent them being committed to version control, mistakes can happen. It is recommended to add some secrets scanning tools (such as Gitleaks or GitSecrets) to your pipeline, which can prevent sensitive values from being committed using pre-commit hooks. Commercial products such as Trufflehog and Gitguardian offer additional features like real-time notifications and integrations into other platforms.

Tools such as Terrascan offer static code analysis to scan for common misconfigurations against Terraform security best practices. An added benefit is that it can be easily integrated into existing CI/CD pipelines. Commercial tools like Snyk and Checkov offer similar functionalities with added features like IDE plugins to help scan code in real-time.