Terraform state locking using DynamoDB (aws_dynamodb_table)?

Terraform state is the backbone of your terraform project for provisioning your cloud infrastructure. When you work on large infrastructure provisioning with Terraform then it is always more than one developer or team of a developer working on the same terraform project.

The problems arise when two developers try to update the same terraform state file which is stored remotely(S3 Bucket). Ideally, the update on the state file should be done incrementally so that when one developer finishes pushing its state file changes another developer can push their changes after taking the update.

But because of the agile working environment, we can not guarantee that incremental updates on terraform state files will be performed one after another. Any developer can update and push terraform state file at any point in time, so there should be some provision to prevent a developer from writing or updating terraform file when it is already being used by another developer.

Why Terraform State Locking is important?- It prevents Terraform state file(terraform.tfstate) from accidental updates by putting a lock on file so that the current update can be finished before processing the new change. The feature of Terraform state locking is supported by AWS S3 and Dynamo DB.

In this blog, we will address the topic on How to implement Terraform state file(terraform.tfstate) locking using AWS S3 bucket and DynamoDB?

Table of Content

1. How Store Terraform state file remotely on S3?
2. Create DynamoDB table on AWS
3. Add AWS DynamoDB Table reference to Backend S3 remote state?
4. Spin one more EC2 instance with same terraform state file
5. Conclusion

1. How Store Terraform state to file remotely on S3?

Before we implement the Dynamo DB locking feature first we need to store the Terraform state file(terraform.tfstate) file remotely on AWS S3 bucket.

I am gonna take a very simple example in which we are going to provision an AWS EC2 machine and store the terraform state file remotely.

Let's start by creating main.tf and we will add the following resource blocks to it -

1. Provider Block
2. AWS Instance resource block(aws_instance) for EC2
3. Backend S3 block
4. Execute terraform script
5. Verify the remote state file

1.1 Provider Block

As we are working on the AWS environment so we will be using AWS provider. So add the following block to your main. tf -

1provider "aws" {
2   region     = "eu-central-1"
3   access_key = var.access_key
4   secret_key = var.secret_key
5}

1.2 AWS Instance resource block(aws_instance) for EC2

After adding the provider block let's add the aws_instance resource block in which we are going to set up the EC2 the machine of type t2.micro -

 1provider "aws" {
 2   region     = "eu-central-1"
 3   access_key = var.access_key
 4   secret_key = var.secret_key
 5}
 6
 7resource "aws_instance" "ec2_example" {
 8    ami = "ami-0767046d1677be5a0"
 9    instance_type = "t2.micro"
10    tags = {
11      Name = "EC2 Instance with remote state"
12    }
13}

1.3 Backend S3 block

(*Note - I have already created an S3 bucket with the name jhooq-terraform-s3-bucket, so make sure to create one for you as well.)

Now after adding the provider and aws_instance block let's add the backend S3 block to my main.tf -

 1provider "aws" {
 2   region     = "eu-central-1"
 3   access_key = var.access_key
 4   secret_key = var.secret_key
 5}
 6
 7resource "aws_dynamodb_table" "state_locking" {
 8  hash_key = "LockID"
 9  name     = "dynamodb-state-locking"
10  attribute {
11    name = "LockID"
12    type = "S"
13  }
14  billing_mode = "PAY_PER_REQUEST"
15}
16
17resource "aws_instance" "ec2_example" {
18    ami = "ami-0767046d1677be5a0"
19    instance_type = "t2.micro"
20    tags = {
21      Name = "EC2 Instance with remote state"
22    }
23}
24
25terraform {
26    backend "s3" {
27        bucket = "jhooq-terraform-s3-bucket"
28        key    = "jhooq/terraform/remote/s3/terraform.tfstate"
29        region     = "eu-central-1"
30    }
31}

2. Create DynamoDB table on AWS

Now for implementing the state locking we need to create a DynamoDB table.

1. Goto your AWS management console and search for DynamoDB onto the search bar.

2. Click on the DynamoDB

3. From the left navigation panel click on Tables

4. Click on Create Table

5. Enter the Table name - "dynamodb-state-locking" and Partition Key - "LockID"

6. Click on Create Table and you can verify the table after the creation

3. Add AWS DynamoDB Table reference to Backend S3 remote state?

After creating the DynamoDB table in the previous step, let's add the reference of DynamoDB table name (dynamodb-state-locking) to backend S3 sate.

1terraform {
2    backend "s3" {
3        bucket = "jhooq-terraform-s3-bucket"
4        key    = "jhooq/terraform/remote/s3/terraform.tfstate"
5        region     = "eu-central-1"
6   dynamodb_table  = "dynamodb-state-locking"
7    }
8}

Your final Terraform main.tf should look like this -

 1provider "aws" {
 2   region     = "eu-central-1"
 3   access_key = var.access_key
 4   secret_key = var.secret_key
 5}
 6
 7resource "aws_dynamodb_table" "state_locking" {
 8  hash_key = "LockID"
 9  name     = "dynamodb-state-locking"
10  attribute {
11    name = "LockID"
12    type = "S"
13  }
14  billing_mode = "PAY_PER_REQUEST"
15}
16
17resource "aws_instance" "ec2_example" {
18    ami = "ami-0767046d1677be5a0"
19    instance_type = "t2.micro"
20    tags = {
21      Name = "EC2 Instance with remote state"
22    }
23}
24
25terraform {
26    backend "s3" {
27        bucket = "jhooq-terraform-s3-bucket"
28        key    = "jhooq/terraform/remote/s3/terraform.tfstate"
29        region     = "eu-central-1"
30       dynamodb_table  = "dynamodb-state-locking"
31    }
32} 

3.1 Apply the above terraform configuration with DynamoDB table

1. The first command we are gonna run is terraform init

2. Now the run the terraform plan command

3. Finally, the terraform apply command

4. Verify the DynamoDB LockID by going into the AWS management console -

4. Spin one more EC2 instance with the same Terraform state file

(*Note- To simulate the locking scenario I am creating another main.tf with the same configuration. I would encourage you to create one main.tf and save the file in some other directory)

To test terraform state locking I will provision one more EC2 machine using the same Terraform state file (jhooq/terraform/remote/s3/terraform.tfstate) stored in my S3 bucket along with the same DynamoDB table (dynamodb-state-locking).

Keep in mind we are still using following two components from previous main.tf

1. S3 Bucket - jhooq-terraform-s3-bucket
2. DynamoDB Table - dynamodb-state-locking
3. Terraform state file - jhooq/terraform/remote/s3/terraform.tfstate

Here is my another main.tf file -

 1provider "aws" {
 2   region     = "eu-central-1"
 3   access_key = var.access_key
 4   secret_key = var.secret_key
 5}
 6
 7resource "aws_instance" "ec2_example" {
 8    ami = "ami-0767046d1677be5a0"
 9    instance_type = "t2.micro"
10    tags = {
11      Name = "EC2 Instance with remote state"
12    }
13}
14
15terraform {
16  backend "s3" {
17    bucket         = "jhooq-terraform-s3-bucket"
18    key            = "jhooq/terraform/remote/s3/terraform.tfstate"
19    encrypt        = true
20    region         = "eu-central-1"
21    dynamodb_table = "dynamodb-state-locking"
22  }
23}

4.1 Run both the terraform files at the same time to simulate the Locking on terraforming state file

On the left side of the screen, you will see the first terraform file(main.tf) which I have created in the Step-1 and on the right-hand side, you will see the terraform file(main.tf) from the Step-4.

**How did I simulate the remote state locking scenario? **

1. I have executed terraform apply on terraform file present on the right-hand side but did not let it finish. While executing terraform apply command I did not type yes when it asks for Do you want to perform these actions? so basically terraform apply command is still running and holding a lock on the remote state file.

2. At the same time I executed the terraform apply on main.tf from Step-4 which you can see on the right side of the screenshot. Since the second main.tf file also referring the same remote state as well as same dynamo db table it will throw en error - Error: Error acquiring the state lock Error message: ConditionalCheckFailedException: The conditional request failed Lock Info ID: 8f014160-8894-868e-529d-0f16e42af405

5. Conclusion

Terraform state file locking is one of the most valuable features offered by terraform for managing the Terraform state file. If you are using the AWS S3 and Dynamo DB then terraform state locking can improve your state management and save your time from unforeseen issues.

Read More - Terragrunt -

1. How to use Terragrunt?