Terraform state file management

Following up on my previous post about storing the terraform state file on a remote object storage location, I would like to talk about the terraform state command

The terraform state command is used for advanced state management. As your Terraform usage becomes more advanced, there are some cases where you may need to modify the Terraform state. Rather than modify the state directly, the terraform state commands can be used in many cases instead.

A typical use case that requires modifying an existing state file is refactoring of existing code (e.g upgrading your code from terraform 0.11x to 0.12x and/or moving from the count construct to the new for_each construct)

Let’s consider this example:

variable “list” {
default = [“linus”, “cheetarah”, “li-shou”]
}

resource random_pet “zoo” {
count = 3
prefix = var.list[count.index]
}

 

 

It’s a simple test case that makes use of the random provider: we define a list of three items, and then use the count construct to deploy all of them with a single resource definition. This was the only way to deploy multiple instances of a resource in terraform 0.11

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
————————————————————————

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# random_pet.zoo[0] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “linus”
+ separator = “-”
}

# random_pet.zoo[1] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “cheetarah”
+ separator = “-”
}

# random_pet.zoo[2] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “li-shou”
+ separator = “-”
}

Plan: 3 to add, 0 to change, 0 to destroy.

————————————————————————

Note: You didn’t specify an “-out” parameter to save this plan, so Terraform
can’t guarantee that exactly these actions will be performed if
“terraform apply” is subsequently run.

 

If we apply this plan, the result will be:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# random_pet.zoo[0] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “linus”
+ separator = “-”
}

# random_pet.zoo[1] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “cheetarah”
+ separator = “-”
}

# random_pet.zoo[2] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “li-shou”
+ separator = “-”
}

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only ‘yes’ will be accepted to approve.

Enter a value: yes

random_pet.zoo[0]: Creating…
random_pet.zoo[1]: Creating…
random_pet.zoo[2]: Creating…
random_pet.zoo[1]: Creation complete after 0s [id=cheetarah-magnetic-woodcock]
random_pet.zoo[2]: Creation complete after 0s [id=li-shou-immense-gnu]
random_pet.zoo[0]: Creation complete after 0s [id=linus-settled-bluegill]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

 

and our state file will contain:

mattiarossi@imacpro-01 ~/D/C/C/TEST> cat terraform.tfstate
{
“version”: 4,
“terraform_version”: “0.12.24”,
“serial”: 1,
“lineage”: “18b586d4-7da1-5dea-8aa8-d78bef761e7d”,
“outputs”: {},
“resources”: [
{
“mode”: “managed”,
“type”: “random_pet”,
“name”: “zoo”,
“each”: “list”,
“provider”: “provider.random”,
“instances”: [
{
“index_key”: 0,
“schema_version”: 0,
“attributes”: {
“id”: “linus-settled-bluegill”,
“keepers”: null,
“length”: 2,
“prefix”: “linus”,
“separator”: “-”
},
“private”: “bnVsbA==”
},
{
“index_key”: 1,
“schema_version”: 0,
“attributes”: {
“id”: “cheetarah-magnetic-woodcock”,
“keepers”: null,
“length”: 2,
“prefix”: “cheetarah”,
“separator”: “-”
},
“private”: “bnVsbA==”
},
{
“index_key”: 2,
“schema_version”: 0,
“attributes”: {
“id”: “li-shou-immense-gnu”,
“keepers”: null,
“length”: 2,
“prefix”: “li-shou”,
“separator”: “-”
},
“private”: “bnVsbA==”
variable “list” {
}
]
}
]
}

Now we have a state file we can query:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state list
random_pet.zoo[0]
random_pet.zoo[1]
random_pet.zoo[2]

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state show random_pet.zoo[0]
# random_pet.zoo[0]:
resource “random_pet” “zoo” {
id = “linus-settled-bluegill”
length = 2
prefix = “linus”
separator = “-”
}

This is an oversimplified example, but let’s say that because of a refactoring we need to move our resource definition from using the count construct to the for_each construct:

variable “list” {
default = [“linus”, “cheetarah”, “li-shou”]
}

resource random_pet “for_each_zoo” {
for_each = toset(var.list)
prefix = each.key
}

If we run a plan, it will say we need to delete the old resources and create three new ones, even if logically they are the same. In a real world use case this would have happened to resources like VMs, attached and configured block volumes and/or network interface, or object storage buckets, and we really wouldn’t want to destroy and recreate them, especially because of an infrastructure code refactor.

 

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.zoo[1]: Refreshing state… [id=cheetarah-magnetic-woodcock]
random_pet.zoo[0]: Refreshing state… [id=linus-settled-bluegill]
random_pet.zoo[2]: Refreshing state… [id=li-shou-immense-gnu]

————————————————————————

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
– destroy

Terraform will perform the following actions:

# random_pet.for_each_zoo[“cheetarah”] will be created
+ resource “random_pet” “for_each_zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “cheetarah”
+ separator = “-”
}

# random_pet.for_each_zoo[“li-shou”] will be created
+ resource “random_pet” “for_each_zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “li-shou”
+ separator = “-”
}

# random_pet.for_each_zoo[“linus”] will be created
+ resource “random_pet” “for_each_zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “linus”
+ separator = “-”
}

# random_pet.zoo[0] will be destroyed
– resource “random_pet” “zoo” {
– id = “linus-settled-bluegill” -> null
– length = 2 -> null
– prefix = “linus” -> null
– separator = “-” -> null
}

# random_pet.zoo[1] will be destroyed
– resource “random_pet” “zoo” {
– id = “cheetarah-magnetic-woodcock” -> null
– length = 2 -> null
– prefix = “cheetarah” -> null
– separator = “-” -> null
}

# random_pet.zoo[2] will be destroyed
– resource “random_pet” “zoo” {
– id = “li-shou-immense-gnu” -> null
– length = 2 -> null
– prefix = “li-shou” -> null
– separator = “-” -> null
}

Plan: 3 to add, 0 to change, 3 to destroy.

————————————————————————

 

What the terraform state command allow us to do is to ‘move around’ resources so that they will not be considered as a change in the plan:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state mv ‘random_pet.zoo[1]’ ‘random_pet.for_each_zoo[“cheetarah”]’
Move “random_pet.zoo[1]” to “random_pet.for_each_zoo[\”cheetarah\”]”
Successfully moved 1 object(s).
mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state mv ‘random_pet.zoo[2]’ ‘random_pet.for_each_zoo[“li-shou”]’
Move “random_pet.zoo[2]” to “random_pet.for_each_zoo[\”li-shou\”]”
Successfully moved 1 object(s).
mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state mv ‘random_pet.zoo[0]’ ‘random_pet.for_each_zoo[“linus”]’
Move “random_pet.zoo[0]” to “random_pet.for_each_zoo[\”linus\”]”
Successfully moved 1 object(s).

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state list
random_pet.for_each_zoo[“cheetarah”]
random_pet.for_each_zoo[“li-shou”]
random_pet.for_each_zoo[“linus”]

 

If we run our plan after the move, there will be no changes to apply:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.for_each_zoo[“linus”]: Refreshing state… [id=linus-settled-bluegill]
random_pet.for_each_zoo[“cheetarah”]: Refreshing state… [id=cheetarah-magnetic-woodcock]
random_pet.for_each_zoo[“li-shou”]: Refreshing state… [id=li-shou-immense-gnu]

————————————————————————

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

This new terraform functionality provides a lot of flexibility when upgrading/updating infrastructure code, as the state management can be programmatically scripted as well.

Previous to this a refactoring would mean manually changing the json state file, with the risk of ending up with an unusable file.

Have fun!

Mattia

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s