Terraform state file management

Following up on my previous post about storing the terraform state file on a remote object storage location, I would like to talk about the terraform state command

The terraform state command is used for advanced state management. As your Terraform usage becomes more advanced, there are some cases where you may need to modify the Terraform state. Rather than modify the state directly, the terraform state commands can be used in many cases instead.

A typical use case that requires modifying an existing state file is refactoring of existing code (e.g upgrading your code from terraform 0.11x to 0.12x and/or moving from the count construct to the new for_each construct)

Let’s consider this example:

variable “list” {
default = [“linus”, “cheetarah”, “li-shou”]
}

resource random_pet “zoo” {
count = 3
prefix = var.list[count.index]
}

 

 

It’s a simple test case that makes use of the random provider: we define a list of three items, and then use the count construct to deploy all of them with a single resource definition. This was the only way to deploy multiple instances of a resource in terraform 0.11

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
————————————————————————

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# random_pet.zoo[0] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “linus”
+ separator = “-”
}

# random_pet.zoo[1] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “cheetarah”
+ separator = “-”
}

# random_pet.zoo[2] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “li-shou”
+ separator = “-”
}

Plan: 3 to add, 0 to change, 0 to destroy.

————————————————————————

Note: You didn’t specify an “-out” parameter to save this plan, so Terraform
can’t guarantee that exactly these actions will be performed if
“terraform apply” is subsequently run.

 

If we apply this plan, the result will be:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# random_pet.zoo[0] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “linus”
+ separator = “-”
}

# random_pet.zoo[1] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “cheetarah”
+ separator = “-”
}

# random_pet.zoo[2] will be created
+ resource “random_pet” “zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “li-shou”
+ separator = “-”
}

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only ‘yes’ will be accepted to approve.

Enter a value: yes

random_pet.zoo[0]: Creating…
random_pet.zoo[1]: Creating…
random_pet.zoo[2]: Creating…
random_pet.zoo[1]: Creation complete after 0s [id=cheetarah-magnetic-woodcock]
random_pet.zoo[2]: Creation complete after 0s [id=li-shou-immense-gnu]
random_pet.zoo[0]: Creation complete after 0s [id=linus-settled-bluegill]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

 

and our state file will contain:

mattiarossi@imacpro-01 ~/D/C/C/TEST> cat terraform.tfstate
{
“version”: 4,
“terraform_version”: “0.12.24”,
“serial”: 1,
“lineage”: “18b586d4-7da1-5dea-8aa8-d78bef761e7d”,
“outputs”: {},
“resources”: [
{
“mode”: “managed”,
“type”: “random_pet”,
“name”: “zoo”,
“each”: “list”,
“provider”: “provider.random”,
“instances”: [
{
“index_key”: 0,
“schema_version”: 0,
“attributes”: {
“id”: “linus-settled-bluegill”,
“keepers”: null,
“length”: 2,
“prefix”: “linus”,
“separator”: “-”
},
“private”: “bnVsbA==”
},
{
“index_key”: 1,
“schema_version”: 0,
“attributes”: {
“id”: “cheetarah-magnetic-woodcock”,
“keepers”: null,
“length”: 2,
“prefix”: “cheetarah”,
“separator”: “-”
},
“private”: “bnVsbA==”
},
{
“index_key”: 2,
“schema_version”: 0,
“attributes”: {
“id”: “li-shou-immense-gnu”,
“keepers”: null,
“length”: 2,
“prefix”: “li-shou”,
“separator”: “-”
},
“private”: “bnVsbA==”
variable “list” {
}
]
}
]
}

Now we have a state file we can query:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state list
random_pet.zoo[0]
random_pet.zoo[1]
random_pet.zoo[2]

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state show random_pet.zoo[0]
# random_pet.zoo[0]:
resource “random_pet” “zoo” {
id = “linus-settled-bluegill”
length = 2
prefix = “linus”
separator = “-”
}

This is an oversimplified example, but let’s say that because of a refactoring we need to move our resource definition from using the count construct to the for_each construct:

variable “list” {
default = [“linus”, “cheetarah”, “li-shou”]
}

resource random_pet “for_each_zoo” {
for_each = toset(var.list)
prefix = each.key
}

If we run a plan, it will say we need to delete the old resources and create three new ones, even if logically they are the same. In a real world use case this would have happened to resources like VMs, attached and configured block volumes and/or network interface, or object storage buckets, and we really wouldn’t want to destroy and recreate them, especially because of an infrastructure code refactor.

 

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.zoo[1]: Refreshing state… [id=cheetarah-magnetic-woodcock]
random_pet.zoo[0]: Refreshing state… [id=linus-settled-bluegill]
random_pet.zoo[2]: Refreshing state… [id=li-shou-immense-gnu]

————————————————————————

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
– destroy

Terraform will perform the following actions:

# random_pet.for_each_zoo[“cheetarah”] will be created
+ resource “random_pet” “for_each_zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “cheetarah”
+ separator = “-”
}

# random_pet.for_each_zoo[“li-shou”] will be created
+ resource “random_pet” “for_each_zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “li-shou”
+ separator = “-”
}

# random_pet.for_each_zoo[“linus”] will be created
+ resource “random_pet” “for_each_zoo” {
+ id = (known after apply)
+ length = 2
+ prefix = “linus”
+ separator = “-”
}

# random_pet.zoo[0] will be destroyed
– resource “random_pet” “zoo” {
– id = “linus-settled-bluegill” -> null
– length = 2 -> null
– prefix = “linus” -> null
– separator = “-” -> null
}

# random_pet.zoo[1] will be destroyed
– resource “random_pet” “zoo” {
– id = “cheetarah-magnetic-woodcock” -> null
– length = 2 -> null
– prefix = “cheetarah” -> null
– separator = “-” -> null
}

# random_pet.zoo[2] will be destroyed
– resource “random_pet” “zoo” {
– id = “li-shou-immense-gnu” -> null
– length = 2 -> null
– prefix = “li-shou” -> null
– separator = “-” -> null
}

Plan: 3 to add, 0 to change, 3 to destroy.

————————————————————————

 

What the terraform state command allow us to do is to ‘move around’ resources so that they will not be considered as a change in the plan:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state mv ‘random_pet.zoo[1]’ ‘random_pet.for_each_zoo[“cheetarah”]’
Move “random_pet.zoo[1]” to “random_pet.for_each_zoo[\”cheetarah\”]”
Successfully moved 1 object(s).
mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state mv ‘random_pet.zoo[2]’ ‘random_pet.for_each_zoo[“li-shou”]’
Move “random_pet.zoo[2]” to “random_pet.for_each_zoo[\”li-shou\”]”
Successfully moved 1 object(s).
mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state mv ‘random_pet.zoo[0]’ ‘random_pet.for_each_zoo[“linus”]’
Move “random_pet.zoo[0]” to “random_pet.for_each_zoo[\”linus\”]”
Successfully moved 1 object(s).

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform state list
random_pet.for_each_zoo[“cheetarah”]
random_pet.for_each_zoo[“li-shou”]
random_pet.for_each_zoo[“linus”]

 

If we run our plan after the move, there will be no changes to apply:

mattiarossi@imacpro-01 ~/D/C/C/TEST> terraform plan
Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

random_pet.for_each_zoo[“linus”]: Refreshing state… [id=linus-settled-bluegill]
random_pet.for_each_zoo[“cheetarah”]: Refreshing state… [id=cheetarah-magnetic-woodcock]
random_pet.for_each_zoo[“li-shou”]: Refreshing state… [id=li-shou-immense-gnu]

————————————————————————

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

This new terraform functionality provides a lot of flexibility when upgrading/updating infrastructure code, as the state management can be programmatically scripted as well.

Previous to this a refactoring would mean manually changing the json state file, with the risk of ending up with an unusable file.

Have fun!

Mattia

 

How to retrieve an OCI Marketplace image id using advanced functionality in the OCI CLI utility

If you ever need to deploy an OCI Marketplace image using your automation, the information that you will have to feed to your script will be:

  "listingId": "ocid1.appcataloglisting.oc1..aaaaaaaam7ewzrjbltqiarxukuk72v2lqkdtpqtwxqpszqqvrm7likfnpt5q",
  "listingResourceId": "ocid1.image.oc1..aaaaaaaaa5u7qtj3j3um4zryg3qkn6if2sqctqjfh46pdq5z56kq6zagg4va",

Let’s say we need to find this information for a specific Marketplace image, like RocketChat:

Screenshot 2020-05-19 at 15.04.55

We can use the OCI cli to retrieve the details needed by terraform, but since there is no direct command to fetch information about a Marketplace listing we are going to use the ‘raw-request’ functionality that the cli provides as a fallback to access the OCI API when needed for advanced use cases

I have created a gist that uses the OCI cli to perform this kind of search, you can find it on GitHub

#!/bin/bash 
#Example string: 'FortiGate Next-Gen Firewall (BYOL)'
export STERM="$1"
#Example version: '6.4.0'
export SVERSION="$2" 
export REGION="uk-london-1" 
export LID=`oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET | jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))' | jq -r '.listingId'` 
oci raw-request --http-method GET --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings/$LID/resourceVersions | jq --arg SVERSION "$SVERSION" '.data[] | select ( .listingResourceVersion | contains($SVERSION))'

The script accepts two parameters

  • the Marketplace image name
  • the Image version needed
mrossi@cloudshell:~ (us-phoenix-1)$ ./getMarketplaceImageId 'RocketChat' '2.4.9'
{
"listingId": "ocid1.appcataloglisting.oc1..aaaaaaaapjvcl7yie67z6ag4hwfgpiegwpy5izkaaq3p3rngvmtmqm7zcaxq",
"listingResourceId": "ocid1.image.oc1..aaaaaaaajq2pfkmd6nl23ouho2ulxldoy2or5zy7dbiv46qhu6tsqm6ftila",
"listingResourceVersion": "2.4.9",
"timePublished": "2020-03-05T13:27:11.940Z"
}

Let’s break down the script:

export STERM="$1"
#6.4.0
export SVERSION="$2" 
export REGION="uk-london-1"

These are initialisation parameters, assigning the search parameters from the script input params, and setting a default OCI region for the API queries

export LID=`oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET | jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))' | jq -r '.listingId'`

The first call to the OCI cli retrieves the Listing id for a given Marketplace image:

mrossi@cloudshell:~ (us-phoenix-1)$ oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET | jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))' | jq -r '.listingId'
ocid1.appcataloglisting.oc1..aaaaaaaapjvcl7yie67z6ag4hwfgpiegwpy5izkaaq3p3rngvmtmqm7zcaxq

Let’s break down the command:

The first part retrieves all the Marketplace listings for a given region:

mrossi@cloudshell:~ (us-phoenix-1)$ oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET 
{
  "data": [
    {
      "displayName": "0Chain Data Privacy Compliance Platform",
      "listingId": "ocid1.appcataloglisting.oc1..aaaaaaaahfwb7mtjpi45wc4waduujja5fdeslfio4r2amuyuz3x62x5rxina",
      "publisherName": "0Chain LLC",
      "summary": "Add data privacy, protection, and private sharing to your business"
    },
....

    {
      "displayName": "RocketChat",
      "listingId": "ocid1.appcataloglisting.oc1..aaaaaaaapjvcl7yie67z6ag4hwfgpiegwpy5izkaaq3p3rngvmtmqm7zcaxq",
      "publisherName": "Rocket.Chat",
      "summary": "The ultimate Open Source communication platform"
    },
    {
      "displayName": "SUSE Linux Enterprise Server 12 SP4 (BYOS)",
      "listingId": "ocid1.appcataloglisting.oc1..aaaaaaaa3mflvqev6iw4jnjibtrlgdz2h75kpmdwgvhifljffv5b5xbcqmxq",
      "publisherName": "SUSE",
      "summary": "SLES 12 SP4"
    },
    {
      "displayName": "SUSE Linux Enterprise Server 12 SP4 (BYOS)",
      "listingId": "ocid1.appcataloglisting.oc1..aaaaaaaaowy672zei2w2dy3s2z4c7kytbrd55cnvox3djurgswguiang32ga",
      "publisherName": "SUSE",
      "summary": "SLES 12 SP4"
    },
 ...
  ],
  "headers": {
    "Connection": "keep-alive",
    "Content-Type": "application/json",
    "Date": "Tue, 19 May 2020 13:31:54 GMT",
    "Transfer-Encoding": "chunked",
    "Vary": "Accept-Encoding",
    "X-Content-Type-Options": "nosniff",
    "opc-next-page": "AAAAAAAAAAAAFCCBIFAUCQKBIFAUCQKHOVUVIY2NOFPTETJNMV5HQX3KKJKHE2DOHBZUWRTRNMZWWODWJIYHM3BTOE3VSVLTNJEGUSCPIVGVM5TGOM2GC3LCFVDHMTLCLA2XA6SXJVCEMRSNNNIHQVCPLFVGSU3DIVIGKTTZI5UE24LUI5KHQLKFOFVVKVLNO5UWS2KQOVWGM4LCOA3HQSLXOI4FATJWLFITAM2DIEZDK3KXMN3C233IJFLEGZZVNQYVATKZJAWUY5DPGFIUUWBVHBZWKSTOOFPUIUKGO5VTCSRZINBVE4JNKI4HKTTZN5SFO4CJOBAUUTDUKVSDKUCXNVVV6ZBTIZMVGYSVKFEHC4BYNJUHQOJQMNGWI6CNGAZFMSTJMFMEUQSCNV3VEZ32NFGWQVCGJFYDOLJTJBXUCVKYM5JWYYKGPFDWC5BSGF2HIZ27KFIEMXZQKVDEUUD2KRHGUVKNJNTFETKHIZGDEWTYGMYHG23CJNSGGMCEINXFKY2XJJWVG4JRGBKGOWSIKFUXKSCCKM3XAZ2EIJHWETLOPFXU23KRNA4UMNLOIQ2FU53CKRUEKWSYGFPUINSVJRWW63DRKVKUKQZVIF5FMSKMOZXG6WS7N5YF62LQMEZE4MLLLBRWQUCDOZUXA23WMJQTGODRJJHG62ZSKZ5HO4ZWGFKWQSLWHFSFQYLCMQYFINRUJRGVAWBWJA3UEVCYKAYFQ6TTJNYVKU2BGI4WSQLXMJDFUV2GPBLGIMDZM5QWY2KMNJTXQ6LCNBPWSSDUOIZE643IL5VG6SD2HBLDKSS2JVVHEVLULBQVE6TMOZWWSN3YJFKWI2TXMRQXQMLWJ5IXCNDSI5LS233SJI4WKSJXIRGTS2RQHFBXGNBVINRWUOCWLFBG26BVGVEECMZZIRDDQYLGJN3FS3DUNQYWGWLLNFHGYVTRONNFK2K2PF2XGT2JMRVWSZ3LG5LFCY2HORMXOODXOREEO3ZZLJVVU6CIKNUGK42DKJLEQTCMKEYXC3DFPBAWMQSNKZXTE5ZTFVHQ",
    "opc-request-id": "843C1ADDCBC74192AFA0499764567CD8/3349DFC98814A8F5F33B1F767B7A0FE6/7D4C870DF934CD46CD94537C86F0969D"
  },
  "status": "200 OK"
}

The second part filters the global result json object and retrieves only information related to the Image we’re interested in:

oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET | jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))'

I am using the jq utility to do that by setting up a filter on the displayName field:

jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))'
mrossi@cloudshell:~ (us-phoenix-1)$ oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET | jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))'
{
"displayName": "RocketChat",
"listingId": "ocid1.appcataloglisting.oc1..aaaaaaaapjvcl7yie67z6ag4hwfgpiegwpy5izkaaq3p3rngvmtmqm7zcaxq",
"publisherName": "Rocket.Chat",
"summary": "The ultimate Open Source communication platform"
}

The third part filters the single listing result json object and retrieves the listingId field, again using the jq utility and assigns it to an utility variable

export LID=`oci raw-request --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings --http-method GET | jq --arg STERM "$STERM" '.data[] | select ( .displayName | contains($STERM))' | jq -r '.listingId'`

Using the collected listing ID we can retrieve the actual information we’re interested in by issuing another API call:

oci raw-request --http-method GET --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings/$LID/resourceVersions | jq --arg SVERSION "$SVERSION" '.data[] | select ( .listingResourceVersion | contains($SVERSION))'

This command filters the API call output using jq to filter only the specific information we’re interested in:

mrossi@cloudshell:~ (us-phoenix-1)$ oci raw-request --http-method GET --target-uri https://iaas.$REGION.oraclecloud.com/20160918/appCatalogListings/$LID/resourceVersions | jq --arg SVERSION "$SVERSION" '.data[] | select ( .listingResourceVersion | contains($SVERSION))'
{
"listingId": "ocid1.appcataloglisting.oc1..aaaaaaaapjvcl7yie67z6ag4hwfgpiegwpy5izkaaq3p3rngvmtmqm7zcaxq",
"listingResourceId": "ocid1.image.oc1..aaaaaaaajq2pfkmd6nl23ouho2ulxldoy2or5zy7dbiv46qhu6tsqm6ftila",
"listingResourceVersion": "2.4.9",
"timePublished": "2020-03-05T13:27:11.940Z"
}

This technique of doing RAW api calls can be extended for any use that the standard cli commands do not cover, while the filtering through jq can be used for manipulating any json output. The OCI cli has an internal filtering mechanism that doesn’t play well with the output from the raw request subcommand

Have fun !

Mattia

USE OCI OBJECT STORAGE TO STORE TERRAFORM STATE FILES

download.png

Introduction

Terraform allows you to deploy infrastructure very quickly, and you can find thousands of script/modules/snippets on the web to get you started in a snap.

Once you move from the “Let’s try this at home” phase to a more “Let’s do this for a real environment” oriented phase you will have to make a lot of decisions about how you are going to maintain your infrastructure and how you are going to manage change, possibly in a non destructive way.

Nothing like dropping a production environment on a Friday evening while you are testing a small change in your dev environment will make you realise how badly you need this when moving past the Terraform learning phase.

One important decision during this transition to real world use of terraform is the location where you are going to store a Terraform project state file.

In this tutorial you will learn how to configure OCI Object storage as a remote backend for a sample terraform project, this will allow you to replicate the setup for your projects.

What is a terraform state file?

Terraform must store state information about your managed infrastructure and configuration. This state is used by Terraform to map real world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures.

This state is stored by default in a local file named “terraform.tfstate”, but it can also be stored remotely, which is essential when more than one person is working on the same project. With remote state, Terraform writes the state data to a remote data store, which can then be shared between all members of a team, encrypted, versioned and secured. Terraform supports storing state in Terraform CloudHashiCorp Consul, Amazon S3, Alibaba Cloud OSS, and other providers.

If you want to dig deeper into why Terraform needs a state file to work you can have a look here: https://www.terraform.io/docs/state/purpose.html

Terraform remote state file and OCI support

OCI object storage is not supported natively for storing state files, but since it has an AWS S3 compatibility layer we can use it all the same

Requirements

An Active Oracle Cloud Tenancy

If you do not have one, you can enroll here for a trial subscription that includes $300 of free credits

An Oracle Cloud Shell Instance

The OCI Cloud Shell is a web browser-based terminal accessible from the Oracle Cloud Console. Cloud Shell is free to use (within monthly tenancy limits), and provides access to a Linux shell, with a pre-authenticated Oracle Cloud Infrastructure CLI and other useful tools for following Oracle Cloud Infrastructure service tutorials and labs. Cloud Shell is a feature available to all OCI users, accessible from the Console. Your Cloud Shell will appear in the Oracle Cloud Console as a persistent frame of the Console and will stay active as you navigate to different pages of the Console. You can find detailed instructions on how to get started with the OCI Cloud Shell here

cloudshell-1.png

Create an initial working dir:

mkdir fk-terraform-objectstorage
cd fk-terraform-objectstorage/

A test terraform project

git clone https://github.com/mattiarossi/terraform-oci-default-vcn/


Cloning into 'terraform-oci-default-vcn'...
remote: Enumerating objects: 61, done.
remote: Counting objects: 100% (61/61), done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 75 (delta 27), reused 44 (delta 14), pack-reused 14
Unpacking objects: 100% (75/75), done.


cd terraform-oci-default-vcn/examples/vcn_default/

The test project will deploy a simple VCN in a compartment of your choice, configurable in the project variables, and will store the state file in an OCI Object storage Bucket that needs to be created in advance

An OCI Object Storage Bucket

The bucket needs to be created in the Compartment that is designated on OCI to be the target for the AWS S3 compatibility layer. You can check the active S3 compartment by using the following command:

oci os ns  get-metadata --output=table
+-------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+-----------+
| default-s3-compartment-id                                                           | default-swift-compartment-id                                                        | namespace |
+-------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+-----------+
| ocid1.compartment.oc1..aaaaaaa...4nxhlvbbbbbbb3kpmffof6vba                          | ocid1.compartment.oc1..aaaaaaa...hlvbbbbbbbbpmffof6vba                              | mytenancy |
+-------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------+-----------+

We will be using a bucket named tf-fk-test-01

This command will create a bucket named tf-fk-test-01 in the AWS s3 compatibility layer compartment:

oci os bucket create --compartment-id ocid1.compartment.oc1..aaaaaaa...4nxhlvb3luoh3kpmffof6vba --name tf-fk-test-01

A properly configured AWS S3 compatibility environment

In order to be able to access the Object Storage bucket using the S3 compatibility layer, you need to generate a set of access credentials, and the oci cli command needed for creating these needs to know the current user ocid:

export ME=mattia.rossi@myemail.com (my tenancy username)
oci iam user list --query "data[?\"name\"=='$ME']".{"name:\"name\",id:\"id\""} --output table

+------------------------------------------------------------------------------+--------------------------+
| id                                                                           | name                     |
+------------------------------------------------------------------------------+--------------------------+
| ocid1.user.oc1..aaaaaaa..................................tojgy2qybdkcayvnxsq | mattia.rossi@myemail.com |
+------------------------------------------------------------------------------+--------------------------+

export OCIUID=ocid1.user.oc1..aaaaaaa..................................tojgy2qybdkcayvnxsq

The following command will reuse the ocid variable to create the credentials needed to setup the AWS compatibility layer (please note that these are examples only and you will need to insert the proper secrets generated by the oci cli in the export command

oci iam customer-secret-key create --user-id $OCIUID --display-name 'key-tf-test' --query "data".{"AWS_ACCESS_KEY_ID:\"id\",AWS_SECRET_ACCESS_KEY:\"key\""} --output=table

+------------------------------------------+----------------------------------------------+
| AWS_ACCESS_KEY_ID                        | AWS_SECRET_ACCESS_KEY                        |
+------------------------------------------+----------------------------------------------+
| fd9bcbb0.........................93e7139 | kS/...............................PRDoGX1NY= |
+------------------------------------------+----------------------------------------------+

export AWS_ACCESS_KEY_ID=fd9bcbb0.........................93e7139
export AWS_SECRET_ACCESS_KEY=kS/...............................PRDoGX1NY=

Environment variables

The only other environment variable needed by the project for authenticating is the tenancy ocid, that can be set up using this command:

export TF_VAR_tenancy_ocid=`cat /etc/oci/config  | grep tenancy | uniq | cut -d '=' -f 2`

Terraform project config

Prepare one variable file named “terraform.tfvars” with the configuration information. The content of “terraform.tfvars” should look something like the following:

$ cat terraform.tfvars
# Region
region = "eu-frankfurt-1"

# Compartment
compartment_ocid = ""

# VCN Configurations
vcn_display_name = "testVCN"
vcn_cidr = "10.0.0.0/16"

Use the following script to retrieve the ocid of the compartment where you want to deploy the test VCN:

export COMPARTMENT=my-compartment
oci iam compartment list --query "data[?\"name\"=='$COMPARTMENT']".{"name:\"name\",id:\"id\""} --output=table
+-------------------------------------------------------------------------------------+-----------------------+
| id                                                                                  | name                  |
+-------------------------------------------------------------------------------------+-----------------------+
| ocid1.compartment.oc1..aaaaaaaabmc54lgslm..........................3hygseg6qeh5pvwq | my-compartment        |
+-------------------------------------------------------------------------------------+-----------------------+

Edit the file vcn_default.tf, and update the terraform backend section to match your OCI setup:

terraform {
  backend "s3" {
    endpoint                    = "https://__mytenancy__.compat.objectstorage.__region__.oraclecloud.com"
    skip_metadata_api_check     = true
    skip_region_validation      = true
    force_path_style            = true
    skip_credentials_validation = true
    bucket                      = "__mybucket__"
    key                         = "__key__"
    region                      = "__region__"
  }
}

where:

  • __mytenancy__ is your tenancy name
  • __region__ is the region where you created the Object Storage bucket
  • __mybucket__ is the name of the bucket you created
  • key is the name of the bucket entry that will hold your terraform state file (for example: terraform/state/oci/vcn/testVCN/terraform.tfstate)

Then apply the example using the following commands:

terraform init
terraform plan
terraform apply

Init:

mattia_ros@cloudshell:vcn_default (eu-frankfurt-1)$ terraform init
Initializing modules...
- vcn in ../..

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "oci" (hashicorp/oci) 3.74.0...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.oci: version = "~> 3.74"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Plan:

mattia_ros@cloudshell:vcn_default (eu-frankfurt-1)$ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

module.vcn.data.oci_identity_availability_domains.this: Refreshing state...

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# module.vcn.oci_core_default_route_table.this will be created
+ resource "oci_core_default_route_table" "this" {
+ defined_tags = (known after apply)
+ display_name = (known after apply)
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ manage_default_resource_id = (known after apply)
+ state = (known after apply)
+ time_created = (known after apply)

+ route_rules {
+ cidr_block = (known after apply)
+ description = (known after apply)
+ destination = "0.0.0.0/0"
+ destination_type = (known after apply)
+ network_entity_id = (known after apply)
}
}

# module.vcn.oci_core_internet_gateway.this will be created
+ resource "oci_core_internet_gateway" "this" {
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ display_name = (known after apply)
+ enabled = true
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ state = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
}

# module.vcn.oci_core_subnet.this[0] will be created
+ resource "oci_core_subnet" "this" {
+ availability_domain = "okVf:EU-FRANKFURT-1-AD-1"
+ cidr_block = "192.168.0.0/28"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ dhcp_options_id = (known after apply)
+ display_name = "Default Subnet okVf:EU-FRANKFURT-1-AD-1"
+ dns_label = "subnet1"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ ipv6virtual_router_ip = (known after apply)
+ prohibit_public_ip_on_vnic = (known after apply)
+ route_table_id = (known after apply)
+ security_list_ids = (known after apply)
+ state = (known after apply)
+ subnet_domain_name = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
+ virtual_router_ip = (known after apply)
+ virtual_router_mac = (known after apply)
}

# module.vcn.oci_core_subnet.this[1] will be created
+ resource "oci_core_subnet" "this" {
+ availability_domain = "okVf:EU-FRANKFURT-1-AD-2"
+ cidr_block = "192.168.0.16/28"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ dhcp_options_id = (known after apply)
+ display_name = "Default Subnet okVf:EU-FRANKFURT-1-AD-2"
+ dns_label = "subnet2"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ ipv6virtual_router_ip = (known after apply)
+ prohibit_public_ip_on_vnic = (known after apply)
+ route_table_id = (known after apply)
+ security_list_ids = (known after apply)
+ state = (known after apply)
+ subnet_domain_name = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
+ virtual_router_ip = (known after apply)
+ virtual_router_mac = (known after apply)
}

# module.vcn.oci_core_subnet.this[2] will be created
+ resource "oci_core_subnet" "this" {
+ availability_domain = "okVf:EU-FRANKFURT-1-AD-3"
+ cidr_block = "192.168.0.32/28"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ dhcp_options_id = (known after apply)
+ display_name = "Default Subnet okVf:EU-FRANKFURT-1-AD-3"
+ dns_label = "subnet3"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ ipv6virtual_router_ip = (known after apply)
+ prohibit_public_ip_on_vnic = (known after apply)
+ route_table_id = (known after apply)
+ security_list_ids = (known after apply)
+ state = (known after apply)
+ subnet_domain_name = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
+ virtual_router_ip = (known after apply)
+ virtual_router_mac = (known after apply)
}

# module.vcn.oci_core_vcn.this will be created
+ resource "oci_core_vcn" "this" {
+ cidr_block = "192.168.0.0/25"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ default_dhcp_options_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_list_id = (known after apply)
+ defined_tags = (known after apply)
+ display_name = "TEST-FK-VCN"
+ dns_label = "vcn"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ is_ipv6enabled = (known after apply)
+ state = (known after apply)
+ time_created = (known after apply)
+ vcn_domain_name = (known after apply)
}

Plan: 6 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

Apply:

mattia_ros@cloudshell:vcn_default (eu-frankfurt-1)$ terraform apply
module.vcn.data.oci_identity_availability_domains.this: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# module.vcn.oci_core_default_route_table.this will be created
+ resource "oci_core_default_route_table" "this" {
+ defined_tags = (known after apply)
+ display_name = (known after apply)
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ manage_default_resource_id = (known after apply)
+ state = (known after apply)
+ time_created = (known after apply)

+ route_rules {
+ cidr_block = (known after apply)
+ description = (known after apply)
+ destination = "0.0.0.0/0"
+ destination_type = (known after apply)
+ network_entity_id = (known after apply)
}
}

# module.vcn.oci_core_internet_gateway.this will be created
+ resource "oci_core_internet_gateway" "this" {
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ display_name = (known after apply)
+ enabled = true
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ state = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
}

# module.vcn.oci_core_subnet.this[0] will be created
+ resource "oci_core_subnet" "this" {
+ availability_domain = "okVf:EU-FRANKFURT-1-AD-1"
+ cidr_block = "192.168.0.0/28"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ dhcp_options_id = (known after apply)
+ display_name = "Default Subnet okVf:EU-FRANKFURT-1-AD-1"
+ dns_label = "subnet1"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ ipv6virtual_router_ip = (known after apply)
+ prohibit_public_ip_on_vnic = (known after apply)
+ route_table_id = (known after apply)
+ security_list_ids = (known after apply)
+ state = (known after apply)
+ subnet_domain_name = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
+ virtual_router_ip = (known after apply)
+ virtual_router_mac = (known after apply)
}

# module.vcn.oci_core_subnet.this[1] will be created
+ resource "oci_core_subnet" "this" {
+ availability_domain = "okVf:EU-FRANKFURT-1-AD-2"
+ cidr_block = "192.168.0.16/28"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ dhcp_options_id = (known after apply)
+ display_name = "Default Subnet okVf:EU-FRANKFURT-1-AD-2"
+ dns_label = "subnet2"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ ipv6virtual_router_ip = (known after apply)
+ prohibit_public_ip_on_vnic = (known after apply)
+ route_table_id = (known after apply)
+ security_list_ids = (known after apply)
+ state = (known after apply)
+ subnet_domain_name = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
+ virtual_router_ip = (known after apply)
+ virtual_router_mac = (known after apply)
}

# module.vcn.oci_core_subnet.this[2] will be created
+ resource "oci_core_subnet" "this" {
+ availability_domain = "okVf:EU-FRANKFURT-1-AD-3"
+ cidr_block = "192.168.0.32/28"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ defined_tags = (known after apply)
+ dhcp_options_id = (known after apply)
+ display_name = "Default Subnet okVf:EU-FRANKFURT-1-AD-3"
+ dns_label = "subnet3"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ ipv6virtual_router_ip = (known after apply)
+ prohibit_public_ip_on_vnic = (known after apply)
+ route_table_id = (known after apply)
+ security_list_ids = (known after apply)
+ state = (known after apply)
+ subnet_domain_name = (known after apply)
+ time_created = (known after apply)
+ vcn_id = (known after apply)
+ virtual_router_ip = (known after apply)
+ virtual_router_mac = (known after apply)
}

# module.vcn.oci_core_vcn.this will be created
+ resource "oci_core_vcn" "this" {
+ cidr_block = "192.168.0.0/25"
+ compartment_id = "ocid1.compartment.oc1..aaaaaaaan2lhk5eqy2hvum45ahjsvgs2sz7x72ur5kw4euwe6my2226mk5ia"
+ default_dhcp_options_id = (known after apply)
+ default_route_table_id = (known after apply)
+ default_security_list_id = (known after apply)
+ defined_tags = (known after apply)
+ display_name = "TEST-FK-VCN"
+ dns_label = "vcn"
+ freeform_tags = (known after apply)
+ id = (known after apply)
+ ipv6cidr_block = (known after apply)
+ ipv6public_cidr_block = (known after apply)
+ is_ipv6enabled = (known after apply)
+ state = (known after apply)
+ time_created = (known after apply)
+ vcn_domain_name = (known after apply)
}

Plan: 6 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

module.vcn.oci_core_vcn.this: Creating...
module.vcn.oci_core_vcn.this: Creation complete after 0s [id=ocid1.vcn.oc1.eu-frankfurt-1.amaaaaaavyxeuaqaavuidjkp2un5v7fgodw63pytlkaxvrfs5coucgfqmgfa]
module.vcn.oci_core_internet_gateway.this: Creating...
module.vcn.oci_core_subnet.this[2]: Creating...
module.vcn.oci_core_subnet.this[1]: Creating...
module.vcn.oci_core_subnet.this[0]: Creating...
module.vcn.oci_core_subnet.this[0]: Creation complete after 0s [id=ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaa2feksp66yxjrj2twxypjbuk5blefju2ghkk34z4hvex57obkxzqa]
module.vcn.oci_core_subnet.this[2]: Creation complete after 1s [id=ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaadqbslze7krqigudqg6ajhxhxddpmhvaottievimyk6jdxz754pta]
module.vcn.oci_core_subnet.this[1]: Creation complete after 1s [id=ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaaoqrhcrv743sdt6norse7wql7rmin3ca6amoigqonvsejuoavnjtq]
module.vcn.oci_core_internet_gateway.this: Creation complete after 2s [id=ocid1.internetgateway.oc1.eu-frankfurt-1.aaaaaaaau4sdjijm4b3miu2hwf5lcbkyb7eaea62qg7dpaar2udn4u4zhfnq]
module.vcn.oci_core_default_route_table.this: Creating...
module.vcn.oci_core_default_route_table.this: Creation complete after 0s [id=ocid1.routetable.oc1.eu-frankfurt-1.aaaaaaaa5ag3j4kbkb763c3qorcowy5r3swyileytlyhagkv7mqragxysgma]

Apply complete! Resources: 6 added, 0 changed, 0 destroyed.

Outputs:

default_dhcp_options_id = ocid1.dhcpoptions.oc1.eu-frankfurt-1.aaaaaaaabzs2tgdpnq6jrd2p67lks44qovk272uxuoi3kgciib62fyflgyja
default_route_table_id = ocid1.routetable.oc1.eu-frankfurt-1.aaaaaaaa5ag3j4kbkb763c3qorcowy5r3swyileytlyhagkv7mqragxysgma
default_security_list_id = ocid1.securitylist.oc1.eu-frankfurt-1.aaaaaaaadm7oi73fwjx2kiiqbc7ucsuvqcmxfjis27at4p3tyhuopod5tqoq
internet_gateway_id = ocid1.internetgateway.oc1.eu-frankfurt-1.aaaaaaaau4sdjijm4b3miu2hwf5lcbkyb7eaea62qg7dpaar2udn4u4zhfnq
subnet_ids = [
"ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaa2feksp66yxjrj2twxypjbuk5blefju2ghkk34z4hvex57obkxzqa",
"ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaaoqrhcrv743sdt6norse7wql7rmin3ca6amoigqonvsejuoavnjtq",
"ocid1.subnet.oc1.eu-frankfurt-1.aaaaaaaadqbslze7krqigudqg6ajhxhxddpmhvaottievimyk6jdxz754pta",
]
vcn_id = ocid1.vcn.oc1.eu-frankfurt-1.amaaaaaavyxeuaqaavuidjkp2un5v7fgodw63pytlkaxvrfs5coucgfqmgfa

Now let’s check the local terraform state file:

mattia_ros@cloudshell:vcn_default (eu-frankfurt-1)$ cat .terraform/terraform.tfstate 
{
"version": 3,
"serial": 1,
"lineage": "1250a1d2-3e20-124b-7c25-524f67566052",
"backend": {
"type": "s3",
"config": {
"access_key": null,
"acl": null,
"assume_role_policy": null,
"bucket": "tf-fk-test-01",
"dynamodb_endpoint": null,
"dynamodb_table": null,
"encrypt": null,
"endpoint": "https://mytenancy.compat.objectstorage.eu-frankfurt-1.oraclecloud.com",
"external_id": null,
"force_path_style": true,
"iam_endpoint": null,
"key": "terraform/state/oci/vcn/testVCN/terraform.tfstate",
"kms_key_id": null,
"lock_table": null,
"max_retries": null,
"profile": null,
"region": "eu-frankfurt-1",
"role_arn": null,
"secret_key": null,
"session_name": null,
"shared_credentials_file": null,
"skip_credentials_validation": true,
"skip_get_ec2_platforms": null,
"skip_metadata_api_check": true,
"skip_region_validation": true,
"skip_requesting_account_id": null,
"sse_customer_key": null,
"sts_endpoint": null,
"token": null,
"workspace_key_prefix": null
},
"hash": 1758364997
},
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {},
"depends_on": []
}
]
}

And let’s check the Object storage bucket:

oci os object list -bn tf-fk-test-01 --output=table
+------+--------------------------+---------------------------------------------------+-------+----------------------------------+---------------+
| etag | md5 | name | size | time-created | time-modified |
+------+--------------------------+---------------------------------------------------+-------+----------------------------------+---------------+
| None | PRXsobIN......efUorHfg== | terraform/state/oci/vcn/testVCN/terraform.tfstate | 14161 | 2020-05-08T18:43:19.428000+00:00 | None |
+------+--------------------------+---------------------------------------------------+-------+----------------------------------+---------------+
prefixes: []

.. and we’re done, you have configured terraform to store state files remotely on OCI.

If you want to use the same pattern outside the OCI cloud shell you will need to configure the oci provider to use the usual API key/fingerprint combo

Have Fun!

Mattia

Create a custom OCI image using Packer and the OCI Cloud Shell

Today we would like to show you how to bake a custom image in OCI using packer, a tool that complements terraform in creating Infrastructure as a code deployment.
We will use an OCI standard Oracle Autonomous Linux Image, and create a custom one that consists of the original image, plus the Oracle instant client libraries and tools like sqlplus and impdp/expdp. At the end of the process, a new custom image will be available for you to deploy in your OCI tenancy.

What is Packer?

Packer is an open-source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image. A machine image is a single static unit that contains a pre-configured operating system and installed software which is used to quickly create new running machinesPacker supports natively creating OCI images by the way of the oracle-oci builder

Required components:

An Active Oracle Cloud Tenancy

If you do not have one, you can enroll here for a trial subscription that includes $300 of free credits

An Oracle Cloud Shell Instance

The OCI Cloud Shell is a web browser-based terminal accessible from the Oracle Cloud Console. Cloud Shell is free to use (within monthly tenancy limits), and provides access to a Linux shell, with a pre-authenticated Oracle Cloud Infrastructure CLI and other useful tools for following Oracle Cloud Infrastructure service tutorials and labs. Cloud Shell is a feature available to all OCI users, accessible from the Console. Your Cloud Shell will appear in the Oracle Cloud Console as a persistent frame of the Console and will stay active as you navigate to different pages of the Console. You can find detailed instructions on how to get started with the OCI Cloud Shell herecloudshell-1.png

mrossi@cloudshell:~ (us-phoenix-1)$ mkdir packer-cloud-test
mrossi@cloudshell:~ (us-phoenix-1)$ cd packer-cloud-test/

An Existing public subnet in an existing Virtual Cloud network

In order to create a custom image, packer needs to deploy an OCI instance, and then connect to it and apply the customizations defined in the configuration file, so you will need to deploy a VCN and set it up so that you are able to ssh into it from the cloud console instance,
For an example of how to do that using terraform, you can use this tutorial: OCI cli Practice 4: Create another VCN with one public subnet

Packer

Download and uncompress the Packer binary

mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$ wget "https://releases.hashicorp.com/packer/1.5.4/packer_1.5.4_linux_amd64.zip"
--2020-03-05 12:59:15--  https://releases.hashicorp.com/packer/1.5.4/packer_1.5.4_linux_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.25.183, 2a04:4e42:6::439
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.25.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 48051150 (46M) [application/zip]
Saving to: ‘packer_1.5.4_linux_amd64.zip’

100%[===============================================================================================================================================>] 48,051,150  53.2MB/s   in 0.9s   

2020-03-05 12:59:16 (53.2 MB/s) - ‘packer_1.5.4_linux_amd64.zip’ saved [48051150/48051150]

mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$ unzip packer_1.5.4_linux_amd64.zip
Archive:  packer_1.5.4_linux_amd64.zip
  inflating: packer                  
mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$ ./packer
Usage: packer [--version] [--help] <command> [<args>]

Available commands are:
    build       build image(s) from template
    console     creates a console for testing variable interpolation
    fix         fixes templates from old versions of packer
    inspect     see components of a template
    validate    check that a template is valid
    version     Prints the Packer version

mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$

cloudshell-2.png

Packer build file

Download the Packer build file from here

mrossi@cloudshell:oci-packer-customimage (us-phoenix-1)$ wget https://gist.githubusercontent.com/mattiarossi/8321dc84e305b59a7969cf7bf856909f/raw/33ed77d20a42cc41b75a63178ec0cf0e48992a6e/packer-oci-autonomous-instantclient.json
--2020-03-05 12:04:15--  https://gist.githubusercontent.com/mattiarossi/8321dc84e305b59a7969cf7bf856909f/raw/33ed77d20a42cc41b75a63178ec0cf0e48992a6e/packer-oci-autonomous-instantclient.json
Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 151.101.196.133
Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|151.101.196.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1698 (1.7K) [text/plain]
Saving to: ‘packer-oci-autonomous-instantclient.json’

100%[==============================================================================================================================================================================================================================>] 1,698       --.-K/s   in 0s      

2020-03-05 12:04:15 (7.32 MB/s) - ‘packer-oci-autonomous-instantclient.json�� saved [1698/1698]

mrossi@cloudshell:oci-packer-customimage (us-phoenix-1)$ cat packer-oci-autonomous-instantclient.json
{
  "variables": {
      "user_ocid":"{{env `user_ocid`}}",
      "tenancy_ocid": "{{env `tenancy_ocid`}}",
      "fingerprint":"{{env `fingerprint`}}",
      "private_key_path":"{{env `private_key_path`}}",
      "availability_domain": "{{env `availability_domain`}}",
      "region": "{{env `region`}}",
      "base_image_ocid": "{{env `base_image_ocid`}}",
      "compartment_ocid": "{{env `compartment_ocid`}}",
      "subnet_ocid": "{{env `subnet_ocid`}}"
  },  

  "builders": [
     {
      "user_ocid":"{{user `user_ocid`}}",
      "tenancy_ocid": "{{user `tenancy_ocid`}}",
      "fingerprint":"{{user `fingerprint`}}",
      "key_file":"{{user `private_key_path`}}",
      "availability_domain": "{{user `availability_domain`}}",
      "region": "{{user `region`}}",
      "base_image_ocid": "{{user `base_image_ocid`}}",
      "compartment_ocid": "{{user `compartment_ocid`}}",
      "image_name": "autonomous-instantclient-demo",
      "shape": "VM.Standard2.1",
      "ssh_username": "opc",
      "ssh_password": "aaaaatrbfgyreyuuyreb",
      "subnet_ocid": "{{user `subnet_ocid`}}",
      "type": "oracle-oci"
    }
  ],
  "provisioners": [
     {
      "type": "shell",
      "inline": [
        "sudo -u root yum-config-manager --add-repo  http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64",
        "sudo -u root yum -y erase oracle-instantclient18.3-basic",
        "sudo -u root yum -y install oracle-instantclient19.5-devel     oracle-instantclient19.5-tools     oracle-instantclient19.5-jdbc     oracle-instantclient19.5-sqlplus",
        "sqlplus -v",
        "/usr/lib/oracle/19.5/client64/bin/imp HELP=Y"
      ],
      "pause_before": "5s"
     }
  ]
}

Setup Environment Variables needed by Packer:

The Packer build file uses some environment variables to deploy the custom image in the proper OCI tenancy.
You will have to follow these instructions to create valid API keys:
Once you have successfully created valid API keys, you will have the following details handy:

  • user_ocid : ID of the user that has been configured with valid API keys
  • tenancy_ocid: ID of the tenancy where the Custom image needs to be created, if it is the same as the Cloud console tenancy use the provided script to retrieve it directly
  • fingerprint: your API key Fingerprint
  • private_key_path: a copy of the private key that has been configured for API access

In addition to that, some other env variables will need to be set up to point the Packer builder to the correct OCI environment

  • compartment_ocid : ID of the Compartment that will host the temporary Instance
  • base_image_ocid : ID of the base image that will be used as a source, in our case: Autonomous Linux 7
  • subnet_ocid : ID of the Subnet that will host

You can use this shell file as an example

mrossi@cloudshell:oci-packer-customimage (us-phoenix-1)$ cat setup_oci_vars.sh
export user_ocid="ocid1.user.oc1..<your user ocid>"
export tenancy_ocid="`cat /etc/oci/config  | grep tenancy | uniq | cut -d "=" -f 2`"
export compartment_ocid="ocid1.compartment.oc1..<id of the compartment that will host the temporary instance>"
export fingerprint="<your API key fingerprint>"
export private_key_path="<path to your OCI api private key>"
export region="us-phoenix-1"
#Use `oci iam  availability-domain list` to get a list ov the availability domains in the current region
export availability_domain="zTzD:PHX-AD-1"

#OCID of the base image
#Use:
#oci compute image list --compartment-id $compartment_ocid | jq '.data[] | {image:."display-name", id:."id"}| select ( .image | contains("Autonomous"))'
# to get a list of the available ocids in the current region, if you need to deploy an image in a different region, adjust the oci cli command or get the id from the Web console
export base_image_ocid="ocid1.image.oc1.phx.aaaaaaaa3ocrbp42shmxitpk5nqphpptnnvtk2grrkzfnap4idejfcpv2q4a"

#Subnet id of a suitable OCI Subnet, the packer client needs to be able to SSH to it in order to deploy additional configuration
#Use:
#oci network subnet list --compartment-id $compartment_ocid --vcn-id $vcn_id | jq '.data[] | {subnet:."display-name", id:."id"}'
#to get a list of subnets in a given VCN
export subnet_ocid="ocid1.subnet.oc1.phx....v7q7ybja"

And, once all fields have been updated, you need to source it:

mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$ . setup_oci_vars.sh

mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$ env | grep -e ocid -e key
compartment_ocid=ocid1.compartment.oc1..aaaa...f6vba
private_key_path=/tmp/oci_api_key.pem
base_image_ocid=ocid1.image.oc1.phx.aaa...jfcpv2q4a
user_ocid=ocid1.user.oc1..aaaaaaaavpt2mna...wfucqlqutg2na
tenancy_ocid=ocid1.tenancy.oc1..aaaa...3u3shj3kxq
subnet_ocid=ocid1.subnet.oc1.phx.aaaaa...bha44siv7q7ybja

Run packer

Now we are ready to run Packer and create our custom image

mrossi@cloudshell:packer-cloud-test (us-phoenix-1)$ ./packer build packer-oci-autonomous-instantclient.json
oracle-oci: output will be in this color.

==> oracle-oci: Creating temporary ssh key for instance...
==> oracle-oci: Creating instance...
==> oracle-oci: Created instance (ocid1.instance.oc1.phx.anyhqljtu24ak7acd3vzqmhazefic7zo6bebsbqz3xvmnq7e23vqexiex2hq).
==> oracle-oci: Waiting for instance to enter 'RUNNING' state...
==> oracle-oci: Instance 'RUNNING'.
==> oracle-oci: Instance has IP: 158.101.23.210.
==> oracle-oci: Using ssh communicator to connect: 158.101.23.210
==> oracle-oci: Waiting for SSH to become available...
==> oracle-oci: Connected to SSH!
==> oracle-oci: Pausing 5s before the next provisioner...
==> oracle-oci: Provisioning with shell script: /tmp/packer-shell958521454
    oracle-oci: Loaded plugins: langpacks
    oracle-oci: adding repo from: http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64
    oracle-oci:
    oracle-oci: [yum.oracle.com_repo_OracleLinux_OL7_oracle_instantclient_x86_64]
    oracle-oci: name=added from: http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64
    oracle-oci: baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64
    oracle-oci: enabled=1
    oracle-oci:
    oracle-oci:
    oracle-oci: Loaded plugins: langpacks, ulninfo
    oracle-oci: Resolving Dependencies
    oracle-oci: --> Running transaction check
    oracle-oci: ---> Package oracle-instantclient18.3-basic.x86_64 0:18.3.0.0.0-2 will be erased
    oracle-oci: --> Processing Dependency: oracle-instantclient18.3-basic >= 18.3.0.0.0 for package: python-cx_Oracle-7.0-1.0.1.el7.x86_64
    oracle-oci: --> Running transaction check
    oracle-oci: ---> Package python-cx_Oracle.x86_64 0:7.0-1.0.1.el7 will be erased
    oracle-oci: --> Processing Dependency: python-cx_Oracle = 7.0 for package: python-oci-cli-2.9.0-1.el7.noarch
    oracle-oci: --> Running transaction check
    oracle-oci: ---> Package python-oci-cli.noarch 0:2.9.0-1.el7 will be erased
    oracle-oci: --> Processing Dependency: python-oci-cli for package: al-config-1.0-3.el7.noarch
    oracle-oci: --> Running transaction check
    oracle-oci: ---> Package al-config.noarch 0:1.0-3.el7 will be erased
    oracle-oci: --> Finished Dependency Resolution
    oracle-oci:
    oracle-oci: Dependencies Resolved
    oracle-oci:
    oracle-oci: ================================================================================
    oracle-oci:  Package                            Arch       Version           Repository
    oracle-oci:                                                                            Size
    oracle-oci: ================================================================================
    oracle-oci: Removing:
    oracle-oci:  oracle-instantclient18.3-basic     x86_64     18.3.0.0.0-2      @al7     220 M
    oracle-oci: Removing for dependencies:
    oracle-oci:  al-config                          noarch     1.0-3.el7         @al7      37 k
    oracle-oci:  python-cx_Oracle                   x86_64     7.0-1.0.1.el7     @al7     1.3 M
    oracle-oci:  python-oci-cli                     noarch     2.9.0-1.el7       @al7     9.9 M
    oracle-oci:
    oracle-oci: Transaction Summary
    oracle-oci: ================================================================================
    oracle-oci: Remove  1 Package (+3 Dependent packages)
    oracle-oci:
    oracle-oci: Installed size: 232 M
    oracle-oci: Downloading packages:
    oracle-oci: Running transaction check
    oracle-oci: Running transaction test
    oracle-oci: Transaction test succeeded
    oracle-oci: Running transaction
    oracle-oci:   Erasing    : al-config-1.0-3.el7.noarch                                   1/4
    oracle-oci:   Erasing    : python-oci-cli-2.9.0-1.el7.noarch                            2/4
    oracle-oci:   Erasing    : python-cx_Oracle-7.0-1.0.1.el7.x86_64                        3/4
    oracle-oci:   Erasing    : oracle-instantclient18.3-basic-18.3.0.0.0-2.x86_64           4/4
    oracle-oci:   Verifying  : python-oci-cli-2.9.0-1.el7.noarch                            1/4
    oracle-oci:   Verifying  : python-cx_Oracle-7.0-1.0.1.el7.x86_64                        2/4
    oracle-oci:   Verifying  : al-config-1.0-3.el7.noarch                                   3/4
    oracle-oci:   Verifying  : oracle-instantclient18.3-basic-18.3.0.0.0-2.x86_64           4/4
    oracle-oci:
    oracle-oci: Removed:
    oracle-oci:   oracle-instantclient18.3-basic.x86_64 0:18.3.0.0.0-2
    oracle-oci:
    oracle-oci: Dependency Removed:
    oracle-oci:   al-config.noarch 0:1.0-3.el7         python-cx_Oracle.x86_64 0:7.0-1.0.1.el7
    oracle-oci:   python-oci-cli.noarch 0:2.9.0-1.el7
    oracle-oci:
    oracle-oci: Complete!
    oracle-oci: Loaded plugins: langpacks, ulninfo
    oracle-oci: Resolving Dependencies
    oracle-oci: --> Running transaction check
    oracle-oci: ---> Package oracle-instantclient19.5-devel.x86_64 0:19.5.0.0.0-1 will be installed
    oracle-oci: --> Processing Dependency: oracle-instantclient19.5-basic >= 19.5.0.0.0 for package: oracle-instantclient19.5-devel-19.5.0.0.0-1.x86_64
    oracle-oci: ---> Package oracle-instantclient19.5-jdbc.x86_64 0:19.5.0.0.0-1 will be installed
    oracle-oci: ---> Package oracle-instantclient19.5-sqlplus.x86_64 0:19.5.0.0.0-1 will be installed
    oracle-oci: ---> Package oracle-instantclient19.5-tools.x86_64 0:19.5.0.0.0-1 will be installed
    oracle-oci: --> Running transaction check
    oracle-oci: ---> Package oracle-instantclient19.5-basic.x86_64 0:19.5.0.0.0-1 will be installed
    oracle-oci: --> Finished Dependency Resolution
    oracle-oci:
    oracle-oci: Dependencies Resolved
    oracle-oci:
    oracle-oci: ================================================================================
    oracle-oci:  Package                          Arch   Version      Repository           Size
    oracle-oci: ================================================================================
    oracle-oci: Installing:
    oracle-oci:  oracle-instantclient19.5-devel   x86_64 19.5.0.0.0-1 yum.oracle.com_repo_OracleLinux_OL7_oracle_instantclient_x86_64
    oracle-oci:                                                                           598 k
    oracle-oci:  oracle-instantclient19.5-jdbc    x86_64 19.5.0.0.0-1 yum.oracle.com_repo_OracleLinux_OL7_oracle_instantclient_x86_64
    oracle-oci:                                                                           1.5 M
    oracle-oci:  oracle-instantclient19.5-sqlplus x86_64 19.5.0.0.0-1 yum.oracle.com_repo_OracleLinux_OL7_oracle_instantclient_x86_64
    oracle-oci:                                                                           686 k
    oracle-oci:  oracle-instantclient19.5-tools   x86_64 19.5.0.0.0-1 yum.oracle.com_repo_OracleLinux_OL7_oracle_instantclient_x86_64
    oracle-oci:                                                                           818 k
    oracle-oci: Installing for dependencies:
    oracle-oci:  oracle-instantclient19.5-basic   x86_64 19.5.0.0.0-1 yum.oracle.com_repo_OracleLinux_OL7_oracle_instantclient_x86_64
    oracle-oci:                                                                            51 M
    oracle-oci:
    oracle-oci: Transaction Summary
    oracle-oci: ================================================================================
    oracle-oci: Install  4 Packages (+1 Dependent package)
    oracle-oci:
    oracle-oci: Total download size: 55 M
    oracle-oci: Installed size: 236 M
    oracle-oci: Downloading packages:
    oracle-oci: --------------------------------------------------------------------------------
    oracle-oci: Total                                               44 MB/s |  55 MB  00:01
    oracle-oci: Running transaction check
    oracle-oci: Running transaction test
    oracle-oci: Transaction test succeeded
    oracle-oci: Running transaction
    oracle-oci:   Installing : oracle-instantclient19.5-basic-19.5.0.0.0-1.x86_64           1/5
    oracle-oci:   Installing : oracle-instantclient19.5-devel-19.5.0.0.0-1.x86_64           2/5
    oracle-oci:   Installing : oracle-instantclient19.5-jdbc-19.5.0.0.0-1.x86_64            3/5
    oracle-oci:   Installing : oracle-instantclient19.5-tools-19.5.0.0.0-1.x86_64           4/5
    oracle-oci:   Installing : oracle-instantclient19.5-sqlplus-19.5.0.0.0-1.x86_64         5/5
    oracle-oci:   Verifying  : oracle-instantclient19.5-devel-19.5.0.0.0-1.x86_64           1/5
    oracle-oci:   Verifying  : oracle-instantclient19.5-basic-19.5.0.0.0-1.x86_64           2/5
    oracle-oci:   Verifying  : oracle-instantclient19.5-jdbc-19.5.0.0.0-1.x86_64            3/5
    oracle-oci:   Verifying  : oracle-instantclient19.5-tools-19.5.0.0.0-1.x86_64           4/5
    oracle-oci:   Verifying  : oracle-instantclient19.5-sqlplus-19.5.0.0.0-1.x86_64         5/5
    oracle-oci:
    oracle-oci: Installed:
    oracle-oci:   oracle-instantclient19.5-devel.x86_64 0:19.5.0.0.0-1
    oracle-oci:   oracle-instantclient19.5-jdbc.x86_64 0:19.5.0.0.0-1
    oracle-oci:   oracle-instantclient19.5-sqlplus.x86_64 0:19.5.0.0.0-1
    oracle-oci:   oracle-instantclient19.5-tools.x86_64 0:19.5.0.0.0-1
    oracle-oci:
    oracle-oci: Dependency Installed:
    oracle-oci:   oracle-instantclient19.5-basic.x86_64 0:19.5.0.0.0-1
    oracle-oci:
    oracle-oci: Complete!
    oracle-oci:
    oracle-oci: SQL*Plus: Release 19.0.0.0.0 - Production
    oracle-oci: Version 19.5.0.0.0
    oracle-oci:
==> oracle-oci:
==> oracle-oci: Import: Release 19.0.0.0.0 - Production on Thu Mar 5 11:02:37 2020
==> oracle-oci: Version 19.5.0.0.0
==> oracle-oci:
==> oracle-oci: Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
==> oracle-oci:
==> oracle-oci:
==> oracle-oci:
==> oracle-oci: You can let Import prompt you for parameters by entering the IMP
==> oracle-oci: command followed by your username/password:
==> oracle-oci:
==> oracle-oci:      Example: IMP SCOTT/TIGER
==> oracle-oci:
==> oracle-oci: Or, you can control how Import runs by entering the IMP command followed
==> oracle-oci: by various arguments. To specify parameters, you use keywords:
==> oracle-oci:
==> oracle-oci:      Format:  IMP KEYWORD=value or KEYWORD=(value1,value2,...,valueN)
==> oracle-oci:      Example: IMP SCOTT/TIGER IGNORE=Y TABLES=(EMP,DEPT) FULL=N
==> oracle-oci:                or TABLES=(T1:P1,T1:P2), if T1 is partitioned table
==> oracle-oci:
==> oracle-oci: USERID must be the first parameter on the command line.
==> oracle-oci:
==> oracle-oci: Keyword  Description (Default)       Keyword      Description (Default)
==> oracle-oci: --------------------------------------------------------------------------
==> oracle-oci: USERID   username/password           FULL         import entire file (N)
==> oracle-oci: BUFFER   size of data buffer         FROMUSER     list of owner usernames
==> oracle-oci: FILE     input files (EXPDAT.DMP)    TOUSER       list of usernames
==> oracle-oci: SHOW     just list file contents (N) TABLES       list of table names
==> oracle-oci: IGNORE   ignore create errors (N)    RECORDLENGTH length of IO record
==> oracle-oci: GRANTS   import grants (Y)           INCTYPE      incremental import type
==> oracle-oci: INDEXES  import indexes (Y)          COMMIT       commit array insert (N)
==> oracle-oci: ROWS     import data rows (Y)        PARFILE      parameter filename
==> oracle-oci: LOG      log file of screen output   CONSTRAINTS  import constraints (Y)
==> oracle-oci: DESTROY                overwrite tablespace data file (N)
==> oracle-oci: INDEXFILE              write table/index info to specified file
==> oracle-oci: SKIP_UNUSABLE_INDEXES  skip maintenance of unusable indexes (N)
==> oracle-oci: FEEDBACK               display progress every x rows(0)
==> oracle-oci: TOID_NOVALIDATE        skip validation of specified type ids
==> oracle-oci: FILESIZE               maximum size of each dump file
==> oracle-oci: STATISTICS             import precomputed statistics (always)
==> oracle-oci: RESUMABLE              suspend when a space related error is encountered(N)
==> oracle-oci: RESUMABLE_NAME         text string used to identify resumable statement
==> oracle-oci: RESUMABLE_TIMEOUT      wait time for RESUMABLE
==> oracle-oci: COMPILE                compile procedures, packages, and functions (Y)
==> oracle-oci: STREAMS_CONFIGURATION  import streams general metadata (Y)
==> oracle-oci: STREAMS_INSTANTIATION  import streams instantiation metadata (N)
==> oracle-oci: DATA_ONLY              import only data (N)
==> oracle-oci: VOLSIZE                number of bytes in file on each volume of a file on tape
==> oracle-oci:
==> oracle-oci: The following keywords only apply to transportable tablespaces
==> oracle-oci: TRANSPORT_TABLESPACE import transportable tablespace metadata (N)
==> oracle-oci: TABLESPACES tablespaces to be transported into database
==> oracle-oci: DATAFILES datafiles to be transported into database
==> oracle-oci: TTS_OWNERS users that own data in the transportable tablespace set
==> oracle-oci:
==> oracle-oci: Import terminated successfully without warnings.
==> oracle-oci: Creating image from instance...
==> oracle-oci: Creating image from instance...
==> oracle-oci: Image created.
==> oracle-oci: Terminating instance (ocid1.instance.oc1.phx.anyhqljtu24ak7acd3vzqmhazefic7zo6bebsbqz3xvmnq7e23vqexiex2hq)...
==> oracle-oci: Terminated instance.
Build 'oracle-oci' finished.

Verify our Custom image is available

We can use either the console or the oci cli to check the availability of our newly created image:

oci compute image list  --compartment-id $compartment_ocid | jq -r '.data | map(select(."time-created" != null)) | sort_by(."time-created")[]| [."display-name",."operating-system",."time-created"]'

...

[
  "CentOS-6.10-2020.02.24-0",
  "CentOS",
  "2020-02-24T20:57:32.240000+00:00"
]
[
  "autonomous-instantclient-demo",
  "Oracle Autonomous Linux",
  "2020-03-05T11:02:39.026000+00:00"
]


webconsole-1.png

 

This pattern applies to an infinite set of possibilities to automate the creation of custom images in OCI (and other Cloud and Legacy Platforms), the limit is your fantasy (and possibly the time you have to play with it …..) Have Fun!
Mattia