How to use Terraform Data sources?
Terraform data sources can be beneficial if you want to retrieve or fetch the data from the cloud service providers such as AWS, AZURE, and GCP. Most of the time when we use Terraform along with AWS/AZURE/GCP then we always send data in terms of instructions or configuration.
But what if you want to get the information(arn, tags, owner_id, etc.) back from the cloud service provider AWS/AZURE/GCP?
Answer - We need to use the data sources to get the resource information back.
So Terraform Data Sources are a kind of an API that fetches the data/information from the resources running under the cloud infra and sending it back to terraform configuration for further use.
In this blog, we will look at the example in which we are going to create an aws_instance
resource and then create a data source to fetch some of the information associated with the aws_instance
.
Table of Content
- Create an aws_instance
- Define a data source
- Create Output variable for data source
- Apply the final terraform configuration along with data source and output values
- Fetching only specific attribute using data source
1. Create an aws_instance
The motive of this exercise is to create an aws_instance
and then create a data source to fetch all the possible Data Source: aws_instance
attributes.
Let's first write the terraform configuration for starting a t2.mirco
aws_instance.
(*Note- Replace the access_key and secret_key with your AWS account. Click here to know how to generate the access_key and secret_key )
1provider "aws" {
2 region = "eu-central-1"
3 access_key = "AKIATQ37NXB2JMXVGYPG"
4 secret_key = "ockvEN1DzYynDuKIh56BVQv/tMqmzvKnYB8FttSp"
5}
6
7resource "aws_instance" "ec2_example" {
8
9 ami = "ami-0767046d1677be5a0"
10 instance_type = "t2.micro"
11
12 tags = {
13 Name = "Terraform EC2"
14 }
15}
2. Define a data source
Now we have created our aws_instance in Step 1, let's add the data source to the existing terraform configuration.
Here is the data source configuration for fetching all the information of aws_instance -
1data "aws_instance" "myawsinstance" {
2 filter {
3 name = "tag:Name"
4 values = ["Terraform EC2"]
5 }
6
7 depends_on = [
8 "aws_instance.ec2_example"
9 ]
10}
Key points to pay attention for -
- filter: Although we have created only one instance but still we have used filter because in a production-like environment you might have multiple aws_instance running, so you need to filter the instance anyhow. And since we have tagged our aws_instance with the name Terraform EC2 so we are going to use the same name inside the filter also.
- depends_on: The second important parameter is depends_on because
data source
does not know by its own which resource it belongs to, so we are going to add the depends_on parameter.
3. Create Output variable for data source
So far in Step 1 and Step 2 we have created the aws_instance and data source, now let's create an output value
so that we can see all the information fetched or retrieved by the data source.
Here is the terraform configuration for the output value -
1output "fetched_info_from_aws" {
2 value = data.aws_instance.myawsinstance
3}
Key points to pay attention for -
- We have linked the output value to the data source which we have created in Step 2.
- To link the output value we are going to use the data source name .i.e. -
data.aws_instance.myawsinstance
4. Apply the final terraform configuration along with data source and output values
Alright now I am assuming you have gone through all the 3 steps(Step 1,Step 2, and Step 3), so here is our final terraform configuration including aws_instance, data source, and output values
1provider "aws" {
2 region = "eu-central-1"
3 access_key = "AKIATQ37NXB2JMXVGYPG"
4 secret_key = "ockvEN1DzYynDuKIh56BVQv/tMqmzvKnYB8FttSp"
5}
6
7resource "aws_instance" "ec2_example" {
8
9 ami = "ami-0767046d1677be5a0"
10 instance_type = "t2.micro"
11
12 tags = {
13 Name = "Terraform EC2"
14 }
15}
16
17data "aws_instance" "myawsinstance" {
18 filter {
19 name = "tag:Name"
20 values = ["Terraform EC2"]
21 }
22
23 depends_on = [
24 "aws_instance.ec2_example"
25 ]
26}
27
28output "fetched_info_from_aws" {
29 value = data.aws_instance.myawsinstance.public_ip
30}
You can simply run the following terraform command to create your aws_instance -
1terraform init
1terraform plan
1terraform apply
Here is the output after applying to terraform configuration -
1Outputs:
2
3fetched_info_from_aws = {
4 "ami" = "ami-0767046d1677be5a0"
5 "arn" = "arn:aws:ec2:eu-central-1:242396018804:instance/i-0eda1c6a59790eb7d"
6 "associate_public_ip_address" = true
7 "availability_zone" = "eu-central-1c"
8 "credit_specification" = tolist([
9 {
10 "cpu_credits" = "standard"
11 },
12 ])
13 "disable_api_termination" = false
14 "ebs_block_device" = toset([])
15 "ebs_optimized" = false
16 "enclave_options" = tolist([
17 {
18 "enabled" = false
19 },
20 ])
21 "ephemeral_block_device" = tolist([])
22 "filter" = toset([
23 {
24 "name" = "tag:Name"
25 "values" = tolist([
26 "Terraform EC2",
27 ])
28 },
29 ])
30 "get_password_data" = false
31 "get_user_data" = false
32 "host_id" = tostring(null)
33 "iam_instance_profile" = ""
34 "id" = "i-0eda1c6a59790eb7d"
35 "instance_id" = tostring(null)
36 "instance_state" = "running"
37 "instance_tags" = tomap(null) /* of string */
38 "instance_type" = "t2.micro"
39 "key_name" = ""
40 "metadata_options" = tolist([
41 {
42 "http_endpoint" = "enabled"
43 "http_put_response_hop_limit" = 1
44 "http_tokens" = "optional"
45 },
46 ])
47 "monitoring" = false
48 "network_interface_id" = "eni-0ffc9d62eafcafcbc"
49 "outpost_arn" = ""
50 "password_data" = tostring(null)
51 "placement_group" = ""
52 "private_dns" = "ip-172-31-9-122.eu-central-1.compute.internal"
53 "private_ip" = "172.31.9.122"
54 "public_dns" = "ec2-3-122-249-219.eu-central-1.compute.amazonaws.com"
55 "public_ip" = "3.122.249.219"
56 "root_block_device" = toset([
57 {
58 "delete_on_termination" = true
59 "device_name" = "/dev/sda1"
60 "encrypted" = false
61 "iops" = 100
62 "kms_key_id" = ""
63 "tags" = tomap({})
64 "throughput" = 0
65 "volume_id" = "vol-0fce01580b0175da8"
66 "volume_size" = 8
67 "volume_type" = "gp2"
68 },
69 ])
70 "secondary_private_ips" = toset([])
71 "security_groups" = toset([
72 "default",
73 ])
74 "source_dest_check" = true
75 "subnet_id" = "subnet-2183316d"
76 "tags" = tomap({
77 "Name" = "Terraform EC2"
78 })
79 "tenancy" = "default"
80 "user_data" = tostring(null)
81 "user_data_base64" = tostring(null)
82 "vpc_security_group_ids" = toset([
83 "sg-272bd157",
84 ])
85}
Here is the screenshot from aws -
- aws_instance is up and running -
- aws_instance details (you can verify the output from step 4) -
5. Fetching only specific attribute using data source
Now let's one step more further and instead of fetching all the attributes of the aws_instance let's only fetch the public_ip
.
You only need to update the output value
configuration -
1output "fetched_info_from_aws" {
2 value = data.aws_instance.myawsinstance.public_ip
3}
Here is the public_ip
information fetched by the data source -
Read More - Terragrunt -
Posts in this Series
- Securing Sensitive Data in Terraform
- Boost Your AWS Security with Terraform : A Step-by-Step Guide
- How to Load Input Data from a File in Terraform?
- Can Terraform be used to provision on-premises infrastructure?
- Fixing the Terraform Error creating IAM Role. MalformedPolicyDocument Has prohibited field Resource
- In terraform how to handle null value with default value?
- Terraform use module output variables as inputs for another module?
- How to Reference a Resource Created by a Terraform Module?
- Understanding Terraform Escape Sequences
- How to fix private-dns-enabled cannot be set because there is already a conflicting DNS domain?
- Use Terraform to manage AWS IAM Policies, Roles and Users
- How to split Your Terraform main.tf File into Multiple Files
- How to use Terraform variable within variable
- Mastering the Terraform Lookup Function for Dynamic Keys
- Copy files to EC2 and S3 bucket using Terraform
- Troubleshooting Error creating EC2 Subnet InvalidSubnet Range The CIDR is Invalid
- Troubleshooting InvalidParameter Security group and subnet belong to different networks
- Managing strings in Terraform: A comprehensive guide
- How to use terraform depends_on meta argument?
- What is user_data in Terraform?
- Why you should not store terraform state file(.tfstate) inside Git Repository?
- How to import existing resource using terraform import comand?
- Terraform - A detailed guide on setting up ALB(Application Load Balancer) and SSL?
- Testing Infrastructure as Code with Terraform?
- How to remove a resource from Terraform state?
- What is Terraform null Resource?
- In terraform how to skip creation of resource if the resource already exist?
- How to setup Virtual machine on Google Cloud Platform
- How to use Terraform locals?
- Terraform Guide - Docker Containers & AWS ECR(elastic container registry)?
- How to generate SSH key in Terraform using tls_private_key?
- How to fix-Terraform Error acquiring the state lock ConditionalCheckFiledException?
- Terraform Template - A complete guide?
- How to use Terragrunt?
- Terraform and AWS Multi account Setup?
- Terraform and AWS credentials handling?
- How to fix-error configuring S3 Backend no valid credential sources for S3 Backend found?
- Terraform state locking using DynamoDB (aws_dynamodb_table)?
- Managing Terraform states?
- Securing AWS secrets using HashiCorp Vault with Terraform?
- How to use Workspaces in Terraform?
- How to run specific terraform resource, module, target?
- How Terraform modules works?
- Secure AWS EC2s & GCP VMs with Terraform SSH Keys!
- What is terraform provisioner?
- Is terraform destroy needed before terraform apply?
- How to fix terraform error Your query returned no results. Please change your search criteria and try again?
- How to use Terraform Data sources?
- How to use Terraform resource meta arguments?
- How to use Terraform Dynamic blocks?
- Terraform - How to nuke AWS resources and save additional AWS infrastructure cost?
- Understanding terraform count, for_each and for loop?
- How to use Terraform output values?
- How to fix error configuring Terraform AWS Provider error validating provider credentials error calling sts GetCallerIdentity SignatureDoesNotMatch?
- How to fix Invalid function argument on line in provider credentials file google Invalid value for path parameter no file exists
- How to fix error value for undeclared variable a variable named was assigned on the command line?
- What is variable.tf and terraform.tfvars?
- How to use Terraform Variables - Locals,Input,Output
- Terraform create EC2 Instance on AWS
- How to fix Error creating service account googleapi Error 403 Identity and Access Management (IAM) API has not been used in project before or it is disabled
- Install terraform on Ubuntu 20.04, CentOS 8, MacOS, Windows 10, Fedora 33, Red hat 8 and Solaris 11