XenonStack Recommends

Deployment Automation

Infrastructure as Code using Terraform - IaC Complete Guide

Gursimran Singh | 25 May 2023

Infrastructure as Code using Terraform

Introduction to Terraform

Earlier, it was a challenging and hectic job to manage IT infrastructures. The System Administrator had to manually manage all the underlying services, hardware, and software needed for the entire application system to work.

In recent years, the IT industry has seen a revolution in terms of tools and technologies that work towards automating the process of deploying and managing the Infrastructure. And this is what we term it as IaC.

Writing Infrastructure in configuration files or code is referred to as Infrastructure as code. This code acts as the single source of truth for all the components required in the Infrastructure, such as Load Balancers, Virtual Machines, Gateways, etc.
Various IaC tools are available, each solving its purpose, but now the question comes: which tool will best suit your goal.
Some of the popular IaC tools are:-

  1. Terraform
  2. Puppet
  3. Ansible
  4. Chef
  5. Cloud formation
The Infrastructure a Code helps DevOps teams use technologies to manage and configure infrastructure. Taken From Article, Principles of Infrastructure as Code

What is Terraform?

It is a tool used for changing, building, and versioning infrastructure efficiently and safely. It is used for managing existing and popular service providers or custom in-house solutions. The components running for a single application or entire data center Configuration files describe it. An execution plan is generated by it, which defines what it will do to reach the desired state and then executes it to build the described infrastructure. By the time configuration changes, it can determine what changed and create incremental execution plans that can be applied. Infrastructure Terraform manages low-level components such as compute instances and networking, as well as high-level components such as DNS entries, SaaS features, etc.
  1. Infrastructure as code.
  2. It is a tool to manage virtual server life cycles (AWS, VMWare, etc.)
  3. It is a tool to manage supporting services (DNS, Email)
  4. It is a tool to manage system services (MySQL, PostgreSQL)
  5. Configuration files can be HCL or JSON.
  6. Created by Hashicorp (Vagrant et al.)
  7. Written in Go

Why Infrastructure as Code is important?

The main goal of DevOps is to perform more efficient software delivery and to make delivery efficient and quick. We need some tools like it, which help companies with infrastructure as a code and automation. It is changing the DevOps world by changing the way infrastructure is managed and making it faster and more efficient for executing DevOps. There exist technologies like Ansible, Chef, or Puppet for automating and provisioning software. It also uses the same law, “infrastructure as code,” but it targets the automation of infrastructure itself. Complete cloud infrastructure like instances, IPs, volumes, and networking can be defined easily.

Why should you use it to manage your Infrastructure?

The below mentioned are the usages of it:

An orchestrator and not an Automation Tool

What do we mean by that? Automation focuses on a single task, whereas orchestration means creating a workflow and combining multiple automation tasks.

It follows a Declarative approach and not a Procedural

In a declarative approach, you will tell what you need and not how it is to be done. Just say what you want in your Infrastructure, and it will manage all the necessary steps to get things done.

Multiple Provider Support

It comes with the feature to provide support for multiple public cloud service providers like Google Cloud Platform, AWS, and Azure.

Kubernetes also follows IaC principles which mean all of our deployment is written in the form of code. Taken From Article, IaC Principles for Kubernetes Configuration Management

How to manage your Infrastructure as Code using Terraform?

It is a tool that lets you manage, build, and version your Infrastructure using configuration files. It is a convenient tool that enables you to deploy a wide range of resources, such as physical servers, networking interfaces, and Load balancers, on various platforms such as AWS, GCP, and Azure. The same code needs to be replicated in different environments, ensuring that the same Infrastructure is applied everywhere.

For example, say you are working with AWS as your cloud provider and would like to spin up several EC2 instances of a specific type. You would define the type and number of instances in a configuration file, and it would use that to communicate with the AWS API to create those instances. The same file could then be used to adjust the configuration, for example, increasing or decreasing the number of instances. Because Infrastructure typically includes many more components than just compute instances, we use a feature called modules to combine multiple infrastructure components into large, reusable, and shareable chunks.

What are the features of Infrastructure as Code?

The features are listed below:

Infrastructure as a Code

A high-level configuration syntax is used for describing support. It allows a blueprint of our data center to be versioned and treated it you would any other code. Also, infrastructure can be shared and re-used.

Execution Plans

It has a “planning” step from which it generates an execution plan. This execution plan shows what it will do when it is applied. This eliminates any surprises when it manipulates infrastructure.

Resource Graph

A graph of all resources is built by it and parallelizes the creation and modification of any non-dependent resources. By doing so, it builds infrastructure as efficiently as possible, and operators get insight into dependencies in their infrastructure.

Change Automation

With minimal human interaction, complex changesets can be applied to infrastructure, what it will change, and in what order can be known by the previously mentioned execution plan and resource graph, which helps in avoiding many possible human errors.

How does Infrastructure as Code work?

It is split into two parts: core and plugins. Its core communicates with its plugin. Its plugins expose an implementation for specific services such as AWS bash.

What is its Core?

It’s a binary written in Go programming language. The compiled binary corresponds to the CLI terraform.
The core is responsible for :

Reading the configuration files, i.e., IaC.
State management of various resources.
Construction of resource graph.
Execution of plan.
Communication with plugins.

What are the Plugins?

Plugins are executable binaries written in Go.

Requirements for Terraform setup

Terraform binary
Configuration files (in which you write IaC)
State file

Terraform configuration files (*.tf)

You write every configuration in files with an extension as .tf. You’ll be mostly dealing with resources in configuration files. Resources are the services or components of your Infrastructure, such as Load Balancers, Virtual machines, etc. And you pass various arguments in a resource you want to set up.
For example:

main.tf
resource "aws_instance" "web" {
ami = data.aws_ami.ubuntu.id
instance_type = "t3.micro"
tags = {
Name = "HelloWorld"
}
}

The above configuration file main.tf, defines a resource, i.e., EC2 instance in AWS. Inside the resource block, we have passed the desired arguments as:
ami- AMI id to be issued for our EC2.
Instance_type
Tags - tags for our EC2
This is how we can define other resources, as well.
We can now use it apply to deploy this resource on AWS.

What is Terraform state?

The most crucial part of it is its state file (terraform.tfstate), which stores all the information related to the deployed Infrastructure and mapping between various components. It then refers to this state file while deploying Infrastructure to check dependencies among resources.

Why is a state file required?

While deploying resources, for example, EC2, as in the example above, you don't specify the ARN or instance ID for the EC2. It gets dynamically assigned, so it needs to store all such information so that internally, it can detect changes if you update the instance_type for the EC2 in the future.

State file is the single source of truth for your Infrastructure, so you need to keep this very safe and ensure that your team always refers to the latest version of terraform.tfstate file to avoid conflicts. By default, the state file is stored locally, but keeping it locally is only beneficial if just one developer is working on managing the Infrastructure. However, the best practice is to keep file in some remote backend such as S3 bucket.

Following example shows the backend configuration as S3 bucket.

terraform {
backend "s3" {
bucket = "mybucket"
key = "path/to/my/key"
region = "us-east-1"
}
}

Backend Initialisation

An initialization step needs to be performed every time there is a change in backend configuration; the command used is it into.
This command does the following:
Initializes backend
Configures its modules (will cover later).

Workflow

Write IAC, i.e., configuration files.
See what changes it will reflect using its plan.
Deploy those changes using it and apply

Plan

The plan will represent a list of changes that your IAC will make. It doesn't actually affect anything, and it just shows what new will be created/deleted/modified. Therefore, it is safe to use it plans multiple times.

Interpreting its Plan

Resources can be easily interpreted by their names. For example,
Syntax: <Resource type>.< resource name>
Example, aws_instance.this.
Resources inside a module can be interpreted by:
Syntax: module.<module_name>.<Resource type>. < resource name>
Example module.my_virtual_machines.aws_instance.this
Its plan shows five actions for the resources:
[+] Add - Creation of new resource.
[ - ] Delete - Removal of existing resource.
[~] Modify in-place - Existing resources will be modified, and some new parameters will be added to them. It plans will also show you which new parameters will be modified/ added.
[- / +] Terraform will delete and re-create the same resource. This happens when it's not possible to modify a parameter in place.

Sample Plan

An execution plan has been generated and is shown below.

20:10:18 Resource actions are indicated with the following symbols:
20:10:18 + create
20:10:18
20:10:18 Terraform will perform the following actions:
20:10:18
20:10:18 # aws_cloudwatch_log_group.log_group_containers[0] will be created
20:10:18 + resource "aws_cloudwatch_log_group" "log_group_containers" {
20:10:18 + arn = (known after apply)
20:10:18 + id = (known after apply)
20:10:18 + name = "/eks/dev-scipher-fx-eks-cluster-2/containers"
20:10:18 + retention_in_days = 14
20:10:18 + tags = {
20:10:18 + "Environment" = "dev"
20:10:18 + "Service" = "EKS Cluster"
20:10:18 + "Terraform" = "true"
20:10:18 }
20:10:18 }
20:10:18
20:10:18 # aws_cloudwatch_log_group.log_group_host[0] will be created
20:10:18 + resource "aws_cloudwatch_log_group" "log_group_host" {
20:10:18 + arn = (known after apply)
20:10:18 + id = (known after apply)
20:10:18 + name = "/eks/dev-scipher-fx-eks-cluster-2/host"
20:10:18 + retention_in_days = 14
20:10:18 + tags = {
20:10:18 + "Environment" = "dev"
20:10:18 + "Service" = "EKS Cluster"
20:10:18 + "Terraform" = "true"
20:10:18 }
20:10:18 }

In the above example, the plan shows:
Two new Cloudwatch groups being created in AWS.
Log groups retention_in_days being set to 14 days
Tags being added to log groups

Terraform Apply

On applying a given plan, it locks the state. It will perform all the actions stated in the planning stage and deploy those resources. Some resources might take time; for example, AWS DocumentDB may take up to 15 minutes. Until that, it will block all other operations.
On completion, it updates the state file.

Modules

All the configuration files, together with a folder, are called a module. Modules allow us to create a reusable code that can be referred to again with the required parameters. Consider it as just like functions in any programming language where we just have to call the process anywhere and pass the necessary set of parameters.
Using a module in a configuration file is similar to using resources, as we have seen above; the only difference is the clause “module.”

module "moduleName" {
source = "module/path"
\CONFIG
/
}

The source path tells where the module can be found.
CONFIG block consists of a set of variables or arguments that can be passed to the module and belong to the module.
For example:
If the main module has a variable log_retention_period, we can pass its value differently for different environments, such as:
For Dev environment:

module "cloud_watch_log_groups" {
source = "module/path"
Log_retention_period = 14}

Similarly for the Prod environment:
module "cloud_watch_log_groups" {
source = "module/path"
Log_retention_period = 21
}

Thus, as you can see, we can reuse the same module to deploy Infrastructure for multiple environments.

What are the Advantages of Infrastructure as Code?

  1. Orchestration, not merely Configuration: For server configuration, all the configuration management tools were created, and to manage and to install the software already existed in the server / virtual machine where their primary goal.
  2. It focuses more on Aerver Provisioning: When the complete cloud infrastructure is considered as code, and all the parameters were combined in a configuration file, all the members of the team can easily collaborate on them as they would do any other code.
  3. Multi-Provider - The most versatile feature of it is that it supports multi-cloud orchestration like AWS, Azure, OpenStack, etc. and also premises deployments. It is beneficial when using two different resources from two various cloud providers at the same time.
  4. Immutable Infrastructure - When using Chef, Puppet, Salt runs any software updates on servers; this often leads to configuration drift when differences in the configuration lead to bugs that lead to security breaches. It solves this issue by using an immutable infrastructure approach in which every change in configuration leads to a separate configuration snapshot which means de-provisioning the old one and deployment of a new one. Updating the development environment in this way goes smooth and bug-free and returning the old configuration is easy, like rolling back to a specific version.
  5. Syntax - HCL (HashiCorp Configuration Language) is a custom language that is used by it.
  6. Dry Runs - It uses a command known as “terraform plan,” which creates an execution plan; by using this execution plan, we can check whether the set of changes meets the expectation without performing any changes to real resources or state. For example, by running a plan before committing a change to version control, check whether it will work as expected or not.
  7. Client-only Architecture - It eliminates the need for additional checks for provisioning the infrastructure, which leverages the cloud provider’s API. In Ansible, this is done by connecting through SSH, but with limitations. It works on APIs and has a wide variety of options, which helps to make it more secure, reliable, and easy to use.
  8. Super Portability - There is only a single tool and single language for describing the infrastructure that is used for multiple cloud providers. The problem of migrating to vendors is not a problem.

What are the Common Use Cases of IaC using Terraform?

The Use Cases are listed below:

Heroku App Setup

Heroku is a PaaS for hosting web applications. Developers create an application and then attach add-ons like a database or email provider. Its best feature is the ability to scale the number of dynos or workers elastically. It is used for codifying the setup required for a Heroku application, ensuring that all necessary add-ons are available. But it can go even further -
  1. Configuring DNSimple to set a CNAME,
  2. By setting Cloudflare as CDN for the app.

Multi-Tier Applications

One of the pervasive patterns is N-tier architecture. A widespread 2-tier architecture is a pool of web servers that uses a database tier. More tiers can be added for API servers, caching servers, routing meshes, etc. To scale tiers independently and provide a separation of concerns, this pattern is used. It is an optimal tool for building and managing these infrastructures. Every tier is described as a collection of resources and automatically handles the dependencies between each tier. Before the web servers are started, the database tier is available will ensure by it and that the load balancers are aware of the web nodes. By using it, each tier can be scaled easily by modifying a single count configuration value because the creation and provisioning of a resource are codified and automated, elastically with load becomes trivial.

Self-Service Clusters

In a specific organizational size, managing a large and growing infrastructure becomes very challenging for a centralized operations team. Instead, to make “self-serve” infrastructure more attractive, using tooling provided by the central operations team allows product teams to manage their infrastructure. By using it, the ability of how to build and scale a service can be codified in a configuration. To enabling customer teams to use the configuration as a black box, configurations can be shared within an organization and to manage their services it is used as a tool.

AI enabled the IT Infrastructure to be flexible, intangible and on-demand. Click to explore about, AI in IT Infrastructure Management

Terraform Comparison with Chef, Ansible, Puppet and Salt

On a machine that already exists, the configuration management tools install and manages. It does the same, and also it allows us to focus on bootstrapping and initializing resources. Configuration management tools can also be used along with it to configure things inside a virtual machine. It makes the configuration management tool to be used to set up a resource once it has been created by using provisioners. When using Docker containers for running applications, these are self-sufficient and will contain the whole configuration of the application. Tools like Chef, Puppet, Ansible, etc. is not needed. But still, need something to manage infrastructure because the container will run anyway in the top of a server/virtual machine. It is used to create infrastructure for containers to run on. Tools such as Chef, Ansible, Puppet, etc. are used as IAS or Infrastructure as Code, but it is best for this because it can even maintain the state of infrastructure.

How to implement Terraform using AWS Services?

There are following steps to implement Terraform (By using AWS)

Setting up AWS Account

(Infrastructure across different types of cloud providers like AWS, Google Cloud, Azure, DigitalOcean, and many others) Cloud hosting services provided by AWS were reliable and scalable. AWS is one of the most popular cloud infrastructure providers.

Installations 

Find the supported package it for the system and download it. It is downloaded as a zip archive, and then the package is unzipped. After installing, verify the installation by opening a terminal session and checking that it is available. Deploying a single server - HCL language is used for writing it code in files with the extension “.tf.” HCL is a declarative language that describes the infrastructure that is wanted, and it will find out how to create it. Infrastructure across different platforms or providers can be created by it, like AWS, Google Cloud, Azure, DigitalOcean, and many others. Configuring the provider is the first step for using it. Deploying a single web server: In the next step, run a web server on this instance.

Deploying a Cluster of Web Servers

To running a single server can be a good start, but a single server can be a single point of failure. If the server is overwhelmed by heavy traffic or the server crashes, users can no longer access the site. To eliminate this run, a cluster of servers, which are routed around servers that go down and on the base of traffic, adjust the size of cluster up or down. There is a lot of work for managing such a cluster manually. Fortunately, AWS will take care of this by using the Auto Scaling Group (ASG).

Deploying a Load Balancer

One more problem to solve before launching the ASG: there are many Instances, for this needs a load balancer to distribute traffic across all of them. There is a lot of work in creating a load balancer that is highly available and scalable. AWS will let take care of this by using an Elastic Load Balancer (ELB) Clean up - After doing experimenting with it, remove all the created resources so AWS doesn’t charge for them. It keeps track of created resources, and cleanup is a breeze.

Command-line interface for Terraform?

The command-line interface (CLI) is used for controlling Terraform.
  1. apply - Use for changes or builds infrastructure
  2. console - Use for interactive console
  3. destroy - Use for impair it – managed infrastructure
  4. fmt - Use for rewriting config files into canonical format
  5. get - Use for downloading and installing modules for configuration
  6. graph - Use for creating a visual graph of its resources
  7. import - Use for importing existing infrastructure into it
  8. init - Use for initializing existing or new it configuration
  9. output - Use for reading output from a state file
  10. plan - Use for generating and showing an execution plan
  11. providers - Use for printing tree of providers used in the configuration
  12. push - Use for uploading this it module to it Enterprise to run
  13. refresh - Use for updating local state file against real resources
  14. show - Use for inspecting it state or plan
  15. taint - Use for manually marking resources for recreation
  16. untaint - Use for manually unmarking a resource as tainted
  17. validate - Use for validating its files
  18. version - Use for printing its version
  19. workspace - Use for workspace management
Java vs Kotlin
IaC helps teams to perform automated configuration steps for building, maintaining Cloud infrastructure with best practices. C Infrastructure as Code Platform

What are the Best Practices of Infrastructure as Code?

The best practices for Infrastructure as Code are:

Code Structure

Writing it code better by having several files split logically like this:

  • main.tf - call modules, locals, and data-sources to create all resources
    variables.tf - contains declarations of variables used in main.tf
    outputs.tf - includes outputs from the resources created in main.tf
    terraform.tfvars: will be used to pass values to variables.

Remote Backend

Use a remote backend such as S3 to store tfstate files. Backends have two main features: state locking and remote state storage. Locking prevents two executions from happening at the same time. And remote state storage allows you to put your state in a remote yet accessible location.

Naming Conventions

This practice is described below:

General Conventions

Use _ (underscore) instead of - (dash) in all resource names, data source names, variable names, and outputs. Beware that existing cloud resources have many hidden restrictions in their naming conventions. Only use lowercase letters and numbers.

Resource and Data Source Arguments

Do not repeat resource type in resource name (not partially, nor completely):
Good: resource "aws_route_table" "public" {}
Bad: resource "aws_route_table" "public_route_table" {}
Include tags argument, if supported by resource as the last real argument, followed by depends_on and lifecycle, if necessary. All of these should be separated by a single empty line.

ReadME

Generate README for each module with input and output variables

Name
Description
Type
Default
Required

instance_count
Number of EC2 instances
String
1
NO
welcome_page
Content to be displayed at home page
String
-
YES

Conclusion

To learn more about Configuration management and Compliance and governance, and provisioning IT infrastructure, we recommend the following steps :