Terraform Your Way to High Availability: Deploying a Full Stack AWS Architecture

 

https://www.nilebits.com/blog/2024/09/terraform-your-way-to-high-availability-deploying-a-full-stack-aws-architecture/

We will look at how to use the open-source infrastructure-as-code tool Terraform to do this in this extensive guide. For companies looking to offer dependable services, putting in place a strong, highly available architecture on AWS is essential. Ensuring that every component is designed for high availability, we'll cover everything from setting up a Virtual Private Cloud (VPC) to deploying an Elastic Load Balancer (ELB), Auto Scaling Groups (ASGs), and a relational database.

Introduction to Terraform and High Availability

Terraform is a powerful tool developed by HashiCorp that allows you to define, preview, and deploy cloud infrastructure using a high-level configuration language. It supports multiple cloud providers, including AWS, Azure, Google Cloud, and many more, making it a versatile choice for infrastructure management.

High availability (HA) refers to systems that are dependable and operate continuously without failing for a long period. In the context of AWS, this involves setting up architectures that can withstand failures by distributing workloads across multiple Availability Zones (AZs) and ensuring redundancy at all levels.

Setting Up Your Terraform Environment

Before we dive into creating AWS resources with Terraform, let's set up our environment:

1. Install Terraform: Start by installing Terraform on your local machine. You can download the appropriate version for your operating system from the Terraform download page.

   # For MacOS
   brew install terraform

   # For Windows
   choco install terraform

2. Configure AWS CLI: Ensure you have the AWS CLI installed and configured with your AWS credentials. This will allow Terraform to interact with your AWS account.

   # Install AWS CLI
   pip install awscli

   # Configure AWS CLI
   aws configure

Defining the VPC

A Virtual Private Cloud (VPC) is the foundational component of your AWS infrastructure. It provides a logically isolated network that you can launch your resources into. Below is the Terraform configuration for creating a VPC with multiple subnets for high availability.

provider "aws" {
  region = "us-east-1"
}

resource "aws_vpc" "main" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
  tags = {
    Name = "main-vpc"
  }
}

resource "aws_subnet" "public_subnet_1" {
  vpc_id = aws_vpc.main.id
  cidr_block = "10.0.1.0/24"
  availability_zone = "us-east-1a"
  map_public_ip_on_launch = true
  tags = {
    Name = "public-subnet-1"
  }
}

resource "aws_subnet" "public_subnet_2" {
  vpc_id = aws_vpc.main.id
  cidr_block = "10.0.2.0/24"
  availability_zone = "us-east-1b"
  map_public_ip_on_launch = true
  tags = {
    Name = "public-subnet-2"
  }
}

resource "aws_subnet" "private_subnet_1" {
  vpc_id = aws_vpc.main.id
  cidr_block = "10.0.3.0/24"
  availability_zone = "us-east-1a"
  tags = {
    Name = "private-subnet-1"
  }
}

resource "aws_subnet" "private_subnet_2" {
  vpc_id = aws_vpc.main.id
  cidr_block = "10.0.4.0/24"
  availability_zone = "us-east-1b"
  tags = {
    Name = "private-subnet-2"
  }
}

This configuration defines a VPC with a /16 CIDR block, two public subnets in different Availability Zones for high availability, and two private subnets. This setup allows us to distribute our resources across multiple AZs to ensure redundancy.

Setting Up an Internet Gateway and NAT Gateway

To allow internet access to the instances in our public subnets and restrict internet access to instances in our private subnets, we need to set up an Internet Gateway (IGW) and a NAT Gateway.

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.main.id
  tags = {
    Name = "main-igw"
  }
}

resource "aws_route_table" "public_rt" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
  tags = {
    Name = "public-route-table"
  }
}

resource "aws_route_table_association" "public_subnet_1_assoc" {
  subnet_id = aws_subnet.public_subnet_1.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_route_table_association" "public_subnet_2_assoc" {
  subnet_id = aws_subnet.public_subnet_2.id
  route_table_id = aws_route_table.public_rt.id
}

resource "aws_eip" "nat_eip" {
  vpc = true
}

resource "aws_nat_gateway" "nat_gw" {
  allocation_id = aws_eip.nat_eip.id
  subnet_id = aws_subnet.public_subnet_1.id
  tags = {
    Name = "nat-gateway"
  }
}

resource "aws_route_table" "private_rt" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat_gw.id
  }
  tags = {
    Name = "private-route-table"
  }
}

resource "aws_route_table_association" "private_subnet_1_assoc" {
  subnet_id = aws_subnet.private_subnet_1.id
  route_table_id = aws_route_table.private_rt.id
}

resource "aws_route_table_association" "private_subnet_2_assoc" {
  subnet_id = aws_subnet.private_subnet_2.id
  route_table_id = aws_route_table.private_rt.id
}

Creating Security Groups

Security groups act as virtual firewalls for your instances, controlling inbound and outbound traffic. Below is a sample Terraform configuration for creating security groups for public and private instances:

resource "aws_security_group" "public_sg" {
  vpc_id = aws_vpc.main.id

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "public-sg"
  }
}

resource "aws_security_group" "private_sg" {
  vpc_id = aws_vpc.main.id

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "private-sg"
  }
}

Deploying EC2 Instances

Next, we will deploy EC2 instances in our public and private subnets. The public EC2 instance could serve as a bastion host for SSH access, while the private EC2 instances could run application servers.

resource "aws_instance" "bastion" {
  ami = "ami-0c55b159cbfafe1f0" # Amazon Linux 2 AMI (HVM)
  instance_type = "t2.micro"
  subnet_id = aws_subnet.public_subnet_1.id
  security_groups = [aws_security_group.public_sg.name]
  tags = {
    Name = "bastion-host"
  }
}

resource "aws_instance" "app_server_1" {
  ami = "ami-0c55b159cbfafe1f0" # Amazon Linux 2 AMI (HVM)
  instance_type = "t2.micro"
  subnet_id = aws_subnet.private_subnet_1.id
  security_groups = [aws_security_group.private_sg.name]
  tags = {
    Name = "app-server-1"
  }
}

resource "aws_instance" "app_server_2" {
  ami = "ami-0c55b159cbfafe1f0" # Amazon Linux 2 AMI (HVM)
  instance_type = "t2.micro"
  subnet_id = aws_subnet.private_subnet_2.id
  security_groups = [aws_security_group.private_sg.name]
  tags = {
    Name = "app-server-2"
  }
}

Implementing an Elastic Load Balancer (ELB)

To distribute traffic across multiple EC2 instances, we will set up an Elastic Load Balancer (ELB).

resource "aws_lb" "app_lb" {
  name = "app-lb"
  internal = false
  load_balancer_type = "application"
  security_groups = [aws_security_group

.public_sg.id]
  subnets = [
    aws_subnet.public_subnet_1.id,
    aws_subnet.public_subnet_2.id
  ]
  tags = {
    Name = "app-lb"
  }
}

resource "aws_lb_target_group" "app_tg" {
  name = "app-tg"
  port = 80
  protocol = "HTTP"
  vpc_id = aws_vpc.main.id
  health_check {
    interval = 30
    path = "/"
    protocol = "HTTP"
    timeout = 3
    healthy_threshold = 3
    unhealthy_threshold = 3
  }
  tags = {
    Name = "app-tg"
  }
}

resource "aws_lb_listener" "app_lb_listener" {
  load_balancer_arn = aws_lb.app_lb.arn
  port = "80"
  protocol = "HTTP"

  default_action {
    type = "forward"
    target_group_arn = aws_lb_target_group.app_tg.arn
  }
}

resource "aws_lb_target_group_attachment" "app_server_1" {
  target_group_arn = aws_lb_target_group.app_tg.arn
  target_id = aws_instance.app_server_1.id
  port = 80
}

resource "aws_lb_target_group_attachment" "app_server_2" {
  target_group_arn = aws_lb_target_group.app_tg.arn
  target_id = aws_instance.app_server_2.id
  port = 80
}

Configuring Auto Scaling Groups

Auto Scaling Groups (ASGs) automatically adjust the number of EC2 instances in response to changes in demand. This is essential for maintaining high availability and optimizing costs.

resource "aws_launch_configuration" "app_lc" {
  image_id = "ami-0c55b159cbfafe1f0" # Amazon Linux 2 AMI (HVM)
  instance_type = "t2.micro"
  security_groups = [aws_security_group.private_sg.name]
  associate_public_ip_address = false
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_autoscaling_group" "app_asg" {
  desired_capacity = 2
  max_size = 3
  min_size = 1
  vpc_zone_identifier = [
    aws_subnet.private_subnet_1.id,
    aws_subnet.private_subnet_2.id
  ]
  launch_configuration = aws_launch_configuration.app_lc.name

  target_group_arns = [aws_lb_target_group.app_tg.arn]

  health_check_type = "ELB"
  health_check_grace_period = 300

  lifecycle {
    create_before_destroy = true
  }

  tags = [
    {
      key                 = "Name"
      value               = "app-asg"
      propagate_at_launch = true
    },
  ]
}

Setting Up a Highly Available Database with Amazon RDS

Amazon Relational Database Service (RDS) provides a managed relational database that is easy to set up, operate, and scale. For high availability, we will deploy a multi-AZ RDS instance.

resource "aws_db_instance" "app_db" {
  allocated_storage    = 20
  storage_type         = "gp2"
  engine               = "mysql"
  engine_version       = "8.0.28"
  instance_class       = "db.t2.micro"
  name                 = "appdb"
  username             = "admin"
  password             = "YourStrongPassword"
  parameter_group_name = "default.mysql8.0"
  publicly_accessible  = false
  multi_az             = true
  skip_final_snapshot  = true
  vpc_security_group_ids = [aws_security_group.private_sg.id]
  db_subnet_group_name = aws_db_subnet_group.main.name
  tags = {
    Name = "app-db"
  }
}

resource "aws_db_subnet_group" "main" {
  name = "main-subnet-group"
  subnet_ids = [
    aws_subnet.private_subnet_1.id,
    aws_subnet.private_subnet_2.id
  ]
  tags = {
    Name = "main-subnet-group"
  }
}

Deploying CloudWatch for Monitoring and Alerts

Monitoring is crucial for maintaining the health of your AWS environment. AWS CloudWatch provides monitoring and logging services for AWS resources.

resource "aws_cloudwatch_log_group" "app_log_group" {
  name              = "/aws/app"
  retention_in_days = 7
}

resource "aws_cloudwatch_metric_alarm" "cpu_alarm" {
  alarm_name          = "high-cpu-usage"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/EC2"
  period              = "120"
  statistic           = "Average"
  threshold           = "80"

  dimensions = {
    InstanceId = aws_instance.app_server_1.id
  }

  alarm_actions = [aws_sns_topic.alarm.arn]
}

resource "aws_sns_topic" "alarm" {
  name = "alarm-topic"
}

resource "aws_sns_topic_subscription" "alarm_subscription" {
  topic_arn = aws_sns_topic.alarm.arn
  protocol  = "email"
  endpoint  = "your-email@example.com"
}

Conclusion

Deploying a high-availability full-stack architecture on AWS using Terraform involves multiple components working in harmony to ensure redundancy, scalability, and reliability. By following the steps outlined in this guide, you can set up a robust infrastructure that can handle varying loads and provide continuous service even in the face of failures.

References:

1. Terraform Documentation
2. AWS VPC Documentation
3. AWS EC2 Documentation
4. AWS RDS Documentation
5. AWS CloudWatch Documentation

By leveraging Terraform and AWS, you can build highly available infrastructures that are resilient, scalable, and easy to manage. The code examples provided here can be customized to suit specific requirements and further optimized to meet business needs. Happy deploying!

https://www.nilebits.com/blog/2024/09/terraform-your-way-to-high-availability-deploying-a-full-stack-aws-architecture/

Understanding Terraform Drift Detection and Remediation

 

https://www.nilebits.com/blog/2024/07/terraform-drift-detection/

Introduction to Terraform and Infrastructure as Code (IaC)

We now manage and deploy infrastructure in a completely new way thanks to Infrastructure as Code (IaC). Consistent and repeatable infrastructure deployment is made possible by IaC through the use of configuration files. One of the industry's most widely used IaC tools is Terraform, which was created by HashiCorp. Users may collaborate, automate, and version infrastructure as code thanks to this feature.

However, maintaining infrastructure with Terraform is not without its challenges. One of the main issues is drift in the infrastructure. Infrastructure drift is the term for when the actual state of your infrastructure differs from the state that is defined in your Terraform setup. This page discusses Terraform drift detection and repair, providing code samples, thorough explanations, and suggested practices for effectively managing infrastructure drift.

What is Infrastructure Drift?

Infrastructure drift happens when changes are made to your infrastructure outside of Terraform's control. These changes can be intentional or accidental and may occur due to:

• Manual changes made by administrators directly in the cloud console.
• Changes made by other automation tools or scripts.
• Modifications resulting from cloud provider updates or changes in service behavior.

Drift can lead to inconsistencies, unexpected behavior, and security vulnerabilities. Therefore, detecting and remediating drift is crucial to maintaining the desired state of your infrastructure.

How Terraform Manages State

Before diving into drift detection, it's essential to understand how Terraform manages state. Terraform uses a state file to keep track of the infrastructure it manages. This state file is a critical component, as it maps the configuration files to the real-world resources.

The state file is usually stored locally or remotely in a secure storage backend, such as AWS S3, HashiCorp Consul, or Terraform Cloud. Terraform uses this state file during operations to plan and apply changes to your infrastructure.

Here's an example of a simple Terraform configuration and the corresponding state file:

# main.tf
provider "aws" {
  region = "us-west-2"
}

resource "aws_instance" "example" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
}

After running terraform apply, Terraform creates a state file (terraform.tfstate) that looks something like this:

{
  "version": 4,
  "terraform_version": "1.0.0",
  "resources": [
    {
      "mode": "managed",
      "type": "aws_instance",
      "name": "example",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 1,
          "attributes": {
            "ami": "ami-0c55b159cbfafe1f0",
            "instance_type": "t2.micro",
            "id": "i-1234567890abcdef0",
            "tags": null
          }
        }
      ]
    }
  ]
}

The state file is used by Terraform to map resources in your configuration to real-world resources. Any changes made outside of Terraform's control can lead to drift.

Detecting Drift in Terraform

The command "terraform plan" is included into Terraform and may be used to identify drift. Terraform compares the desired state specified in your configuration files with the present state of your infrastructure when you run terraform plan. Terraform will indicate any differences that it finds.

Here's how you can use terraform plan to detect drift:

terraform plan

The output will show any differences between the actual state and the desired state. If there's no drift, the output will indicate that no changes are needed. If there is drift, the output will show the necessary changes to reconcile the state.

For example:

# terraform plan output
...
  ~ aws_instance.example
      instance_type: "t2.micro" => "t2.small"
...

In this example, the instance type has changed from t2.micro to t2.small, indicating drift.

Automating Drift Detection

Manually running terraform plan to detect drift is not always practical, especially in large or complex environments. Automating drift detection can help ensure that drift is identified and remediated promptly.

One approach to automate drift detection is to use CI/CD pipelines. Tools like Jenkins, GitHub Actions, GitLab CI, or CircleCI can be used to run terraform plan on a scheduled basis or whenever a change is made to the configuration files.

Here's an example of how you can set up a drift detection pipeline using GitHub Actions:

# .github/workflows/terraform-drift-detection.yml
name: Terraform Drift Detection

on:
  schedule:
    - cron: '0 0 * * *' # Run daily at midnight

jobs:
  drift-detection:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          terraform_version: 1.0.0

      - name: Initialize Terraform
        run: terraform init

      - name: Run Terraform Plan
        run: terraform plan -detailed-exitcode

In this example, the GitHub Actions workflow runs terraform plan daily at midnight. The -detailed-exitcode flag ensures that the workflow fails if there are any changes detected, which can then trigger notifications or further actions.

Remediating Drift in Terraform

Once drift is detected, the next step is remediation. Remediation involves updating the Terraform configuration to match the desired state or applying changes to the infrastructure to bring it back in line with the configuration.

There are two primary approaches to remediation:

1. Update Configuration Files: If the drift represents a desired change, update the Terraform configuration files to reflect the new state. After updating the configuration, run terraform apply to update the state file.

   # Update main.tf
   resource "aws_instance" "example" {
     ami           = "ami-0c55b159cbfafe1f0"
     instance_type = "t2.small" # Updated instance type
   }

   # Apply changes
   terraform apply

2. Revert Changes: If the drift represents an unintended change, run terraform apply to revert the changes and bring the infrastructure back to the desired state.

   terraform apply

In both cases, Terraform will update the state file to match the desired state.

Best Practices for Managing Drift

Managing drift effectively requires a combination of best practices and tooling. Here are some best practices to consider:

1. Use Remote State: Store your Terraform state file in a remote backend to ensure consistency and accessibility across your team.
2. Implement Version Control: Use version control systems like Git to track changes to your Terraform configuration files.
3. Automate Testing and Validation: Use CI/CD pipelines to automate testing, validation, and drift detection.
4. Restrict Manual Changes: Minimize manual changes to your infrastructure by enforcing the use of Terraform for all changes.
5. Regular Audits: Perform regular audits of your infrastructure to detect and remediate drift promptly.
6. Leverage Infrastructure Monitoring: Use infrastructure monitoring tools to detect changes in real-time and alert you to potential drift.

Code Example: Full Workflow

Let's walk through a full workflow example of managing drift with Terraform. This example will include a Terraform configuration, automation of drift detection, and remediation.

1. Terraform Configuration:

   # main.tf
   provider "aws" {
     region = "us-west-2"
   }

   resource "aws_instance" "example" {
     ami           = "ami-0c55b159cbfafe1f0"
     instance_type = "t2.micro"
   }

2. Initialize Terraform:

   terraform init

3. Apply Configuration:

   terraform apply

4. Automate Drift Detection: Create a GitHub Actions workflow:

   # .github/workflows/terraform-drift-detection.yml
   name: Terraform Drift Detection

   on:
     schedule:
       - cron: '0 0 * * *' # Run daily at midnight

   jobs:
     drift-detection:
       runs-on: ubuntu-latest
       steps:
         - name: Checkout repository
           uses: actions/checkout@v2

         - name: Set up Terraform
           uses: hashicorp/setup-terraform@v1
           with:
             terraform_version: 1.0.0

         - name: Initialize Terraform
           run: terraform init

         - name: Run Terraform Plan
           run: terraform plan -detailed-exitcode

5. Remediation: If drift is detected (e.g., instance type changed), update the configuration and apply changes:

   # Update main.tf
   resource "aws_instance" "example" {
     ami           = "ami-0c55b159cbfafe1f0"
     instance_type = "t2.small" # Updated instance type
   }

   # Apply changes
   terraform apply

https://www.nilebits.com/blog/2024/07/terraform-drift-detection/