Terraform, AWS, and Docker: Building a Cloud-Native Testbed

Terraform Lab 1: AWS S3 and Docker Nginx Testbed

This is a basic Terraform lab we did to simulate a small customer environment. This lab provides a hands-on introduction to infrastructure as code (IaC) by guiding us through the deployment of a simple but practical cloud and container-based testbed. By the end of this lab, we gained valuable experience in using Terraform to manage resources across both AWS and a local Docker environment.

This lab was a helpful test for us. Also, it can be duplicated by IT professionals who are new to IaC or looking to expand their Terraform skills. Whether you're a cloud engineer, DevOps practitioner, or infrastructure architect, this lab will teach you how to automate the provisioning and management of cloud services and containerized applications using Terraform. You'll learn how to create an S3 bucket in AWS, run a Docker Nginx web server, and handle common troubleshooting scenarios.

The key outcomes of this lab include:

• Understanding the Terraform workflow and syntax for defining infrastructure resources

• Deploying and configuring an AWS S3 bucket using Terraform

• Running a Docker Nginx container on a local Ubuntu workstation

• Troubleshooting common Terraform and Docker-related issues

To get the most out of this lab, you should have a basic understanding of cloud computing concepts, AWS services, and Docker containers. Experience with command-line tools and text editors will also be helpful. Let's dive in and start automating your infrastructure!

The architecture of this Terraform lab consists of the following key components:

Laptop (Ubuntu):

• Runs Terraform and the AWS CLI to manage cloud resources

• Serves as the central control point for infrastructure deployment and management

Workstation (Ubuntu, 32GB RAM, 1TB SSD):

• Hosts the Docker container running the Nginx web server

• Allows remote access to the Docker daemon on port 2375 for management

AWS:

• Hosts the S3 bucket created by the Terraform configuration

• Provides cloud infrastructure for future extensions, such as EC2 instances

The overall architecture follows a hybrid approach, where the local workstation runs the Docker container while the cloud-based AWS S3 bucket is managed through Terraform. This setup allows for a self-contained, local development environment while still leveraging cloud-based resources.

The key design decisions include:

• Terraform for Infrastructure as Code (IaC): Terraform is used to define and manage the infrastructure, enabling version control, collaboration, and consistent deployments.

• Docker for Local Development: Running the Nginx web server in a Docker container on the local workstation provides a portable, isolated, and reproducible development environment.

• AWS S3 Bucket: The S3 bucket is deployed in AWS to showcase the ability to manage cloud resources using Terraform, laying the foundation for future extensions.

• Remote Docker Access: Enabling remote access to the Docker daemon on the workstation allows the Terraform configuration running on the laptop to control the local Docker container.

This architecture allows for a seamless integration of local and cloud-based resources, enabling IT professionals to practice and develop their infrastructure as code skills using Terraform, Docker, and AWS.

Prerequisites

Before deploying this Terraform lab, ensure you have the following:

1. Terraform: Install Terraform on your local machine. You can download it from the official website.

2. AWS CLI: Install and configure the AWS Command Line Interface (CLI) on your local machine. Follow the AWS CLI installation guide for your operating system.

3. Docker: Install Docker on your Ubuntu workstation. Refer to the Docker installation guide for Ubuntu.

4. AWS Account: You'll need an AWS account with permissions to create an S3 bucket.

Deployment Steps

1. Clone the Repository:

   git clone https://github.com/Cloud-Ops-Dev/Terraform-Lab-1.git
   cd Terraform-Lab-1
   

2. Configure Docker Remote Access:

   • On your Ubuntu workstation, edit the Docker daemon configuration file (/etc/docker/daemon.json) and add the following lines:

     {
       "hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]
     }
     

   • Restart the Docker service: sudo systemctl restart docker

3. Set up Terraform Variables:

   • Rename the terraform.tfvars.example file to terraform.tfvars.

   • Open the terraform.tfvars file and update the following variables:

     • aws_region: The AWS region where you want to create the S3 bucket.

     • bucket_prefix: The prefix for the unique S3 bucket name.

     • docker_host: The IP address or hostname of your Ubuntu workstation, followed by :2375.

4. Deploy with Terraform:

   • Initialize the Terraform working directory:

     terraform init
     

   • Review the Terraform plan:

     terraform plan
     

   • Apply the Terraform configuration:

     terraform apply
     

   • When prompted, type yes to confirm the deployment.

Verification

1. Verify the S3 Bucket:

   • Go to the AWS Management Console and navigate to the S3 service.

   • Verify that the S3 bucket with the name you specified in the bucket_prefix variable has been created.

2. Verify the Nginx Container:

   • Open a web browser and navigate to http://<your_docker_host_ip>:8081.

   • You should see the default Nginx welcome page, indicating that the Nginx container is running successfully.

Configuration Options

• aws_region: The AWS region where the S3 bucket will be created.

• bucket_prefix: The prefix for the unique S3 bucket name.

• docker_host: The IP address or hostname of your Ubuntu workstation, followed by :2375.

Troubleshooting

Refer to the "Troubleshooting Highlights" section in the original README for common issues and resolutions.

Common Errors and Solutions

• AWS CLI Install: Fixed missing awscli package by using AWS's official installer.

• Docker Service: Resolved daemon failure by correcting systemd config syntax.

• Terraform Errors:

  • Updated provider from hashicorp/docker to kreuzwerker/docker.

  • Fixed S3 bucket naming with lowercase random strings.

  • Resolved Docker port conflict by switching from 8080 to 8081.

  • Terraform Provider Error: Fixed duplicate AWS provider issue by consolidating into a single provider block.

Debugging Tips and Techniques

• Terraform State Inspection: Use terraform state list to view the current state of your resources.

• Docker Container Logs: Check the logs of your Docker containers using docker logs <container_name> for troubleshooting.

• AWS CLI Debugging: Enable verbose logging for the AWS CLI by setting the AWS_CLI_LOG_LEVEL environment variable to debug.

• Terraform Dry Runs: Run terraform plan before applying changes to preview the changes and catch any issues.

Configuration Gotchas

• Bucket Naming: S3 bucket names must be globally unique, so using a random suffix is recommended to avoid conflicts.

• Docker Host Configuration: Ensure the Docker daemon on your workstation is configured to allow remote access on the specified port (e.g., port 2375).

• Provider Versions: Keep your Terraform provider versions up-to-date to avoid compatibility issues.

Cleanup and Teardown

To clean up the resources created by this Terraform configuration, run the following command:

terraform destroy

This will remove the S3 bucket and the Nginx Docker container. Make sure to review the plan before applying the destroy operation to ensure you're removing the intended resources.

This Terraform lab demonstrates a versatile infrastructure that can be applied in various real-world scenarios for small-to-medium businesses (SMBs). Here are some key considerations:

Real-world Scenarios

• Web Application Hosting: The Nginx Docker container can serve as a web server for hosting a company's public-facing website or web application. This approach allows for easy scalability, portability, and rapid deployment of new versions.

• Content Hosting and Delivery: The S3 bucket can be used to store and serve static content like images, documents, and web assets. This provides a cost-effective, highly available, and scalable content delivery solution.

• Backup and Storage: The S3 bucket can also be used for backups, archives, or general file storage, leveraging the durability and security of AWS cloud infrastructure.

• Development and Testing: This infrastructure can be easily replicated to create isolated environments for software development, testing, and experimentation, helping to streamline the software delivery lifecycle.

Cost Considerations and Optimization

• AWS S3 Pricing: S3 pricing is based on storage, data transfer, and requests, making it a cost-effective solution for SMBs with variable storage and bandwidth needs. Optimizing S3 usage, such as by implementing lifecycle policies, can help minimize costs.

• Docker Container Efficiency: Containerization with Docker allows for efficient resource utilization, as containers share the host's operating system, reducing the need for dedicated virtual machines. This can lead to lower hardware and maintenance costs.

• Terraform for Infrastructure as Code: By using Terraform, the infrastructure can be easily replicated, scaled, and modified as needed, reducing the time and effort required for managing the environment.

When to Use This Approach vs Alternatives

This Terraform-based infrastructure is well-suited for SMBs that:

• Require a flexible, scalable, and cost-effective cloud-based solution for web hosting, content delivery, or storage.

• Want to leverage the benefits of infrastructure as code for faster deployment, better collaboration, and easier maintenance.

• Have the technical expertise or are willing to invest in upskilling their IT team to work with Terraform, Docker, and cloud services like AWS.

Alternatives to this approach may include:

• Managed web hosting services (e.g., WordPress.com, Squarespace) for simpler web hosting needs.

• Traditional on-premises server infrastructure for businesses with specific security or compliance requirements.

• Fully-managed cloud services (e.g., AWS Elastic Beanstalk, Azure App Service) for a more hands-off infrastructure management experience.

Business Value and ROI Considerations

The key business value and return on investment (ROI) of this Terraform-based infrastructure include:

• Improved Agility and Scalability: The ability to rapidly provision, modify, and scale infrastructure components as needed can help SMBs respond quickly to changing business requirements.

• Cost Optimization: The use of cost-effective cloud services, containerization, and infrastructure as code can lead to lower overall infrastructure costs and better resource utilization.

• Enhanced Reliability and Availability: The durability and high availability of AWS S3 and the portability of Docker containers can improve the reliability and uptime of the infrastructure.

• Increased Efficiency and Productivity: Automating infrastructure management with Terraform can free up IT resources, allowing them to focus on more strategic initiatives.

• Improved Collaboration and Versioning: The use of version control (e.g., Git) for infrastructure code can enhance collaboration and facilitate easier troubleshooting and rollbacks.

By leveraging this Terraform-based infrastructure, SMBs can gain a competitive edge through improved agility, cost savings, and a more reliable and efficient IT environment.