Ansible Real-World Project: Deploy a Production Web App from Zero (Terraform + Ansible + CI/CD)

Introduction

Most Ansible tutorials stop at "install nginx". This one does not. We are going to build what a real team ships:

 1                     ┌─────────────┐
 2   users ──HTTPS──▶  │  lb-1       │  nginx · Let's Encrypt · upstream = app servers
 3                     └──────┬──────┘
 4                 ┌──────────┴──────────┐
 5          ┌──────▼──────┐       ┌──────▼──────┐
 6          │  app-1      │       │  app-2      │   Flask "Notes API" · gunicorn · systemd
 7          └──────┬──────┘       └──────┬──────┘   rolling deploy, one server at a time
 8                 └──────────┬──────────┘
 9                     ┌──────▼──────┐
10                     │  db-1       │   PostgreSQL 14 · password from Ansible Vault
11                     └─────────────┘
  • Terraform creates the four servers (or Docker gives you four containers on your laptop, for free)
  • Ansible configures everything with four roles: common, postgres, app, nginx_lb
  • Ansible Vault keeps the database password out of git
  • A rolling deploy with serial: 1 and a health gate, so users never see a 502
  • GitHub Actions runs that deploy on every push to main

And the whole thing works the same way on Docker, Google Cloud, AWS and Azure. Pick the one you have access to; the playbooks do not know the difference.

Everything lives in the project-01-production-web-app folder of my Ansible course repo: github.com/rahulwagh/ansible-zero-to-expert-masterclass.

In this post:

  1. Where to run it: laptop or cloud
  2. The application we deploy
  3. Inventory: one variable makes it portable
  4. The four roles
  5. site.yml: the whole stack from zero
  6. deploy.yml: zero-downtime rolling deploy
  7. CI/CD with GitHub Actions
  8. HTTPS with Let's Encrypt
  9. Troubleshooting

If some of the building blocks below are new to you (inventory groups, group_vars, templates, handlers, Vault), they are covered step by step in my course: Ansible Zero to Hero on Udemy. This project is what you build after it.

1. Where to run it: laptop or cloud

1git clone https://github.com/rahulwagh/ansible-zero-to-expert-masterclass.git
2cd ansible-zero-to-expert-masterclass/project-01-production-web-app
3make deps                     # Galaxy collections: community.postgresql, community.general
4echo ansible-demo > .vault_pass   # the demo vault password (gitignored); change it later with ansible-vault rekey

Option A: Docker on your laptop (free)

infra/docker/ starts four Ubuntu 22.04 containers that run real systemd and sshd. Ansible treats them exactly like VMs. The only trick worth knowing: the containers share the host's cgroup namespace (cgroup: host), because without it systemd cannot supervise forking services such as PostgreSQL. The lab.sh script handles that, and falls back to plain docker run if your Docker Compose is older than 2.15.

1make lab-up
1==> Creating SSH key infra/keys/lab
2==> Building the node image (first run ~2 min)
3==> Starting lb-1 app-1 app-2 db-1
4==> Ping
5app-1 | SUCCESS => {"ping": "pong"}
6app-2 | SUCCESS => {"ping": "pong"}
7db-1  | SUCCESS => {"ping": "pong"}
8lb-1  | SUCCESS => {"ping": "pong"}

Port map: localhost:8080 is lb-1's port 80. SSH is 2231–2234.

Option B: Google Cloud, AWS or Azure (Terraform)

Each infra/<cloud>/ folder provisions the same four servers, opens 22/80/443 from the internet and everything between the servers inside the VPC, installs your public key for an ansible user, and, this is the useful part, prints a ready-made inventory:

1ssh-keygen -t ed25519 -f infra/keys/lab -N ""
2cd infra/gcp                                  # or infra/aws · infra/azure
3cp terraform.tfvars.example terraform.tfvars  # gcp: project_id · azure: subscription_id · aws: nothing needed
4terraform init && terraform apply
5cd ../..
6./scripts/gen-inventory.sh gcp                # writes ansible/inventory/cloud.ini
7make ping INV=cloud

The generated inventory looks like this:

 1[loadbalancers]
 2lb-1  ansible_host=34.68.10.21 private_ip=10.20.0.2
 3
 4[appservers]
 5app-1 ansible_host=34.68.10.22 private_ip=10.20.0.3
 6app-2 ansible_host=34.68.10.23 private_ip=10.20.0.4
 7
 8[dbservers]
 9db-1  ansible_host=34.68.10.24 private_ip=10.20.0.5
10
11[all:vars]
12ansible_user=ansible
13ansible_ssh_private_key_file=../../infra/keys/lab
14ansible_python_interpreter=/usr/bin/python3

Four small VMs cost roughly $0.05–0.08 per hour on any of the three clouds. terraform destroy when you are done.

The 7 commands you will use, explained visually

Everything in this project is driven by seven short commands. Each one is a line in the project's Makefile that stands in for a longer command. The explorer below draws what each command does, one box at a time: press Next (or the → key) to reveal the next step, ↓ to jump to the next command, and ⤢ Present for fullscreen. The left column lists only the files that command reads, creates or keeps secret.

command 1 of 7

make lab-up

runs

Files this command touches

readscreatessecret, not in git
→ next step · ← back · ↓ ↑ jump between commands · space auto-play
You typeIt really runs
make depsansible-galaxy collection install -r requirements.yml -p collections
make lab-up./infra/docker/lab.sh up
make pingansible -i inventory/docker.ini all -m ping
make siteansible-playbook -i inventory/docker.ini site.yml
make deploy VERSION=1.1.0ansible-playbook -i inventory/docker.ini deploy.yml -e app_version=1.1.0
make smokeansible-playbook -i inventory/docker.ini smoke-test.yml
make lab-down./infra/docker/lab.sh down

Add INV=cloud to any of the Ansible ones (make site INV=cloud) to run against inventory/cloud.ini instead of the Docker lab.

2. The application we deploy

app/app.py is a small Flask API, but it is a real one: it talks to PostgreSQL, it has a /health endpoint the load balancer and the deploy use, and it prints its version and which server answered, so you can watch a rolling deploy happen in the browser.

EndpointPurpose
GET /HTML page: version, server name, DB status, note count
GET /health{"status":"ok","version":"1.0.0","server":"app-1","db":"ok"}
GET /api/notes · POST /api/notesnotes stored in PostgreSQL

It is configured entirely through environment variables, which Ansible writes into /opt/notes-api/.env:

1DATABASE_URL=postgresql://notes:<vault-secret>@db-1:5432/notes
2APP_VERSION=1.0.0

3. Inventory: one variable makes it portable

Every host carries a private_ip. On the clouds it is the VPC address. On the Docker lab it is simply the container name, because Docker's embedded DNS resolves db-1 and app-1 on the compose network:

1[appservers]
2app-1 ansible_host=127.0.0.1 ansible_port=2232 private_ip=app-1

Every place that needs "how do servers reach each other" uses that variable and nothing else:

1# group_vars/all/main.yml
2db_host: "{{ hostvars[groups['dbservers'][0]]['private_ip'] }}"
1# roles/nginx_lb/templates/notes-api.conf.j2
2upstream notes_api {
3{% for host in groups['appservers'] %}
4    server {{ hostvars[host]['private_ip'] }}:{{ app_port }} max_fails=1 fail_timeout=5s;
5{% endfor %}
6}

That is the whole secret to "works on four platforms".

4. The four roles

common

Base packages, hostname, and /etc/hosts entries so every server can reach the others by name. Note the two container-aware conditions: hostname is skipped on Docker (the container already has it) and /etc/hosts is only written when private_ip is an actual IP.

postgres

 1- name: Allow the app servers to connect (any host on the same network, password auth)
 2  ansible.builtin.lineinfile:
 3    path: "/etc/postgresql/{{ postgres_version }}/main/pg_hba.conf"
 4    line: "host    {{ db_name }}    {{ db_user }}    samenet    scram-sha-256"
 5  notify: Restart postgresql
 6
 7- name: Apply config changes before creating the database
 8  ansible.builtin.meta: flush_handlers
 9
10- name: Database user (password comes from Ansible Vault)
11  community.postgresql.postgresql_user:
12    name: "{{ db_user }}"
13    password: "{{ db_password }}"
14  become_user: postgres
15  no_log: true

Two details that bite people: samenet in pg_hba.conf means "any address in the server's own subnets", which is exactly right for a VPC and for a Docker network; and flush_handlers forces the PostgreSQL restart to happen before we try to create the user, instead of at the end of the play.

The password lives in group_vars/all/vault.yml, encrypted:

1ansible-vault view ansible/inventory/group_vars/all/vault.yml
2# vault_db_password: "N0tes-Demo-Pa55"

app

Creates a system user, copies the code, builds a virtualenv, writes .env (mode 0640, it carries the secret), installs a systemd unit for gunicorn, and then the important part:

 1- name: Apply pending restarts now
 2  ansible.builtin.meta: flush_handlers
 3
 4- name: Wait for the app to listen
 5  ansible.builtin.wait_for:
 6    port: "{{ app_port }}"
 7
 8- name: Health check must pass before Ansible moves to the next app server
 9  ansible.builtin.uri:
10    url: "http://127.0.0.1:{{ app_port }}/health"
11    status_code: 200
12    return_content: true
13  register: health
14  retries: 10
15  delay: 3
16  until: health.status == 200 and (health.json.status | default('')) == 'ok'

If the new version does not come up healthy, the play fails on that server and the other app server is still serving the old version. That is the whole safety net.

nginx_lb

Templates the upstream block above, validates with nginx -t, reloads, and optionally runs certbot (section 8).

5. site.yml: the whole stack from zero

1- hosts: all           ; become: true ; roles: [common]
2- hosts: dbservers     ; become: true ; roles: [postgres]
3- hosts: appservers    ; become: true ; serial: 1 ; roles: [app]
4- hosts: loadbalancers ; become: true ; roles: [nginx_lb]
1make site            # Docker lab
2make site INV=cloud  # any cloud
1PLAY RECAP *********************************************************************
2app-1 : ok=17  changed=11  unreachable=0  failed=0
3app-2 : ok=17  changed=11  unreachable=0  failed=0
4db-1  : ok=11  changed=9   unreachable=0  failed=0
5lb-1  : ok=12  changed=8   unreachable=0  failed=0

Open http://localhost:8080 (or lb-1's public IP). Refresh a few times and the "served by" line alternates between app-1 and app-2. Run make site a second time and every task reports changed=0: the playbooks are idempotent.

6. deploy.yml: zero-downtime rolling deploy

Bump the version and deploy while a loop hits the load balancer twice a second:

1make deploy VERSION=1.1.0
1TASK [app : Report] ****************************************************
2    "msg": "app-1 is healthy on version 1.1.0"
3TASK [Show which server and version the load balancer just returned] ***
4    "msg": "LB → app-2 running 1.0.0"        ← app-2 still serving while app-1 was restarted
5...
6    "msg": "app-2 is healthy on version 1.1.0"
7    "msg": "LB → app-1 running 1.1.0"

The request loop during that deploy on my machine: 86 requests, 0 failures, versions seen 1.0.0 then 1.1.0. Three things make that work together:

  1. serial: 1 in deploy.yml: one app server at a time.
  2. The health gate in the app role: Ansible does not move on until /health says ok.
  3. In nginx, max_fails=1 fail_timeout=5s plus proxy_next_upstream error timeout http_502 http_503 http_504: a request that lands on the restarting server is retried on the healthy one.

make smoke sends ten requests through the load balancer and prints which server answered and which versions were seen.

7. CI/CD with GitHub Actions

.github/workflows/deploy-project-01.yml (at the repo root) turns a push to main into that rolling deploy. It needs three repository secrets:

SecretValue
SSH_PRIVATE_KEYcontents of infra/keys/lab
INVENTORY_INIcontents of ansible/inventory/cloud.ini
VAULT_PASSWORDyour vault password

The job installs Ansible and the collections, writes the key, inventory and vault password from the secrets, then:

1ansible-playbook -i inventory/ci.ini deploy.yml -e app_version=${GITHUB_SHA::7}
2ansible-playbook -i inventory/ci.ini smoke-test.yml

concurrency: deploy-production guarantees two pushes never deploy at the same time. Change a line in app/app.py, push, and refresh the browser.

8. HTTPS with Let's Encrypt

On a cloud server with a DNS record pointing at lb-1:

1# ansible/inventory/group_vars/all/main.yml
2enable_ssl: true
3domain_name: notes.example.com
4letsencrypt_email: you@example.com
1make site INV=cloud

The nginx_lb role installs certbot and runs certbot --nginx --redirect, guarded by creates: so it only ever runs once. Renewal is handled by the systemd timer certbot installs.

9. Troubleshooting

SymptomFix
Attempting to decrypt but no vault secrets foundCreate .vault_pass at the repo root (echo ansible-demo > .vault_pass). ansible.cfg points at it.
systemctl start postgresql hangs forever on DockerThe containers must share the host cgroup namespace. Use make lab-up (it handles it) rather than a hand-written docker run.
App health check fails with a DB errorCheck db_host resolves from the app server (getent hosts db-1) and that pg_hba.conf got the samenet line.
nginx returns 502 for everyonemake smoke to see which upstream is down; journalctl -u notes-api on the app server.
Terraform: file("../keys/lab.pub") not foundCreate the key first: ssh-keygen -t ed25519 -f infra/keys/lab -N "".
IPs changed after a stop/start in the cloud./scripts/gen-inventory.sh <cloud> again.

Conclusion

Four roles, two playbooks, one variable that makes it portable, and a deploy that never shows users an error. If you can run this project end to end, you can automate real infrastructure with Ansible.

Posts in this series