AI Caught My Cloud Failure Before I Did

Mar 9, 2026 · 12 min read ·

Last Modified : Mar 9, 2026

Share on:

What Are We Building?
Architecture Overview
Prerequisites
Project Structure
Step 1: Multi-Cloud Infrastructure with Terraform
Step 2: The Flask App — CrashLab API
Step 3: Running Kestra with Docker
Step 4: AI Health Check Flow
Step 5: AI Log Monitor Flow
Step 6: CloudWatch Agent for Log Streaming
How It All Works Together
Conclusion

AI Caught My Cloud Failure Before I Did

What if your infrastructure could diagnose its own failures and tell you exactly what went wrong — before you even check your phone?

That is exactly what we are building in this tutorial. We will provision real multi-cloud infrastructure on AWS and GCP using Terraform, deploy a Flask app that is intentionally broken, and use Kestra as the workflow orchestrator to schedule health checks every hour. When the app fails, an AI agent powered by Google Gemini 2.5 Flash steps in, diagnoses the root cause, and fires a Slack alert — all automatically.

No fake demos. The app generates real errors. The AI catches them.

What Are We Building?

Here is the full picture:

A Flask app called CrashLab API runs on an AWS EC2 instance. It intentionally throws HTTP 500 errors 30% of the time to simulate real-world database failures and generates variable-latency responses.
Terraform provisions the entire multi-cloud infrastructure — VPC, subnets, security groups, and EC2 on AWS, plus a VPC network, firewall rules, Cloud NAT, and a VM on GCP — with a single terraform apply.
Kestra runs locally via Docker as the workflow orchestrator. It schedules two flows: one for AI-powered health checks and one for log monitoring.
Google Gemini 2.5 Flash is the AI model embedded in a Kestra AI Agent task. When the health check fails, Gemini diagnoses the likely cause and suggests an immediate fix — in a single line.
Slack Incoming Webhooks receive the AI-generated diagnosis as a real-time alert.
AWS CloudWatch Agent streams app logs from /app/logs/app.log on the EC2 instance to CloudWatch for retention and visibility.

Architecture Overview

 1┌─────────────────────────────────────────────────────────────┐
 2│                     Kestra (Docker)                         │
 3│                                                             │
 4│  ┌──────────┐    ┌──────────────┐    ┌───────────────────┐ │
 5│  │Scheduler │───▶│ health_check │───▶│   Flow Complete   │ │
 6│  │Every Hour│    │HTTP GET/health    │   App is Healthy  │ │
 7│  └──────────┘    └──────┬───────┘    └───────────────────┘ │
 8│                         │ ❌ Failure                        │
 9│                         ▼                                   │
10│                  ┌──────────────┐    ┌───────────────────┐ │
11│                  │ ai_diagnosis │───▶│   slack_alert     │ │
12│                  │Gemini 2.5    │    │  AI Diagnosis     │ │
13│                  │Flash (AI)    │    │  → Slack          │ │
14│                  └──────────────┘    └───────────────────┘ │
15└─────────────────────────────────────────────────────────────┘
16         │ monitors                          │ alerts
17         ▼                                   ▼
18┌─────────────────┐                  ┌──────────────┐
19│  AWS EC2        │                  │  Slack       │
20│  CrashLab API   │                  │  Channel     │
21│  Flask :5000    │                  └──────────────┘
22│  ~30% errors    │
23└─────────────────┘

Prerequisites

Before starting, make sure you have the following installed and configured:

Terraform >= 1.5
Docker (for running Kestra locally)
AWS CLI configured with valid credentials
GCP CLI (gcloud) authenticated
A Slack Incoming Webhook URL
A Google AI Studio API key for Gemini

Project Structure

 1workflow-orchestrator/
 2├── terraform/
 3│   ├── providers.tf          # AWS ~5.x + GCP ~5.x providers
 4│   ├── variables.tf          # Root variables
 5│   ├── main.tf               # Calls AWS and GCP modules
 6│   ├── outputs.tf            # Prints both VMs' IPs and URLs
 7│   ├── terraform.tfvars      # Your actual values (git-ignored)
 8│   ├── app/
 9│   │   ├── app.py            # Flask CrashLab API
10│   │   └── requirements.txt
11│   ├── scripts/
12│   │   ├── aws-startup.sh.tpl
13│   │   └── gcp-startup.sh.tpl
14│   └── modules/
15│       ├── aws/              # VPC, IGW, subnet, SG, EC2
16│       └── gcp/              # VPC network, firewall, router, NAT, VM
17└── kestra/
18    ├── ai-health-check.yml   # AI-powered health check flow
19    └── ai-log-monitor.yml    # Log monitoring flow

Step 1: Multi-Cloud Infrastructure with Terraform

Providers Configuration

terraform/providers.tf configures both cloud providers:

 1terraform {
 2  required_providers {
 3    aws = {
 4      source  = "hashicorp/aws"
 5      version = "~> 5.0"
 6    }
 7    google = {
 8      source  = "hashicorp/google"
 9      version = "~> 5.0"
10    }
11  }
12}
13
14provider "aws" {
15  region = var.aws_region
16}
17
18provider "google" {
19  project = var.gcp_project_id
20  region  = var.gcp_region
21}

AWS Module — VPC, Subnet, Security Group, EC2

The AWS module provisions a complete networking stack and an EC2 instance. Key resource: a security group that opens port 5000 for the Flask app:

 1# terraform/modules/aws/main.tf (excerpt)
 2
 3resource "aws_vpc" "main" {
 4  cidr_block           = var.vpc_cidr
 5  enable_dns_hostnames = true
 6  enable_dns_support   = true
 7}
 8
 9resource "aws_internet_gateway" "main" {
10  vpc_id = aws_vpc.main.id
11}
12
13resource "aws_subnet" "public" {
14  vpc_id                  = aws_vpc.main.id
15  cidr_block              = var.subnet_cidr
16  map_public_ip_on_launch = true
17  availability_zone       = "${var.aws_region}a"
18}
19
20resource "aws_security_group" "flask_app" {
21  name   = "flask-app-sg"
22  vpc_id = aws_vpc.main.id
23
24  ingress {
25    from_port   = 5000
26    to_port     = 5000
27    protocol    = "tcp"
28    cidr_blocks = ["0.0.0.0/0"]
29  }
30
31  ingress {
32    from_port   = 22
33    to_port     = 22
34    protocol    = "tcp"
35    cidr_blocks = ["0.0.0.0/0"]
36  }
37
38  egress {
39    from_port   = 0
40    to_port     = 0
41    protocol    = "-1"
42    cidr_blocks = ["0.0.0.0/0"]
43  }
44}
45
46resource "aws_instance" "flask_app" {
47  ami                    = "ami-0084a47cc718c111a"  # Ubuntu 22.04 eu-central-1
48  instance_type          = "t2.micro"
49  subnet_id              = aws_subnet.public.id
50  vpc_security_group_ids = [aws_security_group.flask_app.id]
51  key_name               = var.key_name
52
53  user_data = templatefile("${path.module}/../../scripts/aws-startup.sh.tpl", {
54    app_py = file("${path.module}/../../app/app.py")
55  })
56}

The startup script uses templatefile() to inject app.py directly into the VM at boot — no S3 bucket, no Ansible, no SSH required.

GCP Module — Network, Firewall, VM

The GCP module mirrors the AWS setup with a VPC network, three firewall rules (SSH, Flask, internal), Cloud Router, Cloud NAT, and an e2-micro VM:

 1# terraform/modules/gcp/main.tf (excerpt)
 2
 3resource "google_compute_network" "main" {
 4  name                    = var.network_name
 5  auto_create_subnetworks = false
 6}
 7
 8resource "google_compute_subnetwork" "main" {
 9  name          = var.subnet_name
10  ip_cidr_range = var.subnet_cidr
11  region        = var.gcp_region
12  network       = google_compute_network.main.id
13}
14
15resource "google_compute_firewall" "allow_flask" {
16  name    = "allow-flask-5000"
17  network = google_compute_network.main.name
18
19  allow {
20    protocol = "tcp"
21    ports    = ["5000"]
22  }
23
24  source_ranges = ["0.0.0.0/0"]
25}
26
27resource "google_compute_instance" "flask_app" {
28  name         = var.instance_name
29  machine_type = "e2-micro"
30  zone         = "${var.gcp_region}-a"
31
32  boot_disk {
33    initialize_params {
34      image = "ubuntu-os-cloud/ubuntu-2204-lts"
35    }
36  }
37
38  network_interface {
39    subnetwork = google_compute_subnetwork.main.id
40    access_config {}
41  }
42
43  metadata = {
44    startup-script = templatefile("${path.module}/../../scripts/gcp-startup.sh.tpl", {
45      app_py = file("${path.module}/../../app/app.py")
46    })
47  }
48}

Root Main and Outputs

terraform/main.tf wires both modules together:

 1module "aws" {
 2  source      = "./modules/aws"
 3  aws_region  = var.aws_region
 4  key_name    = var.aws_key_name
 5  vpc_cidr    = "10.0.0.0/16"
 6  subnet_cidr = "10.0.1.0/24"
 7}
 8
 9module "gcp" {
10  source         = "./modules/gcp"
11  gcp_project_id = var.gcp_project_id
12  gcp_region     = var.gcp_region
13  network_name   = "flask-network"
14  subnet_name    = "flask-subnet"
15  subnet_cidr    = "10.1.0.0/24"
16  instance_name  = "flask-app-vm"
17}

After terraform apply, the outputs give you both public IPs:

1terraform output aws_ec2_public_ip   # paste into Kestra flow input
2terraform output gcp_vm_public_ip

Step 2: The Flask App — CrashLab API

The Flask app is intentionally designed to generate real failures. It runs as a systemd service on both VMs and exposes four endpoints.

Intentional Chaos — 30% Error Rate

 1@app.route("/api/data")
 2def get_data():
 3    # Simulate ~30% DB timeout errors for Kestra AI monitoring demo
 4    if random.random() < 0.3:
 5        logger.error("Database timeout: connection pool exhausted after 30s")
 6        return jsonify({"error": "Database timeout", "code": "DB_TIMEOUT"}), 500
 7
 8    logger.info("Data request successful")
 9    return jsonify({
10        "status": "success",
11        "data": {
12            "id": random.randint(1, 1000),
13            "value": round(random.uniform(10.0, 99.9), 2),
14            "timestamp": datetime.utcnow().isoformat(),
15        },
16    })

Every 3 in 10 requests to /api/data returns a real HTTP 500 with a "Database timeout" error logged to /app/logs/app.log. This gives Kestra's AI agent something real to detect and diagnose.

Health Check Endpoint

1@app.route("/health")
2def health():
3    logger.info("Health check requested")
4    return jsonify({"status": "healthy", "timestamp": datetime.utcnow().isoformat()})

The /health endpoint is the target of the Kestra health check flow. It always returns 200 while the app is running — the failure scenario is when the app process itself dies.

Log Endpoint — No SSH Needed

 1@app.route("/logs")
 2def get_logs():
 3    log_file = "/app/logs/app.log"
 4    lines = int(request.args.get("lines", 50))
 5    try:
 6        with open(log_file, "r") as f:
 7            content = f.readlines()
 8        recent = "".join(content[-lines:])
 9        return jsonify({"lines": lines, "logs": recent})
10    except FileNotFoundError:
11        return jsonify({"lines": 0, "logs": ""}), 200

This endpoint returns the last N lines of the app log over plain HTTP. Kestra reads this endpoint instead of using SSH — making the log monitoring flow much simpler and more portable.

The app also has a /api/slow endpoint that randomly delays responses between 100ms and 2 seconds to simulate a slow upstream service:

1@app.route("/api/slow")
2def slow_endpoint():
3    delay = random.uniform(0.1, 2.0)
4    if delay > 1.5:
5        logger.warning(f"Slow response detected: {delay:.2f}s latency")
6    time.sleep(delay)
7    return jsonify({"status": "success", "latency_ms": round(delay * 1000)})

Step 3: Running Kestra with Docker

Kestra runs locally as a single Docker container. This is the fastest way to get started:

1docker run --pull=always -it -p 8080:8080 --user=root \
2  --name kestra --restart=always \
3  -v kestra_data:/app/storage \
4  -v /var/run/docker.sock:/var/run/docker.sock \
5  -v /tmp:/tmp \
6  kestra/kestra:latest server local

Once running, open http://localhost:8080 in your browser. You will see the Kestra UI where you can create and manage flows.

Key Kestra concepts used in this project:

Concept	Description
`tasks`	Ordered list of tasks that run on success
`errors`	Tasks that run only when any task in `tasks` fails
`triggers`	Schedules, webhooks, or events that start the flow
`inputs`	Dynamic parameters passed at runtime or via trigger
`pluginDefaults`	Default config applied to all tasks of a given type

Step 4: AI Health Check Flow

This is the heart of the project. The full flow YAML:

 1id: ai-health-check
 2namespace: devops.monitor
 3
 4inputs:
 5  - id: aws_host
 6    type: STRING
 7    description: "Public IP of the AWS EC2 instance"
 8
 9triggers:
10  - id: every_hour
11    type: io.kestra.plugin.core.trigger.Schedule
12    cron: "0 */1 * * *"
13    inputs:
14      aws_host: "YOUR_EC2_PUBLIC_IP"
15
16tasks:
17  - id: health_check
18    type: io.kestra.plugin.core.http.Request
19    uri: "http://{{ inputs.aws_host }}:5000/health"
20    method: GET
21
22  - id: ai_diagnosis
23    type: io.kestra.plugin.ai.agent.AIAgent
24    systemMessage: |
25      You are an SRE. The Flask application health check has just failed on AWS EC2.
26      Respond in a SINGLE LINE with no line breaks, using this format:
27      Cause: <likely cause> | Fix: <immediate remediation step>      
28    prompt: |
29      The Flask app at {{ inputs.aws_host }}:5000 failed its health check.
30      Diagnose the most likely cause and provide an immediate remediation step.      
31
32errors:
33  - id: slack_alert
34    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
35    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
36    payload: |
37      {
38        "text": "🔴 *Flask App Health Check Failed (AWS EC2)*\n{{ outputs.ai_diagnosis.output }}"
39      }      
40
41pluginDefaults:
42  - type: io.kestra.plugin.ai.agent.AIAgent
43    values:
44      provider:
45        type: io.kestra.plugin.ai.provider.GoogleGemini
46        modelName: gemini-2.5-flash
47        apiKey: "{{ secret('GEMINI_API_KEY') }}"
48        configuration:
49          logRequests: true
50          logResponses: true
51          responseFormat:
52            type: TEXT

How the Kestra Errors Block Works

This is the key design decision in the flow:

The health_check task makes a plain HTTP GET request to /health:5000.
Kestra marks the task as FAILED automatically if the response is non-2xx.
When a task fails, Kestra skips all remaining tasks in the tasks block and jumps directly to the errors block.
The errors block runs regardless of which task failed — no string matching, no if conditions needed.

This is the Kestra-recommended pattern for alerting, as described in the official docs.

1tasks:
2  health_check  ──── ✅ 200 OK ──────────────────▶ Flow Complete
3                │
4                └─── ❌ non-200 / timeout ──▶ errors block
5                                                    └─▶ slack_alert

The AI Agent Task — Google Gemini 2.5 Flash

The ai_diagnosis task uses the io.kestra.plugin.ai.agent.AIAgent plugin. The AI is configured with two prompts:

systemMessage: Sets the persona — an SRE who must respond in a single line with Cause: | Fix: format.
prompt: Gives the failure context — the host that failed and what it was doing.

The pluginDefaults block at the bottom of the YAML applies the Gemini provider configuration to all AIAgent tasks in the flow. This keeps the task declaration clean and lets you swap providers (OpenAI, Anthropic, etc.) in one place.

Supported providers that work with Kestra AI Agent:

Provider	Models
Google Gemini	gemini-2.5-flash, gemini-1.5-pro
OpenAI	gpt-4o, gpt-4o-mini
Anthropic	claude-3-5-sonnet, claude-3-haiku

Slack Alert with AI Diagnosis

The slack_alert task in the errors block sends the AI-generated diagnosis to Slack:

1errors:
2  - id: slack_alert
3    type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
4    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
5    payload: |
6      {
7        "text": "🔴 *Flask App Health Check Failed*\n{{ outputs.ai_diagnosis.output }}"
8      }

A typical Slack message produced by this flow looks like:

1🔴 Flask App Health Check Failed
2Cause: Flask process crashed or port 5000 is not listening |
3Fix: SSH into the EC2 instance and run: sudo systemctl restart flask-app

Step 5: AI Log Monitor Flow

The second Kestra flow runs on the same hourly schedule and generates real traffic against the app to produce log entries for analysis:

 1id: ai-log-monitor
 2namespace: devops.monitor
 3
 4triggers:
 5  - id: every_hour
 6    type: io.kestra.plugin.core.trigger.Schedule
 7    cron: "0 */1 * * *"
 8
 9inputs:
10  - id: aws_host
11    type: STRING
12    defaults: "YOUR_EC2_PUBLIC_IP"
13
14tasks:
15  # 1. Confirm the app is healthy first
16  - id: health_check
17    type: io.kestra.plugin.core.http.Request
18    uri: "http://{{ inputs.aws_host }}:5000/health"
19    method: GET
20
21  # 2. Hit /api/data to generate real traffic and errors in the log
22  - id: ping_api
23    type: io.kestra.plugin.core.http.Request
24    uri: "http://{{ inputs.aws_host }}:5000/api/data"
25    method: GET

The ping_api task deliberately hits /api/data which has the 30% error rate. This means on average, 3 out of 10 Kestra executions will produce an ERROR line in the app log — giving you real noise to monitor.

Step 6: CloudWatch Agent for Log Streaming

The CloudWatch Agent streams /app/logs/app.log from the EC2 instance to AWS CloudWatch Logs for retention and visibility.

Step 1: Install the agent

1curl -fsSL https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb \
2  -o /tmp/amazon-cloudwatch-agent.deb
3
4sudo dpkg -i /tmp/amazon-cloudwatch-agent.deb || sudo apt-get install -f -y

Step 2: Create the configuration

 1sudo mkdir -p /opt/aws/amazon-cloudwatch-agent/etc
 2
 3sudo tee /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json > /dev/null <<'EOF'
 4{
 5  "logs": {
 6    "logs_collected": {
 7      "files": {
 8        "collect_list": [
 9          {
10            "file_path": "/app/logs/app.log",
11            "log_group_name": "/flask-app/ec2/app-logs",
12            "log_stream_name": "{instance_id}",
13            "timezone": "UTC",
14            "timestamp_format": "%Y-%m-%d %H:%M:%S,%f"
15          }
16        ]
17      }
18    }
19  },
20  "agent": {
21    "region": "eu-central-1",
22    "logfile": "/var/log/amazon-cloudwatch-agent.log"
23  }
24}
25EOF

Step 3: Start the agent

1sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
2  -a fetch-config \
3  -m ec2 \
4  -s \
5  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

Step 4: Verify

1sudo systemctl status amazon-cloudwatch-agent

After this, your Flask app logs will appear in the AWS CloudWatch Logs console under the log group /flask-app/ec2/app-logs. You can create CloudWatch Alarms on top of these logs as an additional alerting layer.

How It All Works Together

Here is the full end-to-end sequence:

terraform apply provisions an AWS EC2 instance and a GCP VM. The startup scripts install Python, create the systemd service, and start the Flask app automatically — no manual SSH required.
Kestra starts in Docker. You import the two YAML flows from the kestra/ directory via the UI or REST API.
Every hour, the ai-health-check flow fires. It hits http://<EC2_IP>:5000/health.
If the app is healthy, the flow completes silently. Nothing happens. No noise.
If the app is down (process crashed, port not listening, non-200 response), Kestra immediately jumps to the errors block:
- The ai_diagnosis AI Agent task sends the failure context to Gemini 2.5 Flash.
- Gemini responds with a one-line root cause analysis and remediation step.
- The slack_alert task fires with the AI diagnosis embedded in the Slack message.
In parallel, the ai-log-monitor flow generates real traffic and reads logs directly over HTTP from the /logs endpoint — no SSH, no CloudWatch query, no Lambda.
CloudWatch Agent streams the app log continuously to AWS CloudWatch Logs for longer retention and for building dashboards or metric-based alarms.

The AI is the key piece: it is sandwiched between failure detection and human notification. It wakes up only when something breaks, produces an actionable one-liner, and goes back to sleep.

Conclusion

We built a fully automated AI-powered cloud monitoring system that:

Provisions real multi-cloud infra on AWS and GCP with a single terraform apply
Runs a chaos-engineered Flask app that generates real errors for the AI to catch
Uses Kestra's errors block for clean, declarative failure handling — no if-else chains
Embeds Google Gemini 2.5 Flash as an AI Agent that only runs when the app fails
Sends an AI-written incident diagnosis to Slack automatically
Streams logs to CloudWatch for observability beyond the Kestra execution window

The most important lesson: the AI does not replace your alerting system. It makes your alerts smarter. Instead of getting a Slack message that says "health check failed", you get one that says "Cause: OOM kill on t2.micro | Fix: restart flask-app service and increase swap".

That is the difference between knowing something is broken and knowing how to fix it.

Happy Building!

Posts in this series