ovh_gridscale_test_task/README.md

# Technical Writeup: OVH Gridscale, Caddy & Prometheus Infrastructure

## Overview

This document provides a comprehensive technical writeup of the infrastructure deployment and configuration management workflow. The project provisions cloud resources on **OVH / Gridscale** using **Terraform** and automates host configuration using **Ansible**.

---

## 1. Initial Infrastructure Setup & Technical Caveats

### 1.1 Terraform Provider & Resource Foundation
The initial configuration (`main.tf`, `variables.tf`, `resources.tf`) established authentication with the official Gridscale provider (`gridscale/gridscale`). Infrastructure resources were defined using the **Debian 13** image template and SSH key authentication.

### 1.2 Key Failures & Solutions

During initial testing and deployment, several technical hurdles were encountered and resolved:

1. **SSH Connection Timeouts (Public Network Requirement)**:
   * **Issue**: Initial attempts to connect via SSH to the server's public IPv4 address timed out (`ssh: connect to host ... port 22: Operation timed out`).
   * **Root Cause & Fix**: Gridscale instances require explicit attachment to the public network. Because no dynamic network UUID data lookup was provided in the task specification, the public network UUID (`5557a73b-31ee-4b1f-aa15-7789ad6ae04c`) was hardcoded directly into the server's network block in `resources.tf`.

2. **Unpowered Server Instances**:
   * **Issue**: Newly created instances remained offline and unreachable after `terraform apply`.
   * **Root Cause & Fix**: Gridscale creates servers in an unpowered state by default. Setting `power = true` explicitly in `gridscale_server` resources ensured servers automatically boot upon creation.

3. **Ansible SSH Host Key Verification Failures**:
   * **Issue**: Ansible fact-gathering failed with `Host key verification failed`.
   * **Root Cause & Fix**: Cloud IPs change dynamically on fresh deployments. Added `ansible_ssh_extra_args='-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'` to the `[all:vars]` block in `inventory.tf` to bypass interactive host key verification prompts seamlessly.

---

## 2. Network Topology & Firewall Security

### 2.1 Multi-Server Topology
The architecture consists of two dedicated nodes:
* **`caddy01`**: Web server hosting Caddy.
* **`monitoring01`**: Telemetry node hosting Prometheus.

### 2.2 Dual-Interface Network Isolation & Inbound Firewall Rules
To secure management endpoints and telemetry data, instances were configured with dual network interfaces:

1. **Public Network**:
   * Attached to Gridscale public network UUID `5557a73b-31ee-4b1f-aa15-7789ad6ae04c`.
   * **Firewall Rules (`rules_v4_in`, `rules_v6_in`)**:
     * Allow TCP port `22` (SSH).
     * Allow TCP port `80` (HTTP).
     * Block all other inbound public traffic.

2. **Private Network (`gridscale_network.network_internal`)**:
   * Internal DHCP subnet: `192.168.121.0/27` (Gateway: `192.168.121.1`, DNS: `192.168.121.2`, Reserved: `192.168.121.0/31`).
   * **Firewall Rules**:
     * On `caddy01`, allows inbound TCP port `2019` (Caddy Admin API / Metrics) **only** from the internal CIDR `192.168.121.0/27`. Public access to port 2019 is strictly denied at the firewall layer.

---

## 3. Dynamic Ansible Inventory Generation

### 3.1 Automated Inventory Build (`inventory.tf`)
To eliminate manual IP tracking, Terraform's `local_file` resource generates `inventory.ini` dynamically upon `terraform apply`.

### 3.2 Dynamic IP Interpolation
- Extracts auto-assigned private IP addresses from resource state: `${gridscale_server.server_caddy.network[1].auto_assigned_ip}`.
- Automatically exports host groups (`[caddy]`, `[monitoring]`) and internal IP variables (`privat_ip_caddy`, `privat_ip_monitoring`) for Ansible playbooks.
- Exposes outputs `caddy_public_ip`, `monitoring_public_ip`, `caddy_private_ip`, and `monitoring_private_ip` in `outputs.tf`.

---

## 4. Ansible Playbook Orchestration (`playbook.yml`)

### 4.1 Base System Maintenance & Idempotent Reboot (`all`)
* **Package Updates**: Executes `apt update` and `upgrade: dist`.
* **Idempotent Reboot Check**: Uses `ansible.builtin.stat` to check for `/var/run/reboot-required`. The reboot task triggers *only* when required (`when: reboot_required_file.stat.exists == true`), avoiding unnecessary reboot delays during repeat runs.
* **Prerequisites**: Installs `debian-keyring`, `debian-archive-keyring`, `apt-transport-https`, and `curl`.

### 4.2 Caddy Web Server Configuration (`caddy`)
* **Official Deb822 Repository**: Per Caddy official documentation, configured Caddy's Debian repository using `ansible.builtin.deb822_repository` (`https://dl.cloudsmith.io/public/caddy/stable/deb/debian` with official GPG key verification).
* **Metrics & Admin API Security**:
  * Configured `Caddyfile` with `/metrics` and bound the Admin API to `:2019`.
  * Set `origins http://{{ privat_ip_caddy }}:2019` so Caddy validates origin headers and only processes authorized internal requests.
* **Service Handlers**: Uses Ansible handlers to restart the `caddy` service automatically upon `Caddyfile` updates.

### 4.3 Prometheus Monitoring Setup (`monitoring`)
* **Installation**: Installs the `prometheus` package on `monitoring01`.
* **Private Scrape Target**: Configured `prometheus.yml` to scrape Caddy metrics target `{{ privat_ip_caddy }}:2019` strictly over the internal private network (`192.168.121.0/27`).
* **Service Handlers**: Triggers `Restart Prometheus service` automatically when `prometheus.yml` changes.

---

## 5. Execution Guide

### 5.1 Provision Infrastructure (Terraform)
```bash
# Initialize Terraform providers
terraform init

# Apply infrastructure changes
terraform apply -auto-approve
```

### 5.2 Run Ansible Provisioning

```bash
# Provision Caddy Web Server
uv run ansible-playbook -i inventory.ini -l caddy playbook.yml

# Provision Monitoring Server
uv run ansible-playbook -i inventory.ini -l monitoring playbook.yml

# Provision All Servers
uv run ansible-playbook -i inventory.ini playbook.yml
```

---

## 6. Modular Jinja2 Templating & Prometheus Alerting Rules

### 6.1 Refactoring Inline Configurations to Jinja2 Templates
To improve maintainability and separate configuration data from playbook logic, inline file content blocks in `playbook.yml` were refactored into dedicated Jinja2 template files located in the `templates/` directory:

* **[templates/Caddyfile.j2](file:///Users/doctor/Git/ovh_gridscale_test_task/templates/Caddyfile.j2)**: Contains Caddy server block definitions and metrics setup.
* **[templates/prometheus.yml.j2](file:///Users/doctor/Git/ovh_gridscale_test_task/templates/prometheus.yml.j2)**: Contains global scrape configuration and imports rule files via `rule_files: [/etc/prometheus/alert_rules.yml]`.
* **[templates/alert_rules.yml.j2](file:///Users/doctor/Git/ovh_gridscale_test_task/templates/alert_rules.yml.j2)**: Defines Prometheus alert rules for monitoring Caddy service availability and error rates.

The playbook tasks were updated to use `ansible.builtin.template` instead of `ansible.builtin.copy`.

### 6.2 Variable Naming Refactoring
Standardized variable naming from `privat_ip_*` to `private_ip_*` across:
* `inventory.tf` (Terraform dynamic inventory builder)
* `inventory.ini` (Generated inventory)
* `playbook.yml` & Jinja2 templates (`{{ private_ip_caddy }}`)

### 6.3 Prometheus Alert Rules & Raw Jinja Escaping
Configured alerting rules in `templates/alert_rules.yml.j2` using Jinja `{% raw %}` ... `{% endraw %}` blocks to prevent Ansible from misinterpreting Prometheus template variables (such as `{{ $labels.instance }}` and `{{ $value }}`):

1. **`Caddy Down`**: Triggers a critical alert if `up{job="caddy"} == 0` for 1 minute.
2. **`CaddyHighHTTP4xxErrorRateService`**: Triggers a critical alert if the 4xx HTTP error rate exceeds 5% over a 3-minute window (with total request count > 0).
   * *Source*: Rule specification adapted from [Awesome Prometheus Alerts (Caddy Rules)](https://samber.github.io/awesome-prometheus-alerts/rules/proxies-load-balancers-and-service-meshes/caddy/).

### 6.4 Verification
Verified playbook execution and rule rendering:
```bash
uv run ansible-playbook -i inventory.ini -l monitoring playbook.yml
```

### 6.5 Automated Playbook Execution via Terraform (`null_resource`)
To fully automate the end-to-end deployment, a `null_resource` provisioner was integrated into `inventory.tf`. It executes automatically as the final step of `terraform apply` after server creation and inventory file generation:

```hcl
resource "null_resource" "ansible_provisioner" {
  depends_on = [
    local_file.ansible_inventory,
    gridscale_server.server_caddy,
    gridscale_server.server_monitoring
  ]

  provisioner "local-exec" {
    command = "uv run ansible-playbook -i ${local_file.ansible_inventory.filename} playbook.yml"
  }
}
```

---

## 7. End-to-End Environment Setup & Deployment Workflow

### 7.1 Environment Prerequisites & Credentials (`.env`)
Copy the `.env.example` template and export your Gridscale API credentials into your shell session:

```bash
# Copy template environment file
cp .env.example .env

# Edit .env with your credentials, then export variables
source .env
```

### 7.2 Python Environment Initialization & Dependency Setup (`uv`)
Initialize Python environment and install Ansible using `uv`:

```bash
# Initialize uv virtual environment and lockfile
uv init --bare

# Add Ansible dependency
uv add "ansible>=14.2.0"
```

### 7.3 Infrastructure Provisioning & Automated Execution
Initialize Terraform and apply the configuration. Terraform will provision the servers, dynamically construct `inventory.ini`, and automatically trigger the Ansible playbook via the `null_resource`:

```bash
# Initialize Terraform providers
terraform init

# Apply infrastructure changes (automatically provisions servers & runs Ansible)
terraform apply -auto-approve
```

*(Optional)* Execute Ansible playbooks manually via `uv`:
```bash
# Run Ansible playbook for all hosts
uv run ansible-playbook -i inventory.ini playbook.yml

# Target specific host groups
uv run ansible-playbook -i inventory.ini -l caddy playbook.yml
uv run ansible-playbook -i inventory.ini -l monitoring playbook.yml
```