AI Agent DevOps Server

1Full stack architecture

Complete DevOps stack for AI agents deployed on a single EC2 g4dn instance with GPU acceleration.

Important system paths

# DevOps runtime
/opt/devops-server/
  app/                    # FastAPI code
    main.py
    config.py
    routers/              # health, evaluate, compare, traces
    utils/                # ollama_judge, langfuse_client
  venv/                   # Python 3.12 with deps
  logs/                   # api.log, api-error.log
  docker-compose.yml      # Langfuse + PostgreSQL + Redis

systemd services

devops-api.service    -> uvicorn app.main:app on :8000
docker.service        -> manages Langfuse, PostgreSQL, Redis containers
ollama.service        -> LLM judge runtime on :11434

2API endpoints

REST API documented with OpenAPI. Swagger UI available at /docs.

Request schemas

# POST /evaluate
{
  "question": "string",
  "answer": "string",
  "context": "string (optional)",
  "agent_name": "string (optional)",
  "trace_id": "string (optional)"
}

# POST /compare
{
  "question": "string",
  "answer_a": "string",
  "answer_b": "string",
  "context": "string (optional)"
}

# POST /traces
{
  "name": "string",
  "input": "string",
  "output": "string",
  "metadata": {"key": "value"}
}

3curl examples

Healthcheck

curl -s http://localhost:8000/health | jq

Evaluate an answer

curl -X POST http://localhost:8000/evaluate \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is the capital of France?",
    "answer": "The capital of France is Paris.",
    "context": "France is a country in Europe. Its capital city is Paris.",
    "agent_name": "geography-agent"
  }' | jq

Compare two versions

curl -X POST http://localhost:8000/compare \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What is recursion?",
    "answer_a": "Recursion is when a function calls itself.",
    "answer_b": "Recursion is a programming technique where a function calls itself to solve smaller instances of the same problem.",
    "context": "Recursion is a fundamental concept in computer science."
  }' | jq

Create a trace

curl -X POST http://localhost:8000/traces \
  -H "Content-Type: application/json" \
  -d '{
    "name": "test-trace",
    "input": "What is Python?",
    "output": "Python is a high-level programming language.",
    "metadata": {"agent": "test", "version": "1.0"}
  }' | jq

4Evaluation metrics

How LLM-as-Judge works

1. You submit a question, an answer, and optional context
2. The API sends a structured prompt to the judge LLM (llama3.1:8b)
3. The LLM returns a numeric score from 0 to 1
4. The API parses the score and returns it as JSON

Available metrics

Change the judge model

Edit /opt/devops-server/app/config.py:

judge_model: str = "llama3.1:8b"

Recommended alternatives:

• llama3.2:3b - faster, lower quality
• qwen2.5:14b - slower, higher quality
• mistral:7b - balanced option

After change: sudo systemctl restart devops-api

5Langfuse observability

What is Langfuse

Langfuse is an open-source observability platform for LLM applications. It stores traces (inputs, outputs, scores) and provides a UI to browse, filter and chart them.

Access

http://<IP>:3000

Initial setup (first time only)

1. Open Langfuse UI
2. Create admin account (email + password)
3. Create organization and project
4. Go to Settings - API Keys - Create new API key
5. Copy Public Key and Secret Key

Configure API keys

sudo -u ubuntu tee -a /opt/devops-server/.env > /dev/null <<'EOF'

LANGFUSE_PUBLIC_KEY=pk-lf-xxxxxxxxxxxxxxxx
LANGFUSE_SECRET_KEY=sk-lf-xxxxxxxxxxxxxxxx
EOF

sudo systemctl restart devops-api

Container architecture

devops-langfuse-server   Langfuse UI and API      :3000
devops-postgres          Metadata storage         :5432 (internal)
devops-redis             Cache and queues         :6379 (internal)

6systemd services

Basic commands

# Status
sudo systemctl status devops-api docker ollama

# Restart the API
sudo systemctl restart devops-api

# Restart Docker containers
sudo docker restart devops-langfuse-server devops-postgres devops-redis

# Check if enabled at boot
sudo systemctl is-enabled devops-api docker ollama

devops-api service structure

# /etc/systemd/system/devops-api.service
[Unit]
Description=AI Agent DevOps Server API
After=network.target docker.service
Wants=docker.service

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/opt/devops-server
Environment="PATH=/opt/devops-server/venv/bin:/usr/local/bin:/usr/bin:/bin"
ExecStart=/opt/devops-server/venv/bin/uvicorn \
  app.main:app --host 0.0.0.0 --port 8000 --workers 1
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

7Logs and monitoring

API logs

sudo journalctl -u devops-api -f
sudo journalctl -u devops-api -n 100 --no-pager
tail -f /opt/devops-server/logs/api.log
tail -f /opt/devops-server/logs/api-error.log

Container logs

sudo docker logs -f devops-langfuse-server
sudo docker logs -f devops-postgres
sudo docker logs -f devops-redis
sudo docker stats

GPU monitoring

nvidia-smi
watch -n 2 nvidia-smi
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Ports

sudo ss -tulpn | grep -E ':(3000|5432|6379|8000|11434)'

8Troubleshooting

Quick diagnostic

# Services
systemctl is-active devops-api docker ollama

# Ports
sudo ss -tulpn | grep -E ':(3000|5432|6379|8000|11434)'

# API health
curl -s http://localhost:8000/health | jq

# Ollama models
ollama list

# Docker containers
sudo docker ps

# GPU
nvidia-smi

9Backup and restore

What to back up

• /opt/devops-server/app/ - your customized code
• /opt/devops-server/.env - configuration including Langfuse keys
• PostgreSQL volume devops-server_postgres-data - all traces and scores
• Redis volume devops-server_redis-data - cache and queues

Full backup

sudo tar -czf /tmp/devops-backup-$(date +%Y%m%d).tar.gz \
  /opt/devops-server/app \
  /opt/devops-server/.env

sudo docker run --rm \
  -v devops-server_postgres-data:/data \
  -v /tmp:/backup \
  alpine tar czf /backup/devops-postgres.tar.gz -C /data .

scp ubuntu@your-ip:/tmp/devops-backup-*.tar.gz ./

Restore

sudo tar -xzf devops-backup-YYYYMMDD.tar.gz -C /
sudo systemctl restart devops-api
sudo docker restart devops-langfuse-server devops-postgres