Challenge: CS2Coach automatically turns Counter-Strike 2 match recordings into ready-to-watch highlight reels — parsing a demo file, scoring and selecting the key moments, rendering fragments and stitching them into an annotated video, with no human reviewer in the loop. At a target of 1000 matches a day, the core engineering problem was resource cost: automated rendering out of a game engine is extremely GPU-heavy, and renting that much cloud GPU capacity would have been prohibitively expensive. The infrastructure had to be cheap, reliable, and horizontally scalable at the same time — while still hitting strict speed and quality targets.
Solution: Working as part of a 3-engineer team, I designed the infrastructure and orchestration layer against an agreed technical spec: a central scheduler and Redis-backed task queue coordinating a hybrid pool of 20+ local PCs and GPU servers, with capability-aware routing sending heavy renders to strong nodes and light segments to weaker machines. On top of the scheduler I built the reliability layer — heartbeats, acknowledgement-based task delivery, exponential backoff and a dead-letter queue — plus the CI/CD pipeline, containerized agent rollout, and Prometheus/Grafana observability needed to run it safely in production.
Result: The system sustains 1000+ match analyses a day at under 6 minutes average processing time, with an error rate held below 2% and 99% target availability — all at close to zero cloud GPU spend, since rendering runs on a self-owned machine pool instead of rented infrastructure. Adding a new node to the pool is a one-command, one-click operation, and releases ship with blue/green deploys that roll back in under 5 minutes if something goes wrong.
Each match travels through a fixed pipeline: API Gateway → Redis task queue → Scheduler → Parser/Analytics → Render Pool (20+ PCs + GPU servers) → Stitcher → Storage + webhook callback. The scheduler is the central decision point — it reads agent capabilities and current load from Redis and routes each segment to whichever node can actually handle it, rather than round-robining blindly across a pool where every machine is different.
The 20 local PCs had different CPUs and GPUs, could drop offline at any moment, and sat behind an unstable network link — the opposite of a clean, homogeneous cloud fleet. I used Redis as the broker rather than a dedicated message queue like RabbitMQ, since Redis was already needed for status tracking and rate limiting, and its throughput comfortably covered this volume. Agent health is tracked via Redis TTL keys: when a key expires, the scheduler knows instantly that a machine dropped, with no separate health-check service to maintain. Jobs are routed by weighted capability — GPU type and core count — so heavy renders land on strong nodes and light segments go to weaker PCs, with a priority queue (a Redis sorted set) so urgent jobs never sit behind the general backlog. The result: the pool behaves as a single system, and a newly added machine gets picked up automatically with zero downtime.
At 1000 matches a day, an agent crash or dropped connection could silently hang or lose a task — unacceptable at that volume. I built an acknowledgement model with a visibility timeout: a task only counts as done after an explicit ack, otherwise it's automatically returned to the queue and re-assigned. Retries use exponential backoff (30s → 2m → 10m) so a transient failure doesn't flood the system with instant retries, and a dead-letter queue catches tasks that fail three times — separating genuinely broken demo files from systemic infrastructure issues instead of looping forever. Redis persistence (AOF) plus graceful shutdown means the queue survives a scheduler restart and an agent always finishes its current job before going offline. The outcome held error rate below 2%, with a single agent failure no longer affecting the overall processing stream.
A single Redis instance and a single API instance couldn't sustain 1000+ tasks a day with realistic peaks. I moved to Redis Cluster to spread load across shards and remove the single point of failure, and put nginx/HAProxy in front of multiple API instances for horizontal scaling and automatic traffic shedding away from unhealthy instances via health checks. I load-tested against a full 1000-matches-per-day simulation plus a stress test at 80 matches/hour — double the expected peak — and tuned PostgreSQL with added indexes and batched status updates based on what the profiling showed. The system held target load with average match processing time under 6 minutes.
Manually configuring each of the 20 machines — headless CS2, GPU-enabled FFmpeg, the agent, networking — would have been slow and error-prone. I built a Dockerized agent image bundling every dependency, so every machine in the fleet runs an identical environment with no manual library-version fiddling, plus an installer script that sets up the environment, pulls the image, and writes scheduler and networking config in a single command. Before a machine joins the live pool it runs a dry-run test task, so a misconfigured node never enters the production stream, and fleet-wide updates ship centrally via image re-deploy rather than touching each PC by hand. Adding a new machine came down to running one script.
The pool mixed NVIDIA, AMD and Intel GPUs, and parallel rendering could easily exhaust video memory on any of them. I used FFmpeg with automatic encoder detection (NVENC / VCE / QSV) so one pipeline covers all three vendors instead of maintaining three separate ones, defaulting to hardware encoding rather than CPU-based libx264 — software encoding would have been far too slow to hit the processing-time budget, and is kept only as a fallback. A cap of 5–8 concurrent processes per GPU, tuned empirically and configurable per card model, combined with live VRAM monitoring via nvidia-smi and equivalents, keeps every node encoding at 1080p/60fps without exceeding its memory limits regardless of GPU vendor.
Early on there was no visibility into the system — when something broke, it was hard to tell which agent a task was stuck on or at which stage. I set up Prometheus and Grafana dashboards covering the API, the queues, and every individual agent (CPU/GPU load, job counts), with threshold alerts on error rate, queue growth and agent offline events, so problems surface as an alert instead of a user complaint. Structured logs with an end-to-end correlation ID let a single match's path be reconstructed across every module in one query. Releases go out via blue/green deploys with a canary stage rolling to 10% of traffic first, alongside sandboxed containers, TLS and secrets managed in Vault — giving a rollback path under 5 minutes if a release needs to be reverted.
