Inference at the Edge: Why Real-Time AI Can't Live in the Cloud

In the cloud, inference scales horizontally. Need more throughput? Add more GPUs. Latency too high? Move to a closer region. The cloud model is elastic by design.

At the venue edge, none of that applies.

Fixed-budget inference#

Every OZ VI Venue has a fixed compute envelope: a known GPU, a known power budget, and a known thermal ceiling. The AI models that run on-venue must fit within that envelope, not occasionally, but every frame, every second, every match day.

OZ Cortex is the runtime that enforces this contract.

What Cortex manages#

• Model registry: which models are deployed, which versions, which configurations
• Pipeline orchestration: multi-model execution order, data flow between stages
• Latency budgets: per-model and per-pipeline time allocations with hard deadlines
• Resource allocation: GPU memory, compute cycles, and I/O bandwidth partitioning
• Health monitoring: inference latency tracking, anomaly detection, automatic degradation

Optimization is not optional#

A model that runs in 50ms on a cloud A100 might take 200ms on a venue GPU. Cortex manages the optimization pipeline: quantization, pruning, operator fusion, and batch tuning, all validated against accuracy thresholds before deployment.

The goal is not maximum accuracy. The goal is maximum accuracy within the latency budget. A model that misses its deadline is worse than a model that runs faster with slightly lower precision.

Deterministic execution#

Cortex guarantees that the inference pipeline completes within its time window. If a model exceeds its budget, the runtime doesn't wait. It escalates, degrades gracefully, or skips the frame. The venue keeps operating. The control loop does not stall.

This is the fundamental difference between cloud AI and edge AI: the edge does not have the luxury of retrying.