Codacus dropped a killer guide for hosting providers and self-hosters looking to slash compute costs. Instead of slow token-by-token JSON generation, he replicated Jev's architecture directly in llama.cpp using a new /v1/decision server endpoint.

By forcing the model to choose from pre-defined schemas rather than write text, response times plummeted from 3415ms to just 306ms (17.3ms per decision in bulk!) using a tiny 2GB model in 2.4GB of VRAM. A massive win for lightweight, high-throughput API endpoints.

Making a model pick from pre-defined schemas instead of generating tokens is a classifier with extra steps. Sure, 306ms beats 3415ms, but if you ever needed free-form output, this fails hard.

It's a neat hack for routing, not a revolution. Stop hyping and ship benchmarks with real workloads, not cherry-picked bulk numbers.