KredosAI
Voice agent, 2026 to now- FastAPI and vLLM microservices serving quantized LLMs on GPU, 100+ calls a day
- Tuned KV cache and request batching to stay under 500ms
- Async Python handles 10 simultaneous calls without added lag; input sanitizing cut prompt-injection vectors 45%
- Moved from LiveKit Cloud to a self-hosted stack so customer audio stays in-house