Kokoro TTS
Gemma 4 26B-A4B QAT (full 262,144-token context)
Generation settings
Whisper large-v3-turbo
PaddleOCR-VL-1.6
Specialized OCR model — use the trained "OCR:" prompt; custom instructions make results worse.
Qwen3-Embedding-4B (CPU)
Runs on CPU only, doesn't touch GPU/VRAM. 2560-dimensional vectors.
Silero VAD (CPU)
Voice activity detection — returns timestamps of detected speech segments. Runs on CPU only.
Stack Information
Hardware
Single VPS, Tesla V100S-PCIE-32GB (Volta, compute capability 7.0), 43GB system RAM, 394GB disk.
Volta lacks BF16/FP8 tensor cores and current vLLM builds ship with zero compiled kernels for compute capability 7.0. Everything here runs on llama.cpp/CTranslate2-based stacks instead, which still compile Volta kernels.
Models & services
| Service | Model | Compute | Port | Notes |
|---|---|---|---|---|
| Chat | Gemma 4 26B-A4B QAT | GPU | 8000 | MoE, 128 experts (8 active + 1 shared), 3.8B active / 26B total params, full 262,144-token context, ~102 tok/s |
| Text-to-speech | Kokoro TTS (82M) | GPU | 8880 | 9 languages, 68 voices, ~260 words/sec, ~88x real-time on short text |
| Speech-to-text | Whisper large-v3-turbo | GPU | 9000 | faster-whisper (CTranslate2), ~9.7x real-time |
| OCR | PaddleOCR-VL-1.6 | GPU | 8002 | 0.9B params, document layout + reading-order understanding, not just character recognition |
| Embeddings | Qwen3-Embedding-4B | CPU | 8006 | 2560-dim vectors, llama.cpp CPU server |
| Voice activity detection | Silero VAD | CPU | 8005 | Custom FastAPI wrapper, returns speech segment timestamps |
Gateway & routing
nginx (port 80) is the single public entry point. It routes
/v1/chat/completions, /v1/audio/speech, /v1/audio/transcriptions,
and OCR requests to LiteLLM (port 4000), which fans out to the GPU model backends above.
Embeddings (/v1/embeddings) and VAD (/vad) are proxied directly
to their own containers, bypassing LiteLLM, since they don't fit its chat/audio request shape.
Everything else (/) goes to this playground.
Known tradeoffs
- Plain HTTP, no TLS — a self-signed certificate was tried and rejected by default client certificate verification (no real domain is pointed at this VPS's IP, so Let's Encrypt isn't an option). Traffic including the LiteLLM key travels unencrypted.
- No rate limiting anywhere in the stack — the LiteLLM key has unrestricted access, and the embeddings/VAD routes have no auth at all.
- VRAM is the tight resource, not RAM/CPU/disk — Gemma 4 26B-A4B at full context plus PaddleOCR-VL, Kokoro, and Whisper leaves only a few GB of headroom on the 32GB card.