Continuing my experiments with hosting models locally on the Docker + Ollama stack.
Tried different models for different jobs — from lightweight ones for quick answers to heavier ones for working with code. On consumer hardware the quality is noticeably behind cloud APIs, but for isolated scenarios or experimentation it's a perfectly workable option.
Tried hosting language models locally — interesting both from a privacy standpoint and for offline scenarios.
The core stack: Docker Desktop for isolation and a reproducible environment + Ollama as the runtime for running the models. A container with a model comes up with a single command, the environment config doesn't sprawl across the system — and the sheer fact that an LLM is spinning on my own hardware without any clouds still surprises me a little.