Kindi: Modern Local AI on a $15 GPU

Most local AI advice starts with buying a newer GPU. Kindi starts with the one sitting in the closet. It is a custom inference engine from Amir Husain that runs modern small language models on old NVIDIA cards—including a 2014 Quadro K420 with 1 GB of memory that sells used for around $15. The current Kindi 0.2.0 release has grown well beyond that first experiment: it now runs 2B, 4B, 9B, and 27B models across five card setups, from Kepler-era Quadros to the GeForce RTX 3080. (Kindi project page)

What Kindi can do

Kindi is not a general-purpose model runner like Ollama or LM Studio. It is a hardware-specific inference engine for Prism ML’s Ternary-Bonsai-2 family: 2B, 4B, 9B, and 27B models with ternary weights. The 27B model file is only about 5.9 GB, which is how it fits on an 8 GB GTX 1080 with a 65,536-token context—or on two 4 GB Quadro K1200 cards working together as one 8 GB pool. (Kindi page)

It exposes an OpenAI-compatible server at http://localhost:8093/v1, so chat apps, scripts, and coding agents can talk to it without modification. It supports streaming, tool calls, configurable thinking, prompt caching, parked conversations, and checkpointing for long agent sessions. The default output is token-for-token identical to llama.cpp for the same model file; Kindi also offers a faster “fast head” that matches the exact head 99.4% of the time in the project’s tests. (GitHub)

That makes the practical use cases pretty clear: a private assistant on an old office PC, a small coding agent on a cheap workstation, long-document Q&A on a GTX 1080, or a 2B model on a 1 GB card for short decisions and simple tool tasks. The 27B model on a GTX 1080 passed 63 of 65 agent tasks in Kindi’s own benchmark; the 9B passed 26 of 32 programming and repair tasks on a single K1200. (Kindi page)

What’s new in 0.2.0

The big story is the jump from “a fun 2B demo on one ancient card” to a small model family across multiple hardware tiers.

• 2B to 27B models instead of just 2B.
• Five card setups: Quadro K420 1 GB and 2 GB, Quadro K1200, two K1200 cards, GTX 1080, and RTX 3080.
• Tensor parallelism across two K1200 cards, so the 27B model can run on 8 GB total.
• Speculative decoding on the RTX 3080, using the 2B model as a drafter for the 27B and reaching up to 99.9 tok/s on agent prompts at 190 W, or 108 tok/s at 380 W.
• CPU fallback for machines with no NVIDIA card: the 2B model runs on the CPU with a 24,576-token context.
• One-command installer, Docker packaging, and an OpenAI-compatible API.
• Long-context work: 65,536 tokens for 27B on a GTX 1080, and about 85,000 tokens on an RTX 3080 with the installer’s settings—up to 103,680 without the drafter.

Source: Kindi 0.2.0 feature and hardware tables

Hardware and model compatibility

This is the part every DIY user should read before downloading.

Source: Kindi compatibility table

The platform requirements are important: Linux x86-64, an NVIDIA driver, and Docker. The K420 needs the older NVIDIA 470 driver branch. This is not a macOS app, not a Windows app, and not an AMD engine. If you have an Apple Silicon Mac, Kindi is not for you. If you have an old Linux workstation with a supported NVIDIA card, it is very much for you. (Kindi page, GitHub)

How it compares to llama.cpp, vLLM, Ollama, and LM Studio

Kindi’s pitch is not “better than every inference engine.” It is “much faster on hardware the mainstream engines either don’t support or run poorly.”

The benchmark gap is dramatic on old hardware. On a Quadro K420, Kindi generates the 2B model at 20.8 tok/s versus 1.65 tok/s for llama.cpp. On a single K1200, it hits 81.0 tok/s with the 2B, 39.2 with the 4B, and 23.4 with the 9B—versus 20.8, 8.7, and 5.9 for llama.cpp. Two K1200 cards run the 27B at 13.0 tok/s versus 1.69. On a GTX 1080, Kindi does 27.9 tok/s with the 27B at short context; llama.cpp does 19.4 and runs out of memory at 64K. (Kindi performance tables)

Those are the author’s measurements, using the same card, model file, and context for both engines. They are not independent third-party benchmarks. But the direction is consistent: Kindi’s custom kernels and quantization make old cards useful in a way generic runtimes do not.

Is it good for the DIY and home user?

Yes—if you fit the hardware profile.

The ideal Kindi user has an old Dell or HP workstation, a spare GTX 1080, or a cheap Quadro K1200 lying around, and is comfortable with Linux and Docker. For that person, Kindi is a genuinely fun project: a private AI server with no account, no subscription, and no cloud dependency. The K420 is not going to replace ChatGPT, but it can handle short text decisions, simple agent steps, and small coding tasks. The 27B on a GTX 1080 or RTX 3080 is much more useful for real coding and long documents.

The ideal Kindi user does not have a Mac, an AMD GPU, a Windows-only workflow, or a need for high-concurrency serving. Kindi runs one request at a time, supports only the Ternary-Bonsai-2 family, and expects Linux. For everyone else, Ollama, LM Studio, or llama.cpp will be a better daily driver. (GitHub limits)

The fine print

Kindi is free, with commercial licenses available. The project page and GitHub list AGPL-3.0; the launch video description said it was free but “not currently open source.” If you plan to ship it inside a closed product or hosted service, check the current license terms rather than relying on the video. The underlying Ternary-Bonsai-2 models are published separately by Prism ML. (Kindi page, GitHub license)

Bottom line

Kindi 0.2.0 is one of the more interesting local-AI projects of the year because it attacks a problem most vendors ignore: the huge installed base of old, low-VRAM NVIDIA hardware. It will not replace llama.cpp for flexibility or vLLM for modern serving. But if you have a supported card and a Linux box, it can turn hardware that was headed for the recycling bin into a private, OpenAI-compatible AI server.

For the DIY crowd, that is a very appealing trade: a little setup work in exchange for a fast, offline model on hardware you already own.

References

1. Kindi — Amir Husain
2. Kindi on GitHub
3. Kindi launch video — “Your Old GPU Can Run Modern AI Agents”
4. Deep Tech with Amir Husain — YouTube channel
5. Kindi 0.2: Fastest Inference Across Four NVIDIA GPU Families
6. How To Get Work Done with Kindi and a $15 GPU — field manual