I'm making LLMs work on small and old devices

🤏 smolmaxxing Every device around you is more capable than it looks. TV boxes, old phones, single-board computers and routers all ship with CPUs, RAM and flash that sit idle most of the day. smolmaxxing is about unlocking that capability: finding the biggest language model each device can really run, then pushing it further with custom kernels, smarter memory use and weight streaming. Every claim is measured on the hardware itself. Device #1 is a Xiaomi TV Box S (3rd Gen): Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable), eMMC flash, Android 14, no root. On it, this repo runs models from 260 K to 16 B parameters: Qwen3.5-0.8B: 16.8 tok/s (2.2× the stock 7.5; measured in the evening of 11-05, see §17). Qwen3.5-2B: 7.7 tok/s in real chat. The box's speed drifts by ~10 % over a day, so every step is measured back to back: §16 gave +11 %, §17 another +7 %. An 8B mixture-of-experts model (Moon-Beam-8B-A1B): 7.2 tok/s in real chat, its experts paged from flash on demand and prefetched one layer ahead (§19). A dense 1-bit 8B model (Bonsai-8B): 5.2 tok/s in real chat, up from 1.2: fully resident in a "headless" mode plus a lookup-table kernel (§19). A 16B mixture-of-experts model (Ling-mini-2.0): 5.1 tok/s in real chat, coherent over 600-word answers, the same expert pager applied unchanged (§20). Twice Moon-Beam's size, but not better per byte: bigger only helps on this box when the model is also better. A dense 9B model: 0.27 tok/s, streaming most of its weights from flash. Device #2 is a first-generation iPad Air (2013): Apple A7, 2 cores, NEON without dotprod, 1 GB of RAM, iOS 12.5.7, jailbroken, CPU only. A 13-year-old tablet runs a 1.7B-parameter model at 24-26 tok/s (18-22 in long chats), a 4B one at about 10 tok/s, an 8B mixture of experts at about 3.5 tok/s (5 with half its experts, at a quality cost), and a 16B one at 2.6-4.5 tok/s, with their experts paged from flash, from a chat page in its own Safari. See Results on device #2. Every step is written up experiment by experiment in EXPERIMENT.md (E1-E160). Devices The techniques aren't tied to one box. The 1-bit lookup-table kernel needs only base NEON, as the iPad shows; any ARM chip with dotprod gets the other kernels; and any Linux or Android device with more model than RAM can use the streaming engine. Scoreboard "First run" is the first time each model ran on the box: 0.8B: the stock Q4_K_M file. 2B: the stock file, streamed from flash because it didn't fit in RAM. Its first build that fit ran at 4.8, so from there the gain is ×1.6. 4B: the stock file, which thrashed. 8B dense: the last 0.1 GB streamed from flash. 8B MoE: the first expert-pager run, without routing bias. 9B: streamed. 16B: started on the Moon-Beam recipe, so its gain is small. On this box, the 2B is the best speed-quality balance. Ling-mini-2.0 shows that bigger isn't automatically better: it is twice Moon-Beam's size, but scores worse per byte and runs slower. What we built, grouped by the bottleneck it attacks (details and numbers in What's in here): Results on device #1 Quality is English perplexity on a fixed 22 KB text, converted to bits per byte so models with different tokenizers compare fairly (lower is better). Speed is generation on the TV, measured either through llama-server with real chat requests ("chat") or with llama-bench ("bench"). * measured on the pruned head's positions (1.3 % of tokens fall outside it), so not exactly comparable; the stock full-vocabulary Q4_0 file scores 1.024. Other small models, stock files (screening, §15 E54): Granite 4.0 H 1B was the one model close to the 2B's quality. Rebuilt our way (E62), it runs at 5.29 tok/s, below the 2B's 6.55 at slightly lower quality: it carries about twice the 0.8B's weight bytes per token. Physics of the box: one process can hold ~0.96 GB before Android starts killing things (~1.15 GB in headless mode, with the launchers and Google services force-stopped), and flash reads top out at 278 MB/s, with a lone ~1 MB random read costing ~6 ms. A model that has to stream X MB per token can never beat 278 / X tokens per second. That, not the CPU, is what limits everything above 2B, and why a mixture-of-experts model, which touches ~1 B of its 8 B weights per token, is the way past it. 10 tok/s at 8 B? Measured, not guessed (E98): Moon-Beam reaches 10.4 tok/s with 2 experts per token, but the text turns to gibberish; the best coherent setting is ~7-8 tok/s. Dense Qwen3-8B is Bonsai-8B at 1 bit, and the smallest Qwen MoE (35B-A3B) touches 3× Moon-Beam's weights per token. Results on device #2 A 2013 iPad Air: Apple A7, 2 cores, NEON without dotprod, 1 GB of RAM, iOS 12.5.7, jailbroken with Amethyst. Everything runs on the CPU with 2 threads and a 1,024-token context, cross-compiled from a Mac. The write-up is §21 of EXPERIMENT.md, and build steps are in ipad-air/README.md. Only the Bonsai and Moon-Beam rows were measured on the latest build. The other rows predate the attention and sampler work below. "Can't load": iOS 12 lets a process map at most about 1.7 GB, so stock llama.cpp fails on the 2.6 and 5.5 GB files before running a token. "First run" for Moon-Beam is the first expert-pager run (pool 8, no bias). What we built on the iPad, grouped by the bottleneck it attacks: How Bonsai-1.7B got from 2.75 to about 25 tok/s (same model file throughout): In long chats (400-token answers), the three turns went from 15.4 / 7.6 / 8.2 to 21.8 / 17.9 / 18.4 tok/s through the attention work alone. The pruning threshold that holds long chats above 17 costs +0.42 % perplexity at 1,024 tokens of context, near the ±0.3 % noise floor; GGML_FA_EPS=0.0001 trades that back for about 15 tok/s. An 8B model, through an expert pager. iOS 12 caps a process at about 1.7 GB of mapped address space (1.3 GB for the web server's user), so Moon-Beam-8B's 2.6 GB file can't even be mapped, and keeping fewer experts to make it fit more than doubles its perplexity (§21.7). Dense Bonsai-8B runs at 0.08 tok/s from flash. So the model file is rewritten with its experts last, only the 474 MB before them is mapped, and each layer keeps 14 experts in RAM, reading the others from flash with pread when the router picks them. Results match the unpaged model exactly. A small routing bias toward experts already in RAM (+0.3 % perplexity, inside the noise) brought it to 2.4-2.8 tok/s in about 600 MB (§21.8). Round 2 (§21.9) queues flash reads by when they're needed, prefetches the next layer's likely experts, and runs the experts through a lookup-table kernel for Q2_K: 3.4 tok/s, 40× the dense 8B, at +0.7 % perplexity on the full eval text. The flash is still the limit: each token reads about 53 MB at 266 MB/s. Wiring weights in RAM kernel-panicked the iPad, like a pool of 16 did before. Round 3 (§21.10) evicts by decayed use frequency instead of recency, for 3.5 tok/s at no quality cost. 5 tok/s takes 4 experts per token instead of 8 (+27 % perplexity); choosing the count per token adaptively turned out no better than a fixed count. A 16B model (§21.11): Ling-mini-2.0, 16.3B parameters with 256 small experts per layer, runs through the same pager at 2.6 tok/s with unchanged routing, 3.3 at +2.6 % perplexity and 4.5 with a strong cache bias, in under 580 MB. Its 5.5 GB file was assembled from two published quantizations over HTTP range requests and streamed straight to the iPad, since the Mac had no room for it. It is twice Moon-Beam's size at the same speed, but not a better model (+3 % perplexity). Limits worth knowing. Bonsai is a 1-bit model and still gets facts wrong (in one test it called Mercury the closest planet to Earth). About 700 MB is the practical memory ceiling: 1-bit weights don't compress, so iOS starts killing processes past it, which is why the 4B needs a 512-token context. The jailbreak doesn't survive a reboot. The fast output head adds 2-3 s to the first prompt after each launch. What's in here Kernels for Cortex-A55 (ARMv8.2 dotprod, no i8mm), as a patch on llama.cpp v0.8.0, each with an exactness test against a scalar reference: Memory tricks: Pruned LM head: keep only the ~49-66 K most useful vocabulary rows, chosen by token frequency on English, code and 7 other languages. The other logits are scattered to −inf. Saves 0.2-0.5 GB and misses ~1 % of English tokens. Repack cache on flash (GGML_REPACK_CACHE): repacked weights are written to a file once and mapped afterwards. Models load in seconds. Headless mode (tools/tv-llm-mode.sh on|off): a small foreground activity, then the launchers, Google services and other heavy apps are force-stopped, and svc power stayon keeps the animated screensaver (17 % of a core) away. Nothing persistent except stay-awake, which off restores. MemAvailable ~1000 → ~1300 MB. A fork-free guard (tools/tv-watch.c) replaced a shell loop that started ~25 processes a second and cost 10 % of the speed. Expert pager for mixture-of-experts models (ggml-cpu/moe-pager-tv.inc): Experts live on flash: each expert tensor is overlaid with empty anonymous memory, and right after a layer's routing the missing experts are read with parallel O_DIRECT straight into their own addresses; a per-layer LRU pool (43 of 128 experts) keeps the rest. Kernels see an ordinary tensor. The GGUF's own expert bytes are hole-punched once the repack cache holds them, so the model costs one copy of its weights on flash. Cache-aware routing (GGML_MOE_BIAS): a small bonus for experts already in RAM cuts misses from 13 % to ~2 % for a measured quality cost (arXiv 2412.00099 has the idea on other models). Fewer experts per token (LLAMA_MOE_USED=6) is the other dial. Warm start from a list of each layer's most used experts, and pre-gated prefetch: layer L+2's router applied to layer L's hidden state, predicting what the router will actually pick (sigmoid, the model's own selection bias and the cache bonus), loads ~60 % of misses before they are needed. Why answers looped, and the fix: 2-bit experts break long-form coherence after ~250-300 words, and routing bias amplifies it. DRY sampling only hides it (the model paraphrases its loop); a repeat penalty of 1.15 over 256 tokens brings degenerate long answers from 12/13 to 1/13 on the TV (E99, E100). Weight streaming for models bigger than RAM (ggml-cpu/stream-tv.inc): What stays in RAM: a chosen part of the weights stays resident. The rest lives in "stream zones". Prefetch: a hook on every matmul records the order the weights are used. Reader threads then fetch zone weights ahead of time with parallel 8 MB O_DIRECT reads straight into the weights' own addresses, at the flash's full 278 MB/s. Correctness: every compute thread waits until its weight is completely loaded, so a wrong prediction costs time, never correctness. Drop-behind: each zone weight is freed right after use. Verify mode: GGML_STREAM_VERIFY=1 compares every streamed tensor with the file. It found two silent-corruption bugs, which are fixed and documented in E48 and E50. Speculative decoding: with the model's own MTP head, verifying 2 tokens costs about one pass over the weights when the pass is flash-bound, so it pays off (+69 %). On the in-RAM 2B it lost (E31). Profiling (env switches in the patch): GGML_OPPROF=1 per-op table, GGML_OPPROF=2 a node-by-node, per-thread trace of one graph (tools/optrace.py), LLAMA_UBATCH_PROF / GGML_CPU_PROF phase timings. They showed where a small model's time really goes (E57): ~71 % Q4_K matmuls, and only ~2 ms per token of barriers and scheduling. Tooling: Streaming GGUF rewriter: adds a pruned head, extra tensors and metadata in one sequential pass, so Hugging Face → Mac → TV pipelines never store the model on the Mac. MTP fetcher: range-reads only the MTP tensors from safetensors. Memory guard: wraps every heavy run on the TV. Benchmark and perplexity harnesses, a model screener that streams any Hugging Face GGUF to the TV and scores it (tools/screen-small.sh), and a live token-stream web UI. Quick start You need: the TV box (or another ARMv8.2 Android device) with network ADB, adb, zig (for static aarch64 musl builds), cmake + ninja, python3, and uv (for numpy). export ADB_SERIAL= :5555 # every script talks to the box through this adb connect $ADB_SERIAL # 1. llama.cpp v0.8.0 + the patch, cross-compiled for the TV git clone https://github.com/ggml-org/llama.cpp third_party/llama.cpp cd third_party/llama.cpp && git checkout 8642541 && git apply ../../patches/llama.cpp-v0.8.0-tvbox.patch cmake -S . -B build-tv -G Ninja -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 \ -DCMAKE_C_COMPILER=$PWD/../../zigcc/cc -DCMAKE_CXX_COMPILER=$PWD/../../zigcc/c++ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_EXE_LINKER_FLAGS=-static -DBUILD_SHARED_LIBS=OFF -DGGML_NATIVE=OFF \ -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16 -DGGML_OPENMP=OFF -DLLAMA_OPENSSL=OFF -DLLAMA_BUILD_TESTS=OFF ninja -C build-tv llama-server llama-cli llama-bench llama-perplexity llama-quantize cd ../.. adb -s $ADB_SERIAL shell mkdir -p /data/local/tmp/llm/patched25 /data/local/tmp/llm/patched adb -s $ADB_SERIAL push third_party/llama.cpp/build-tv/bin/llama-{server,cli,bench,perplexity} /data/local/tmp/llm/patched25/ adb -s $ADB_SERIAL push third_party/llama.cpp/build-tv/bin/llama-quantize /data/local/tmp/llm/patched/ # used by tools/tv-quant.sh zig cc -target aarch64-linux-musl -O2 -static -o gguf_recut tools/gguf-recut/gguf_recut.c && adb -s $ADB_SERIAL push gguf_recut /data/local/tmp/llm/ # 2. models: rebuilt from their Hugging Face sources straight onto the TV (see below) tools/rebuild-models.sh 2b # also: 08 4b b8 tb8 q9 mb lm caches # 3. the live chat page (default: the 2B on patched25 with fp16 state), then open http:// :8090 from any device on the network tv-llm-ui/start.sh tv-llm-ui/start.sh stop Other models: # Moon-Beam-8B-A1B MoE, ~7.2 tok/s (first: `tools/rebuild-models.sh mb`, needs a native Mac build and ~9 GB on the Mac while it runs). # start.sh turns on headless mode, the expert pager, routing bias, prefetch and the repeat penalty; POOL / BIAS / EXPERTS override them. MODEL=mb-q2-h60.gguf tv-llm-ui/start.sh # Ling-mini-2.0 16B MoE, ~5.1 tok/s (first: `tools/rebuild-models.sh lm`) MODEL=lm2-q2-h60.gguf tv-llm-ui/start.sh # Bonsai-8B 1-bit fully resident, ~5.2 tok/s (first: `tools/rebuild-models.sh b8`; the repack cache is written on first start) MODEL=b8-q1-hml.gguf tv-llm-ui/start.sh sh /data/local/tmp/llm/tv-llm-mode.sh off # on the TV, afterwards: launcher and screensaver back # Qwen3.5-0.8B, ~16 tok/s (first: `tools/rebuild-models.sh 08`, needs a native Mac build of llama-quantize) MODEL=q08-q4k-h65p8.gguf tv-llm-ui/start.sh # The 8B / 9B numbers below were measured with the build of that time (patched16); the current build has the same # code plus §15-17. GGML_CHUNK_GUIDED=0 in SENV restores the old matmul scheduling. No fp16 state with MTP. # Bonsai-8B 1-bit with streaming and no headless mode, ~1.2 tok/s (first run: `tools/rebuild-models.sh b8 caches`) MODEL=b8-q1-hml.gguf BIN=patched25 EXTRA="-ctk q8_0 -ctv q8_0 -fa on -cram 0" \ SENV="GGML_REPACK_CACHE=/data/local/tmp/llm/cache/b8c GGML_REPACK_RESIDENT_MB=800 GGML_STREAM_MODE=cache GGML_STREAM_WINDOW_MB=64" tv-llm-ui/start.sh # Qwen3.5-9B with MTP, ~0.27 tok/s MODEL=q9-iq2-h65-mtp.gguf BIN=patched25 EXTRA="-nr -ctxcp 0 -cram 0 -ub 16 -b 64 --spec-type draft-mtp --spec-draft-n-max 1" \ SENV="LLAMA_STREAM_RESIDENT_MB=600 LLAMA_STREAM_PIN=output.,blk.32. GGML_STREAM_WINDOW_MB=64" tv-llm-ui/start.sh Everything on the box lives in /data/local/tmp/llm and runs as the adb shell user. The only system setting touched is stay-awake in headless mode (svc power stayon), which tv-llm-mode.sh off turns back off. adb shell rm -rf /data/local/tmp/llm removes it all. Model files: not in git, rebuildable The model files (0.5-4 GB each) are not in this repo. tools/rebuild-models.sh recreates every one from public Hugging Face files, streaming them through tools/gguf_build.py onto the TV. The SHA-256s below are of the files used in the experiments. Rebuilds of the 4B, Bonsai-8B and 9B files were checked byte-identical. Ready-made downloads. The best file of each size class is on Hugging Face exactly as the TV runs it. They need the patched llama.cpp above; stock llama.cpp can't load the pruned head. Each card has the run command and the measured numbers: Download with hf download hyperionaidev/ --local-dir models/, then adb push the files to /data/local/tmp/llm/, with moe-warm-*.txt next to them. That replaces the 08 / 2b / 4b / mb / b8 / lm targets. For the 8B and 16B models, the first start writes the repack cache. iPad Air downloads. Every model run on the iPad, where to get it, and its speed there. The three MoE files are our own builds, laid out for the iPad's expert pager and lookup-table kernel: they need the iOS 12 patch (ipad-air/), and any other llama.cpp loads them but gets scrambled experts. The rest are used exactly as published; the Bonsai models' fast layouts are built in RAM at load time. 53 experiments in all (E108-E160, §21): The head-id lists in data/ were made by tools/vocab_select.py (Qwen3.5 tokenizer) and tools/head_select_tok.py (any tokenizer, via llama-tokenize), from the corpus tools/fetch_corpus.py downloads (Wikipedia + Project Gutenberg). Layout Lessons On this box the 2B at 4 bits beats every bigger model at 1-2 bits. Post-training quantization below ~2.5 bits breaks normal models (E22, E23, E35). Models trained for 1-bit or ternary hold up, but at 4-8 B they don't fit without streaming. Past RAM, flash bandwidth sets the speed. The kernel's LRU is the worst possible policy for a cyclic scan bigger than the cache: it re-reads everything every token. Keep a fixed resident set and stream the rest yourself. Batching is free when you are flash-bound. Prompts and speculative verification cost about one weight pass, whatever their size. Measure, then verify. Two streaming bugs gave plausible speeds and wrong numbers; only comparing tensors with the file, and perplexity with a Mac reference, caught them. Small models are mostly vocabulary. Qwen3.5-0.8B's 248K-row head is a third of its weights; pruning it to 65K rows plus all-Q4_K took it from 7.5 to 12.1 tok/s (E55). Small models also lose more to 4 bits (+7 % perplexity vs +2.3 % for the 2B). Simulate before you write a kernel. Activation sparsity (E58) and a low-rank head (E60) both looked promising on paper; a quality simulation on the Mac killed each in under an hour. On an in-order core, count NEON ops as well as bytes. Tighter packing saved 20 % of the bytes and cost 2.5× the ops; undoing it made the 4B 33 % faster in chat (E76, E77). The 4-bit kernel is the opposite case, limited by RAM (E27), so the trick does not help the 2B or 0.8B. Check that every big matrix reaches the fast kernel. The 2B's head (65,668 rows, ~10 % of each token's weights) silently ran on the generic kernel for weeks because ggml only repacks row counts divisible by 8; four extra rows gave +5.8 % (E71, E72). The load log's repack buffer size gives it away. Most of the RAM was Android's, not the model's. Force-stopping the launchers and Google services for the duration (nothing persistent) let Bonsai-8B sit fully in RAM: 1.2 → 3.6 tok/s before any kernel work (E80). An animated screensaver and a guard that forked 25 times a second each cost another 10-15 % (E82, E88). For a mixture-of-experts model the flash is a cache miss, not a stream. A ~6 ms miss per expert means the pool hit rate decides the speed; routing toward cached experts and prefetching what the next router will pick turned 4.4 into 7.2 tok/s. Perplexity can't see loops. It scores one token at a time on clean text; the 2-bit model scored fine and still fell into loops in 12 of 13 long answers. Score long free-running generations too (tools/loop_stats.py), and compare models with different tokenizers in bits per byte, never in perplexity. Past 9B, count resident bytes, not parameters. The ceiling for a mixture-of-experts model here is the non-expert weights that must stay in RAM plus a useful expert pool. Ling-mini-2.0 (16B) keeps 279 MB resident and runs at 5 tok/s; Qwen3.6-35B-A3B would need ~870 MB before caching a single expert (§20). Guard the box. After two early kernel panics, every heavy run goes through a wrapper that kills it below 180 MB of free memory. License MIT. The llama.cpp patch is a derivative of llama.cpp (MIT). Model weights keep their own licenses, and the evaluation texts are Wikipedia excerpts (CC BY-SA 4.0). Credits Built on llama.cpp and llama2.c. Models from Qwen, unsloth (GGUFs and importance matrices), PrismML (Bonsai), Google (Gemma) and stories260K. The evaluation texts are excerpts from Wikipedia (CC BY-SA 4.0). The 🤏 logo is the pinching-hand emoji from Google's Noto Emoji (Apache 2.0).