Inkling-2-Mini in 5.4 GB Memory
LookAtMyQuant LookAtMyQuant is a quantization compiler. It measures how much each stored piece of a model hurts the output when compressed. It then picks a code for each piece so the whole model fits a size budget with the least damage. On Gemma 4 E2B, its GGUF has 12× lower KL than Q4_K_M at the same file size. The method and every result are on the site. The code is under the MIT license. LookAtMyQuant 2: Qwen3.5-4B in 1.2 GB LookAtMyQuant 2 takes Qwen3.5-4B to 1.23 GB, 2.3 bits per weight. The file solves 330 of 540 exact-answer math problems. Prism ML's Bonsai 2 Ternary 4B solves 328 at 1.20 GB, and the original solves 359. Its KL from the original on held-out text is 0.45 nats, against 0.59 for Bonsai. The recipe is post-training quantization with trellis codes, calibrated on the model's own worked math solutions with digit tokens weighted 10×, and with bits allocated per row by Fisher information. 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The page has the findings, the full recipe and its cost, about $34 of GPU time. The recipe's code is here, and the next section runs it. The 1.23 GB file itself is the lm2-qwen35-4b-1.2gb release, packed for the Apple Silicon runtime. Qwen3.8-27B in 7.2 GB The same recipe takes Qwen3.8-27B from 53.7 GB to 7.2 GB, 2.1 bits per weight, calibrated and distilled on English prose and chat, code and math. The file solves 401 of the 540 math problems, as many as the original, and Unsloth's UD-IQ2_XXS GGUF of the same size solves 348. It also scores above that GGUF on HumanEval (73.8% against 67.1%), MMLU (78.8% against 75.0%) and GSM8K (94.5% against 92.7%). Beside it sits a 4-bit DSpine drafter (1.02 GB, converted from tencent/Qwen3.8-27B-DSpine-b16 with runtime_mlx/convert_dspine.py and packed with pack_drafter.py), which roughly doubles decode speed on math and code on Apple silicon. Inkling-2-Mini in 5.4 GB and 8.9 GB The same compiler and proximal fine-tuning (SQYFR), extended to a 26.4B physical / 5.4B active 3×-looped MoE (L6–17 looped 3× = 48 logical visits, 128 experts top-8), compresses thinkingmachines/Inkling-2-Mini from 18.89 GB of text weights to 5.38 GB (1.63 bits/weight) and 8.87 GB (2.69 bits/weight) using 4 KB SRAM hashed block-vector trellis codes (ht2/ht3), row trellis codes (rtr1/rtr4), and a tr2 floor on the tail lm_head bucket (head.7). Evaluated on the same BF16 teacher, openai/gsm8k (test[:200], greedy, reasoning_effort="none"), all 164 executable openai/openai_humaneval problems, 4 Vision VQA tests (with the untouched 637 MB BF16 vision tower), and 16+13-language TextMix windows: At 5.38 GB, Inkling-2-Mini-LookAtMyQuant-5.4GB outscores Unsloth's 9.14 GB UD-Q2_K_XL on both GSM8K and HumanEval at 41% smaller size (up from 76/200 and 24/164 untuned). At 8.87 GB, Inkling-2-Mini-LookAtMyQuant-8.9GB matches the BF16 original on GSM8K and cuts KL 2.34× on English/chat/code and 29.6% across all 13 languages below UD-Q2_K_XL. Both run directly through lamq and runtime_mlx with custom Apple Silicon Metal kernels, optionally paired with the 178.9 MB 4-bit DSpark drafter: on M5 Max, 5.4 GB decodes at 73.2 tok/s serial and 125.2 tok/s median (141.8 tok/s math, 129.0 tok/s reasoning, 121.4 tok/s code, 102.3 tok/s prose; 1,037 tok/s prefill at T=512, 5.83 GB active RAM), and 8.9 GB decodes at 63.1 tok/s serial and 101.3 tok/s median (133.9 tok/s reasoning, 118.9 tok/s math; 838 tok/s prefill at T=512, 9.32 GB active RAM). Install and use the Apple Silicon runtime On Apple silicon, install the runtime package directly (this installs mlx, mlx-lm, numpy, and huggingface-hub without the PyTorch compiler dependencies): pip install git+https://github.com/ShinMegamiBoson/lookatmyquant.git From a local clone, pip install -e . installs the lamq command, or run python -m runtime_mlx without installing. Command line. Pass either a local model folder or a Hugging Face repo ID: lamq generate shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \ -p "What is 17 times 23? Show the steps." lamq generate shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \ --drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter \ -p "Write a Python function that merges overlapping intervals." --stats lamq chat shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB \ --drafter shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter lamq generate shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-5.4GB \ --drafter shinmegamiboson/Inkling-2-Mini-LookAtMyQuant-drafter \ -p "What is 17 times 23? Show the steps." --stats lamq generate reads stdin when -p is omitted, supports --raw, --system, --thinking, --temp, --mode linear|chat-fork|copy-wide, --no-adaptive, and --json. lamq info
inspects a packed model's format and dimensions without loading weights. Python integration. runtime_mlx.load returns an (model, tokenizer) pair compatible with mlx_lm.generate and mlx_lm.stream_generate (auto-detecting opgraph-packed-1 models such as Qwen3.5-4B / Qwen3.8-27B and lamq.json exports such as Gemma 4 E2B): from mlx_lm import generate from runtime_mlx import load model, tok = load("shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB") prompt = tok.apply_chat_template( [{"role": "user", "content": "What is 17 times 23? Show the steps."}], add_generation_prompt=True, enable_thinking=False, ) print(generate(model, tok, prompt, max_tokens=200)) For streaming, multi-turn chat, or speculative decoding through one interface, use Runtime: from runtime_mlx import Runtime rt = Runtime.load( "shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB", drafter="shinmegamiboson/Qwen3.8-27B-LookAtMyQuant-7.2GB-drafter", ) for chunk in rt.stream("Solve 3x + 4y = 31 and 5x - 2y = 17.", max_tokens=256): print(chunk.text, end="", flush=True) Run the compiler on SmolLM2-360M in three commands Install the dependencies and download the model and WikiText-2. The code reads SmolLM2-360M's config when it starts, so do this first. pip install -r requirements.txt python -c "from huggingface_hub import snapshot_download as d; d('HuggingFaceTB/SmolLM2-360M')" python -c "from datasets import load_dataset as l; l('Salesforce/wikitext', 'wikitext-2-raw-v1')" Measure. This compresses one piece at a time, leaves the rest exact, and records the KL to the original model. The full set for this model, 2,312 measurements, took about 50 minutes on an L40S. python compiler.py measure --model HuggingFaceTB/SmolLM2-360M --part linear --layers 0-3 --out results/batch/my_measurements.json Allocate. This picks one code per piece, with the least summed KL inside the budget. It needs no GPU. This command uses the SmolLM2-360M measurements in this repository, with a budget of 3.044 bits per weight. python compiler.py allocate --sens 'results/batch/ccSens*.json' --target disk --items linear --bits 3.044 --out results/compiler/plans/my_plan.json The plan it writes has the same widths as results/compiler/plans/sm_lin_3044.json. Build and score. This quantizes the model to the plan, one matrix at a time. It then reports perplexity and KL to the original on WikiText-2 test. python quantize.py --program plan --model HuggingFaceTB/SmolLM2-360M --plan results/compiler/plans/my_plan.json --out results/my_run.json Our run of that plan is in results/batch/cc_sm_lin_3044.json. It reached perplexity 16.64 and KL 0.0926, against an original perplexity of 15.31. Add --layers 1 --calib 8 --eval 2 for a quick check that everything works. Run the LookAtMyQuant 2 recipe on Qwen3.5-4B Each step is a named job; python run_batch.py --list prints its command. The jobs need Qwen3.5-4B in models/Qwen3.5-4B on the volume and the text mix (modal run --detach modal_fetch.py --text). Every file they make goes to lm2/ on the volume. modal run --detach modal_app.py --jobs lm2Text modal run --detach modal_app.py --jobs lm2Calibration,lm2Solutions modal run --detach modal_app.py --jobs lm2RowImportance modal run --detach modal_app.py --jobs lm2Build modal run --detach modal_app.py --jobs lm2Finetune modal run --detach modal_app.py --jobs lm2Math,lm2Math216,lm2TextKL lm2BuildSmoke, lm2FinetuneSmoke and lm2MathSmoke run the same code on two layers, 60 steps and 36 problems, in a few minutes each. The row plan. lm2Build reads results/compiler/rows/q35_rowplan_1p2GB.json, which gives 1 to 4 bits to each of 1,520 row groups. row_allocation.py made it. Its build-control command builds and packs a control with a fixed split of rows, prepare-rows allocates rows inside the control's packed bytes with rowgroup_alloc.py, and build-rows builds under the plan and checks the packed size against it. The codebooks. The four e-means codebooks the build decodes with are in cache/codes/. --trellis-fit lloyd gives the earlier ones, and a plan made before the change keeps them unless the flag says otherwise. Recent equal-byte recipe improvements. Two zero-byte additions are available in the compiler and fine-tuning scripts: Softmax-Fisher token table ranking (--embed-order sfisher-mix in compiler.py measure and quantize.py). Ranks tied embedding/output buckets by a 50/50 blend of output Softmax-Fisher E[p_v (1 - p_v)] and corpus frequency (and untied output heads by Softmax-Fisher alone). Mid-training equal-byte row reallocation (--realloc-at 1000 in finetune.py). Paired with --rebuild-at 1000, measures each layer's continuous-vs-stored proximal KL gap KL(all P) - KL(layer = W) to shift bits toward deep unabsorbed layers while keeping exact file size (--realloc-max-delta 1). Combined with sfisher-mix on Qwen3.5-4B at identical 1.23 GB (2.297 bpp), WikiText-2 KL fell from 0.425 to 0.387 (−9.0%) and 13-source KL from 0.447 to 0.378 (−15.5%, improving all 13 sources). Where these jobs differ from our run. Our GSM8K solutions were written earlier, with thinking always off. Our text KL on 13 sources used 52 windows picked from the text mix by a separate tool; --sources mix takes the mix's own evaluation windows, on which our file scores 0.44 where the page reports 0.45. Large models run on Modal, one GPU per job run_batch.py names every job, and modal_app.py runs them. modal_fetch.py downloads models and the text mix used for calibration and evaluation. modal run modal_app.py::setup modal run --detach modal_fetch.py --text modal run --detach modal_fetch.py --models google/gemma-4-E2B-it modal run --detach modal_app.py --jobs ccMG4E2Noise,ccMG4E2Spot,ccMG4E2EmbKv The runs behind the site used these machines, each with 8 CPU cores: gguf_compiler.py makes plans out of llama.cpp's own tensor types, so the result runs in llama.cpp as it is. public_eval.py scores public quantizations the same way as our plans. Both run inside the image in modal_gguf.py, which builds llama.cpp with CUDA. The Apple Silicon runtime reads Lloyd–Max and trellis plans quantize.py --export
writes a plan in the format runtime_mlx/ reads. A plan with trellis codes also needs --trellis-rows 16,8, which runs each trellis path along a row of its matrix, so that decoding an entry reads only that row. On Apple silicon, pip install . installs mlx and mlx-lm; requirements.txt holds the PyTorch compiler dependencies. On Gemma 4 E2B at 2.5 GB, the trellis plan has 26% lower KL than the Lloyd–Max plan and decodes about 22% slower. runtime_mlx/test_trellis_gen.py and runtime_mlx/test_trellis.py check the trellis kernels against the matrices the quantizer built. python runtime_mlx/bench.py lamq path/to/export A drafter makes the 1.2 GB file generate 1.5 to 2 times faster on Apple silicon runtime_mlx/fast_q35.py decodes the packed 1.2 GB Qwen3.5-4B file one token at a time, straight from its trellis codes. runtime_mlx/drafter_q35.py adds speculative decoding. A small drafter proposes the seven tokens after the model's latest one in a single pass. The model then checks all eight positions in one pass over its coded weights (runtime_mlx/block_q35.py) and keeps the proposals that match its own choice, so the text is exactly its own greedy text. On an M5 Max, over 30 held-out prompts of up to 256 tokens, generation with the drafter was 2.0 times as fast: 176 tokens a second against 88, timed in alternation in one process on 2026-11-11. A check kept 4.3 tokens on average. Over eight everyday prompts it was 1.5 times as fast. The gain follows how predictable the text is: 2.5 times on algebra, 2.0 on code, 1.2 on chat and 1.05 on a story, and one of the 30 held-out prompts ran slower with the drafter. The text matched plain greedy decoding on every prompt. These are speeds after the first token; reading the prompt takes about 0.1 s either way. Two later changes are in this code. Each was measured against the version before it on 40 paired generations: proposing from the 96,033 most common tokens was 6.0% faster, and picking the winning token inside the packed product was 5.3% faster. Linear mode also enables self-calibrating adaptive fallback by default (adaptive=True; pass --no-adaptive to disable): when rolling acceptance drops below the measured break-even ratio of drafted-check latency to serial-step latency (accounting for cache flush and drafter context replay), it pauses drafting for short exact serial bursts and resumes once acceptance recovers, preserving bit-exact greedy parity while avoiding slowdowns on low-acceptance prose. The drafter is a DSpine model, trained with SpecForge from the published DFlash drafter for Qwen3.5-4B on the 1.2 GB file's own greedy answers. modal_drafter.py runs the training as one Modal job, and it cost $42. On 300 held-out prompts the trained drafter gets 4.65 tokens per check, against 4.30 for the stock DFlash drafter. pack_drafter.py stores the drafter at 4 bits a weight in the build's own code, with no calibration data: 464 MB against 1.83 GB at 16 bits. On 60 held-out prompts the 4-bit drafter kept 4.35 tokens per check against 4.38, and the text was the same. drafter_q35.py loads either form and writes the packed one out dense as it loads. The 4-bit drafter is the lm2-drafter-4bit release. It runs beside the packed 1.2 GB file, the lm2-qwen35-4b-1.2gb release. gh release download lm2-qwen35-4b-1.2gb --repo ShinMegamiBoson/lookatmyquant -D q35-1.2gb gh release download lm2-drafter-4bit --repo ShinMegamiBoson/lookatmyquant -D drafter-4bit python runtime_mlx/bench_drafted.py q35-1.2gb drafter-4bit The benchmark defaults to linear verification. Use --mode chat-fork to try a second candidate at an ambiguous early position, or --mode copy-wide for known contiguous copies. Copy mode uses the source cursor and tunes the gate/up and down projections for 16-slot checks. Both modes verify all emitted tokens with the packed target. To benchmark the chat fork, run: python runtime_mlx/bench_drafted.py q35-1.2gb drafter-4bit --mode chat-fork --prompts chat For a copy benchmark, pass --mode copy-wide --heldout path/to/copy-prompts.json. The JSON entries carry source, prompt_len, and input_ids from the packed model's tokenizer. The checks, and the packer for a drafter of your own: python runtime_mlx/test_fast_q35.py LAMQ_Q35_PACKED=path/to/packed python runtime_mlx/test_block_q35.py python runtime_mlx/check_drafter.py path/to/packed path/to/drafter python runtime_mlx/bench_drafted.py path/to/packed path/to/drafter --heldout path/to/drafter/heldout/heldout.json --take 30 python pack_drafter.py path/to/drafter path/to/drafter-4bit runtime_mlx/pack_dense.py writes the packed file from the fine-tuned build's dense weights, and runtime_mlx/modal_pack.py runs it on Modal. check_drafter.py compares the drafter in MLX with SpecForge's own model on reference blocks that drafter/reference.py writes. Where everything is The tests need SmolLM2-360M and WikiText-2 Each test file runs on its own from the repository root, and all seven finish in under a minute. python tests/test_frame.py python tests/test_grammar.py python tests/test_codes.py python tests/test_elements.py python tests/test_nested.py python tests/test_rowgroup_alloc.py python tests/test_reallocation.py runtime_mlx/test_core.py checks the runtime against the quantizer's math. Run python runtime_mlx/make_ref.py first, which writes 1.6 GB of reference files. runtime_mlx/test_fast_q35.py and runtime_mlx/test_block_q35.py check the Qwen3.5 kernels against NumPy products of independently unpacked codes, with no model. When LAMQ_Q35_PACKED names a packed file, the second also checks a block of tokens against decoding them one at a time. When LAMQ_Q38_PACKED names the packed Qwen3.8-27B file, it runs the same checks on a model whose output head is stored apart from its embedding. What this repository leaves out Model weights. Everything downloads from Hugging Face. The Qwen3.5-4B jobs expect that model in models/Qwen3.5-4B on the volume. Benchmark copies. eval_tasks.py downloads its benchmarks. On an offline machine it reads exported copies from data/bench/, which are not here. The Modal images run offline, so cache the benchmarks on the volume first. The script that assembled site/data.json. It was a set of one-off steps. The source numbers are all in results/. LookAtMyQuant 2's dense weights and result files. The 1.2 GB file is a release in the packed format. The dense weights the benchmarks ran on are not published, and neither are the per-problem results behind site2/data.json or the script that writes the page's numbers from them. The drafter at 16 bits. The drafter as trained is 1.8 GB and is not published. The 4-bit one is a release file, and modal_drafter.py trains it.