Deploying language models (LMs) on the edge presents unique memory, latency, and power challenges absent in high-performance computing.
KV-cache compression emerges as a promising approach to address these challenges.
However, the effectiveness of current KV-cache compression methods has not been thoroughly examined on edge devices.
Moreover, we show that existing evaluation methods do not always involve the KV-cache under test.
We propose EdgeLLM-Nano, a measurement study of KV-cache compression using a nano-scale decoder-only LM (transformer) testbed that has 8.39M parameters, i.e., 33.6 MB, to enable end-to-end examination of each backend, seed, and power sample.
Our results give a negative verdict for a polar-guided key-cache quantizer (round-to-nearest (RTN) control causes an order of magnitude less damage), as well as a consistency-anchored progressive weight pruner (one-shot magnitude-based pruning performs as well as, if not better).
We also introduce Long-Context Decode Fidelity (LCDF) evaluation, a token-by-token examination of teacher-forced log-losses using the KV-cache under test.
Unlike perplexity that gives the same 5.461 score to all methods, LCDF distinguishes different KV-cache compression backends thanks to its paired statistics across seeds and effect sizes.
Our companion Raspberry Pi 5 measurement shows all compression backends we tested to be less energy-efficient per token compared to uncompressed FP16 inference.
Except for KIVI that is close to lossless in terms of LCDF, our findings indicate KV-cache compression can be improved.
LCDF offers a simple and effective way of shedding light on KV-cache compression methods, whose negative impacts have been overlooked by perplexity.