uv: Deduplicate all files in the wheel cache

Uh oh! There was an error while loading. Please reload this page. Notifications You must be signed in to change notification settings Fork 3.5k Star 89.3k Conversation ## Summary When content hashing is enabled, we currently allocate and zero a new 64 KiB buffer for every file we copy and hash during streaming extraction. This PR reuses one buffer across the wheel instead. For the PyTorch wheel used in the benchmarks, that reduces buffer allocations for hashing from 11,120 to one, while keeping the buffer size at 64 KiB per active wheel. The following measurements compare #21327 at `a188b8e833aef3c3b4b60a32ed9fafe6ac74186a` with this optimization applied on top, before moving the change onto `main`. They are not measurements against `main`. The Linux benchmarks alternate base and candidate, using pinned wheels served over local HTTP with content-addressed caching enabled: | Cold install | #21327 | #21327 + buffer reuse | Change | | --- | ---: | ---: | ---: | | AnyIO | 110 ms | 107 ms | -2.6% | | SymPy | 845 ms | 775 ms | -8.3% | | NumPy | 627 ms | 567 ms | -9.5% | | PyTorch CPU | 6.50 s | 5.99 s | -7.8% | | 14-package environment, concurrency 4 | 6.95 s | 6.47 s | -7.0% | The individual results above use 16 paired rounds; the full environment uses 12. AnyIO, SymPy, and NumPy were repeated after an initial 20-pair run: the initial AnyIO timings were noisy, while the initial SymPy and NumPy improvements were 7.8% and 6.9%. All original samples were retained. Cached installs and local-wheel controls showed no consistent change. Across the initial runs, repeats, and controls, we measured 672 installs, excluding warmups and cache priming. Co-authored-by: Charlie Marsh Reviewers Assignees Labels Projects Milestone Development Successfully merging this pull request may close these issues. Uh oh! There was an error while loading. Please reload this page. 3 participants