Code to generate new solver performance difference statistics

# Generate a run with the 500 best and worst performers from data.csv
# Change some code to get just the 1000 worst, more useful for testing, less for generating plots
# NOTE: data is cached automatically. Aborting this script and restarting it resumes where it left off,
# and diffs etc can be generated by running the script on multiple nightly versions.
# gotta use nightlies, we use RUSTFLAGS=... so stable/beta wont work. local compiles do work with +stage1 and toolchain linking
# (see the dev guide)
export NUM=1000; python run.py +nightly-2026-08-19 -c $NUM -p 32
# generate a csv export. --html for a html export.
python run.py +someversiondoesntmatter -c 1000 -p 16 --csv --no-test

data.csv is initial data on 20k crates gathered by @lqd. The whole point of the code in this repository is to quikcly iterate on that list, and get diffs on samples of the original run to see progress made since then. This is much cheaper (I can run ~1000 crates per hour locally) than doing a full perf run again across all crates, every time we changed something.

See also https://blog.rust-lang.org/2026/08/21/enabling-next-solver-on-nightly/ and my elaboration on perf in https://donsz.nl/blog/new-solver-performance.