Local model runs on the LLMeter coding suite

Local model runs on MacBook Pro M5 Pro, 64 GB

Loading runs…

Priority
Thinking

Run details

One row per run: model build, runtime and machine, then the suite score (0 to 100, with the easy, medium and hard parts), generation speed and wall time.

Model Tier Quant Runtime Thinking Hardware Score Easy / Med / Hard Tok/s Time Parallel Date Status
Loading data…

Suite: local-coding-v1, ten short Python tasks with hidden tests and partial credit (easy weight 1, medium 2, hard 3). Score is the weighted share of hidden tests passed, 0 to 100. Tokens/sec is the average generation speed across the run's tasks. Limits per task: 4,000 output tokens and 5 minutes with thinking off; 12,000 tokens (thinking included) and 15 minutes with thinking on. An answer cut off at a limit scores 0, so a low thinking-on score can mean the model spent its budget on thinking, not that its answer was wrong. Results are read from data/local-runtime-results.csv (one row per task) and the model list from data/local-test-models.csv; a row marked planned, not installed or cannot run has no numbers; the reason a model cannot run is in the model list.