Local model runs on the LLMeter coding suite
Local model runs on MacBook Pro M5 Pro, 64 GB
Loading runs…
Run details
One row per run: model build, runtime and machine, then the suite score (0 to 100, with the easy, medium and hard parts), generation speed and wall time.
| Model | Tier | Quant | Runtime | Thinking | Hardware | Score | Easy / Med / Hard | Tok/s | Time | Parallel | Date | Status |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Loading data… | ||||||||||||
Suite: local-coding-v1, ten short Python tasks with hidden tests and partial credit (easy weight 1, medium 2, hard 3). Score is the weighted share of hidden tests passed, 0 to 100. Tokens/sec is the average generation speed across the run's tasks. Limits per task: 4,000 output tokens and 5 minutes with thinking off; 12,000 tokens (thinking included) and 15 minutes with thinking on. An answer cut off at a limit scores 0, so a low thinking-on score can mean the model spent its budget on thinking, not that its answer was wrong. Results are read from data/local-runtime-results.csv (one row per task) and the model list from data/local-test-models.csv; a row marked planned, not installed or cannot run has no numbers; the reason a model cannot run is in the model list.
Result per task
One column per run, best first. Each cell is the share of the hidden tests gained over the starter files, with the tests passed under it. An answer cut off at the token limit ("truncated") or the time limit ("timeout") scores 0.