Ario Barin Ostovary

Software Engineering, University of Waterloo

which-llm

Artificial Analysis needs to step up their game.

Picking an LLM from memory is a great way to recommend last month's model at last month's price. I kept doing that, got annoyed, and built which-llm.

The skill joins Artificial Analysis benchmarks with OpenRouter prices, context windows, modalities, and provider slugs, then checks the snapshot into the repository. The first query works offline and the daily workflow refreshes the evidence without needing an API key from the person using the skill.

which-llm Pareto chart plotting intelligence index against benchmark task cost from the July 31, 2026 refreshed snapshot

This chart came straight from which-llm's cost-intel frontier command. The interesting models sit toward the upper-left: more measured intelligence for less benchmark task cost. That cost is not the same thing as your production API bill, and the tool says so explicitly, because pretending one benchmark can predict every workload would be nonsense.

I kept the interface small: pick ranks candidates, compare puts named models side by side, profile explains one model, resolve finds provider slugs, and frontier generates plots like this one. The output is evidence, not a magic answer, so the agent still has to say what it is optimizing for and explain the tradeoff.

The part I like most is that it refuses to bluff. If the snapshot is stale, it stops. If no model matches, it shows the nearest constraints you could relax. That is much more useful than a confident recommendation built from vibes.