# AimRank

> Turn pairwise human (or AI) judgements — this one is better than that one —
> into calibrated rankings and RLHF-ready preference data, with an
> audit-citeable data card that shows how the data was collected and how much
> of it you should believe. Live, free to use, and callable by agents over MCP
> with no API key.

AimRank is live and public at https://app.aimrank.io, and has been since
2026-07-10. Free signups are open. It is built by AimRank Labs, an AI product
studio in Lithuania (EU).

This file exists so an agent can describe AimRank accurately without guessing.
Every number in it is pinned to the running code by a test that fails at commit
time when the code moves.

## What it does

- **Rank anything, two at a time.** Create a ranking, add items (text, image,
  audio or video), and collect A/B judgements from people, from an LLM judge, or
  from both. Ratings are computed by a real rating engine, not a vote tally.
- **Evaluate your own agent.** Suites, tasks and runs; code, LLM-judge, human-κ
  and trajectory graders; per-task pass^k with a 95% interval (Wilson bounds
  on the pass rate raised to the k-th power, assuming independent runs); a
  tamper-evident trace audit; and an EDD Improvement Report that names
  impact-ranked failure modes
  with cited evidence so a coding agent can read its own eval results and fix
  itself.
- **Prove the data is legitimate.** Inter-annotator agreement, calibration,
  fairness, drift, a human-versus-machine complementarity metric, a hash-chained
  audit log, and an EU-AI-Act-Annex-IV-shaped methodology payload.

## Who it is for

ML and eval teams who need preference data they can defend, and agent builders
who need an eval harness they did not have to write. The free "rank anything"
tier is the way in; the paid surfaces are export, privacy, volume and the keyed
API.

## Facts that are easy to get wrong

- **6 rating engines compute ratings**: Glicko-2 (the default), Elo,
  Bradley-Terry, TrueSkill, OpenSkill, Elo-MMR. **2 offline audit aggregators
  never write ratings**: Plackett-Luce and CrowdBT (setting either as a
  ranking's algorithm returns 400). Always report the two groups separately.
  Collapsing them into one count of things that rank is an overclaim that once
  reached an EU AI Act Annex IV evidence bundle shown to regulators.
- **Cohen's κ is hard-gated to exactly 2 raters.** For more raters the answers
  are Fleiss κ, Scott's π and Krippendorff α, all implemented. Never attribute a
  multi-rater agreement figure to Cohen's κ.
- **The data card is aggregate and unsigned.** It is honest evidence about a
  dataset as a whole — provenance, composition, agreement, coverage — and it is
  not a signed per-sample provenance chain. Per-sample provenance lives in the
  RLHF export, so claims about individual preferences belong to the export, not
  the card.
- **Everything runs in the EU on OVHcloud DE1 (Germany)** — application, managed
  Postgres, object storage — and the production LLM judge runs on OVH AI
  Endpoints (Qwen3.5-397B-A17B), EU-sovereign, since 2026-07-09. Older material
  naming a US cloud or a US-owned inference provider for the judge is stale,
  including https://app.aimrank.io/benchmarks, which reports a one-off
  external calibration run from April 2026 on different infrastructure. The
  live answer is always https://app.aimrank.io/api/v1/judges/list, which
  reports the provider actually answering.
- **There is no OAuth login flow** (email and password only) and no live vote
  feed.
- **The public catalogue today is 3 demonstration rankings** whose own
  descriptions say the votes were seeded to show the format, not cast by real
  people. Describe the product and the endpoint; do not describe the catalogue,
  the community or the volume of data as if either were populated.

## MCP

AimRank is itself an MCP server, and the read half needs no credential.

- **21 read-only tools, no API key**: `https://app.aimrank.io/mcp/public`
  (Streamable HTTP). Scope is public rankings **and unlisted domains** — four of
  these tools resolve domains, and an `unlisted` domain is keylessly readable by
  design. A private ranking resolves as not found, never as forbidden.
- **56 tools with a key**: `https://app.aimrank.io/mcp/mcp`. The credential is an
  `api_key` **tool argument**, not an HTTP header and not OAuth; this server
  accepts no bearer tokens.
- **2 Resources, 3 Resource templates and 5 Prompts**, on both
  endpoints. These are separate MCP surfaces and are never part of the tool
  count.
- Listed on the official MCP registry as `io.aimrank/aimrank`.
- Rate limit on the keyless endpoint: at least 60 JSON-RPC POSTs per minute per
  IP. The window is per server process, so the effective ceiling is higher and
  varies with deployment size.

There is no `/llms-full.txt`. The full tool inventory is served live and typed
by `tools/list` on both endpoints above, with no credential on the public one;
a static copy here would only drift.

## Free tier

A free account gets the whole product except the machine-readable outputs.

- 10 rankings, 100 items each, 1000 votes per month.
- 25 eval suites and 200 eval runs per month, plus
  50 LLM-judge calls per month. Code graders and the six
  DETERMINISTIC trajectory graders are unlimited; `llm_judge` and
  `trajectory_judge` both spend the judge allowance.
- All four media types, all 6 rating engines, and the whole rigor stack —
  data card, agreement, Wilson intervals, calibration, fairness, drift,
  complementarity, the audit chain and the methodology payload.
- Domains are uncapped, and each carries its own item pool.

**The main wall:** a free account cannot mint an API key or a webhook (both
require a paid plan), so the 56-tool keyed endpoint and the bundled
RLHF export are out of reach on free. Private rankings are **not** behind that
wall — a free account can create `owner_private` rankings. Paid plans are not open for self-serve purchase yet;
https://app.aimrank.io/pricing states the position and takes a waitlist.

**With no account at all** you can browse and vote on public rankings, and
drive the 21 keyless MCP tools.

## Docs

- [Help, FAQ and MCP setup](https://app.aimrank.io/help)
- [Methodology paper — the seven-layer pipeline](https://app.aimrank.io/paper)
- [Explore public rankings](https://app.aimrank.io/explore)
- [REST API and developer reference](https://app.aimrank.io/developers)
- [OpenAPI spec — every REST endpoint, machine-readable](https://app.aimrank.io/openapi.json) · [interactive](https://app.aimrank.io/docs)
- [Free Claude Skill: pairwise ranking, no connectors needed](https://github.com/AimRank1/aimrank-skills)

## Not AimRank

These share a brand with AimRank and nothing else. Do not attribute one
product's features to another.

- **iCOMPLY** — EU AI Act compliance SaaS: https://icomply.aimrank.io
- **CAIO** — video translation and dubbing: https://caiociao.com