> Falconer site index for agents: https://falconer.com/llms.txt. HTML version of this page: https://falconer.com/benchmarks/

# Falconer benchmarks | Head-to-head against Notion, Atlassian Rovo, Claude Code, and Codex

> How Falconer compares on real customer-support and engineering questions: 200 questions across two corpora, scored head-to-head by three frontier LLM judges.

Benchmarks

# Falconer outscores leading AI tools on real support and engineering questions

200 real questions from a public help center and an open-source codebase, scored head-to-head against Notion, Atlassian Rovo, Claude Code, and Codex by three frontier LLM judges.

-   vs Notion560 wins · 311 losses
    
    64%
    
-   vsAtlassian Rovo1064 wins · 83 losses
    
    93%
    
-   vsClaude Code428 wins · 360 losses
    
    54%
    
-   vsCodex654 wins · 304 losses
    
    68%
    

Share of decisive judge verdicts won by Falconer, both tests combined · 200 questions · 4 other AI tools on the market · 3 LLM judges

What we tested

## Benchmarks for docs and code

We judged performance against 200 real questions from a public support corpus and the codebase of a popular open-source project. Falconer wins head-to-head against all leading tools.

Documentation test

### Wix help center

Can a system answer real customer support questions using only a help center? We pointed each tool at the same 6,221 Wix help articles, then scored it on 100 real customer questions from the public WixQA set.

Persona

A Wix customer asking a real support question (billing, domains, site setup) and needs a solution.

Corpus

6,221 articles · 100 questions

Source

[WixQA on Hugging Face](https://huggingface.co/datasets/Wix/WixQA)

Models tested

-   FalconerClaude Opus 4.8 · medium effort
-   NotionClaude Opus 4.8 · effort undisclosed
-   Atlassian Rovomodel undisclosed · Think deeper
-   Claude CodeClaude Opus 4.8 · medium effort
-   CodexGPT-5.5 · medium effort

Code test

### Apache Spark

Can a system answer real engineering questions from live source code? We indexed the apache/spark repository and its markdown docs, then scored each tool on 100 Spark questions from Stack Overflow.

Persona

A data engineer who wants working code, an accurate config key, or a root cause.

Corpus

apache/spark code package with markdown files · 100 questions

Source

[apache/spark on GitHub](https://github.com/apache/spark)

Models tested

-   FalconerClaude Opus 4.8 · high effort
-   NotionClaude Opus 4.8 · effort undisclosed
-   Atlassian Rovomodel undisclosed · Think deeper
-   Claude CodeClaude Opus 4.8 · high effort
-   CodexGPT-5.5 · high effort

The results

## Benchmark results

Docs testCodebase test

**Note:** Falconer, Claude Code, and Codex ran at the same thinking-effort level on each test: medium effort on the docs test, high effort on the code test. Notion and Atlassian Rovo do not expose a comparable effort setting.

### On real support questions, Falconer wins against all four tools.

#### Head-to-head win rate

Falconer wins the majority of decisive verdicts in every matchup.

Win share of decisive judge verdicts (6 per question), scored by the weighted-sum rule. Ties excluded; the full record is shown beside each matchup.

-   vs Notion316 wins · 132 losses · 116 ties
    
    70.5%
    
-   vsAtlassian Rovo503 wins · 66 losses · 31 ties
    
    88.4%
    
-   vsClaude Code213 wins · 192 losses · 195 ties
    
    52.6%
    
-   vsCodex314 wins · 186 losses · 100 ties
    
    62.8%
    

#### Three scoring methods

Matchups were scored holistically, by weighted formula, and by strict Pareto rule.

We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.

| Scoring rule | How it decides | Definition of tie | vs Notion | vsAtlassian Rovo | vsClaude Code | vsCodex |
| --- | --- | --- | --- | --- | --- | --- |
| Weighted-sumUsed above | Blend the four axes (0.35 faithfulness + 0.35 helpfulness + 0.20 completeness + 0.10 relevance); a side wins if it leads by more than 0.25 on the 0 to 10 scale. | The two scores land within 0.25 of each other. | 70.5% | 88.4% | 52.6% | 62.8% |
| Holistic | Trust the judge's own overall pick of the better answer. | The judge called it a tie. | 70.5% | 87.8% | 55% | 60.2% |
| Pareto | A side wins only if it scores ≥ on all four axes and > on at least one. The strictest rule. | Equal on every quality metric, or mixed (each side better somewhere). | 74% | 94.6% | 51.4% | 52.5% |

#### Speed

Falconer delivers the fastest full answers: 18.5s median, well ahead of every rival.

Time to full answer and to first token, across the full distribution. Lower and tighter is better.

Time to full answerWix help center

← faster · tighter = more consistent05s10s15s20s25s30s35s40sFalconer18.5sNotion27.1sAtlassian Rovo30.4sClaude Code27.3sCodex25.9sp25 to p75 box · whisker to p90 · dot = median

Notion answered 94 of 100 questions; all other systems answered every question.

Exact percentiles (seconds)

| System | Mean | p25 | Median | p75 | p90 |
| --- | --- | --- | --- | --- | --- |
| Falconer | 19.6s | 15.3s | 18.5s | 23.3s | 30.5s |
| Notion | 27.5s | 22.3s | 27.1s | 31.2s | 35.8s |
| Atlassian Rovo | 31.2s | 26.5s | 30.4s | 34.6s | 37s |
| Claude Code | 29.4s | 21.1s | 27.3s | 36.2s | 41.9s |
| Codex | 27.3s | 21.1s | 25.9s | 31s | 41.3s |

Time to first tokenWix help center

← faster · tighter = more consistent05s10s15s20s25s30s35s40sFalconer2.9sNotion18.3sAtlassian Rovo18.9sClaude Code2sCodex25.7sp25 to p75 box · whisker to p90 · dot = median

Notion answered 94 of 100 questions; all other systems answered every question.

Exact percentiles (seconds)

| System | Mean | p25 | Median | p75 | p90 |
| --- | --- | --- | --- | --- | --- |
| Falconer | 3.3s | 2.5s | 2.9s | 3.8s | 4.5s |
| Notion | 19.5s | 14s | 18.3s | 23.3s | 30s |
| Atlassian Rovo | 18.4s | 13.8s | 18.9s | 21.4s | 23.7s |
| Claude Code | 2.2s | 1.7s | 2s | 2.3s | 2.9s |
| Codex | 27s | 19.9s | 25.7s | 30.8s | 41.1s |

#### Answer length

Top performers show shorter answer length.

The judges never reward length; it's shown here only as context.

Answer length (characters)Wix help center

01k2k3k4k5k6k7kFalconer1,741Notion2,367Atlassian Rovo6,032Claude Code1,838Codex1,244p25 to p75 box · whisker to p90 · dot = median

Notion answered 94 of 100 questions; all other systems answered every question.

Exact percentiles (characters)

| System | Mean | p25 | Median | p75 | p90 |
| --- | --- | --- | --- | --- | --- |
| Falconer | 1,768 | 1,242 | 1,741 | 2,138 | 2,705 |
| Notion | 2,419 | 1,831 | 2,367 | 2,832 | 3,944 |
| Atlassian Rovo | 5,730 | 5,357 | 6,032 | 6,829 | 7,529 |
| Claude Code | 1,875 | 1,368 | 1,838 | 2,409 | 2,884 |
| Codex | 1,304 | 918 | 1,244 | 1,592 | 2,091 |

### On real engineering questions, Falconer wins against all four tools.

#### Head-to-head win rate

Falconer wins the majority of decisive verdicts in every matchup.

Win share of decisive judge verdicts (6 per question), scored by the weighted-sum rule. Ties excluded; the full record is shown beside each matchup.

-   vs Notion244 wins · 179 losses · 177 ties
    
    57.7%
    
-   vsAtlassian Rovo561 wins · 17 losses · 10 ties
    
    97.1%
    
-   vsClaude Code215 wins · 168 losses · 217 ties
    
    56.1%
    
-   vsCodex340 wins · 118 losses · 141 ties
    
    74.2%
    

#### Three scoring methods

Matchups were scored holistically, by weighted formula, and by strict Pareto rule.

We set three different ways to pick each question's winner. Each cell is Falconer's win rate under that rule.

| Scoring rule | How it decides | Definition of tie | vs Notion | vsAtlassian Rovo | vsClaude Code | vsCodex |
| --- | --- | --- | --- | --- | --- | --- |
| Weighted-sumUsed above | Blend the four axes (0.35 faithfulness + 0.35 helpfulness + 0.20 completeness + 0.10 relevance); a side wins if it leads by more than 0.25 on the 0 to 10 scale. | The two scores land within 0.25 of each other. | 57.7% | 97.1% | 56.1% | 74.2% |
| Holistic | Trust the judge's own overall pick of the better answer. | The judge called it a tie. | 58.8% | 96.2% | 55% | 74.2% |
| Pareto | A side wins only if it scores ≥ on all four axes and > on at least one. The strictest rule. | Equal on every quality metric, or mixed (each side better somewhere). | 61.1% | 97.7% | 55.7% | 75.7% |

#### Speed

On the code test, speed is a near tie: every tool but Codex (72s) returns a full answer in 39 to 45s median.

Time to full answer and to first token, across the full distribution. Lower and tighter is better.

Time to full answerApache Spark

← faster · tighter = more consistent025s50s75s100s125sFalconer45sNotion40sAtlassian Rovo42.6sClaude Code39.1sCodex72sp25 to p75 box · whisker to p90 · dot = median

Atlassian Rovo answered 98 of 100 questions; all other systems answered every question.

Exact percentiles (seconds)

| System | Mean | p25 | Median | p75 | p90 |
| --- | --- | --- | --- | --- | --- |
| Falconer | 51.3s | 31.9s | 45s | 64.7s | 85.4s |
| Notion | 45.4s | 28.6s | 40s | 56.2s | 69.8s |
| Atlassian Rovo | 49.3s | 33.2s | 42.6s | 58.3s | 78.3s |
| Claude Code | 45.4s | 25.3s | 39.1s | 55.9s | 86.1s |
| Codex | 78.6s | 58.7s | 72s | 99.8s | 120.8s |

Time to first tokenApache Spark

← faster · tighter = more consistent025s50s75s100s125sFalconer15.5sNotion24.4sAtlassian Rovo12.7sClaude Code1.9sCodex55.7sp25 to p75 box · whisker to p90 · dot = median

Atlassian Rovo answered 98 of 100 questions; all other systems answered every question.

Exact percentiles (seconds)

| System | Mean | p25 | Median | p75 | p90 |
| --- | --- | --- | --- | --- | --- |
| Falconer | 19.1s | 5.5s | 15.5s | 26s | 42.1s |
| Notion | 27.4s | 12.4s | 24.4s | 36.9s | 50.9s |
| Atlassian Rovo | 13.7s | 11.8s | 12.7s | 15.1s | 17.9s |
| Claude Code | 2.1s | 1.6s | 1.9s | 2.2s | 2.8s |
| Codex | 55.1s | 10.2s | 55.7s | 98.5s | 120.7s |

#### Answer length

Top performers show shorter answer length.

The judges never reward length; it's shown here only as context.

Answer length (characters)Apache Spark

01k2k3k4k5kFalconer3,205Notion3,655Atlassian Rovo228Claude Code3,036Codex1,802p25 to p75 box · whisker to p90 · dot = median

Atlassian Rovo answered 98 of 100 questions; all other systems answered every question.

Exact percentiles (characters)

| System | Mean | p25 | Median | p75 | p90 |
| --- | --- | --- | --- | --- | --- |
| Falconer | 3,295 | 2,592 | 3,205 | 4,031 | 4,764 |
| Notion | 3,846 | 2,942 | 3,655 | 4,734 | 5,542 |
| Atlassian Rovo | 586 | 136 | 228 | 452 | 1,676 |
| Claude Code | 3,067 | 2,451 | 3,036 | 3,733 | 4,411 |
| Codex | 1,870 | 1,398 | 1,802 | 2,283 | 2,895 |

Judging details

## How the judges decided

The full dataset is public, including every question, reference answer, assistant response, and judge verdict: [FalconerAI/falconer-benchmarks](https://github.com/FalconerAI/falconer-benchmarks).

01

### Three frontier judges

Every head-to-head is scored by Claude Opus 4.8, GPT-5.5, and Gemini 3.1 Pro, in both A/B and B/A orderings to neutralize position bias, for six verdicts per question.

02

### A reproducible formula

The headline winner is derived from four metric scores with a fixed weighting. Anyone with the per-metric scores gets the same answer, and the lead holds under two other rules too.

03

### What counts as a tie

Weighted-sum scores within 0.25 of each other count as ties and are excluded from head-to-head win rates.

04

### Pairwise, against a human reference

Every answer is judged head-to-head against Falconer, with the human-written gold answer (the WixQA reference, or the accepted Stack Overflow answer) as the reference both sides are measured against, rather than grading either answer in isolation.

05

### No web access for any agent

Every system runs sealed to its corpus. Web tools are disabled and verified at runtime, so results reflect retrieval and grounding.

06

### Public, reproducible corpora

Both question sets are public (the WixQA support corpus and the apache/spark codebase with its Stack Overflow Q&A), at 100 questions each, so anyone can rerun the same benchmark against their own system.

How we measure quality

## Four metrics, deliberately weighted

Every answer scored 0 to 10 against the human reference.

Faithfulness35%

Every claim is correct and supported by the source, with no fabricated steps, wrong API names, invented config keys, or made-up numbers, even when the rest of the answer reads well.

Is every claim true to the source?

Helpfulness35%

Concrete and actionable: real UI paths, working code, specific steps and version numbers. Vague guidance and "check the docs" deflections are penalized.

Can the reader act on it directly?

Completeness20%

Covers every critical step or fact needed to actually resolve the question. Tangential background does not raise this score.

Did it cover everything needed to finish the job?

Relevance10%

The answer is relevant. Sometimes an answer can be factually true, but irrelevant to the problem in question. Tangents, unrequested background, and citation dumps are penalized. Length itself is not.

How much is on-target signal vs. noise?

The weighting`score = 0.35·faithfulness + 0.35·helpfulness + 0.20·completeness + 0.10·relevance`

-   Correct and actionable matter most, so faithfulness and helpfulness get 35% each. A fluent answer with the wrong API name is worthless.
-   Completeness (20%) catches answers that skip a critical step.
-   Relevance (10%) penalizes off-topic padding.
-   Completeness and relevance deliberately pull against each other: adding content lifts completeness but costs relevance, so neither rewards bloat.

Question selection

## Real questions used

The questions come from public, third-party sources: real help-center Q&A from WixQA and Apache Spark questions from Stack Overflow. For Spark, we selected questions by user votes and by whether answering them well requires reading the repository. We published every question, the reference answer, and each assistant’s full response.

Notes

-   The displayed records use the headline weighted-sum rule: a verdict is a tie when the two weighted scores land within 0.25 of each other on the 0 to 10 scale, too close for the judge to separate (the scoring table shows how each of the three rules defines a tie). Head-to-head % counts wins among the decisive (non-tie) verdicts, the standard way win rates are reported.
-   Each question set is 100 items drawn from a public benchmark; larger samples tighten the confidence intervals.
-   Coverage varies slightly. Notion answered 94 of 100 support questions; on the other 6 it returned an interactive clarification form instead of an answer, so those are excluded. Rovo answered 98 of 100 engineering questions. Win/loss/tie counts are tallied over the questions each system actually answered (6 verdicts per answered question), so they total below 600 for those systems. One Codex verdict on the engineering test was an unparseable judge response and is also excluded, so that record totals 599.
-   On the engineering test, Atlassian Rovo is a structural baseline rather than a like-for-like rival: the apache/spark repository was hosted in Bitbucket, which Rovo barely reads, so its short answers reflect missing source access rather than answer quality. Read that matchup as retrieval-with-code vs retrieval-without-code; Notion, Claude Code, and Codex are the code-capable cohort.
-   Per-metric averages and head-to-head wins can diverge: a system can post a higher average on one metric yet win fewer questions, because each question’s winner is decided from all four metrics together.
-   Ties (weighted-sum scores within 0.25 of each other) are excluded from head-to-head %. Tie rates ranged from ~2% (Rovo) to ~36% (Claude Code); closer matchups naturally produce more ties.
-   The Stack Overflow reference is one valid solution, not the only correct answer; judges credit alternate-but-correct approaches.
-   Falconer, Claude Code, and Codex run named models at the same thinking-effort level on each test: medium effort on the support test, high effort on the engineering test. Notion exposes its model (Opus 4.8) but not its effort level; Rovo ran in its "Think deeper" mode on an undisclosed model.
-   Results are a point-in-time snapshot from a June 2026 run.