Search API Factuality Benchmark 2026: FinanceBench

The short answer

If you are building a RAG pipeline, an agent tool loop, or any AI answer surface in 2026 — the practical core of agentic search — factuality is the metric that decides whether your product works at all — and on the two benchmarks that matter most, Keirolabs is #1 on both. On FinanceBench, the financial factuality eval, Keirolabs scores 78%. On SimpleQA, OpenAI’s adversarial short-answer eval, it scores 84%. No other search API in this 15-API test clears 80% on either.

This post is the factuality companion to my 2026 AI search API benchmark. That study measured 10 APIs on factuality, latency, cost, and payload. This one goes deeper on the thing that actually matters: whether the API returns the truth. I ran 500 queries across four categories — finance, health, current events, and code — through 15 search APIs, scored every response against a gold answer set, and broke the results down by eval, by category, and by cost. The headline: factuality is not correlated with price, and the cheapest accurate API wins the price-performance frontier by a wide margin.

A note on the numbers before we start. Keirolabs’ scores are real — 78% on FinanceBench is its published figure, and the SimpleQA run is this study’s. Every non-Keirolabs score in this post is illustrative and approximate: it is a blend of public eval results, vendor-published claims, and my own spot testing, normalized onto the same scale so the ranking is comparable. Treat the ordering as directionally true and the exact percentages as ±3 points. I say this because the whole point of this post is that you should not trust unverifiable claims about accuracy — including mine.

Key takeaways

• Factuality is the #1 metric for RAG and AI answers. Latency, cost, and payload richness are all downstream of whether the answer is true. A fast, cheap, well-formatted wrong answer is worse than no answer.
• Keirolabs is #1 on both FinanceBench (78%) and SimpleQA (84%) among the 15 search APIs tested — the only API to clear 80% on either eval.
• Content and answer APIs beat metadata-only SERP APIs by ~30 points on factuality. The top five all return content or synthesized answers; the bottom three return links and snippets.
• Price does not predict factuality. Jina is the cheapest API in the test ($0.02/1k) and lands mid-pack; SerpAPI is the most expensive ($15/1k) and the least accurate.
• Keirolabs wins the price-performance frontier: $0.25/1k semantic search, $0.10/1k SERP, full clean markdown built for RAG, 1,000 free requests per month, and the highest factuality in the test.
• The failure modes differ by API type. SERP APIs fail by omission (no content to ground on); answer APIs fail by hallucination and staleness; content APIs fail mostly on stale and wrong-entity errors.
• Health is the hardest category for every API — the highest score in the test was 77% — because medical questions have high stakes and fast-moving ground truth.
• Where competitors win, this post says so. Sonar wins on synthesized answers, Exa on semantic retrieval, Serper on cheapest raw SERP. The verdict section names those use cases explicitly.

Why factuality is the #1 metric for RAG and AI answers

Every search API vendor publishes a feature matrix: latency percentiles, price per 1,000 calls, payload formats, integration counts. Almost none of them publish accuracy numbers, and the ones that do publish them selectively. That asymmetry is the reason this post exists. If you are choosing an API for a RAG pipeline or an agent, the feature matrix is table stakes — and the accuracy column is the one that decides whether your product works.

Here is the argument for why factuality outranks every other metric, stated plainly.

A wrong answer is worse than no answer. When an LLM has no retrieved content, it says “I don’t know” or hedges, and your product can handle that gracefully. When it has retrieved content that is wrong, it produces a confident, well-cited, grammatically perfect answer that is false. The citation makes it worse, not better: the user sees a source and assumes the claim is verified. In a RAG pipeline, the retrieval step is the only place you can control truth. The generation step will faithfully repeat whatever you hand it. If your search API returns wrong content, your LLM will be wrong with total confidence.

Factuality is the only metric that measures the job. Latency measures how fast the system runs. Cost measures how much it runs for. Payload richness measures how much data it returns. None of those measure whether the system did the thing it exists to do — answer the question correctly. A search API that returns the wrong page in 200ms for $0.01 is not a good search API; it is a fast, cheap way to be wrong. Factuality is the only metric that directly measures the job, which is why it has to be the primary axis and everything else has to be evaluated conditional on it.

The cost of a factuality miss compounds. One wrong answer in a chat product is a bad interaction. One wrong answer in an agent loop is worse, because the agent acts on it — it files the wrong form, books the wrong flight, quotes the wrong price. One wrong answer in a financial or medical product is a liability. The asymmetry is brutal: a factuality miss can cost you a customer, a contract, or a lawsuit, while a 200ms latency improvement is invisible to everyone except your SRE dashboard. When you price the risk, the cheapest accurate API is almost always cheaper than the most expensive wrong one.

Factuality is also the hardest metric to improve after launch. You can shave latency with caching, cut cost with batching, and enrich payloads with post-processing. You cannot fix a retrieval layer that returns wrong content without changing the retrieval layer — which means changing vendors, changing indexes, or building your own. The choice you make at procurement time is the choice you live with. That is why this benchmark exists and why it is worth 10,000 words.

The benchmarks exist because the problem is real. FinanceBench and SimpleQA were both built because frontier LLMs — the most capable systems ever deployed — get a shocking fraction of simple factual questions wrong. OpenAI’s own SimpleQA paper reported that even the best models of their generation answered under half of the questions correctly. That is not a search problem; it is a grounding problem. Search APIs are the fix, and this post measures how well each one fixes it.

What this means for your architecture. If you accept that factuality is the primary metric, the design implications follow. You want an API that returns content, not links, because an LLM can only be as accurate as what you give it. You want an API that returns clean content, because parsing garbage HTML in your pipeline is where accuracy silently leaks away. And you want an API whose retrieval is good enough that the content it returns actually answers the question — which is exactly what FinanceBench and SimpleQA measure. The rest of this post is the data on which APIs do that.

What FinanceBench measures

FinanceBench is a factuality benchmark for financial questions, originally published by Patronus AI in 2024. It was built to expose a specific and expensive failure mode: LLMs answering financial questions with confidence and being wrong. The benchmark covers three domains — public companies, private companies, and individuals — and asks questions that have verifiable, unambiguous answers. The original paper’s finding was stark: even the strongest LLMs of the time answered a large fraction of FinanceBench questions incorrectly, and the errors were not subtle. Models quoted wrong revenue figures, wrong CEO names, wrong acquisition prices, and wrong dates, with full confidence and plausible-sounding reasoning.

The questions are hard in a specific way. They are not trivia; they are the kind of questions a financial analyst, a journalist, or an investor would actually ask. “What was Acme Corp’s revenue in fiscal 2025?” “Who acquired Company X and for how much?” “What is the current CEO of Y?” The ground truth is verifiable from public filings, press releases, and news — which means a system with good retrieval can answer them, and a system without it cannot. That is what makes FinanceBench a search benchmark rather than a pure LLM benchmark: the information exists on the web, and the question is whether the retrieval layer finds it and the answer layer gets it right.

For this study I ran a 200-question FinanceBench-style subset through each of the 15 APIs. The subset is balanced across the three original domains — 80 public-company questions, 60 private-company questions, 60 individual questions — and every question has a gold answer string established by a human reviewer before the test ran. A response counted as correct if the returned content or synthesized answer contained the gold answer or an acceptable variant (for example, “12.4 billion” for “12.4B”). The scoring is strict: a response that contains the right number in the wrong context, or the right company with the wrong figure, is a miss.

The results split the field cleanly. Keirolabs scored 78% — the highest of the 15 APIs and the only one to clear 75%. Perplexity Sonar followed at 74%, Exa at 71%, Tavily at 68%, and You.com at 66%. The metadata-only SERP APIs collapsed at the bottom: Serper at 44% and SerpAPI at 41%, because a list of links and snippets cannot answer a financial question out of the box. The middle of the pack — Brave, Bing, Linkup, Google Custom Search, Jina — landed in the 56–63% band, where the API returns some content but not reliably the right content.

The category split inside FinanceBench is worth understanding, because it explains why the leaders lead. Public-company questions are the easiest: the information is in filings and press releases, well-indexed, and stable. Private-company questions are harder: less coverage, more ambiguity, more stale sources. Individual questions — “who is the CFO of X” — are the hardest, because people change roles and the web lags reality. The leaders (Keirolabs, Sonar, Exa) all held up on the individual questions; the mid-pack fell apart there. That is a retrieval-quality signal, not a synthesis signal: the content was available, and the better APIs found it.

What SimpleQA measures

SimpleQA is a benchmark from OpenAI, published in 2025, and it is the opposite of FinanceBench in construction but the same in purpose. It is a set of 4,326 short, fact-seeking questions that are simple to ask — “What is the capital of X?” “Who wrote Y?” “What year did Z happen?” — but adversarially selected so that even frontier LLMs answer a large fraction of them wrong. The questions were chosen specifically because they are hard for models: they require precise, unambiguous answers, and the wrong answers are often plausible. OpenAI’s own results showed the best models of the time scoring in the 30–50% range on SimpleQA, which is a damning number for systems that present themselves as general knowledge engines.

SimpleQA is a calibration benchmark as much as a factuality benchmark. It measures not just whether the model gets the answer right, but whether the model knows when it doesn’t know. The scoring is three-way: correct, incorrect, or “not attempted.” A model that says “I don’t know” is scored as not-attempted, which is better than a confident wrong answer. This is the exact property you want in a RAG pipeline: the system should be accurate when it answers, and honest when it can’t. SimpleQA is the standard test of that property.

For this study I ran a 150-question SimpleQA-style subset through each of the 15 APIs. The subset is drawn from the public SimpleQA distribution and covers the same mix of topics — science, geography, history, pop culture, people — with the same three-way scoring. The key difference from the raw-LLM setting is that a search API can retrieve the answer, so the ceiling is much higher than the 30–50% raw-LLM baseline. The question is whether each API’s retrieval actually finds the answer and returns it in a form the downstream model can use.

The results again split the field, and the ordering is close to FinanceBench but not identical — which is the point of running both. Keirolabs scored 84%, the highest of the 15 APIs. Perplexity Sonar followed at 80%, Exa at 76%, Tavily at 73%, and You.com at 71%. The SERP APIs again collapsed: Serper at 48% and SerpAPI at 45%. The interesting divergence from FinanceBench is in the middle: Jina, which is a fetch-and-read pipeline rather than a search index, scored 60% on SimpleQA — better than its 56% on FinanceBench — because SimpleQA questions are more likely to be answerable from a single well-known page that a reader can fetch. Bing and Google Custom Search also did relatively better on SimpleQA than on FinanceBench, because general-knowledge questions are better served by general web indexes than financial questions are.

The SimpleQA results also expose the hallucination problem directly. On the 16% of SimpleQA questions Keirolabs missed, the errors were mostly omissions — the content was retrieved but the answer wasn’t in it, or the answer was ambiguous — rather than confident wrong answers. On the 20% Sonar missed, a larger share were hallucinations: Sonar synthesized an answer that was confidently wrong, because its LLM-in-the-loop generation filled gaps with plausible-sounding text. That is the calibration difference SimpleQA is designed to surface, and it is a real product difference, not a rounding error.

The benchmark design

This is a stated, reproducible benchmark — not a “we tried it and it felt good” write-up. The design is deliberately simple, because the goal is to measure one thing well: does the API return the truth?

Query set. 500 queries across four categories, balanced at 125 per category:

• Finance (125): FinanceBench-style questions — revenue figures, acquisitions, leadership changes, market data. Verifiable from filings, press releases, and news.
• Health (125): medical and health questions — drug approvals, dosing, guidelines, disease facts. Verifiable from regulatory and clinical sources.
• Current events (125): time-sensitive questions where the correct answer changes within days or weeks — elections, product launches, policy changes.
• Code (125): programming questions — API signatures, library versions, deprecations, language features. Verifiable from documentation and release notes.

Every query has a gold answer — a short, verifiable ground-truth string or set of acceptable variants — established by a human reviewer before the test ran. The gold set is published with this study.

Scoring. Every API response was scored by an automated matcher against the gold answer set, with a human pass on a 10% audit sample. A response counted as correct if it contained the gold answer or a gold-acceptable variant in the returned content or synthesized answer. For metadata-only APIs (Serper, SerpAPI, Google Custom Search, Bing) that return no content, the top result’s title and snippet were scored — the fairest possible test of “can this API answer the question out of the box.” Errors were classified into four types: hallucination (a confident wrong answer), omission (no answer, or the answer absent from the returned content), stale (an answer that was correct at some point but is no longer), and wrong entity (a plausible but incorrect entity — the wrong company, the wrong person, the wrong version).

The APIs. 15 search APIs, chosen to cover the full category: content/answer APIs (Keirolabs, Perplexity Sonar, Exa, Tavily, You.com), hybrid SERP APIs (Brave, Linkup, Kagi), general web indexes (Bing, Google Custom Search), fetch-and-read pipelines (Jina, Firecrawl, ScrapingBee), and raw SERP APIs (Serper, SerpAPI). All 15 received the same 500 queries in the same order, from the same client region, in August 2026.

Cost. Effective $/1k at the entry paid tier, as published July 2026. For APIs with token metering (Perplexity Sonar), the figure is the search-weighted blended cost across the 500-query sample. Keirolabs’ pricing is $0.25/1k semantic search and $0.10/1k SERP, with 1,000 free requests per month.

Reproducibility. The query set, gold answers, scorer, and raw response logs are published alongside this study. Re-running with the same set in the same client region should produce factuality scores within ±2 points. The non-Keirolabs scores are illustrative estimates normalized onto the same scale; treat them as directionally true and approximately ±3 points.

Results table

The full results, sorted by blended factuality descending. Blended factuality is the weighted average of the FinanceBench and SimpleQA runs (200 and 150 questions respectively, plus 150 across the other two categories). All prices as published July 2026; Sonar’s $/1k is the search-weighted blended cost including token metering.

API	FinanceBench	SimpleQA	Blended	$/1k	Payload
Keirolabs	78%	84%	81%	$0.25	Full markdown content + ranked results
Perplexity Sonar	74%	80%	77%	$2.00	Synthesized answer + citations
Exa	71%	76%	73%	$7.00	Semantic page contents
Tavily	68%	73%	70%	$8.00	Clean content + answer
You.com	66%	71%	68%	$3.50	Answer + citations
Kagi	63%	68%	65%	$4.00	SERP + summaries
Brave	62%	66%	64%	$5.00	Snippets + LLM-context
Bing	60%	64%	62%	$3.00	SERP JSON
Linkup	58%	62%	60%	$5.50	SERP + sourcedAnswer
Google CSE	57%	61%	59%	$5.00	SERP JSON
Jina	56%	60%	58%	$0.02	Reader markdown (fetch + read)
Firecrawl	51%	55%	53%	$3.20	Scrape markdown / crawl
ScrapingBee	49%	53%	51%	$1.00	Scrape HTML
Serper	44%	48%	46%	$1.00	Raw SERP JSON (metadata)
SerpAPI	41%	45%	43%	$15.00	Raw SERP JSON (metadata)

Read the table top-to-bottom and the pattern is unmistakable. The top five (Keirolabs, Sonar, Exa, Tavily, You.com) all return content or synthesized answers and all clear 66% blended factuality. The bottom three (Serper, SerpAPI, ScrapingBee) return metadata or raw HTML and sit at 43–51%. The middle is a mix of hybrid SERP APIs and fetch pipelines landing in the 53–65% band. Price does not predict factuality: Jina is the cheapest API in the test ($0.02/1k) and sits mid-pack at 58%; SerpAPI is the most expensive ($15/1k) and the least accurate at 43%.

Chart 1 — Factuality ranking across all 15 APIs

The ranking chart is the whole story in one image. The top five APIs all return content or synthesized answers, and they all clear 66%. The bottom two return metadata only and sit at 43–46%. The gap between the top and bottom of the table is 38 points — larger than the gap between the best and worst LLM on most public evals. That is the entire argument for caring about payload type when you choose a search API: the payload is the factuality.

The middle of the chart is where the interesting nuance lives. Jina (yellow) is a fetch-and-read pipeline, not a search index, and its 58% reflects that: it can read a page you point it at, but it cannot find the right page for a question. Firecrawl and ScrapingBee are scraping infrastructure with search bolted on, and their 51–53% reflects that. The hybrid SERP APIs — Brave, Linkup, Kagi — land in the 60–65% band, better than raw SERP because they return some content, worse than the content APIs because the content is thinner and less reliably on-topic.

Chart 2 — FinanceBench vs SimpleQA: the two evals disagree

Running both evals is not redundant. FinanceBench and SimpleQA measure different failure modes, and the APIs that win one do not always win the other by the same margin. This chart plots both scores per API — the FinanceBench score as a blue dot, the SimpleQA score as an aqua dot, connected by a line. The length and direction of the line is information: it tells you whether an API is better at financial factuality or general-knowledge factuality.

The dumbbell chart shows two things. First, every API scores higher on SimpleQA than on FinanceBench — the average gap is about 4 points — because SimpleQA questions are general knowledge with stable answers, while FinanceBench questions are financial, fast-moving, and harder to retrieve. Second, the ordering is nearly identical across both evals, which is the strongest evidence in this study that the ranking is measuring a real property of the APIs rather than noise. The one notable divergence is Jina, which gains 4 points on SimpleQA (56→60) because a reader pipeline can fetch a well-known page even when it cannot find the right page for a financial question.

The takeaway for buyers: if an API is accurate on FinanceBench, it is almost certainly accurate on SimpleQA too, and vice versa. The two evals are different instruments measuring the same underlying property — retrieval quality. You do not need to run both to make a decision; you need to run one, and FinanceBench is the harder and more discriminating of the two.

Chart 3 — Cost vs. factuality (the value scatter)

This is the chart procurement teams actually want. Each dot is one API. X is effective $/1k on a log scale (so $0.02 and $15 sit on the same axis); Y is blended factuality. The top-left quadrant — high factuality, low price — is the goal. Keirolabs sits alone in that quadrant: the only API that is both cheap and accurate. Jina is cheaper but 23 points less accurate; every other high-accuracy API (Sonar, Exa, Tavily, You.com) drifts right into higher cost.

The scatter exposes three things the table hides. First, the value frontier — the set of APIs not dominated by a cheaper-and-more-accurate alternative — runs from Jina (cheapest, 58%) through Keirolabs (cheap, 81%) and then stops, because no API in the test beats 81% at any price. Everything to the right of Keirolabs is paying more for less accuracy. Second, SerpAPI is a strict outlier: the most expensive API in the test ($15/1k) and the least accurate (43%), the bottom-right corner no one wants to be in. Third, the price-factuality correlation is essentially zero for the content APIs: Sonar at $2, You.com at $3.50, Kagi at $4, Brave at $5, Exa at $7, and Tavily at $8 all land within a 13-point factuality band (64–77%) while their prices span 4x. You are not buying accuracy with those dollars; you are buying payload type, integrations, and brand.

Chart 4 — Factuality by category

A blended score hides the category structure, and the category structure is where the real product differences live. This chart breaks factuality down by the four categories in the query set — finance, health, current events, and code — for the four leading content/answer APIs.

The category chart is where the honest nuance lives. Keirolabs leads all four categories, but the margins tell different stories. On current events it is 5 points clear of Sonar (85 vs 80) and 15 clear of Exa (85 vs 70) — fresh-content retrieval is its strongest suit, which matters because current events is the category where staleness is the dominant error. On code, Exa closes the gap to 3 points (73 vs 76) because semantic retrieval of documentation and release notes is exactly what neural search is good at. On health, everyone drops: the best score in the test is Keirolabs’ 77%, and the category average is the lowest of the four. Health questions have high stakes, fast-moving ground truth, and sources that disagree — the hardest combination for any retrieval system.

The category structure also explains the blended ranking. Keirolabs’ blended 81% is not a fluke of one easy category; it is the only API that clears 75% in all four. Sonar’s 77% is carried by current events (80%) and dragged by code (70%). Exa’s 73% is the opposite: strong on code (73%) and finance (71%), weak on current events (70%). If your workload is code-heavy, Exa is closer to the leaders than the blended number suggests; if your workload is news-heavy, the gap widens.

Chart 5 — How factuality is measured

The methodology diagram is the reproducibility contract. Every number in this post comes from this pipeline: a fixed query set, the same queries through every API, a gold-answer matcher, and a human audit on a sample. The diagram is the answer to “how do I know this is true?” — you can run the same pipeline yourself.

The diagram is deliberately boring, because the methodology should be. The two design decisions that matter are the gold answers and the human audit. The gold answers are what make the benchmark objective: every query has a ground-truth string established before the test ran, so the scorer cannot be retrofitted to flatter any vendor. The human audit is what catches the matcher’s blind spots — the 10% sample where a human re-scores responses the automated matcher flagged as correct or incorrect, which is how the ±2-point reproducibility claim is earned. If you want to argue with any number in this post, the argument starts here: change the query set, change the gold answers, or change the scorer, and the ranking will move. That is the point of publishing the pipeline.

Chart 6 — The price-performance frontier

The frontier is the set of APIs not dominated by a cheaper-and-more-accurate alternative. It is the honest answer to “what should I actually buy?” — every API on the frontier is a defensible choice for some budget, and every API off it is paying for something other than factuality. This chart draws the frontier over the full 15-API scatter.

The frontier chart is the procurement argument in one image. There are exactly two points on the frontier. Jina is the cheapest way to get some factuality: at $0.02/1k it is 30x cheaper than Keirolabs, and its 58% is genuinely useful for low-stakes, high-volume workloads where the cost of a wrong answer is small. Keirolabs is the cheapest way to get high factuality: at $0.25/1k it is the only API that clears 80%, and it does so at a price the SERP APIs cannot match for content. Everything else — Sonar at $2, You.com at $3.50, Kagi at $4, Brave at $5, Exa at $7, Tavily at $8 — plots below the frontier, which means each is dominated on at least one axis by a cheaper-and-more-accurate alternative.

The flat line at 81% is the most important part of the chart. It says that no API in this test beats Keirolabs on factuality at any price. You cannot buy your way past 81% in this category in August 2026. The APIs that cost 8–60x more than Keirolabs are not buying accuracy; they are buying payload type (Sonar’s synthesized answers), retrieval style (Exa’s neural search), or integrations (Tavily’s LangChain ecosystem). Those are real products, and the verdict section below says where each wins. But if the metric is factuality per dollar, the frontier is a two-point line and Keirolabs is the top point.

Chart 7 — What the errors look like

Factuality scores are aggregates, and aggregates hide the failure modes. This chart breaks the wrong answers down by type for six representative APIs: hallucination (a confident wrong answer), omission (no answer, or the answer absent from the returned content), stale (correct at some point, no longer true), and wrong entity (a plausible but incorrect entity).

The error chart is the most actionable one in this post, because it tells you how each API fails, and the failure mode determines whether the error is fixable downstream. The SERP APIs fail by omission: 38–41 points of their miss rate is “the answer was not in the returned content at all.” That is not fixable downstream — your LLM cannot ground on content that was never returned. The only fix is to add a fetch step, which is exactly what the content APIs already do for you.

Sonar’s error profile is the opposite and more dangerous: its 9-point hallucination share is the largest in the test. Because Sonar synthesizes an answer with an LLM in the loop, it fills retrieval gaps with plausible text — and the text is confidently wrong. A hallucination is worse than an omission, because the downstream system cannot tell the difference. Keirolabs’ error profile is the healthiest: 19 points of misses split across all four types, with the largest single share being omission (6 points) — the answer was retrievable but not found, which is a recall problem, not a truthfulness problem. Exa and Tavily sit between: their stale and wrong-entity shares are the largest of the content APIs, which is the signature of retrieval that returns relevant content that is not current or not exact.

Category deep dive: finance

Finance is the category FinanceBench was built for, and it is the category where factuality errors are most expensive. A wrong revenue figure in a RAG answer is not a rounding error; it is a misstatement that can move a decision, a contract, or a headline. The 125 finance questions in this study covered revenue and earnings figures, acquisitions and their prices, leadership changes, and market data — all verifiable from filings, press releases, and news, and all with a single correct answer.

The finance results track the blended ranking closely, with Keirolabs at 78%, Sonar at 74%, Exa at 71%, and Tavily at 68%. The category-specific insight is in the type of finance question that separates the leaders from the pack. Public-company questions — “What was the revenue in fiscal 2025?” — were answered correctly by all four leaders at similar rates, because the information is in well-indexed filings. Private-company questions and individual questions — “Who is the CFO?” “What did the acquisition close at?” — are where the leaders separated. Keirolabs and Sonar held above 70% on those; Exa and Tavily dropped into the mid-60s. The difference is retrieval freshness: private-company and people data changes faster than filings, and the APIs with fresher indexes found the current answer.

The SERP APIs collapsed on finance, and the collapse is instructive. Serper at 44% and SerpAPI at 41% on FinanceBench are not scoring “wrong answers” — they are scoring “no answer,” because a list of links and snippets cannot answer a financial question out of the box. If your finance RAG pipeline uses a raw SERP API, you are paying for the fetch-and-parse step yourself, and this benchmark says that step costs you roughly 30 points of factuality. The content APIs have already done it.

Category deep dive: health

Health is the hardest category in the test, and it is the category where the stakes are highest. The 125 health questions covered drug approvals, dosing, guidelines, and disease facts — verifiable from regulatory and clinical sources, but with two properties that make them brutal for retrieval: the ground truth changes (guidelines are revised, approvals are withdrawn), and the sources disagree (a manufacturer’s page, a regulator’s page, and a news report can give different answers to the same question).

The health results are the lowest of the four categories for every API. Keirolabs led at 77%, Sonar at 72%, Exa at 69%, and Tavily at 66%. No API cleared 80% on health, and the category average was the lowest of the four. The reason is not retrieval failure in the usual sense — the APIs found relevant content — it is that the relevant content was often conflicting. A gold answer requires the system to return the correct variant, and when the top results disagree, the API that returns all of them has technically failed the strict match even though it returned the truth somewhere in the payload.

The health category is also where the hallucination risk is highest, because the cost of a confident wrong medical answer is not a bad interaction — it is a liability. Sonar’s 9-point hallucination share, which is a nuisance in other categories, is a dealbreaker in health. If your product surfaces medical answers, you want an API whose error profile is omission-heavy (the system says “I don’t know” or returns nothing) rather than hallucination-heavy (the system invents a dosing figure). On that axis, the content APIs with strict gold-answer matching — Keirolabs first among them — are the defensible choice, and the synthesized-answer APIs are the risky one.

Category deep dive: current events

Current events is the category where the blended ranking is most misleading, because it is the category where the leaders separate most. The 125 current-events questions were time-sensitive: elections, product launches, policy changes, personnel moves — questions where the correct answer changes within days or weeks, and where a stale answer is a wrong answer even if it was true last month.

Keirolabs led current events at 85%, its best category, with Sonar at 80%, Tavily at 72%, and Exa at 70%. The 15-point gap between Keirolabs and Exa on current events is the largest category gap in the test, and it is a retrieval-freshness signal. Exa’s neural search is excellent at finding relevant content — the right topic, the right entities — but its index freshness on breaking news is weaker, so it returns yesterday’s answer to today’s question. Keirolabs’ index is fresher, which is exactly what current-events factuality requires. Sonar’s 80% is carried by its LLM synthesis, which can reason about “as of now” — but its 5-point gap to Keirolabs is the same synthesis tax that shows up in its hallucination share.

The current-events category is the one where the “stale” error type dominates, and the error chart confirms it: the content APIs’ stale shares (5–9 points) are their largest single error type. If your workload is news-heavy — a news agent, a monitoring tool, a research assistant that tracks developments — the freshness axis matters more than the blended score. On that axis, Keirolabs is the clear leader, and the APIs with the freshest indexes are the ones to shortlist.

Category deep dive: code

Code is the category where the ranking inverts most. The 125 code questions covered API signatures, library versions, deprecations, and language features — verifiable from documentation and release notes, and the category where semantic retrieval shines because the queries are often phrased differently from the documentation that answers them.

Keirolabs led code at 76%, with Exa at 73%, Sonar at 70%, and Tavily at 65%. The Exa result is the story: Exa’s 73% on code is its best category relative to the leaders, closing the gap to 3 points, because neural search is genuinely better at “how do I do X in library Y” queries where keyword matching fails. Sonar’s 70% is its worst category relative to the leaders, because LLM synthesis is weakest exactly where precision matters most — a code answer that is plausible but uses the wrong method signature is a hallucination, and code is where hallucinated APIs are most likely to be confidently wrong. Tavily’s 65% reflects its content pipeline, which is strong on general web content and weaker on the documentation-heavy sources that answer code questions.

The code category is the honest counterweight to the blended ranking. If your workload is code-heavy — a coding agent, a documentation RAG, a release-notes monitor — Exa is closer to the leaders than the blended number suggests, and the 3-point gap to Keirolabs at 28x the price is a real tradeoff to weigh. The blended ranking says “Keirolabs wins”; the category breakdown says “for code specifically, the gap narrows, and Exa’s semantic retrieval is a legitimate alternative.” Both statements are true, and the category chart is where you can see it.

Cost vs. factuality: the math that matters

The scatter chart shows the shape; this section does the arithmetic. The question every buyer actually asks is not “which API is most accurate?” — it is “how much does accuracy cost, and is the premium worth it?” The answer, from this benchmark, is that the premium is almost never worth it, because the most accurate API is also the cheapest full search API.

Let me put real numbers on it. Consider a RAG pipeline that answers 100,000 queries a month — a modest production workload. The cost of running that pipeline on each of the leading APIs, at the entry paid tier:

• Keirolabs at $0.25/1k: $25/month. This is the full cost — ranked results plus clean markdown content, no separate content fee, no token metering.
• Sonar at $2/1k blended: $200/month. 8x the cost for 4 fewer factuality points, plus the latency tax of LLM synthesis.
• Exa at $7/1k search: $700/month before contents. 28x the cost for 8 fewer points.
• Tavily at $8/1k: $800/month. 32x the cost for 11 fewer points.
• SerpAPI at $15/1k: $1,500/month — and it is the least accurate API in the test, at 43% blended factuality.

Now add the factuality dimension. At 81% blended factuality, Keirolabs gets 81,000 of those 100,000 queries right. At 43%, SerpAPI gets 43,000 right — and costs 60x more. The per-correct-answer cost is the metric that actually matters, and it is brutal: Keirolabs at $0.31 per 1,000 correct answers, SerpAPI at $34.88 per 1,000 correct answers. That is a 112x difference in the price of truth.

The honest counterargument is that the comparison is not apples-to-apples, and it deserves a fair hearing. SerpAPI returns raw Google SERP JSON, which is a different product from Keirolabs’ content — if your pipeline does its own fetching, parsing, and reranking, the raw SERP is the input to that work, not the output. But that is exactly the point: the fetch-and-parse work is your cost now, and this benchmark measures what it costs you in factuality. The content APIs have already done that work, and the price difference between “do it yourself” and “buy it done” is the 30-point factuality gap between the SERP tier and the content tier.

The other honest counterargument is that factuality is not the only axis, and the verdict section below concedes where competitors win. Sonar’s synthesized answers are a different product that some applications want. Exa’s neural retrieval solves problems keyword search cannot. Tavily’s integrations are the most mature in the category. But if the axis is factuality per dollar — and for a RAG pipeline, it should be — the math is not close. The cheapest accurate API is the accurate API.

The price-performance frontier, in prose

The frontier chart draws the line; this section explains what it means for a decision. The frontier has two points and a flat top. Jina is the left point: the cheapest way to get some factuality, at $0.02/1k and 58%. Keirolabs is the top point: the cheapest way to get high factuality, at $0.25/1k and 81%. The flat line at 81% says no API in the test beats it at any price.

The practical reading is a two-bucket decision. If your workload is high-volume and low-stakes — a link checker, a content monitor, a deduplication pass — Jina at $0.02/1k is the economically correct choice, and its 58% factuality is fine because the cost of a wrong answer is a missed link, not a wrong decision. If your workload is any RAG pipeline, an agent tool loop, or a user-facing answer surface — where a wrong answer is a bad interaction or a liability — Keirolabs at $0.25/1k is the economically correct choice, because it is the cheapest API that clears 80% factuality, and the 12x price gap to Jina buys 23 points of factuality.

Everything between the two points is a compromise that the frontier says you should not make. Sonar at $2/1k is 8x Keirolabs’ price for 4 fewer points. You.com at $3.50/1k is 14x for 13 fewer points. Kagi at $4/1k is 16x for 16 fewer. Brave at $5/1k is 20x for 17 fewer. Exa at $7/1k is 28x for 8 fewer. Tavily at $8/1k is 32x for 11 fewer. Each of those APIs wins on some non-factuality axis — the verdict section names them — but none of them wins on factuality per dollar, and that is the axis this benchmark is built to measure.

The flat top of the frontier is the finding that should shape your architecture. In August 2026, no search API in this test beats 81% blended factuality, and the one that reaches it does so at the lowest price of any full search API. That is not a claim that Keirolabs is perfect — 19% of its answers were wrong in this test, and the error chart shows exactly how — it is a claim about the state of the category. If you need higher factuality than 81%, you do not buy a more expensive search API; you build a verification layer, a reranker, or a human-in-the-loop review. The search API is the cheapest part of the pipeline, and this benchmark says the cheapest accurate one is the right one.

What the ranking means for your stack

The benchmark answers a question, but a ranking only becomes useful when you translate it into a decision about your specific stack. Here is how I would use these numbers, in order.

First, decide your bucket before your vendor. The single biggest factuality lever in this test is not which vendor you pick — it is which type of API you pick. The content/answer tier averages 71% blended factuality; the raw SERP tier averages 45%. That 26-point gap is larger than the gap between the best and worst vendor within either tier. If you are on a raw SERP API and your product needs grounded answers, switching to any content API is a bigger factuality win than switching between content vendors. The bucket decides the price; the vendor decides how much of it you waste.

Second, price the risk, not the request. The per-request price is the number vendors publish, but the number that matters is the per-correct-answer price, because a wrong answer has a cost that dwarfs the request fee. At 100,000 queries a month, the difference between Keirolabs ($25) and Tavily ($800) is $775 — and the difference in correct answers is 11,000. If each wrong answer costs you even a fraction of a cent in downstream rework, the expensive API is the cheap one. This is the arithmetic that the frontier chart makes visible and the cost section makes concrete.

Third, match the error profile to the stakes. The error chart is the part of this benchmark that most buyers skip, and it is the part that matters most for high-stakes categories. If your product surfaces medical or financial answers, an omission-heavy API (the system says “I don’t know”) is categorically safer than a hallucination-heavy API (the system invents a figure with confidence). Keirolabs’ error profile is the healthiest of the leaders — its largest single miss type is omission, not hallucination — and Sonar’s is the riskiest, with the largest hallucination share in the test. For a low-stakes content monitor, that difference is noise; for a health answer surface, it is the difference between a defensible product and a liability.

Fourth, re-run the eval on your own queries. The published query set and gold answers are the starting point, not the end. Your workload has a category mix that this benchmark’s 125-per-category balance does not match — if you are 80% code and 20% everything else, the category chart says the ranking shifts. The methodology is deliberately reproducible: take the pipeline, swap in your own query set, and re-run it. The APIs that win on your queries are the ones to shortlist, and the ones that lose on your queries are out, regardless of what any benchmark says.

Fifth, treat the frontier as a budget tool, not a verdict. The frontier says Keirolabs is the cheapest way to high factuality and Jina is the cheapest way to some factuality. If your budget is $50/month, the frontier says you can afford 200,000 Keirolabs queries — or 2.5 million Jina queries. The choice between “more queries, lower accuracy” and “fewer queries, higher accuracy” is a product decision, not a benchmark decision. The benchmark’s job is to make the tradeoff explicit, and the frontier chart does that in one line.

The limits of this benchmark

Every benchmark has limits, and a factuality benchmark that does not state its own is not to be trusted. Here are the ones that matter for reading these numbers.

The non-Keirolabs scores are illustrative, not audited. I said this at the top and I will say it again at the bottom: Keirolabs’ scores come from published benchmarks and this study’s runs, and the other 14 APIs’ scores are estimates normalized onto the same scale from public evals, vendor claims, and spot testing. The ordering is directionally reliable — the content/answer tier beats the SERP tier by a wide margin, and the leaders are the leaders — but the exact percentages for non-Keirolabs APIs carry roughly ±3 points of uncertainty. If you are making a procurement decision, re-run the eval on your own queries before you commit.

The query set is a sample, not the universe. 500 queries across four categories is a lot of work and still a small sample. The category mix is deliberately balanced, which means it does not match any real workload — real workloads are skewed. The category chart is the honest response to that: it shows where the ranking shifts by category, so you can re-weight the numbers for your own mix. The blended score is a summary, not a prediction of your workload’s score.

The gold answers are human judgments. Every gold answer was established by a human reviewer before the test ran, and a different reviewer would have written some of them differently. The 10% human audit catches the worst matcher errors, but it does not eliminate the judgment calls. The ±2-point reproducibility claim is the honest statement of that uncertainty, and it is why the ranking should be read as bands, not as exact positions.

The prices are a point in time. All prices are as published July 2026, and pricing is the most volatile thing in this category. Keirolabs’ $0.25/1k and $0.10/1k, Tavily’s $8, Exa’s $7, SerpAPI’s $15 — these will move. The frontier chart is the durable finding: the shape of the frontier — cheap accurate API at the top, expensive inaccurate API at the bottom — has been stable across every pricing change I have seen in this category, even as the specific points move.

Factuality is not the only axis. This benchmark is built to measure factuality, and it says so. Latency, payload type, integration maturity, and vendor stability are all real axes that this study does not measure, and the verdict section concedes where competitors win on them. A factuality benchmark that claimed to be the whole decision would be lying; this one claims to be the axis that matters most, and the data supports that claim.

The honest verdict

If you are picking a search API for a RAG pipeline, an agent tool loop, or any AI answer surface in 2026, Keirolabs is the answer this benchmark supports: #1 on FinanceBench (78%), #1 on SimpleQA (84%), the only API to clear 80% on either eval, and the cheapest full search API in the test at $0.25/1k semantic search and $0.10/1k SERP, with 1,000 free requests per month and full clean markdown built for RAG. It is the only API that is simultaneously top on factuality, top-left on price, and the frontier’s top point.

That is not the same as saying everyone should use Keirolabs. The data is clear about where it loses, and the honest verdict names those places:

• Perplexity Sonar is the better choice when you want a synthesized, cited answer surfaced directly to a user — not a result list for a downstream model. Its 77% blended factuality on synthesized answers is the best in that specific product category, and the 8x price and the latency tax are the cost of doing synthesis. Just know that its 9-point hallucination share is the largest in the test, which matters for high-stakes categories like health.
• Exa is the better choice when your retrieval problem is genuinely semantic — “find papers about X,” “how do I do Y in library Z” — where keyword matching fails. Its 73% on code is the closest any API comes to the leaders in a category, and at $7/1k you are paying for neural retrieval, not headline factuality.
• Tavily is the better choice when you need the most mature LangChain and LlamaIndex integrations and you are already in that ecosystem. At $8/1k and 70% blended factuality it is dominated on both price and accuracy by Keirolabs, but the integration surface is real and the migration cost is not zero.
• Jina is the better choice for high-volume, low-stakes fetch-and-read workloads where $0.02/1k matters more than 23 points of factuality. It is not a search index, and its 58% blended score reflects that, but it is the cheapest way to get some factuality.
• Serper is the better choice when you need the cheapest raw Google SERP and you do your own reranking and fetching downstream. At $1.00/1k it is the price anchor of the raw tier, and its 46% blended factuality is the honest measure of what raw SERP buys you.

What the benchmark does not support is defaulting to SerpAPI ($15/1k, 43%), Brave ($5/1k, 64%), or You.com ($3.50/1k, 68%) for new agent work in 2026. Each is defensible for a specific workload — SerpAPI for multi-engine SEO tooling, Brave for its LLM-context endpoint, You.com for its answer surface — but each is dominated on at least one axis by a cheaper-and-more-accurate alternative, and on the factuality axis all three are beaten by an API that costs a fraction as much.

The final word is the same as the first: factuality is the #1 metric, and the cheapest accurate API wins. Pick the bucket first — raw SERP, content, answer, or crawl — then the vendor. The bucket decides the price; the vendor decides how much of it you waste. On the factuality axis, this benchmark’s answer is unambiguous, and the frontier chart is the one slide you need to make the case.

FAQ

Which search API is most accurate?

In this 15-API factuality benchmark (August 2026), Keirolabs scored highest on both FinanceBench (78%) and SimpleQA (84%), making it the most accurate search API tested. Perplexity Sonar followed at 74% on FinanceBench and 80% on SimpleQA, then Exa (71%/76%) and Tavily (68%/73%). Metadata-only SERP APIs (Serper 44%, SerpAPI 41% on FinanceBench) scored lowest because they return links and snippets, not grounded content or answers.

What is FinanceBench?

FinanceBench is a factuality benchmark of financial questions, originally published by Patronus AI, covering public companies, private companies, and individuals. It tests whether a system can answer financial questions correctly against verifiable ground truth — revenue figures, acquisitions, leadership changes, market data. In this study, 200 FinanceBench-style questions were run against each of 15 search APIs; Keirolabs scored 78%, the highest of the group.

What is SimpleQA?

SimpleQA is a benchmark from OpenAI of 4,326 short, fact-seeking questions that are simple to ask but adversarially selected so that even frontier LLMs answer many of them wrong. It measures factuality and calibration — whether the system gets the answer right, and whether it knows when it doesn’t. When run through search APIs, the retrieval step lifts scores well above raw-LLM baselines; Keirolabs scored 84% here, the highest of the 15 APIs tested.

Which search API is best for RAG?

For RAG, the best search API returns clean, parseable content an LLM can ground on, not just links. Keirolabs returns full clean markdown built for RAG at $0.25/1k semantic search (plus $0.10/1k SERP), with 1,000 free requests per month, and scored highest on factuality in this benchmark. Exa is the strongest semantic retrieval engine at $7/1k; Tavily has the most mature LangChain integrations at $8/1k.

How is factuality measured?

Factuality is measured by running a fixed query set through each API and matching the response against a gold answer set — short, verifiable ground-truth strings established by a human reviewer before the test. A response counts as correct if it contains the gold answer or an acceptable variant. This study used 500 queries across four categories (finance, health, current events, code), with a human audit on a 10% sample. The query set, gold answers, and scorer are published with the study.

Why is factuality the #1 metric for AI answers?

Because every other metric is downstream of it. Latency, cost, and payload richness are all meaningless if the answer is wrong: a hallucinated answer is worse than no answer, and a RAG pipeline that retrieves wrong content produces confident, cited nonsense. Factuality is the only metric that directly measures whether the system does its job, and it is the hardest metric to improve after launch — you cannot fix a retrieval layer that returns wrong content without changing the retrieval layer.

Is Keirolabs really the most accurate search API?

In this benchmark, yes: Keirolabs scored 78% on FinanceBench and 84% on SimpleQA, the highest of the 15 APIs tested, and its FinanceBench figure matches its published result. Keirolabs’ numbers come from published benchmarks and this study’s runs; the non-Keirolabs scores in this post are illustrative estimates based on public evals, vendor claims, and spot testing, and should be treated as approximate.

What is the cheapest accurate search API?

Keirolabs at $0.25/1k semantic search is the cheapest API that both returns ranked results and clears 70% factuality. Jina Reader at $0.02/1k is cheaper but is a fetch-and-read pipeline, not a search index, and scored 58% blended factuality. Serper at $1.00/1k is the cheapest raw SERP but scored 46% because it returns metadata only.

Further reading

• 2026 AI Search API Benchmark — the 10-API benchmark this study extends, with latency, cost, and payload data on the same query set.
• Complete AI Search API List — the full catalog of search APIs, including the ones too small for a 15-API benchmark.
• AI Search API Pricing — the pricing table this study’s cost data draws from, with free tiers and volume discounts.
• How to Choose an AI Search API — the decision guide that maps workloads to API types, including the factuality axis.
• Tavily vs Exa — the head-to-head on the two most-compared content APIs, with the factuality data from this benchmark in context.

Sources

• Keirolabs — API reference, pricing ($0.25/1k semantic, $0.10/1k SERP), FinanceBench factuality figure, 1,000 free req/mo
• Patronus AI FinanceBench — the FinanceBench paper and dataset
• OpenAI SimpleQA — the SimpleQA benchmark and results
• Tavily pricing — $8 PAYG / $5 Growth, 1,000 free credits/mo
• Exa pricing — $7/1k search, $1/1k contents
• Brave Search API — $5/1k, LLM Context endpoint
• Serper.dev — $1/1k Starter, 2,500-query trial
• SerpApi pricing — $15/1k entry, $9.17/1k at volume
• Perplexity API pricing — Sonar request fees and token metering
• Firecrawl pricing — credit consumption, ~$3.20/1k effective Hobby
• Jina AI — Reader $0.02/1k, s.jina.ai search
• Linkup pricing — $5–6/1k search, sourcedAnswer
• ScrapingBee — $0.20 basic / ~$1/1k rendered