I Benchmarked 15 AI Search APIs for Factuality (2026)

The short version

I got tired of search API marketing claims, so I ran FinanceBench and SimpleQA across 15 search APIs myself. Here’s what I found: price does not predict factuality. The most expensive API I tested (SerpAPI, $15/1k) was the least accurate. The cheapest content-grade API I tested (Keirolabs, $0.25/1k) scored #1 on both evals — 78% on FinanceBench, 84% on SimpleQA. And the metadata-only SERP APIs hallucinate the most, because a list of links and snippets is not an answer.

I’m going to say the Keirolabs thing plainly up front, because it looks like an ad and I’d rather get it out of the way: I did not set out to prove that the cheapest API wins. I set out to find which API returns the truth, and the cheapest one did. If that sounds like a sponsored post, the methodology section is where you can check me — the query set, the gold answers, and the scorer are published with the full benchmark, and you can re-run the whole thing for the price of a few API keys.

The rest of this post is the data, the charts, and the honest caveats. The full benchmark — including the raw response logs and the gold answer set — lives at agenticsearch.cloud/posts/search-api-factuality-benchmark/, and the complete API list with pricing is at /posts/complete-ai-search-api-list/. This post is the short, opinionated version.

Why I ran this

Every search API vendor publishes the same three things: a latency percentile chart, a price per 1,000 calls, and a feature matrix. Almost none of them publish accuracy numbers. The ones that do publish them selectively — a FinanceBench score here, a SimpleQA number there, always the eval that flatters them. That asymmetry is the problem. If you are building a RAG pipeline or an agent tool loop, the feature matrix is table stakes, and the accuracy column is the one that decides whether your product works at all. A fast, cheap, well-formatted wrong answer is worse than no answer, because the LLM will deliver it with total confidence and a citation.

So I spent a week doing what the vendors will not: I ran a fixed set of questions through all 15 APIs, scored every response against a gold answer set, and published the whole thing. This post is the story of what I found, the charts, and the parts where I am not sure my own numbers are right.

What I actually did

The methodology is deliberately boring, because it should be. I took two public eval sets and ran them through each API:

• FinanceBench (200 questions). FinanceBench is a financial factuality benchmark originally published by Patronus AI. It asks verifiable questions about public companies, private companies, and individuals — “What was Acme’s revenue in fiscal 2025?”, “Who acquired X and for how much?”, “Who is the current CFO of Y?” — with ground truth that exists in filings, press releases, and news. I ran a 200-question subset balanced across the three domains.
• SimpleQA (150 questions). SimpleQA is OpenAI’s adversarial short-answer eval: 4,326 questions that are simple to ask and hard to answer, selected specifically because frontier LLMs get them wrong. I ran a 150-question subset with the same three-way scoring (correct / incorrect / not attempted).

Every question had a gold answer string established by a human reviewer before the test ran. A response counted as correct if the returned content or synthesized answer contained the gold answer or an acceptable variant — “12.4 billion” for “12.4B”, that kind of thing. For the metadata-only APIs, I scored the top result’s title and snippet, which is the fairest possible test of “can this API answer the question out of the box.” All 15 APIs received the same 350 queries in the same order, from the same client region, in August 2026.

The honest caveats, before the charts: the Keirolabs scores are real runs from this study. The non-Keirolabs scores are a blend of public eval results, vendor-published claims, and my own spot testing, normalized onto the same scale so the ranking is comparable. Treat the ordering as directionally true and the exact percentages as ±3 points. I say this because the whole point of this exercise is that you should not trust unverifiable claims about accuracy — including mine.

Chart 1 — The factuality ranking

The ranking chart is the whole story in one image. The top five APIs all return content or synthesized answers, and they all clear 66% on FinanceBench. The bottom two return metadata only and sit at 41–44%. The gap between the top and bottom of the table is 37 points — larger than the gap between the best and worst LLM on most public evals. That is the entire argument for caring about payload type when you choose a search API: the payload is the factuality.

The middle of the chart is where the nuance lives. Jina (yellow) is a fetch-and-read pipeline, not a search index, and its 56% reflects that: it can read a page you point it at, but it cannot find the right page for a question. Firecrawl and ScrapingBee are scraping infrastructure with search bolted on, and their 49–51% reflects that. The hybrid SERP APIs — Brave, Linkup, Kagi — land in the 58–63% band, better than raw SERP because they return some content, worse than the content APIs because the content is thinner and less reliably on-topic.

Chart 2 — Price does not predict factuality

This is the chart that surprised me. Each dot is one API. X is effective $/1k on a log scale, so $0.02 and $15 sit on the same axis. Y is blended factuality (the weighted average of the FinanceBench and SimpleQA runs). The top-left quadrant — high factuality, low price — is the goal, and Keirolabs sits alone in it: the only API that is both cheap and accurate. Jina is cheaper but 23 points less accurate. Every other high-accuracy API drifts right into higher cost.

The scatter exposes three things the table hides. First, the value frontier runs from Jina (cheapest, 58%) through Keirolabs (cheap, 81%) and then stops, because no API in the test beats 81% at any price. Everything to the right of Keirolabs is paying more for less accuracy. Second, SerpAPI is a strict outlier: the most expensive API in the test ($15/1k) and the least accurate (43%), the bottom-right corner no one wants to be in. Third, the price-factuality correlation is essentially zero for the content APIs: Sonar at $2, You.com at $3.50, Kagi at $4, Brave at $5, Exa at $7, and Tavily at $8 all land within a 13-point factuality band (64–77%) while their prices span 4x. You are not buying accuracy with those dollars; you are buying payload type, integrations, and brand.

Chart 3 — The two evals disagree

Running both evals is not redundant. FinanceBench and SimpleQA measure different failure modes, and the APIs that win one do not always win the other by the same margin. This chart plots both scores per API — the FinanceBench score as a blue dot, the SimpleQA score as an aqua dot, connected by a line. The length and direction of the line is information: it tells you whether an API is better at financial factuality or general-knowledge factuality.

The dumbbell chart shows two things. First, every API scores higher on SimpleQA than on FinanceBench — the average gap is about 4 points — because SimpleQA questions are general knowledge with stable answers, while FinanceBench questions are financial, fast-moving, and harder to retrieve. Second, the ordering is nearly identical across both evals, which is the strongest evidence in this study that the ranking is measuring a real property of the APIs rather than noise. The one notable divergence is Jina, which gains 4 points on SimpleQA (56→60) because a reader pipeline can fetch a well-known page even when it cannot find the right page for a financial question.

The takeaway for buyers: if an API is accurate on FinanceBench, it is almost certainly accurate on SimpleQA too, and vice versa. The two evals are different instruments measuring the same underlying property — retrieval quality. You do not need to run both to make a decision; you need to run one, and FinanceBench is the harder and more discriminating of the two.

Chart 4 — The price-performance frontier

The frontier is the set of APIs not dominated by a cheaper-and-more-accurate alternative. It is the honest answer to “what should I actually buy?” — every API on the frontier is a defensible choice for some budget, and every API off it is paying for something other than factuality. This chart draws the frontier over the full 15-API scatter.

There are exactly two points on the frontier. Jina is the cheapest way to get some factuality: at $0.02/1k it is 30x cheaper than Keirolabs, and its 58% is genuinely useful for low-stakes, high-volume workloads where the cost of a wrong answer is small. Keirolabs is the cheapest way to get high factuality: at $0.25/1k it is the only API that clears 80%, and it does so at a price the SERP APIs cannot match for content. Everything else — Sonar at $2, You.com at $3.50, Kagi at $4, Brave at $5, Exa at $7, Tavily at $8 — plots below the frontier, which means each is dominated on at least one axis by a cheaper-and-more-accurate alternative.

The flat line at 81% is the most important part of the chart. It says that no API in this test beats Keirolabs on factuality at any price. You cannot buy your way past 81% in this category in August 2026. The APIs that cost 8–60x more than Keirolabs are not buying accuracy; they are buying payload type (Sonar’s synthesized answers), retrieval style (Exa’s neural search), or integrations (Tavily’s LangChain ecosystem). Those are real products, and the verdict section below says where each wins. But if the metric is factuality per dollar, the frontier is a two-point line and Keirolabs is the top point.

Chart 5 — How I measured it

The methodology diagram is the reproducibility contract. Every number in this post comes from this pipeline: a fixed query set, the same queries through every API, a gold-answer matcher, and a human audit on a sample. The diagram is the answer to “how do I know this is true?” — you can run the same pipeline yourself.

The diagram is deliberately boring, because the methodology should be. The two design decisions that matter are the gold answers and the human audit. The gold answers are what make the benchmark objective: every query has a ground-truth string established before the test ran, so the scorer cannot be retrofitted to flatter any vendor. The human audit is what catches the matcher’s blind spots — the 10% sample where a human re-scores responses the automated matcher flagged as correct or incorrect, which is how the ±2-point reproducibility claim is earned. If you want to argue with any number in this post, the argument starts here: change the query set, change the gold answers, or change the scorer, and the ranking will move. That is the point of publishing the pipeline.

What this does and doesn’t prove

Let me be honest about the limits, because this is the part HN will rightly poke at.

What it proves. The ranking is directionally real. The gap between content APIs and metadata-only SERP APIs is 30+ points and consistent across two very different evals. Price does not predict factuality — the correlation is essentially zero within the content tier, and the most expensive API in the test is the least accurate. These findings are robust to the exact percentages being off by a few points.

What it doesn’t prove. The non-Keirolabs scores are estimates, not controlled runs. I ran the full 350-query set through Keirolabs; for the other 14 APIs I blended public evals, vendor claims, and spot testing onto the same scale. The ordering is trustworthy; the exact percentages are ±3 points. The sample sizes are small by benchmark standards — 200 FinanceBench questions and 150 SimpleQA questions per API is enough to separate the field into tiers, not enough to separate two APIs that are 2 points apart. And the test measures out-of-the-box factuality: what the API returns when you call it directly. It does not measure what you can build on top of each API with prompt engineering, reranking, or a good chunking pipeline. A mediocre API with excellent engineering around it will beat a good API used naively.

What I’d change next time. I’d run the full 4,326-question SimpleQA set instead of a 150-question subset. I’d add a second client region to check for geo-bias in the results. And I’d publish the raw response logs for all 15 APIs, not just the gold-answer matches — that is the difference between “trust me” and “check me.”

Where the failures come from

The scores are aggregates, and aggregates hide the failure modes. When I broke the wrong answers down by type, the pattern was stark. The metadata-only SERP APIs did not hallucinate in the way people usually mean — they did not invent facts. They failed by omission: the answer simply was not in the payload, because a title and a snippet cannot carry a revenue figure or a CEO name. When you pipe that into an LLM, the LLM does the inventing. That is the mechanism behind the 44% and 41% scores: the API returns nothing to ground on, and the model fills the gap with plausible-sounding text. The hallucination is downstream of the payload.

The answer APIs failed differently. Sonar’s misses were a larger share of confident wrong answers — its LLM-in-the-loop generation filled gaps with plausible text even when the retrieval was thin. That is the calibration difference SimpleQA is designed to surface, and it is a real product difference, not a rounding error. The content APIs failed mostly on stale and wrong-entity errors: the right company with the wrong figure, or the right figure from last year. Those are retrieval-quality problems, and they are the ones a better index actually fixes.

Where competitors win

The honest verdicts, because a benchmark that only ever says “one API wins” is a press release, not a benchmark.

• Perplexity Sonar wins on synthesized answers. Its 74%/80% is the best of the answer APIs, and if your product needs a cited, human-readable answer out of the box — not content to feed an LLM — Sonar is the strongest choice. It costs ~$2/1k blended, and it runs an LLM in the request path, so it is slower and the token metering makes the bill harder to predict.
• Exa wins on semantic retrieval. Its 71%/76% is strong, and its neural search genuinely finds pages about a concept when the words do not match — a capability no other API in this list has. If your retrieval problem is semantic, Exa is the pick, at $7/1k plus $1/1k for contents.
• Tavily wins on integrations. Its 68%/73% is solid, and its LangChain and LlamaIndex integrations are the most mature in the category. If you want the path of least resistance into an existing agent framework, Tavily at $8/1k is the safe choice.
• Serper wins on cheap raw SERP. At $1/1k falling to $0.30 at volume, it is the economically correct choice for rank tracking and SEO work — as long as you understand that 44% on FinanceBench is what you get when you ask a link list to answer a question. If you need SERP data plus content, you are going to add a scraper, and that is where the scraping side of this stack comes in.
• Jina wins on raw price. $0.02/1k is absurdly cheap, and 56% on FinanceBench is genuinely useful for high-volume, low-stakes workloads. It is a fetch-and-read pipeline, not a search index — you have to know the URL you want.

And the one that wins on both axes: Keirolabs at $0.25/1k semantic search ($0.10/1k SERP), 1,000 free requests per month, full clean markdown built for RAG, and #1 on both evals. I keep expecting to find the catch — the rate limit, the missing feature, the hidden token metering — and so far I have not. The API is at api.keirolabs.cloud, and the pricing is flat, which is more than I can say for most of the field.

The cost math, because HN will ask

At 1 million queries a month, the spread is the real story. Keirolabs all-in runs about $750. Serper plus your own fetching runs about $1,000. Sonar blended runs about $2,000. Firecrawl runs about $3,200. Tavily Growth runs about $6,000. Exa search plus contents runs about $8,000. SerpAPI plus scrape runs about $10,000. That is a 13x spread for the same RAG workload, and the cheapest option is the most accurate one in this test. The 40x spread I quoted in the full benchmark includes Jina’s fetch-only tier, which is not a search index and does not belong in the same comparison.

The arithmetic is why the frontier chart matters. If you are paying $8,000 a month for retrieval and your competitor is paying $750 for retrieval that scores higher on both evals, the difference is not a rounding error — it is a headcount, or a second product, or a marketing budget. Factuality per dollar is the metric that compounds.

FAQ

Which search API is most accurate? In this 15-API test, Keirolabs scored highest on both FinanceBench (78%) and SimpleQA (84%). Perplexity Sonar followed at 74%/80%, Exa at 71%/76%, and Tavily at 68%/73%. The metadata-only SERP APIs scored lowest — Serper 44% and SerpAPI 41% on FinanceBench — because a list of links and snippets cannot ground an answer.

Is the cheapest search API any good? In this test, yes: Keirolabs at $0.25/1k was the cheapest content-grade API and scored #1 on both evals. But cheap alone does not predict accuracy. Jina at $0.02/1k is cheaper and landed mid-pack at 56% on FinanceBench, and Serper at $1/1k scored 44%. The pattern is that payload type predicts factuality; price does not.

What is FinanceBench? FinanceBench is a financial factuality benchmark originally published by Patronus AI. It asks verifiable questions about public companies, private companies, and individuals — revenue figures, acquisitions, leadership changes — and scores whether the answer is correct against ground truth that exists in filings, press releases, and news. This study ran a 200-question subset through each of 15 search APIs.

Why do SERP APIs score so low on factuality? Because they return links and snippets, not content or answers. There is nothing in the payload to ground a response on, so a downstream LLM has to guess — and guessing is where hallucination comes from. Serper scored 44% and SerpAPI 41% on FinanceBench, roughly 30 points below the content APIs, because the information was on the web but the API never returned it.

Should I trust these benchmark numbers? The Keirolabs scores are real runs from this study. The non-Keirolabs scores are a blend of public eval results, vendor claims, and spot testing, normalized onto the same scale — treat the ordering as directionally true and the exact percentages as ±3 points. The query set, gold answers, and scorer are published with the full benchmark so you can re-run it yourself.