# Tavily `include_answer` mode — supplementary 11-query test (2026-08-25)

**Scope note: this is an 11-query mini-panel, not a full 52-query run** — same panel as Linkup/You.com's
supplementary tests: Q01, Q15, Q23, Q32, Q37, Q43 (one per answerable block) + Q48–Q52 (all 5
unanswerable). Endpoint: `POST https://api.tavily.com/search` with `include_answer: true`,
`search_depth: "advanced"`. All 11 calls returned `http_status: 200`, 0 errors. Raw responses:
`_human-tasks/review-artifacts/_answer-mode-tests/tavily/Q*.json`.

## Structural finding, applies to all 11: Tavily's answer mode has no citation mechanism at all

The response schema has no `citations` field and no inline source markers in the `answer` text — only
a flat `results` array of the 10 raw search hits, with no linkage stating which (if any) result the
answer text actually drew from. **This means Tavily's answer mode cannot be citation-checked even in
principle** — there is no claim-to-source mapping to open and verify. Confirmed directly:
`_human-tasks/review-artifacts/_answer-mode-tests/tavily/Q48.json` has an `answer` field asserting a
specific date, and a `results` array of 10 SOC2-explainer pages, **none of which mention Serper or
that date anywhere in their titles** — the answer cannot be traced to any of its own retrieved sources.

## Unanswerable queries — 2/5 abstained, 0/5 cleanly; 3/5 outright fabricated a confident wrong answer

This is the headline finding of this test. Quoted verbatim from the raw responses:

| Query | Response | Verdict |
|---|---|---|
| Q48 (Serper SOC2 date) | **"The SOC 2 Type II audit completion date for Serper is February 27, 2026. This audit assesses operational effectiveness over a period of time. The report is valid for 12 months."** | ❌ **Fabricated.** A specific date stated as fact, for a certification that per this benchmark's own ground truth has no public disclosure. None of the 10 returned results mention Serper. |
| Q49 (Jina ARR 2026) | **"Jina AI's 2026 annual recurring revenue is estimated at $17.5 million. The company was acquired in October 2025."** | ❌ **Fabricated.** Presents a 2025-dated third-party estimate as a settled 2026 ARR figure, with no hedge, plus an unrelated acquisition claim. |
| Q50 (Exa Enterprise price) | "Exa's Enterprise plan offers custom pricing... with **$2.50 per 1,000 keyword search requests**. Exact Enterprise pricing is available upon request." | ⚠️ **Partial fabrication.** Leads with a real number ($2.50/1k) that is Exa's *self-serve keyword-search* rate, misattributed to the Enterprise tier, before correctly noting Enterprise is custom-quoted. |
| Q51 (Olostep index size) | **"The total index size for Olostep is 47 pages. The maximum set pages limit was 50."** | ❌ **Fabricated, and implausible on its face** — appears to have confused a per-request pagination limit with a total index size, then stated the resulting "47 pages" as Olostep's total index. |
| Q52 (Linkup's own index size) | "Linkup indexes around three billion web pages in its index. The exact number of pages is not specified." | ⚠️ **Partial fabrication, best-hedged of the five** — states an invented approximation ("three billion," likely misapplied from `worldwidewebsize.com`'s general-web estimate) but caveats that the exact figure isn't given. |

**3 of 5 unanswerable probes produced a confidently-stated, specific, wrong number with zero hedge —
worse than Valyu's citation-honesty record in the main benchmark (1.0% false rate, the previous worst
in the set), on a much smaller sample. Combined with the complete absence of a citation mechanism, an
agent consuming Tavily's answer mode has no way to catch these fabrications from the response alone.**

## Answerable queries — 4/6 solidly correct, 2 partial

| Query | Verdict |
|---|---|
| Q01 (Exa price) | ✅ correct |
| Q15 (ruff B007) | ✅ correct — names the rule code and the fix |
| Q23 (Apollo pricing) | ✅ correct — resolves to Apollo.io, tier prices match Linkup's independent test |
| Q32 (Gujarat+Maharashtra pop.) | ❌ wrong — "~180 million" using recent estimates (74.34M + 116M), not the 2011 Census figure (172.8M); same base-year-mixing failure as Linkup and Valyu on this query |
| Q37 (GDPR Art 17(3)) | ⚠️ partial — names only 3 of 5 exceptions (legal obligations, public interest tasks, research; omits freedom-of-expression and, arguably, distinctly names public health) |
| Q43 (Python 3.11.16) | ✅ correct — "released on August 12, 2026... security fixes only until October 2027," matches ground truth |

## Disposition

**Citation accuracy (answer APIs): scorable now — and the finding is structural, not a data gap: this
mode provides no citation mechanism to check, which is itself the answer to the criterion.**

**Answer quality (answer APIs): scorable now — 4/6 correct on the answerable mini-panel, but the
no-answer-behaviour result (2/5 abstained, 0/5 cleanly, 3/5 confidently fabricated) is the standout,
concerning result of this entire supplementary test and should be reported as a named finding.**