TC94 — Vectara — R6 Managed RAG v1 (Grounded Generation, Group A shared stimulus) Query: How quickly does Fenwake acknowledge a Priority 1 fault reported outside business hours? Endpoint: POST /v2/corpora/r6-managed-rag-ss-v1/query with generation:{} (Vectara defaults, default prompt template, not overridden) ==================================================================================================== GENERATED ANSWER (verbatim) ==================================================================================================== A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within 90 minutes of the ticket being raised [1]. factual_consistency_score (Vectara's own HHEM-based score): 1.0 ==================================================================================================== CLAIM-BY-CLAIM CHECK against the retrieved passages ==================================================================================================== Claim: "acknowledged by a named duty engineer within 90 minutes of the ticket being raised" -> SUPPORTED verbatim by rank-1 passage (A01, page 4): "A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within 90 minutes of the ticket being raised." -> Nothing added that is not in the passages. -> Nothing dropped that the question asked for ("how quickly" -> answered: 90 minutes). -> The rank-2 passage (P1/P2/P3 acknowledgement-target table: 30 min / 4 hrs / next working day) is in the retrieved context but NOT conflated into the answer. ==================================================================================================== TOP 5 RETRIEVED PASSAGES (default search, default generation context) ==================================================================================================== --- rank 1 | score=0.998649 | doc=A01_field_service_policy_and_charges_uk.pdf | page=4 --- 6.1 Acknowledgement out of hours A Priority 1 fault reported outside business hours is acknowledged by a named duty engineer within 90 minutes of the ticket being raised. --- rank 2 | score=0.917303 | doc=A01_field_service_policy_and_charges_uk.pdf | page=2 --- Priority Acknowledgement target Measured from Priority 1 30 minutes Ticket raised Priority 2 4 working hours Ticket raised Priority 3 The next working day Ticket raised --- rank 3 | score=0.612426 | doc=A01_field_service_policy_and_charges_uk.pdf | page=6 --- Route When Detail Service desk Business hours service@fenwake-instruments.example Service desk telephone Business hours +44 117 496 0188 Out-of-hours pager All other times +44 117 496 0199 --- rank 4 | score=0.535918 | doc=A09_warranty_statement_service_pack.pdf | page=1 --- Where a covered failure occurs, Fenwake repairs or exchanges the instrument at its own option and bears the cost of carriage in both directions. --- rank 5 | score=0.535918 | doc=A08_warranty_terms_sales_pack.pdf | page=1 --- Where a covered failure occurs, Fenwake repairs or exchanges the instrument at its own option and bears the cost of carriage in both directions. ==================================================================================================== Remaining results (rank 6-10, score-only) ==================================================================================================== rank 6: score=0.467210 doc=A01_field_service_policy_and_charges_uk.pdf page=6 rank 7: score=0.248534 doc=A09_warranty_statement_service_pack.pdf page=1 rank 8: score=0.248534 doc=A08_warranty_terms_sales_pack.pdf page=1 rank 9: score=0.144631 doc=A09_warranty_statement_service_pack.pdf page=2 rank 10: score=0.144631 doc=A08_warranty_terms_sales_pack.pdf page=2 ==================================================================================================== APPENDIX — earlier run with CUSTOMIZED generation params (max_used_search_results=5, fabricated citations.url_pattern/text_pattern) — kept for comparison, NOT the primary evidence ==================================================================================================== Same verbatim answer text. factual_consistency_score dropped to 0.2109375 (vs 1.0 on Vectara defaults) — the fabricated citation config appears to have degraded the reported consistency score even though the answer text itself was identical and fully grounded. This is a genuine product observation: the HHEM score is sensitive to generation-config parameters unrelated to the answer text itself, not something to read as a groundedness signal on its own.