---
title: "What we measured, and under what conditions"
description: "Reading times and costs, folio accuracy, search quality, invented citations and latency, as they came out of the benchmark of 6 October 2026, and what those figures do not say."
url: https://scholaris.joseluissaorin.com/en/knowledge/performance
markdown: https://scholaris.joseluissaorin.com/en/knowledge/performance.md
lang: en
alternate_es: https://scholaris.joseluissaorin.com/saber/rendimiento.md
updated: 2026-10-06
author: José Luis Saorín Ferrer (https://joseluissaorin.com)
---

# What we measured, and under what conditions

> Reading times and costs, folio accuracy, search quality, invented citations and latency, as they came out of the benchmark of 6 October 2026, and what those figures do not say.

## Conditions

Everything was measured on **6 October 2026**, from a Mac in Tenerife over a home connection, against the cloud APIs (Gemini 3.5 Flash-Lite and 3.8 Flash, Gemini Transcribe, Gemini Embedding 2 at 1536 dimensions, Jev, Crossref and OpenAlex) with no local GPU. **It was not measured on Cloudflare**, which should be more stable; nor was the latency of "ask". Dollar figures are provider costs, not prices.

## Reading a document

| Document | Before (first Scholaris) | Now | Cost |
| --- | --- | --- | --- |
| Video interview, 54 min | 3 h 39 min | 45 s (30 s without speaker attribution) | $0.41 |
| Video interview (Cortázar), 2 h 2 min | the 83-min ones took 5 h 43 min | 70 s | $0.94 |
| *The Discarded Image*, 245 pp., with old OCR | not recorded | median 85 s over 16 runs (37 to 175 s) | $0.64 |
| *El casamiento en la muerte*, 43 scanned pp., 17th century | not recorded | 27 s | $0.20 |
| *Attention Is All You Need*, 15 digital pp. | not recorded | 12 s | $0.04 |
| *El perseguidor*, 37 digital pp. | not recorded | 15 s | $0.04 |

Times are end to end, conversion included. The first pages are searchable earlier: in under a tenth of a second for a digital PDF, in about 3 to 6 seconds for a scan and in about 3 to 6 seconds for a video. The slow runs of the 245-page book came from the API or the network, not the code. Very short media (one minute) now take a little longer to be ready than before: from 5 to 6 seconds it went to between 8 and 10.

**Economy mode** cut the cost of the *Casamiento* from $0.206 to $0.111 (46 % less), in exchange for taking 6.9 minutes instead of 19.7 seconds. Transcription costs about $0.005 a minute.

## Reading well

- **Character error rate (CER)** on seventeenth-century drama: 0.007 with Gemini 3.8 Flash, against 0.073 for the first Scholaris's reader. Measured on a single hand-transcribed page.
- ***The Discarded Image***: the first Scholaris left 195 pages empty (74,000 characters in all); now there are 358,000 characters and 7 empty pages.
- **Printed folio:** 64 of 64 exact on the pages checked by eye (45 of 45 with the number visible). In *The Discarded Image*, 229 folios read agree with the PDF's labels on 229 of 232 pages.
- **Speakers:** attribution right in 20 of 21 turns and in 23 of 23 in two interviews (25 random turns each, annotated by reading the text).
- **Records:** 55 of 55 checks right across 9 documents after checking against external sources (30 of 55 before).

## Searching well

Over 183 queries and 8,879 relevance judgments. The judgments are "silver": two models and an arbiter made them, not people; they agreed 82 % exactly and 99 % within one point, and of 30 checked by hand, the author agreed with 29.

| System | nDCG@10 | Recall@20 | MRR |
| --- | --- | --- | --- |
| First Scholaris | 0.695 | 0.618 | 0.925 |
| Lexical only | 0.564 | | |
| Visual only | 0.199 | | |
| Semantic only | 0.804 | 0.860 | 0.931 |
| Hybrid, not reranked | 0.818 | 0.884 | 0.942 |
| **Current** (hybrid, reranked) | **0.889** | **0.904** | **0.979** |

By query type (nDCG@10 of the current system): cross-language 0.897; literal 0.880; audio and video 0.882; old spelling 0.814; visual 0.950 (only 4 queries).

**Old spelling:** on Lope de Vega's *El casamiento en la muerte*, with 30 queries, average recall per query went from 32 % to 100 % with the modernised-spelling layer; on modern documents, 250 of 250 searches returned exactly the same.

**Latency:** median of about 650 ms from home, no caches (about 360 ms for the query vector and about 300 ms for the judge); without reranking, about 350 ms with nDCG 0.818. In the measured runs, median and 95th percentile were 687 and 856 ms.

## Citing well

Over 41 claims (30 supported and 11 not): **95.1 % precision, 96.7 % recall, zero invented citations**, zero citations on unsupported claims and the verification verdict right 100 % of the time. Each claim took 998 ms and cost $0.0054.

## What these figures do not say

- They come from a home connection, not Cloudflare, and do not include the latency of "ask".
- 1,594 relevance judgments with an arbiter or a disagreement are still awaiting human review.
- The CER was computed on a single page.
- The benchmark is small and was made by the person who makes Scholaris. If you want to repeat it with your documents, write to us.