Speaker
Description
As Automatic Text Recognition (ATR) pipelines rapidly evolve, the digital humanities are confronted with a highly fragmented landscape of transcription technologies. Researchers today must navigate between specialised, fine-tuned line-level engines, such as Kraken, TrOCR, and the emerging zero-shot capabilities of generalised Vision-Language Models (VLMs). All approaches have different advantages and disadvantages which can be leveraged. However, as generating transcriptions across multiple engines becomes easier, evaluating their true quality on complex historical documents remains a significant hurdle. Traditional string-matching metrics like Character Error Rate (CER) are overly rigid, often penalising minor orthographic variations while failing to assess whether the core semantic and structural meaning of the text was successfully conveyed.
To test the limits of modern evaluation, this talk presents the results of a deliberately "unsound" experiment built on such an inference pipeline. We pitted highly specialised, fine-tuned ATR engines against unconstrained, zero-shot VLMs and deployed a Large Language Model (LLM) as an automated judge to qualitatively assess the resulting transcriptions.
Methodologically, this setup is inherently flawed: it compares apples to oranges in terms of model architecture, while relying on an LLM judge that is intrinsically biased toward modernising historical spellings and hallucinating missing context. Yet, despite these structural flaws, the experiment yielded critical insights. We will discuss the practical realities of deploying multi-engine ATR inference, expose the distinct error patterns unique to VLMs versus „traditional“ models, and demonstrate what the failures of an „LLM as a judge" reveal about the future of benchmarking historical text recognition.