Speaker
Description
This talk presents a scalable pipeline for converting digitised Arabic books from PDFs into structured mARkdown for integration into the OpenITI corpus. The workflow begins by preparing the PDFs and processing their opening pages to extract the bibliographic metadata required to identify works and editions. The documents are then transcribed in parallel by several complementary recognition systems that use different approaches to document OCR and vision-language processing. Each system’s output is converted into a common page-based representation that records its text, lines, structural labels, coordinates and recognition confidence when available. The pipeline aligns the resulting tokens and compares competing readings using weighted agreement across the systems. The selected readings are combined with the available layout and document-structure information to produce a page-aware consensus containing headings, paragraphs, line breaks, footnotes, and the division of poetry into verses and hemistichs. Although currently optimised for printed books, the pipeline also provides an experimental workflow for manuscripts, which require more extensive manual correction.
The resulting consensus and extracted metadata are presented in an interactive review environment in which users can compare alternative readings and correct the bibliographic information, transcription and layout. Reviewed documents are exported both as corpus-ready OpenITI mARkdown and as ground-truth data for evaluation and for fine-tuning locally trainable OCR models.
The pipeline is designed to reduce the time required for OCR and scholarly revision while keeping processing costs limited. Its modular architecture allows individual recognition and processing components to be enabled, disabled or replaced. The talk will describe the workflow from PDF ingestion to final mARkdown export and compare the textual and structural accuracy of the individual recognition approaches with that of the resulting consensus, alongside measurements of processing time and cost. Finally, it will consider how complementary recognition, structured consensus and reusable human correction can support the large-scale, quality-controlled expansion of OpenITI.