Who are citizen archivists?
Many may be surprised to learn that countless dedicated citizen archivists around the globe volunteer their time and expertise to transcribe historical documents on behalf of national archives. Without these individuals, access to these records and the information they contain would be even more limited. Document digitisation, data keying services, and document records management are key elements of Enterprise Information Management (EIM) efforts for public organisations. Their altruism supports public organisations in this endeavour, helping to provide a valuable service to the public.
Estimating the number of historical handwritten documents is difficult due to their vastness, dispersal, and varying definitions. Despite these challenges, historians and archivists estimate the quantity to be enormous, potentially in the billions or even trillions worldwide. Considering the first typewriters weren’t commercially available until 1874, the scale of the task is enormous. While format and readability will vary widely, accurate transcription of historical documents is crucial. It safeguards valuable information, preserving our past for future generations. Transcripts must faithfully mirror the originals, including punctuation and spelling.
A challenging task
Transcribing these records is particularly challenging because handwriting styles have changed over time. Most of us now rarely write in cursive, if at all. Historical documents often utilise styles like secretary script (where “c” can resemble “r”), round script (the basis of modern handwriting), and chancery script (used for business and legal documents). Additionally, many words are no longer in common use, some are spelled differently, and handwriting often displays a forward slope as people have begun to write faster, making letters less distinct.
Automation is key
Automating transcription seems like the obvious solution given the sheer volume of documents. Early attempts used Optical Character Recognition (OCR) and Intelligent Character Recognition (ICR) for initial transcription, followed by manual corrections from data entry clerks. These solutions were costly, error-prone, and had low “straight-through processing rates” (STPs), meaning few documents were transcribed with no intervention.
Cloud computing and AI
Cloud computing and generative AI have revolutionised document “reading,” categorised as Intelligent Document Processing (IDP). IDP uses AI and machine learning to automate the processing of various document types and aims to understand their content like a human. Examples include Google’s Document AI/Google Cloud Vision and Amazon Textract.
XBP Europe has several clients requiring historical document transcription, with an estimated 200 million handwritten documents dating back to the early 1800s. To address this challenge, XBP Europe developed an Intelligent Document Management solution with a core transcription engine that combines proprietary and third-party technologies, including Natural Language Processing (NLP) and cloud-based generative AI. This engine boasts industry-leading STP rates and continuously improves with use, even processing seemingly illegible handwriting.
Humans still have a role to play
While fully automated, human-free processing might be achievable in controlled academic environments, limitations to current AI necessitate human involvement to manage exceptions. A crucial part of the solution is ensuring “human-in-the-loop” processes present easily actionable information for quick and accurate exception handling.While AI development is rapid, and may eventually surpass experienced archivists, the sheer volume, complexity, and diversity of documents guarantee a continued role for citizen archivists for the foreseeable future.
