Home
Production
Descriptions of OCR processing, TEI transformation, and metadata creation
Organization
How to keep hundreds of thousands of files organized.
Repositories
A guide to the different repositories used to store ocr-project data.
GitHub
Used for data files.
outputs Repository
Explains the content of the outputs repository.

Production
Descriptions of OCR processing, TEI transformation, and metadata creation
- Organization
  How to keep hundreds of thousands of files organized.
  - Edition-Section System
    File organization depends on two basic folder types
  - Folder names
    As the OCR workflow passes through its various stages, production moves into specific folders for each stage. Their names and contents are given below:
  - Repositories
    A guide to the different repositories used to store ocr-project data.
    - Google Drive
      Used for image files.
    - GitHub
      Used for data files.
      - outputs Repository
        Explains the content of the outputs repository.
        autoindex folder
        Collects all materials needed for indexing entry files.
        code Folder
        The central repository for program code and transformation scripts.
        digital-editions Folder
        Storage area for editions generated from the master files.
        entry Folder
        Contains the TEI-encoded data after the page files are converted to entry files.
        master Folder
        Contains the master files for creating digital editions.
        metadata Folder
        A collection of files containing metadata for each entry page.
        page Folder
        Pages are individual printed pages in the Encyclopedia.
        records Folder
        A collection of spreadsheets and other documents recording details of the production and analytical work.
    - archive Repository
      Long-term storage of image files
  - Setting Up the Repositories
    Create local copies of the remote repositories
- Page Files
  Explains the procedures we use to get the best quality OCR of each page.
- Entry Files
  Procedures for converting single pages into Encyclopedia entries.
- Master Files
  After creating entry files, we run an automated metadata generation process to add index terms to each entry.

outputs Repository

Explains the content of the outputs repository.

The outputs repository resides on GitHub. It contains the OCR output files and their derivatives, like master file anddigital editions. This repository has the following folders:


analytics	Files used in analyzing all text in the Nineteenth-Century Knowledge Project.
autoindex	Reserved for data related to generating metadata.
code	All Python, XSLT, and related scripts used in processing data in the Knowledge Project.
digital-editions	We use the master file to generate editions in different formats and store these derivative editions here.
documentation	This user manual.
dp-files	Reserved for material from Distributed Proofreaders
entry	Contains files in the second state of the production process, with text from one encyclopedia entry in each file.
master	Contains master files for the creating new editions.
metadata	Files necessary to creating metadata for the master files.
page	Contains files in the first stage of the production process, with text from one printed page in each file.
records	A collection of spreadsheets and other documents recording details of the production and analytical work.