Home
Production
Descriptions of OCR processing, TEI transformation, and metadata creation
Master Files
After creating entry files, we run an automated metadata generation process to add index terms to each entry.
Automated Metadata Procedure
How we create keywords for every entry file.
Generating the Manifest
The manifest stores all of the parameters for HIVE

Production
Descriptions of OCR processing, TEI transformation, and metadata creation
- Organization
  How to keep hundreds of thousands of files organized.
- Page Files
  Explains the procedures we use to get the best quality OCR of each page.
- Entry Files
  Procedures for converting single pages into Encyclopedia entries.
- Master Files
  After creating entry files, we run an automated metadata generation process to add index terms to each entry.
  - Automated Metadata Procedure
    How we create keywords for every entry file.
    - Named Entity Recognition
      Running named entity recognition on entries
    - Generating the Manifest
      The manifest stores all of the parameters for HIVE
    - Run HIVE2
      With the NER output, plain text, and a manifest in place, we are ready to generate the index terms.
    - Add index terms to master files
      Adding index terms to the master files

Generating the Manifest

The manifest stores all of the parameters for HIVE

HIVE2 needs to know which vocabularies to use for each file and what the parameters are for the vocabulary: minimum word count, for example. All of this information is stored in a csv file that includes the names of the all the files to be processed. This is called the “manifest” for HIVE. We create it with a Python script.

Run auto-manifest.py on the batch folder.
1. Provide the edition number (1 or 2 digits) of the batch folder when requested.
2. Provide the letter name when requested.
The script creates a manifest file and saves it to the batch folder. It names the file manifest_ + the letter name.
manifest_A.csv

The batch folder now contains three files for each entry and a single manifest file for the batch.


each entry	full text	kp*.txt
each entry	NER Topics	kp*a.csv
each entry	NER Geo	kp*b.csv
one per folder	manifest	manifest_*.csv