The Ghost Stories Had the Best Data

A Song-dynasty ghost story will tell you where the haunted bridge is. Not vaguely: the exact location, the officials posted nearby, what the local economy ran on, the social rank of everyone who walks onto the page. All of that arrives first. Only once the ledger is complete does the ghost show up.

I spent a semester building a searchable database (opens in new tab) for historian Hsiao-Wen Cheng's research group at Penn. It indexes 2,745 tales from Hong Mai's twelfth-century Yijian zhi (Record of the Listener), a collection of ghost stories and other accounts of the strange that ran to 420 chapters before much of it was lost. I went in expecting the extraction to be the interesting problem. But it kept finding things in the tales: prefectures and counties, reign years, official titles, and in many of them the name of whoever told Hong Mai the story.

The supernatural needs an address

At first all that detail looks like a quirk of Hong Mai's. It follows instead from what the genre has to do.

A supernatural tale only works if the reader accepts the world it interrupts. The haunting is the anomaly, so everything around it has to be ordinary enough to make the anomaly legible, and the author supplies that ordinariness in the same four slots. One of Hong Mai's tales sets a tea shop under the Guanfeng Bridge in Huangzhou, says it belongs to Li Shiliu and that his hired man, Cui San, is still awake one spring night in 1181, and then lets someone knock. The young woman at the door is not human.

Compare the literature that got taken seriously. Poetry and philosophical essays are allusive by design; their skill lies in compression and reference, in saying one thing by gesturing at another, which makes them wonderful and nearly useless if what you want to know is who administered a county. Canonical texts are abstract because abstraction is the achievement. In a ghost story, concreteness is just how the trick works.

Collections like Hong Mai's are classed as biji xiaoshuo, miscellaneous writings and trivial anecdotes. To a historian reconstructing where people lived and how they were ranked, though, the tales are invaluable, largely because nobody was performing significance in them.

What the pipeline was actually for

I would not have noticed how far the pattern ran by reading alone. I noticed because entity extraction turns a corpus into something you can count.

Classical Chinese does not cooperate with modern tooling. There is no punctuation in the original and no spaces between words, and its grammar had left spoken Chinese centuries before Hong Mai wrote. A single character can be a noun, a verb, or a particle depending on context that takes real historical knowledge to resolve. I ran a few off-the-shelf Chinese named-entity recognition (NER) models over sample passages, and the results were bad enough to be funny: a model trained on People's Daily has no idea what to do with a sentence where one character means "to govern" in one clause and is somebody's surname in the next, and where the place name refers to a region that hasn't existed since the Southern Song.

So we went hybrid. Rule-based heuristics took the predictable cases, since reign-era dates follow fixed formats, official titles come from a known set, and place names often end in an administrative suffix like 州 (prefecture) or 縣 (county). For ambiguous names, literary allusions, and places referenced obliquely, we fine-tuned models on a hand-annotated subset. That training data was expensive, because the annotators had to read classical Chinese, but the models trained on it generalized to the rest of the collection better than I'd expected.

The piece that paid off most was cross-referencing every extracted entity against existing historical databases, including Harvard's China Historical GIS for place names. An entity's confidence score came from how many independent sources confirmed it, so a person who also appears in the official dynastic histories scored high and a reference that could be three different people got flagged for review. That step cut manual cataloging time by roughly 70%.

Once the tales had entity counts attached, the distribution was visible. The addresses ran all through a book filed under trivial anecdotes.

I suspect that generalizes past this corpus. Which texts a tradition treats as serious tells you what the tradition valued, and almost nothing about which texts know the most. Here those were different piles.