The Ghost Stories Had the Best Data

Song-dynasty supernatural tales were filed for centuries as minor entertainment. They hold the richest geographic and social detail in the corpus, and the reason is structural.

4 min read
NLPDigital HumanitiesClassical ChineseFlask

A Song-dynasty ghost story will tell you where the haunted bridge is. Not vaguely: the exact location, the officials posted nearby, what the local economy ran on, the social rank of everyone who walks onto the page. All of that arrives first. The ghost shows up afterward, once the ledger is complete.

I spent a semester building a searchable database of 2,700-odd literary works from the tenth through thirteenth centuries — poems, essays, fiction, administrative records — most of which had never been cataloged in a way that let anyone search across them. I went in expecting the interesting problem to be the extraction. The interesting thing turned out to be which texts the extraction kept finding things in.

It was the ghost stories. Consistently, and by a wide margin.

The supernatural needs an address

At first this looks like an accident of taste. It follows instead from what the genre has to do.

A supernatural tale only works if the reader accepts the world it interrupts. The haunting is the anomaly, so everything around the haunting has to be ordinary enough to make the anomaly legible. The author grounds it in a real place, real officials, a working local economy, a family whose standing a contemporary reader could place at a glance. That realism is load-bearing. Without it the ghost has nothing to be strange against.

Compare the literature that got taken seriously. Poetry and philosophical essays are allusive by design; their skill lies in compression and reference, in saying one thing by gesturing at another. Which makes them wonderful, and nearly useless if what you want to know is who administered a county. The canonical texts are abstract because abstraction is the achievement. In the entertainment, concreteness is just how the trick works.

What you get is an inversion of the usual hierarchy. Literary scholars filed these tales for centuries under minor work. For a historian reconstructing where people lived and how they were ranked, they are among the densest sources in the collection, largely because nobody was performing significance in them.

What the pipeline was actually for

I would not have noticed this by reading. I noticed because entity extraction turns a corpus into something you can count.

Classical Chinese does not cooperate with modern tooling. It isn't Mandarin with older vocabulary; it is structurally different in ways that break standard pipelines. There is no punctuation in the original and no spaces between words, and the grammar left spoken Chinese centuries ago. A single character can be a noun, a verb, or a particle depending on context that takes real historical knowledge to resolve. I ran a few off-the-shelf Chinese NER models over sample passages and the results were bad enough to be funny: a model trained on People's Daily has no idea what to do with a sentence where one character means "to govern" in one clause and is somebody's surname in the next, and where the place name refers to a region that stopped existing during the Southern Song.

So we went hybrid. Rule-based heuristics took the predictable cases, since reign-era dates follow fixed formats, official titles come from a known set, and certain character sequences reliably mark a geographic reference. For ambiguous names, literary allusions, and places referenced obliquely, we fine-tuned on a hand-annotated subset. That training data was expensive, because your annotators have to actually read classical Chinese, and it generalized across genres and reigns better than I had expected.

The piece that paid off most was automated cross-referencing against existing historical databases. Every extracted entity carries a confidence score drawn from how many independent sources confirm it: a person who also appears in the official dynastic histories scores high, a reference that could be three different people gets flagged for review. That is where the roughly 70% cut in manual cataloging time came from, by aiming human judgment at the cases that needed it.

Once every text had an entity count attached, the distribution was visible in an afternoon. The tales nobody had bothered to index were the ones carrying the addresses.

I suspect that generalizes past this corpus. Which texts a tradition treats as serious tells you what the tradition valued, and almost nothing about which texts know the most. Here those were different piles.