Is it possible to study something as ill-defined as culture in a quantitative manner? Researchers from Harvard have collaborated with Google and some traditional publishers to answer that question with a qualified "yes." By leveraging a portion of Google's massive library of digitized books, the team has created what they call a "culturome," with which they can track the use of language and terms across hundreds of years. This lets them track not only trends in language and usage, but the rise and fall of celebrities and historic events in the books of many eras. And, thanks to Google, the underlying data has been exposed via a Web interface, allowing others to perform their own analysis.
The authors didn't work with the full complement of Google's digitized texts, but the amount of material they did use is staggering: over 5 million books. They estimate that's about four percent of the books ever published. Google has about three times as many works scanned, but the scan quality and metadata on these—date and location of publication, etc.—isn't uniformly good, so the research has focused on the material with the best quality. Works start appearing at 1500, and include significant contributions in seven languages. They estimate that it would take someone 80 years to read it all, assuming said individual didn't eat or sleep.
Given this data source, the authors performed a simple analysis, identifying the frequency of a word relative to the total number of words produced that year. This compensates for the vast expansion in the printed word. In 1900, about 1.4 billion words a year were being printed; by 2000, that figure had reached 8 billion. With this data in hand, they performed a similar process for what they call "n-grams," or short phrases of up to five words.
This simple procedure is enough to allow them to detect some obvious trends. In the US, the frequency of the word "slavery" peaked during the Civil War, and then had a resurgence during the civil rights movement era. But it also revealed a variety of less obvious phenomena.
For example, they attempted to figure out how many distinct words appeared in their culturome to the numbers in various dictionaries (they did a random sampling to estimate the appearance of dates and other non-words). By 2000, there were over a billion words, thanks to the addition of about 8,500 a year since 1950. Dictionaries only contain about a third of that; the authors (not content to borrow terminology from biology) refer to the rest as "lexical dark matter." The vast majority of this is in the form of low-frequency words. Many of the new arrivals to dictionaries, like "buckyball" and "netiquette," have seen an explosive rise in use. But others ("amplidyne," for example) seem to get added to dictionaries even as their usage is plummeting.
They also looked at the occurrence of specific years, like "1911." Here, most years showed a similar trajectory: soaring use as they occurred, and a rapid decline starting a few years afterwards before tailing off slowly. But the trajectory of these curves changed. 1880 had a half life of 32 years in the literature, while 1973 had a half life of only 10 years. "We are forgetting our past faster with each passing year," the authors conclude.
Researchers also used the biographies present in Wikipedia and Encyclopaedia Britannica to track the trajectories of celebrities, which ended up looking a lot like years. Relative to the 1800s, our celebrities are younger, and the half life of their fame is shrinking, dropping by about 50 years during that period. Actors become famous the earliest, at around age 30, while authors and politicians require progressively more of their lives to reach a peak. There's also bad news for those reading this column out of professional interest. "Science is a poor route to fame," state the authors obviously. "Physicists and biologists eventually reached a similar level of fame as actors, but it took them far longer."
The paper is filled with other nuggets, including a successful detection of censorship in different cultures and the finding that God could use a better publicist. But, thanks to Google labs, n-gram the database is enjoying its own rapid rise to fame online. Google labs has created a Web interface that allows anyone to insert their n-gram of choice. As a result, a whole host of interesting and amusing graphs have been appearing, shared by their creators.
Some of these, however, have pointed out the limits of this sort of analysis. For example, because of its multiple meanings, "punk" has always had a much larger usage than "rock and roll," even though the former is a subgenre of the latter. "Punk rock" gets you a more realistic looking graph, but probably doesn't capture the full extent of its use. There's also more general questions about the approach, such as how well four percent of the total output really captures the culturally significant works, or whether word frequencies can miss the development of significant concepts or events.
Nevertheless, it's hard to escape the sense that many of these trends seem to follow realistic trajectories, and that some of those identified by the authors do reflect changes in the global culture. If nothing else, it's far better than simply being left to speculate about what the overall usage might be.