Zipfs law and my ineptitude, part deux

Sometime ago I tried my hand at proving Zipfs law using Saint Thomas Aquainas Summa Theologia. The attempt was a complete failure, so I wanted to try again (once bitten, twice shy, and you don’t grow up to be a nerd).

This time however I went with Kotlin (an awesome new language for the Java virtual machine) and War and Peace, the classic Russian novel (which I got for free from Project Gutenberg). If Zipfs law holds, then we should see a blue line falling straight and a red line (which is the frequency of the word, multiplied with the relative position of the word) staying almost constant, but as can be seen in the graph, this is not the case, not by a long shot. If we just use the to 100 words the average frequency is 3107, the average adjusted frequency, divided by the total adjustment, is 1427, which means the average frequency is more than twice what it should be.
Word frequency in War and Peace

Even worse for my self-esteem, it doesn’t look like there is a bug in the program as the top ten words are: the and to of a he in that his was. Given that the main person is a male (and at least the couple pages that I read) talks about him in the first person, I don’t think those are unrealistic.

Okay so Tolstoy isn’t British, perhaps if I had used the original Russian it would have been different – but looking in to it, a significant part of the novel is actually in French. Oh well. Lets try with something completely different: Anna Karenina.

Word frequency in Anna Karenina

Still no luck. If Zipfs law holds then we would expect to see the ratio of the average unajusted word over the average adjusted word devided by the adjustment to be one (ie they cancel out) but that isn’t the case for either of the two, so I went ahead and downloaded the top ten books from project Gutenberg. These are the ratios I found:

Title Ratio
The Adventures of Huckleberry Finn 1.9992198058099362
Alice Adventures in Wonderland 1.9851261916381613
Anna Karenina 2.0704857183257914
Beowulf 2.201444236037978
Frankenstein 2.384831717351647
The Metamorphis 1.9659961377948403
Moby Dick 2.1206125367045408
Pride and Prejudice 1.9197390080077206
The Adventures of Sherlock Holmes 2.070056679992902
The Legend of Sleepy Hollow 2.4493014749970023
The Yellow Wallpaper 2.0006322162974524
War and Peace 2.177793662470332

The only possibly remarkable thing here is that the ratio seems to hover around 2.

By digging into Google Scholar I found this old article (pdf link warning), where the author is nice enough to count his own words and their frequency out (check the link for the data, the link will open in a separate window or tab), which caused me to take another look at my data: the graphs and numbers do actually fit relatively nice, once you discount the top 7 or so words.

Word frequency in War and Peace, top cut out

Thank you. Thats the kinda of data we would expect to see. I guess when you cut the outliers, the theory fits the data beautifully.