12.26.09

Crossword Word Frequency

Posted in math, programming at 10:45 am by danvk

In a previous post, I discussed downloading several years’ worth of New York Times Crosswords and categorizing them by day of week. Now, some analysis!

Here were the most common words over the last 12 years, along with the percentage of puzzles in which they occurred:

(you can click column headings to sort.)

So “ERA” appears, on average, in about 23 puzzles per year. How about if we break this down by day of week? Follow me past the fold…

Read the rest of this entry »

Permalink 2 Comments

12.04.09

A new view on Minard’s Napoleon

Posted in web at 5:29 pm by danvk

Charles Minard’s chart of the demise of Napoleon’s Grand Armée is famous for its “brutal eloquence”. Edward Tufte says it “may well be the best statistical graphic ever drawn”.

The width of the line represents the size of Napoleon’s army as it marched to Moscow and then retreated.

I found a version of this visualization put on a Google Map using protovis. This visualization is great! It draws attention to one of the main problems with this famous visualization: it doesn’t give any geographical context. I had no idea where Napoleon’s army started and turned around until I saw this map:

Some things that stand out:

• The Grand Armée was way far away from France at the start of this.
• The march wasn’t as long as I’d imagined. When I think “march across Russia”, the image in my mind goes halfway across Siberia.

Permalink 2 Comments

11.05.09

How long is a day?

Posted in astronomy, math at 5:39 pm by danvk

As we approach the winter solstice, the days get shorter and shorter. There’s a common misconception about how quickly this change happens. Most people know that:

1. The summer solstice (June 21) is the longest day of the year.
2. The winter solstice (December 21) is the shortest day of the year.
3. The days get shorter between Summer and Winter.
4. The days get longer between Winter and Summer.

Many people take these four pieces of information and assume that the day length changes like this over the course of the year:

(The x-axis is the date. The y-axis is length of the day in hours.)

This is consistent with the four pieces of information, but is incorrect! There aren’t many sharp edges like that in Physics. Reality is much smoother:

The length of the day slowly increases as we approach the summer solstice, then slowly decreases as we leave it. This is great — it means that there are lots of long days in the summer. As we get to the autumnal equinox, the rate of change hits a maximum. The same thing happens around the winter solstice, only in reverse.

The summer solstice is the longest day of the year, but not by much! Here’s some day lengths for San Francisco:

The lengths of the days around the solstice differ by only a few seconds! On the other hand, here are some day lengths around the autumnal equinox (September 22):

The length of each day changes by several minutes in September. Over a single week the day gets a whole 15 minutes shorter!

note: the interactive graphs are dygraphs, a JS library I created. Check it out!

Permalink 2 Comments

11.01.09

Healdsburg Half Marathon

Posted in personal, sports at 1:25 pm by danvk

I ran my first half marathon this weekend, the Healdsburg Half in scenic California wine country. The race was held on Halloween, and no small number of runners came in costume. This is a nice twist on the usual “what should I be for Halloween” dilemma. Instead, it’s “what should I be that I can sweat in for 13 miles?”

My first goal was to finish. My second goal was to finish in under two hours. And I did! Final time was 1:54:33.1 (they are apparently very precise about these things!)

A race like this is a field day for data junkies like me, especially when you jog with an iPhone app like RunKeeper. I had it going for the first 10 miles, before my phone ran out of batteries. Here’s the track and mile splits:

I must have picked it up after that — my pace over the remaining non-iPhone miles was 8:14/mile.

Some more stats and thoughts on what to do differently next time:

• My co-worker Jeremy suggested that a good goal for a first half marathon would be a “reverse split”: running the second half faster than the first. I did that, too! The first 6.6 miles took 61 minutes, so the second must have taken 53. I guess I should have run the first half faster!
• I should have charged my iPhone the previous night! More important than recording a track, it let me know exactly how far I’d gone: “6.34 miles” instead of “a few minutes past that six mile marker”.
• I should have brought a jacket with me to the start. The race started before dawn and it was very cold! They even transported stuff to the end of the race for the runners. Something to remember for next time.
• Running with a friend is great and can be good motivation. I ran most of the race with my friend Erica, who shaved a full 20 minutes off her previous half marathon time!
• Erica’s dentist (a former marathoner) told her that she should take a drink at every water station. This was good advice. The only drink station I skipped was the one serving wine samples!

The Healdsburg Half was very well-organized. They had full results (PDF) posted the day of the race. I finished 396/1438 overall, 234/496 amongst men and 40/76 amongst 25-29 men.

Permalink Comments off

10.27.09

Guns, Germs, and Steel

Posted in reviews at 8:50 am by danvk

I recently picked up a copy of Jared Diamond’s Guns, Germs, and Steel, a self-described “short history about everyone for the last 13,000 years.” This book should have been right up my alley: it’s all about history, specifically early human history, which I don’t know much about.

Diamond’s goal is to explain why civilizations on the “Eurasian” continent developed to a much greater extent than they did in Africa, Australia or the Americas. Why did Europeans conquer the Americas and not the other way around? He wants to do this by looking for ultimate causes, not proximate ones like “Europeans had guns, while Americans didn’t.” He wants to explain this with environmental factors, not genetic ones. This seems like a nobel goal and an interesting subject for a book.

After about fifty pages, I was growing increasingly skeptical. He would make a point, present a very specious argument against it, then counter this argument. The old Straw man!

Then came the snippet that did this book in:

On the Chatham Islands, 500 miles east of New Zealand, centuries of independence came to a brutal end for the Moriori people in December 1835. On November 19 of that year, a ship carrying 500 Maori armed with guns, clubs, and axes arrived, followed on December 5 by a shipload of 400 more Maori. Groups of Maori began to walk through Moriori settlements, announcing that the Moriori were now their slaves, and killing those who objected…

Moriori and Maori history constitutes a brief, small-scale natural experiment that tests how environments affect human societies. Before you read a whole book examining environmental effects on a very large scale—effects on human societies around the world for the last 13,000 years—you might reasonably want assurance, from smaller tests, that such effects really are significant…

He’s holding this up as a small-scale example of how environmental factors can take two civilizations in different directions. The Maori (of New Zealand) and the Moriori (of the Chathams) were only separated around AD 1500. But by 1835, the Maori conquered the Moriori using disproportionate force! Diamond argues that they were able to do this because they were a stratified, agricultural society while the Moriori were loosely-organized hunter-gatherers. Such a short time-span! No western influences! Such a clean sociological experiment!

But this is incredibly misleading. The giveaway should have been “armed with guns”. The Maori did not develop guns independently. But I skimmed over this. The bit that did jump out at me was “November 19, 1835″ and “December 5″. How do we have such precise dates for an interaction between native peoples? It was enough to send me racing to the Wikipedia article on The Chathams.

Those 500 Maori came to the Chathams on a British whaling ship. They were armed with British guns. They were told of the existence of the Chathams by the sailors on the same whaling vessel. There are no hints of ultimate causes here. If the British had armed the Moriori instead, things would have turned out very differently.

I try to retain a healthy sense of skepticism when reading any non-fiction book that presents a thesis. But it stops being fun when you know that the author is willing to deliberately mislead.

I’m curious how this book won a Pulitzer Prize. Did any of the reviewers actually read it?

Permalink 3 Comments