A Few More Million Amazonian Monkeys

Update 5: The monkeys recreated every work of Shakespeare and went viral. See the project project postmortem for my thoughts on going viral and what I learned during the project.

Update 6: I created a new visualization of the monkeys’ data.

Update 4: The monkeys recreated “A Lover’s Complaint”. Check out the write up.

Update 3: Welcome Slashdot, Fox News, Engadget and Japanese Engadget. So far, the monkeys have ran through 7.5 trillion 6.5 trillion 5 trillion (2011-09-22) 4 trillion (2011-09-16) 3 trillion (2011-09-10) 2.5 trillion (2011-09-07) 2 trillion (2011-09-05) 1.5 trillion (2011-09-01) 1 trillion (2011-08-28) 515,912,000,000 (2011-08-25) character groups.

In a recent post, I described a recent project to recreate Shakespeare using Hadoop and Amazon EC2. This time, I am going to recreate every work of Shakespeare randomly.

This project comes from one of my favorite Simpsons episodes which has a scene where Mr. Burns brings Homer to his mansion (YouTube Video). One of his rooms has a thousand monkeys at a thousand typewriters. One of the monkeys writes a slightly incorrect line from Charles Dickens “It was the best of times, it was blurst of times.” The joke is a play on the theory that a million monkeys sitting at a million typewriters will eventually produce Shakespeare. And that is what I did (am doing). I created millions of monkeys on Amazon and put them at virtual typewriters (aka Infinite Monkey Theorem).

Less Technical Explanation

Instead of having real monkeys typing on keyboards, I have virtual, computerized monkeys that output random gibberish. This is supposed to mimic a monkey randomly mashing the keys on a keyboard. The computer program I wrote compares that monkey’s gibberish to every work of Shakespeare to see if it actually matches a small portion of what Shakespeare wrote. If it does match, the portion of gibberish that matched Shakespeare is marked with green in the images below to show it was found by a monkey. The table below shows the exact number of characters and percentage the monkeys have found in Shakespeare. The parts of Shakespeare that have not been found are colored white. This process is repeated over and over until the monkeys have created every work of Shakespeare through random gibberish.

Technical Explanation

For this project, I used Hadoop, Amazon EC2, and Ubuntu Linux. Since I don’t have real monkeys, I have to create fake Amazonian Map Monkeys. The Map Monkeys create random data in ASCII between a and z. It uses Sean Luke’s Mersenne Twister to make sure I have fast, random, well behaved monkeys. Once the monkey’s output is mapped, it is passed to the reducer which runs the characters through a Bloom Field membership test. If the monkey output passes the membership test, the Shakespearean works are checked using a string comparison. If that passes, a genius monkey has written 9 characters of Shakespeare. The source material is all of Shakespeare’s works as taken from Project Gutenberg.

The monkeys’ data from Amazon’s cloud is updated on this site every 30 minutes. The images below show green for every character group that was found and white for those that are still missing. The images output is kind of like the animations for defrag utilities. As the monkeys progress through the works, more and more character groups will be found and show green.

The Tabular Output Of What Has Been Found

Every Work Of Shakespeare

All Works of Shakespeare

Progress Through Individual Works Of Shakespeare

A Lovers Complaint

Loves Labours Lost

The Merchant Of Venice

The Tragedy Of Julius Caesar

A Midsummer Nights Dream

Measure For Measure

The Merry Wives Of Windsor

The Tragedy Of King Lear

Much Ado About Nothing

The Tragedy Of Macbeth

Alls Well That Ends Well

The Sonnets

The Tragedy Of Othello Moor Of Venice

As You Like It

The Comedy Of Errors

The Taming Of The Shrew

The Tragedy Of Romeo And Juliet

Cymbeline

The Tempest

The Tragedy Of Titus Andronicus

King Henry The Eighth

The First Part Of King Henry The Fourth

Second Part Of King Henry IV

The First Part Of Henry The Sixth

The Second Part Of King Henry The Sixth

The Third Part Of King Henry The Sixth

The Two Gentlemen Of Verona

King John

The History Of Troilus And Cressida

The Tragedy Of Antony And Cleopatra

The Winters Tale

King Richard III

The Life Of King Henry The Fifth

The Tragedy Of Coriolanus

Twelfth Night Or What You Will

King Richard The Second

The Life Of Timon Of Athens

The Tragedy Of Hamlet Prince Of Denmark

Update: I was running this on a free micro instance (600 MB RAM) from Amazon. Alas, the monkeys needed more RAM than the free micro instance had and the processes get out of memory errors. I have moved the Hadoop server to my home computer which is much faster and has more memory.

Update 2: I updated the Hadoop configuration to have less idle CPU time. This will significantly increase the monkey power and find more character groups.

Update 4: I made a small change to how memory is allocated for the random character groups. It should help speed things up again.