Letter Frequencies in English
January 3, 2006 11:45 pm Data, EnglishI went looking for letter, digram and trigram frequency counts to gauge scoring a word game I’m writing. What I found was disagreement, chaos, and unacceptably small sample sizes.
Whee.
So, I played some word games recently – Boggle and Scrabble among them – whose scoring mechanic hinges largely on board placement and on the rarity of the letters in use (the canonical “Q and Z are ten, J and X are eight” rule.) I decided that I didn’t like the distribution in use, and that it was time to generate a new one.
First step was to decide on a distribution. Scrabble’s a bunch of fun, but in all honesty it touches on a very small part of our vocabularies. Even when you’re playing against very skilled players, certain words just come out more often than others. As I’m sure to eventually start ranting about in another blog entry, I’m into the idea of educational gaming; Scrabble may have originally been invented to fill the inventor’s lack of a job time during the Great Depression, but a huge percentage of its sales these days are in a pseudoeducational capacity by parents hoping to develop their children’s lexical repertoire in a way that’s at least entertaining.
Alfred Mosher Butts, the game’s inventor, did a great job given the tools available at the time. He went as far as to histogram several major newspapers of the time for letter frequency, something which must have been blindingly boring and error prone. Possibly because his scoring and rules were subject to a decade of tinkering before the game finally took off, the rules he ended up writing are in fact quite nicely polished; the game distinguishes itself with a large number of attempted variants and an extremely small number of even moderately successful variants, which gets into another topic I’m sure to blog about sooner or later – namely how to determine when your ruleset is stable via a shaky metaphor regarding minima, maxima, local minima and fitness evaluation.
Anyhoo, Butts did a good job with the tools available to him, but it’s been almost 80 years, and I have better tools than he did. One of the best of those tools is the ability to look back on 80 years of reaction to his game, whereas he had no reflection on anything closer than a crossword puzzle. Given that there’s no competition nor any tendency to intentionally blunt the board to prevent opponent access to powerful squares, the two games aren’t similar enough that one might learn significantly from the other.
What have we learned from Scrabble, Boggle, Upwords and the ilk? What needs to change? What needs to stay? Given that this is now being considered for a videogame, what things can I do that they didn’t have the ability to do? Asploding tiles? Alternate physics? Gold, shiny shiny gold?
One particular game I’ve played a lot recently, and rather enjoyed but thought more could be done with, was Bookworm. Bookworm is basically Boggle, but with mechanics added to prevent stagnation – for example, when you use letters, they’re destroyed, allowing new letters to fall in from the top. Also, because four surfaces are too few and because people get into arguments about diagonals, Bookworm uses a hexagonal board (rendered with squares, but that’s relatively unimportant.) Still, it’s not aimed at my demographic; turns are infinite time, which is great for part of the casual gamer crowd, but combined with how long it takes for the obstacles to start showing up and how suddenly and annoyingly diffcult they become, I’m convinced that this is a very good game mechanic suffering a near-disasterously flawed implementation.
Not that I have a competitor just yet. But still. More to come. I’ve got something I think will be pretty neat underway; I’ve been playing Sid Sackson’s BuyWord, and mulling over Scorched Earth style markets. There’s something maybe new on the horizon.

January 4th, 2009 at 10:57 pm
Yes, to be a competitive Scrabble player much more is required than a good vocabulary (crossword puzzles or just plain reading are better at building vocabulary than Scrabble imo):
1. Ability to recombine letters. For instance, I’m a freak who sees a word and immediately my brain starts making new words using all (or some) of the letters from that word.
2. Making several words in one turn. My husband has trouble with this. He’s a writer with a much better vocabulary than I possess and will make a brilliant word and get pissed that it’s worth like nine points while I’ll play like three first grade words at a time and get sixty points.
3. Like poker, knowing what’s been played on the board, what’s in your hand, and thus what your opponent may be holding.
4. Playing on certain parts of the board while avoiding others so as to limit your opponent’s access to bonus tiles. My husband often gets frustrated because I “screw up” the board. It’s fun
Bookworm is a fun game. It can be played timed and untimed. I stopped playing when I reached over two million points (iirc) and stopped getting new job titles.
My favorite part about Boggle is that the whole family can play it together. But, it rewards speed and I often get hung up trying to make longer words rather than just write down every three letter word that’s there.
I know this blog is years old but I’m interested in seeing your new word game, if you ever get around to making it.
January 6th, 2009 at 5:14 pm
Yeah, well, you can’t. The game suffers the single largest sin any game may suffer: it isn’t fun. I completed it several years ago, and tried it, and it just bored the teeth right out of my mouth, all puns notwithstanding.
It’s kind of a shame – what I had should have been a high action high pressure competitive word game. However, instead, it is fail. And fail, you’ll learn, is no fun at all. (I sure did.)
January 7th, 2009 at 5:24 pm
Hmm…maybe it was boring to you since you invented it. If you change your mind and want an objective test subject, I’m willing to subject myself to being bored by your game
Anyway, it’s good to fail sometimes.
January 9th, 2009 at 7:55 pm
You’d need specialty hardware to load the game onto a GameBoy Advance. I appreciate the offer, but I’ll be up front: the game is just goddamned dreary.
I do have games you can try, but this one outright embarrasses me in retrospect, so it’s not going to be this one.
January 9th, 2009 at 10:25 pm
Sorry if it seemed I was trying to push you, didn’t mean to
I made up a board game in the 5th grade called ‘The Ultimate Journey’ – talk about dreary, lol