The future of software is in algorithms.
I don’t remember where I first read that. And it’s not as innovative, as just predicting a trend unfolding under our eyes. A big part of the drive towards simplicity and better user experience, is moving away from software that has a lot of features, to software that has a lot of smarts.
You can argue that those are different problems with different solutions. That’s true. What’s changing is the rise of software that is based on algorithms more than it is based on features. For every new office online service, there’s a new Amazon recommends service. Vertical search engine, recommendation engine, and recently an explosion of memetrackers. These all live and die on the quality of their algorithm.
Which memetracker is better? They all feed from the same data, the vast expanse we call the blogosphere. They all have more or less the same features: a front page and a feed. What sets them apart is the algorithm that determines what rises to the top. I read Memeorandum daily. And I read Reddit daily. What sets these two apart is the algorithm they use to capture hot topics of discussion.
Then there’s TailRank, which again is not distinguished by any features it adds. All three have a feed, and that feed is the only feature I use. It’s distinguished by serving me different content that is picked up by a different algorithm.
Sooner or later — and I’m going to err on the side of sooner — we’re going to reach a point where algorithms become too complicated to hand tune and improve. That’s the point where, no matter how many smart people you have, you just can’t come up with a better algorithm. But we’re not going to stop there. We’re going to go to the next step: algorithms that create better algorithms.
Imagine that you have an algorithm that finds the most relevant blog posts on a certain topic. Some are very relevant, some are less relevant. Imagine that you can track what readers pay attention to, which of the results they think is more relevant. Now, add an algorithm that compares the results of the first algorithm, the attention given by the user, and uses that to devise an even better algorithm.
Instead of serving your best effort attempt to find relevant search results, you’re serving an algorithm that keeps improving the more people use it. The software keeps improving, but the process is fully automated.
Genetic algorithms is one of the most interesting ways to create self-improving algorithm. You start with a given algorithm, and you let the software mutate it by making small changes, and determining which of these changes improve the algorithm. The improvements survive to the next step. The next step just repeats the first step, making even more changes and narrowing down on the few improved algorithms.
This is all science and theory without a good example. So let me point you at one such example:
This is the home page of Critticall. A programming tool, which enables you to apply the Darwinian force to algorithms! To those fundamental, and to those quite common. Evolution isn’t just a theory. It’s a nature’s technology to make complicated things. And to sharpen those things further, to perform better and better.
The Critticall page could use some human intervention to improve the quality and readability of the text. But it does have some simple examples that are nicely illustrated with animated GIFs. You can see the original and improve algorithms, how little they change but how well it affects the results.
Genetic algorithms are a few years away from influencing the way we search data, interact with other people, and keep track of our daily lifes. But they are the next thing to watch.
ryan king