Happy Sunday and welcome to Investing in AI. I’ll be at HumanX this week, so, if you are at the conference and want to chat let me know.
Today I want to discuss a paper that is a few years old about “The Hardware Lottery.” It’s an important concept that applies to AI in general but in particular the exploration around AGI. The paper is fantastic so mostly below I’m just going to quote the key sections from it, because it is a concept you should understand.
Here is what the hardware lottery means:
This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions.
Why does the hardware lottery matter for AI? Because AI is incredibly GPU focused, and that is driving much of the direction in which AI research is going. This matters because:
Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas.
Dr. Hooker’s analysis of the current state of AI (circa the paper date, but I still think it is accurate) is:
While deep neural networks have clear commercial use cases, there are early warning signs that the path to the next breakthrough in AI may require an entirely different combination of algorithm, hardware and software.
This essay begins by acknowledging a crucial paradox: machine learning researchers mostly ignore hardware despite the role it plays in determining what ideas succeed.
In fact, she argues that the modern day neural network version of AI might have happened sooner were it not for CPUs and the hardware lottery.
Perhaps the most salient example of the damage caused by not winning the hardware lottery is the delayed recognition of deep neural networks as a promising direction of research. Most of the algorithmic components to make deep neural networks work had already been in place for a few decades.
In the near-term, here are her recommendations.
An interim goal should be to provide better feedback loops to researchers about how our algorithms interact with the hardware we do have. Machine learning researchers do not spend much time talking about how hardware chooses which ideas succeed and which fail. This is primarily because it is hard to quantify the cost of being concerned. At present, there are no easy and cheap to use interfaces to benchmark algorithm performance against multiple types of hardware at once.
As an investor, there is a play here – be open to things that violate the tenets of the hardware lottery. If we fund new ideas, even if they take a bit longer to generate economics, we could create the really breakthrough technologies that will get us to AGI. Of course, given the current assumption by many that AGI is just around the corner, maybe that is a difficult bet to take.
Thanks for reading.