When you get inside a well-run hyper-growth company like Meta or Google, conversely, you see a lot of very specific and rigorously-collected and defined second and third derivative metrics that really tell you how well the product is working and what people are doing. Google famously optimised for response time, which no-one else thought was important, and aimed to get people to leave the site quickly, which everyone else thought was bad. A lot of these metrics can also be a positive feedback cycle making the product itself better: when you reformulate a Google search and try again, or click on the third link and do or don’t come back afterwards, you’re giving Google signals that make it better, and that’s a powerful network effect. It’s not clear that any LLM providers are really able to leverage this kind of thing yet, and what they would measure: if I ask a question and don’t try again, was that the right result, was it wrong but I thought it was right, or did I give up and go to Google?
At the other extreme, I think charts comparing generative AI user growth to things like the internet or smartphones need some caution, or context. The original Macintosh started at $7850 and the original iPhone at $800 ($2450 and $499 before adjusting for inflation) where generative AI is ‘just’ a website or an app (as far as the user experiences it). You don’t need to buy a device or wait for your telco to build broadband or 3G, and meanwhile there are billions of people online now instead of tens or hundreds of millions, so yes, it’s grown a lot bigger a lot faster: we’re standing on the shoulders of giants. (This is also why Nvidia can ramp its sales so fast - it’s riding on the contract manufacturing base built over the last few decades). That doesn’t mean this is a bad comparison: as I wrote a long time ago, unfair comparisons are often the best kind, but you do need to know it’s unfair.
Stepping back, Eric Schmidt told Sheryl Sandberg that when you’re getting on a rocket ship, don’t argue about which seat, and this is certainly a rocket ship. The Occam’s Razor is that in the end all of these metrics resolve to money and time. But the fuzziness today also reflects how early and unclear all of this is. We don’t know what the business and the products will be yet, and the right metric will be shaped by that. Mary Meeker’s 1995 report forecast email and web use separately, and she thought email would be bigger, which wasn’t really how this worked.
