Presentation: The Next Generation of AI Products

Transcript

Hilary Mason: I have been wrestling with what to talk about. It really comes down to the moment of chaos we're in and thinking about the complexity and the nuance and how we talk about AI, and how we can reason about what a future we want to build will look like in this moment of a lot of change and uncertainty, and people using the same words to mean different things. This is what leads us to things like this. Recently, this headline popped up. I kept it, including the ads because I felt like it set the stage for today's moment. All of us as technologists, as leaders, are dealing with this feeling of like, everyone's into AI, but what does it even mean? I chose AI products as the frame for this talk, but it's really about trying to narrow in on, what is that thing actually? What is real? What can we do to build really great experiences, businesses? What does that mean for us as technologists, as people building careers who are managing people, leading people? There are so many questions right now and not that many answers. I think as technologists, we know framing the question is the hard part.

Usually, the answers are trivial or impossible. Hopefully, we will come through with some questions. You may be asking, why is someone who makes a video game platform here at QCon AI where we all have very serious jobs and do real things? I will explain that. I did not always make video games. I went to grad school in machine learning when it was so uncool that I once told a boy at a party what I was working on and he turned and walked away.

Then found my way out of academia. It was not for me. I was very mediocre at it. Became the chief scientist at a little social media company here in New York called bit.ly, which was short links on social media, before we had any idea how wrong that was all going to go. It was 2008, 2009. I've worked with a ton of startups, large companies. In 2009, I gave a talk that is still one of the talks I'm most proud of. It was about a set of scripts I wrote when I was a computer science professor to help me answer email. The moral of this story was actually that automating email was wrong, because it ended up automating a message to my mom that got her really upset and she called me. The talk is 5 minutes, but it's really about how communication is humanity. Yes, this is possible. This was from 2009, but the work was a few years before. It was not something that everyone was trying to do every day. The point was that maybe we shouldn't. Here we are.

In 2014, I started a company called Fast Forward Labs. We called it a machine intelligence company because we did not want to use the term AI because it's a marketing term. We built a natural language generation system, this was before transformer models. You could do this before transformer models. In this case, it took New York City real estate ads, and you gave it like, a two bedroom by the park, and it would write this cozy sun-filled home. It turned out cozy correlated with a 400 square foot smaller apartment. We learned all kinds of things about language in real estate back then. This company was really about helping people understand the tech.

I think if I have one super power, it is really understanding the math, understanding the code and the system architecture, and then calling it what it is. That is what I am hoping to bring here today. I'm sharing another example of that work, because this was deep learning for image object recognition. All this work is still online if you want to search for it. We built an image recognizer. You may notice that none of the photos at the end are crabs. Anything that looked like French fries near water was automatically classified as a crab. I'm sharing this with you because a huge part of doing this work well is understanding where it fails and how it fails. These are probabilistic systems.

By definition, they fail. If we can't talk about the failures, then we can't really do a great job working with the tech. That is a theme you will see throughout my story here. That company sold to Cloudera. I was the general manager of their machine learning, AI business unit. We advised and built stuff. It was a very cool job, because we got to work on all kinds of data problems across different industries. I pulled a few things here that were just a couple of my favorites.

The thing we did that saved a client the most money ever was actually taking their spreadsheets of nuts and bolts suppliers. They had agglomerated together four different production-based companies. They're a manufacturer. We're like, you're doing three orders from the same part, what if we combine those orders and actually negotiate a slightly better rate? Or you're ordering interchangeable bolts from two different suppliers, what if we do one? This saved them close to $100 million in the first year of deploying it. It was all spreadsheets and looking at pieces of information, and asking an expert, is this really the same thing, or is this a similar thing? I bring this up because sometimes the greatest opportunities for using our data skills are not in the most interesting models, but are really on the ground thinking about the entire set of considerations that matter.

Another example I wanted to share, and again, this is from maybe 2016, where we were building customer service support for a call center at a big U.S. bank. We took a year of calls into the call center, did speech to text analysis, tried to figure out what people call about, correlated it with stuff going on in their account before they called. In some cases, we're able to proactively send an email saying like, "We think this is going on. You can call or we can handle this for you". Also learned things like just under 1% of the calls were just people who wanted to call and talk to somebody. There wasn't actually an issue in the account at all. They wanted some reassurance. Thinking about ways to provide that that were more effective. The point of this story is to say that the human considerations matter a lot.

Why Talk About AI Products?

This is why we're talking about AI products, even though many of us may spend our days in the code or in the weeds. It is the tech that is changing all of these layers at once. If we really want to be accurate, robust, and optimistic, we have to find it in the socio-technical and everything around that as well. I often talk to boards or CEOs and they'll ask me questions. Some of them are so far down one path, they're like, when can I have the AGI because I want to fire everybody and just be done with it. Some of them are so far on the other side, like, "I don't want to deal with this. Why do I need to care about it?" I love this question because it gets at the heart of what we're talking about here in this whole event. Why do we need to care about AI at all? Here's what AI thinks. I tried Copilot. I was like, please create a slide explaining to an audience of senior engineers and architects why they should care about AI. It wrote some stuff.

Then I said, make me the slide. It wrote some slightly different but still the same stuff. Then I hit the little make me the slide. Then it's like, you got to go log in over here again, for no reason. I logged in. Then it gave me three slides that said nothing about the stuff it had said earlier. This was the first one. It's got a cool circuit board, though. This is the second one. This is the third one. These are all good reasons. They're very boring. Then I went to ChatGPT with the same query, and it made me this slide. I don't even know what this means, "AI didn't kill engineering, it moved the hardest problems back into it". That's a fortune cookie. It sounds great. There's a point here.

Here's what I think. Here's my real answer to this question. Then we're going to get into the product question here. If someone asks you, why do I need to care about AI? The answer is that what we do when we run a business, when we're responsible for a team, we are managing risk. Even if you're managing a small team, you're largely absorbing possibility space and helping your team navigate it. You create clarity so they can do their best work, because you absorb that uncertainty. That's what leadership is. This tech change, unlike NFTs, which might have happened way over there, or a really cool new database, this one affects every layer of the stack at the same time. The technology, while it can be understood mathematically and technically, is not well understood in the market. You have a lot of people learning about it in ways that are not giving them robust mental models for making good decisions. The world is changing around us. What matters is changing around us. The economy is changing around us. Still, we have to make good decisions in all of this.

The thing about AI is that I don't think it's going to replace N% of jobs. That's a vast simplification. It is going to have an impact on how we work and what people do. Even if you don't care about the technology yourself, it is going to change the environment in which you are operating such that your past experience, your knowledge of what's easy and hard, your pattern matching no longer applies. That is to say that as leaders, we manage risk, your risk function needs to change. If you don't understand it, at least a little bit, you will make bad decisions because your priors do not help anymore. It's hard. You either wake up really excited about this kind of stuff or it's just another headache on top of these headaches. That's what leadership is. It is one giant impossible and really fun headache.

What is Generative AI?

What even are we talking about then at the technical level? I'm not going to be very technical. I do like to make sure we are using the same words to mean the same things. This is a slide I wrote in 2014 to explain AI. At the time we had big data. Does anyone remember big data? That was really fun. What that was about was being able to get all of the data in one place and do interesting mathematical operations on it without having to have specialized infrastructure. We had Hadoop in it, like changed a lot of things. When you can do that, you can have an analytics function. You can start asking questions. When you can do that, you can do data science. You can make predictions. You can build models. You can start to understand things in new ways. You can build those models into your products. Now your products have search and recommendations, and that's awesome. You can do machine learning. Models that improve with the introduction of more data that can automate some of that decision-making.

Then AI, it is actually blue and it is a question mark because even at the time, it was a very exciting marketing term. What does it mean? It means whatever we want to call the shiny thing over there. It is still a bit that, though now we talk about generative AI. What I wanted to share this to make the point about is that the math and the code and the data are still the same. All of the things that matter about good data hygiene and good data curation and provenance, they all still matter. They matter even more now than they did before, even if they're not the big shiny thing everyone's excited about. It's still super important.

When we talk about LLMs, this is my favorite picture of what an LLM does. This was the art director I work with. I explained it to him and then he made this. I'm like, it's perfect. I'm never changing it again. What we do with LLMs is essentially take all of the text in the underlying training data, build a compressed representation of the relationships between tokens, and then have a generative function that can generate things. Of course, it's much more complicated than that. I like to just put it down so we have something to refer to, and say like, what is it? We manage the attention to learn the relationships between tokens. We do all kinds of mechanisms to try and make it not be horrible. These are generative representations. What this means is that LLMs are aspirationally an engine for producing generally mid content. That is what they are designed for at best. They take everything in the data and they make something that looks like what is most common in that data. This is a vast simplification, but I told you I like to call things what they are. It is really misleading to say it is a PhD level or whatever. Also, it has a lot of that information in it.

This is another good, evocative metaphor. Science fiction promised us cold logic artificial intelligence. Instead, we have this thing that doesn't quite do what we ask for and is kind of drunk. Again, with a lot of love, when Copilot first came out, I just called it my drunk pair programmer. We jammed. It is great. Also, it's not what we have been brought up to believe AI would look like. All of our metaphors for this are broken. How do we grapple with something that can pass the LSATs but can't count the number of r's in Strawberry? I know we can do that now, but there are always other questions we can ask.

The point is that it can do things that we natively think are incredible intellectual achievements. They are things that have been done before, largely, but it cannot do other things that a child can do. We don't have the metaphors to understand and have a good mental model of what it is. People start to make flawed decisions. They become vulnerable to certain marketing hype. That makes it very hard as technologists to grapple with the reality of what we are building and what matters. We mean a lot of different things, those words collide. Every time I walk in a room, I ask people what they're talking about. Usually, they think I don't know what I'm talking about. I really want to make sure we're using words to mean the same things. I encourage you to ask these questions.

Why Are AI Products Hard?

Why are AI products actually really hard? This is the surface area I wanted to dig into for this talk, because it is the place where everything collides and we start to see the seams and the cracks show. It is really easy to do AI-driven software engineering. It's easy to build agents. It's easy to put a wrap around ChatGPT and make a product there. It is really hard to do this whole thing well. People are forced to think about it in only one slice at a time. A lot of us live down at the data model layer. We're working on algorithms. We're collecting data. We're making that available for analysis, for systems. We're building products. We have an AI API and chat layer, which is where most of our big tech AI companies sit right now. We have an experience layer, people trying to build vertical products on top of the technology. We have distribution. I'm running a video game company now in entertainment.

Distribution is much harder than making your product and all. I'm going to talk about that. We have business: business design, business leadership. That's where boards and executives sit. When you see those headlines, like CEOs investing in AI, even though they're not seeing returns, it is because they're thinking at that high organizational level. They're not thinking about the full stack changing all together all at once. That's why it's hard, because very few of us get the luxury of thinking about it this way.

Definitions and misunderstandings. Again, this is more of the like call it what it is. AI does hallucinate. Some hallucinations just happen to be factually correct. I think this is a good way to frame it so everyone understands that all outputs are essentially equal. The model does not know if it is lying to you. It is incentivized in some cases to do so. As we consider how to work with and what infrastructure is needed around these things, it is good to have that in mind. This does create product and experience problems. AI Darth Vader with a fake James Earl Jones voice was saying really inappropriate things in a Fortnite environment.

Then the other one, which probably has made it around the enterprise AI community a little bit more. Air Canada was forced to honor a refund policy its customer support chatbot just hallucinated. That is good. It means that if you're designing these things, you should expect that to happen. AI is biased. These models definitely do compress the relationship between things and the training data in a way that creates a biased representation of the underlying truth. This is one of my favorite tricks. I am not a dude, probably named Matt, with short hair and maybe glasses. This is just Midjourney. Any of them will do the same thing. You can go ask them, what does this look like? This is, again, the most mediocre output. No offense to all these guys. There's too many of them. Look around the room. This is a beautiful, diverse room of brilliant, creative people. This is not the only kind of bias. One of the issues that I think is underappreciated when we talk about bias, it's easy to focus on the obvious. The bias is also about mediocrity and a lack of taste, and a safe middle or shallowness, and loses a lot of the richness of what we create when we think about art. Is this an awesome painting? It's not my taste.

How does this filter up to product? These are my hot takes. These are the things you can feel free to argue with me about, or you can argue with me about everything, on the internet, out in the hallway. These are strong opinions. Chat is a terrible interface for software. I am not sure it is a good interface for interpersonal communication either. It seems to work well enough for us with all of the other information we're able to convey. We have ended up in a moment where because of the wild success of ChatGPT, which the model existed for some time before the product was built around it. It was the product that made it take off. I'm sure you all remember this. It wasn't even that long ago. That means that everyone who is figuring out an AI product is starting with chat. It is a terrible way to get anything done.

Humans fundamentally have not changed that much in a very long time. This is an old post. It's a decade old, but it's still my favorite Reddit post about why humans have not changed even as our technology has. Is, essentially, I have a device in my pocket with access to the entirety of human knowledge and I use it to look at pictures of cats and get in arguments with strangers. We are the same creatures we were before. We are social creatures who look for social validation. Chat as a single player experience is leading to problems. It is not just a problem for actually effectively and reliably doing work. It is also a problem for people's psychological well-being and health. We should pay attention to that. Try to build products that are pro-social, that help us connect with other human beings. That help us go outside. This photo has nothing to do with AI. I just took it in the park, and wanted to show you how beautiful New York can be for that one hour when the snow falls and before it gets dirty.

Another thing that comes up a lot are agents. What are agents? This is a metaphor chosen to make you think you're talking about a competent decision-making person. They are systems that take action on a person's behalf. When an investor says agent, they're seeing dollar signs because they see software they can sell the same way we sell labor. It's an economic statement. When an engineer says an agent, we're talking about 1000 LLM calls in a trench coat with context management and process management. We can think about, where does the human judgment fit in? How do we manage that context to be able to do really good work? There are now really wonderful examples of agent systems that we can use and build on and talk about that, too. It's trying to understand, what is it?

Where Does This Leave Us in 2025?

Where does this leave us in this moment? We're still searching for the right user experience for AI. I think if you take any one thing away, other than I'm going to go get some coffee and think about building great products, it is that no one has found it yet. It is not chat, it is something else. What it is I can't say. Here's an example from a story game on Hidden Door. We work with authors. We make role-playing meets fanfiction style games where you get to be a character. You can do anything. How do you show that to someone? The very first version of this we built, you could just type in whatever you wanted your character to do in the world, and people got writer's block. They got anxiety about their poor spelling. They didn't know what to do.

If you show them three examples of the kinds of things you can do in this moment, one of them is hard, one of them is easy, one of them is a free action. People are like, I can either click one of these or I can type what I want. This is the kind of UX trick that we are all sort of discovering independently. I'm so excited for one of you to figure out the ideal form of this, because it's not chat. It requires a two-stage submission. There's another affordance here that I'll talk about in a bit around safety and keeping Darth Vader from saying horrible words to children. It is still really hard to figure this out.

We have to think a lot about the quality. Whether we're in enterprise or startups, these questions will be entirely different. These are just examples. This is not formal advice. You always have to ask, what vendors are we using? Are we locked in to anyone? Can we easily swap in new models and validate them? What do our evaluations look like? I can share that at Hidden Door, every story somebody plays on our system goes through an evaluation step where there is both quantitative evaluation and qualitative with an LLM matching criteria that we can update evaluation. I get an alert if a story goes below a certain level of quality. I hope that those robust evaluation metrics are part of every conversation. What guardrails exist to keep this thing from hallucinating new policies we're going to have to honor or saying something inappropriate or not up to our spec? Or even just failing.

If anyone is using many of these vendors, you know that they don't always have the best uptime. How much does it cost to operate? A lot of the time, people build prototypes and then the prototypes are really cool but they look at like, we're making a thousand calls to a very expensive model and it ends up costing more than we're gaining in utility out of this prototype so let's shut it down. When really there are other architectural paths you could go for the same utility and functionality without that cost burden. Managing through that requires knowing what you're doing. I'll talk about that again in a bit. Then because this is again the frame on product, how do people feel using this product? Here's another example of me trying to make a picture of you all via AI. This one is Gemini. I am an equal opportunity critic so I love and pick on everybody. Can you make me a picture? It gave me this gray box that was loading, and so I left it in the little window for a while, and I went back and it was still loading.

Then I think I went back like an hour later and was like, you ok? Gemini is my buddy. They're like, "Yes, I'm great. I made this image for you". I'm like, "There is no image, it's just the gray box". It's like, "Yes, I put the placeholder image in there". I was like, "Can I actually have the image?" It's like, "No, I don't do that". I did not feel good in this interaction with this product experience. Again, I feel safe punching up a bit here at Google's team because the underlying models are fantastic, but the product experience is confusing. Like I know what I'm doing and I know what I'm asking for, and it's still taking me three tries to be told that, no, I actually have to go to another interface where I probably have to log in again to actually do the thing I want to attempt. Yes, I pay for this. This also isn't the free version that they're just putting the scraps of compute on.

This is the moment where product is the hard part and the experience is the hard part because we have to bring together all of these different pieces at the same time. All of our old metrics for what good looks like may not be appropriate anymore. Like, is number of chat messages the thing we're optimizing for? Is it the most or the fewest? Because I can think of cases for both. Here's an example of, I had my kiddo at the Met Museum. In the Egyptology wing they have an amazing exhibit open right now on divinity. 9-year-olds get bored, and he pointed at some hieroglyphics and was like, what does that even say? I'm like, I don't actually know. We took a picture, we asked ChatGPT, and it told us what it said. I know it is wrong. It is 100% wrong. It doesn't matter. It's like, it's a funerary poem. It served the purpose in that moment. Accuracy didn't matter. Directional accuracy mattered because it gave us something to learn and get excited about.

If we really wanted to do more with that, we could go learn it, consult an expert. I bring this up because everything we have learned in machine learning, precision, recall, accuracy in thinking about product analytics like daily active users and how many interactions does someone have? We have to question all of these metrics for evaluation as well right now. I think this complexity is why we get these product experiences that just are not joyful. The challenge right at this moment for people building these systems is context management. What I mean by that is really, what information are you using for what task and what purpose? What model are you hitting with it? Where does the response come out? What format is it? How is that verified? What system does that go in to? How do we manage that whole process? When you think about whether you call it an agent, several queries in a row that share context, things happening in parallel, being reintegrated.

This is one of those problems where we are in that exciting moment where there's some really cool open-source work, lots of people doing homegrown work because those things don't quite map to their problems. A few general-purpose systems. I'm a huge fan of Claude Code because of the context management, it is not the underlying model, are doing it well for a purpose. We've built one at Hidden Door around content creation, not for LLMs to write the content for our games, but for our narrative designers to sharpen their vision and then find existing content we already have that can be used in new stories.

Here's just my little bit of what is working in context management today. This is very much a work in progress. It is asking questions that we didn't have to ask before. What metrics even matter? How are we going to know if someone is happy with the experience they are having? What guardrails do we use and why do those matter? Are we using RAG? Are we using a database? Do we not even care? Are we just like, take the wheel, let's go. Are our architectures flexible enough to work today, but also adapt when things change? Because the rate of change is very high. I have been building the technology at Hidden Door for over 5 years now. I can tell you that we have vastly simplified aspects of our original architecture.

Originally, our whole system could run without LLMs. We do use them in the critical flow at this point, but we could still yank them out if we wanted to. Having that component-based, task-based architecture lets us very easily swap things in. How are we going to manage our operations? What is the difference between a prototype, a production-designed architecture, and something that is actually running operationally? How do we keep that going and understand when it's going off the rails? This is one example I wanted to share from our work, only just a very practical hands-on keyboard example. How do we handle people who want to swear or have their character do wildly inappropriate things? I'll give you a bit of context.

At Hidden Door, we work with authors, filmmakers, people who are incredible creative world builders, fictional worlds, and we make fanfic role-playing games in those worlds. We use a ton of machine learning and AI to do this. Each of these creators has a different sense of what is appropriate for their world or not. We have a romance writer who really wants very tasteful lovemaking scenes, and she writes them very well and wants that to happen. We have the Wizard of Oz, which is appropriate for kids, or like a Scooby-Doo meets Call of Cthulhu go on a wild adventure to crack some mysteries. Really fun for younger folks. Each world has its own standards. How do we make sure nobody types the standard format of like, I F around and find out? Here's an example of what happens when you try that in a world that does not allow it. We have some worlds that do allow it. Our system first has a database of somewhere around 40,000 English-language words and phrases, each of which is also enriched with metadata.

If you type something like a coffee cup, it knows that a coffee cup is a vessel that contains liquid. It is a small item. It is usually made out of this set of materials. It shows up in these genres of works that's from this time period for technical development, all in a database. It's the most boring thing. Then we check every bit of language typed in against that database. If we don't know what it is, we run a little quick translation into PG-13 English and check that against the database. You will get, I mess around and find out, if you really want to try that. It then presents it back to the player and says, is this what you mean? As a side effect, for one thing, we do not have the problem of people doing inappropriate stuff.

For another, we have players playing in languages that are not English using translation. They can type in whatever language they want and the little translation process translates it to English for them. They can read English and they can submit what they want, which is a huge affordance that we had not planned for at all. This is also super cheap to run and very effective. I love databases. I highly recommend very simple approaches like that. Here's another example I wanted to share because I am also a big fan of NotebookLM and its RAG on top of documents. Again, working on a game, we put in all of the now public design docs from The Sims from 1997, and we can ask questions about it. It is great. It pulls in the citation of where in the document it found the information. I assume you all know this is a great tool. Go play with it.

The other question I would encourage you to ask if you're thinking about going from, we've made a prototype with a big, expensive LLM and now we want to see what we can get the cost to, is to say, can we turn a generation problem into a ranking problem? By that I mean, can you pre-generate almost everything you need and then use embeddings to say, this is the thing that is most relevant to the thing we're looking for. We use this trick everywhere. In our Hidden Door system our operational costs are actually very low because much of the content already exists in a generic form. Then we just do a very little LLM pass at the end to make it very custom for the current moment of the game. This does require knowing what good looks like on the product side. It is the post-prototype phase.

What Do the Teams Building These Systems Look Like in 2030?

This is one that I think is important to consider as senior technologists. The technology itself is changing what it is to be a senior technologist. You are more important than ever because writing code in the syntax doesn't matter. Being able to figure out what a great, high-performing team looks like now will be the difference between excellence and mediocrity. I wrote an article about this in a data science context a long time ago, but I think the principle still applies, which is that we all know what failure in bad teams look like. Hopefully, none of you are dealing with that right now. If you are, there is better out there. Go find it, create it. Mediocrity and excellence often look the same for a long time. It is figuring out the difference that is something I'm very obsessed with in terms of team design. I always say that the job of the CEO is in large part to make sure that the machine of the people are able to do their best work. Being a great engineer has never been about knowing syntax. It is also something that is not, I think, very commonly understood.

It is the difference between saying I'm hiring a Java programmer versus I am hiring somebody to do this work and we use Java for this. Once you know N languages, you should be able to work in any of them. The syntax was never the important part. This is Codex. I love Codex too. Great tool for writing very well-scoped little bits of code. The code is not the hard part. The question is the hard part, the problem statement, the answer is usually trivial. Navigating through a technology career today is very different than it was before. I would encourage you to look up this paper. It is a paper in which these folks took pairs of programmers working in the same company on the same team, one of whom adopted GenAI programming tools enthusiastically and one of whom did not.

Then they looked at what their differing attitudes and ideas were about using the tools in their workflow. I don't think any of this is all that surprising, but it is always nice. I always look for the things I already know to be true and then I look to learn the things that question my assumptions. You find that it is people who think about realistic ideas of what they can do and not do, keep trying, who don't give up quickly, who are a little bit more resilient and flexible. These are probably traits we tend to look for anyway. It speaks a bit to like, who are the people who are succeeding with this tech and aren't in the same organizations, which is super interesting.

I did want to take a moment and just say, we are having a collective existential career crisis in engineering at the moment. It is the same thing I've been talking about, all of this tooling means we can be incredibly productive. I mean that in the sense, I think it is better for people like me who are in a job that is largely around context switching than it is if I were still in a full-time hands-on keyboard engineering role. Because it is the ability to form really good questions, to know what good looks like when you see an answer to the question, and then to coordinate all those pieces coming together, that is the skill here. Whereas in many organizations, like I was talking to a firmware engineer who was like, I'm a staff technologist at a major company that makes big GPUs that we all use. I used to be the cool kid because I just knew everything about the language. I knew all the little tricks and the optimizations. Now all the junior engineers are able to do all that stuff, and I'm not cool anymore.

That is the existential crisis of the profession in the moment, because good judgment, systems thinking and design has never been more important than in a world where you can have 12 things implemented and they can all be crap. You need to know which one is not. You need to know what good is. You need to model good for the systems to be able to mimic that. We are not currently able to articulate that. Our hiring processes have not broadly adapted. We still have people coming in thinking their job is in this box.

Then you say, actually I've torn the box away. Now you have to think about the user experience. You have to make all these judgment calls. This is probably obvious, but it is, what makes a great engineer today? I'm going to say it comes down to that product focus, that ability to consider the software they're writing in the context of the broader complex system, in the context of the team they're on, in the context of the business they're working within and what the overall goals are. To think about that poor person at the end who just really wants a picture for her slide presentation and now has to go through four steps of being lied to. Please.

This is really exciting because I've been working as a technologist for 25 years. I got my first full-time adult programming job at the end of 1999. We have seen changes. We have seen the whole big data phase. We've seen changes in our infrastructure. I even gave up using vi six years ago. We have gone through a tooling process where we sit in organizations, how important we are. We've seen the differences in philosophy between people who are data first versus people who are software first.

By that, I mean probabilistic mindsets versus discrete engineering mindsets. It looks like you're doing the same thing but you are not thinking about it the same way. How do we get you to work in a harmonious and happy accord? Now we have this new thing and we have to answer questions, like, what does it mean to be a junior software engineer in this world? What do the best engineers of the future look like? Maybe they're not the people who can memorize every command line option for awk. Maybe they look different. How do we get there and what do great teams look like? Again, this is a presentation of questions and provocations and not answers. Because I don't think any of us have all of the answers, but it is one of the most exciting times to be an engineer ever. I did try to sum this up. If you're going to tweet something, this is my fortune cookie. The idea is saying we need to be able to give people trust, autonomy, that they can handle that complexity, and we need to be really rigorous in our process for how we manage our machines. They're different. It is still super cool.

What Can You Build When You Rethink the Business Across the Entire Stack?

This is going way back to the beginning. What can we do when you can rethink everything across that entire stack? I will say as someone who has worked in large organizations and startups, it has also never been a better time to build something yourself. If you need to hear it, you can hear it, because anyone who can think in those complex systems ways right now has a really big opportunity space. There's so much unexplored surface area, new business models, new product experiences.

The cost functions of things are changing. It's a really fun time to be doing this stuff. I'm going to tell you what I'm building because I think it's great. You can personalize things. You can generate content at the moment someone experiences it rather than pre-writing everything. You can have single-use software. I think back to when I got married, I had to hand write a script to lay out the postscript for the place cards just the way I wanted it and I remember thinking, I don't know how anyone gets married if you can't program, but now everyone can do that. We can have context come from our environments, from our digital environments, from our physical environments that can make our software more responsive to what we want. So much fun stuff you can do.

What we're doing at Hidden Door is creating a space where we work with creators. We want to pay our favorite authors more money, our favorite filmmakers more money because we love the worlds they create and want them to keep doing that forever, and so we license those worlds from them. We create automatically with our narrative design team these role-play meet fanfic experiences in those worlds, and then as people pay for those, we give money back to the creators. We work with a ton of amazing authors. This is our sci-fi panel we just launched. We also have a bunch of romance and a whole bunch of holiday stuff coming including a Charles Dickens Christmas carol. You get to make a character. You get to share it with your friends. This was our Comic-Con ad campaign. I was very much hoping this wasn't actually illegal. I wasn't sure but we were taping them up all over the Javits Center. You get to play in different worlds in story which is a mix of mostly text and a little bit of art. All of our art is created by actual artists and it's assembled dynamically by an algorithm. We don't use diffusion models. We make a lot of really fun content so those worlds keep living.

This is just one example of a business that has become possible because of the technology. It takes the joys of playing a game like Dungeons & Dragons and gives you the ability to do it on your phone, and then quickly make a little story moment, send it to a friend who can build on that and send it back. We are having a blast and this would not be possible without this tech. I know it is weird and fun, but I hope you're fans of things so you get it, and if not, go build your own awesome stuff because we are in this moment where also we have this choice of lean in to the exploitative side of AI. I'd say, don't do that, go the other way. Try and build really great things. It is a really exciting moment to do that and to think about what it means to be a technologist building great things for the next 5 years, 10 years.

See more presentations with transcripts