Why AI and Drones Won’t Bring the Apocalypse

Prefer to read? Check out the Audio Transcript

Zachary Karabell: What could go right? I’m Zachary Karabell, the founder of The Progress Network, and this is my podcast. And it’s called What Could Go Right? because we’re trying to look at, well, what could go right, given that we live in a world where everybody is chronically looking at what could go wrong.

And we are going to talk today with someone whose topic area is absolutely suited to both of those questions, and who has addressed one of the pressing issues of our day, namely: Will all this new technology kill us? Should we be as scared as many of us are?

Now, that doesn’t seem like a “what could go right” question. That seems like the opposite. But it is a question posed in light of, are we overdoing our sense of doom, and is that clouding our ability to see a sense of possibility?

And this emerging world, not just of AI, but of drones, which current conflicts over the past couple of years have really shown autonomous technology du jour. I mean, we talk about Waymo and we talk about cars, but drones really have solidified a clear sense of what this future world is going to look like, particularly in areas of war. Although, as we’ll talk about, not just that.

So today I’m going to speak with Sarah Kreps, who is a professor at Cornell University, was once upon a time a U.S. Air Force officer, who was around early in her career when the Predator drone was first deployed.

She’s written a number of books, including one that is forthcoming called Harnessing Disruption: Building the Tech Future Without Breaking Society, which is a great subtitle. Great title, actually. We’ll talk a bit about that, even though you’ll have to wait for the book for a bit — unless you’re listening to this a bit later, which is also possible.

So on that note: Hi, Sarah.

Sarah Kreps: Hi, Zachary. It’s great to be here.

Zachary Karabell: You’re kind of in both the academic and professional business of trying to assess not just where we are with a new technology, but where we’re going.

And one of the things you’re grappling with — or the world is grappling with — is rapidly changing, expanding drone technology, and you fuse it with rapidly changing and expanding artificial intelligence technology. Not necessarily the LLM aspect of artificial intelligence, but really the different AI agents. And you’ve written a lot about this and thought about all this.

And we’re having this conversation also on the heels of this very bizarre, I don’t know if you’d call it a battle, but something went on, obviously, between Anthropic, which develops the Claude AI models, and the Pentagon, and Anthropic’s concern that its models would be used in ways that it, in a corporate sense, found unethical by the Pentagon in terms of what’s targeted.

One of the, I don’t know if irony is the right word, but the fact is, even though that happened before the Iran conflict, Claude’s models are embedded in Pentagon systems anyway. So it actually was used that way, whether or not either Anthropic or the Pentagon wanted to.

Given that the models were used that way anyway, is it too late? I mean, are there any guardrails? Do we need guardrails? What would guardrails even look like?

Sarah Kreps: Yeah. I found that debate between the Pentagon and Anthropic — which is obviously still going on — to be a bit surprising in a way because Anthropic became involved with the Pentagon and with this company, Palantir, which is a largely military contractor organization, in 2024.

So for the last year and a half, they’ve been developing Claude to be integrated into a military system. And so it was a little bit surprising that then they said that they weren’t comfortable with whether this would be used in lawful ways. So I just wondered what they thought was going on when they got involved in this project.

And I also couldn’t help but think about something that had happened in 2018, when Google had been in a somewhat similar situation. They were developing what was called Project Maven, which is now the Maven support system that Anthropic supports, and Google had essentially gotten out of that contract because 3,000 Google employees had said basically what Anthropic said this time: We don’t want to be in this business of things that we don’t think are lawful or that we’re not comfortable with.

Zachary Karabell: And weirdly enough, this makes me less negative. Human beings develop tools. If you’re the person inventing one of those tools, the idea that you can actually predetermine what uses those tools will be used for once they’re released into the world, I just feel is silly.

Sarah Kreps: I think I agree with you. I don’t think that’s how Anthropic sees it. I think that is exactly the point, which is they want that control, and I think that’s why Google got out of that business in 2018.

So Google was able to get out of that business because there was no one else to step in. And one of the things that’s, I think, different now is there are so many AI companies that this makes it tricky for Anthropic because they might say, hey, we think that we can do this better, more ethically, more legally. But that sort of principled view of it only goes so far, because as they stepped away, OpenAI says, hey, we’re happy to do this. And if OpenAI doesn’t do it, there are five or 10 or 20 or 100 others that are willing to step in and do it.

So I think that the whole landscape has changed so much in the last, whatever that is, six to eight years.

Zachary Karabell: But isn’t it a little bit like if you and I had invented a light bulb and I said, I don’t want to sell my light bulb to anyone who’s going to turn it on between the hours of 12 and 2 because I believe firmly that human circadian rhythms mean that you should be asleep in the dark then?

Sarah Kreps: Yeah. I think that’s like what it’s saying. But I think this is why, in a way — and I think we’ll get to this — I wrote this book, because it did remind me so much of what happened with the nuclear project 80 years ago, which is why I think it’s very interesting and ironic that Dario used to give new employees Richard Rhodes’s The Making of the Atomic Bomb book because he also saw these parallels.

And I think that they think that they can do what the Manhattan Project scientists did. And there are debates about how much they were able to maintain agency over what happened once this product — your light bulb was, in this case, the nuclear weapon — was released because there were a lot of scientists who said, we don’t want to use the atomic bomb on Japan. That’s not what we got into this for. But the U.S. government said, it’s not your product anymore. This is now in our hands.

And then at some point there was a debate about whether to develop the hydrogen bomb. And so what I think is very interesting is you see these similar debates. You see these scientists who have these ideas, and they get involved in this kind of path dependence, technological innovation, and then they kind of get to this point where this is captured, in a way, by a government agency who is much more powerful than they are because — and I think this is what we’re seeing with the Pentagon — the national security imperative is a pretty powerful trump card.

And so what can Anthropic say? And this is how the courts have viewed it, which is, well, national security, and that then basically ends the conversation. And so I think, in a way, the answer, if these companies or individuals are uncomfortable with it, is just not to get involved with it in the first place. 

I think where Anthropic thought they were different is that if this was going to be developed, they wanted to do it better. They wanted to do it more ethically. And that’s where they ran into this problem at the end, which is they developed what they hoped would be a better and more ethical product, but then there were these questions about this end use and whether Anthropic could say, you can’t use this light bulb between 1 and 4, and the Pentagon says, you gave us this light bulb. What do you mean we can’t use it between 1 and 4?

Zachary Karabell: My take has always been, if any scientist who was working on the Manhattan Project was under any illusions that this weapon wouldn’t be militarized, they were engaged in a massive act of delusion and denial in the service of really wanting to pursue this line of scientific inquiry. You know, they wanted to see what could be done to harness the power of the atom. But if they thought for one minute that that would not then be utilized by the military … . And there were people who told themselves that story unequivocally, who were involved in the Manhattan Project.

And as you referenced, there was a subsequent debate with Edward Teller. Oppenheimer was on the other side of that, losing, about the development of the hydrogen bomb. And of course the argument then was, if we don’t, someone will, therefore we should, which is exactly the same debate now about AI, right?

Sarah Kreps: Mm-hmm. Mm-hmm.

Zachary Karabell: If we don’t, the Chinese will, and the Chinese are like, if we don’t, the Americans will. And if Anthropic says, well, if we don’t, then OpenAI will.

The only point in human history, I think, where there was a collective act of recognizing we’re not going to develop certain things was certain conventions around biological weapons that have somewhat held.

Sarah Kreps: Yeah. I know there are a lot of debates about that in different disciplines. So is it a question of whether that was an ethical decision, or the kind of rationalist, economics argument would say the militaries just did not find any utility to biological or chemical weapons. 

And, you know, we saw this in 1915 when these militaries were in trench warfare and these chemical weapons are blowing back into their own soldiers’ faces. That’s not a very useful weapon. And so maybe that doesn’t undermine the argument, which is that these militaries — whether they were moved by ethics, or whether the ethics were reinforced by the absolute inutility of these weapons — maybe that doesn’t matter. But I think you’re right. These are exceptions.

Zachary Karabell: No, I mean, that is a good point. It’s one thing to say we all saw the ethical wisdom of not using chemical and biological weapons, which you kind of want to be the truth. As opposed to, if we had figured out how to use them without blowback — literally, in the case that you just talked about — we would have. But we couldn’t, so we didn’t.

Sarah Kreps: Yeah. How much does the mechanism matter? I guess the mechanism matters because it points to whether this can replicate in other cases. And I think the reason why nuclear weapons were different is that they were such a deterrent that you couldn’t get other countries to agree not to develop nuclear weapons because there was a real consensus that even if you don’t use nuclear weapons, there’s a deterrent value to having them in an arsenal. And I don’t think that that’s true with chemical and biological weapons.

Zachary Karabell: So what about drones, and what about drone swarms? The assumption now, I think, is we’ve unleashed this new autonomous technology, which is going to be infused by AI, that we’re loosely calling drones, that can be developed by non-state actors, state actors, less expensively deployed for huge amounts of disruption for little amounts of effort.

And whether that means a country like Ukraine being able not just to credibly resist an invasion by Russia but potentially turn the tide, I mean, who knows? We’ll see. But there are some indications now that that is going to have another chapter that’s mostly drone-driven.

But if that becomes ubiquitous, you could imagine a scenario where human beings start realizing there’s no defense. The only actual defense is some sort of collective agreement not to use them. Do you see that as a possibility?

Sarah Kreps: I think that’s more likely when you have a scenario that is so overwhelmingly destructive the way nuclear weapons were. And I think the problem with drones, and I would say AI as well, is that, A, it’s really hard to ensure compliance. Back to your circularity of, well, if they’re going to develop it, we’re going to develop it. And if we can’t know that they’re not going to develop it, well then we have to.

I mean, a little bit like the Elon Musk thing where he said, let’s do a moratorium on AI development, and everyone sort of scoffed at this, and then he said, well, I guess if no one else is going to observe a moratorium, we’re going to accelerate our development. And this is, I think, again, why AI governance is close to impossible. You know, it’s in no one’s economic self-interest to issue and observe any kind of moratorium on AI.

And I think that’s true for drones too, and that the costs are not consequential enough that they can deter use the way we saw with nuclear weapons. And so I think those differences are really important.

Zachary Karabell: Or maybe just not yet. Meaning, if you knew for a fact that every individual, every state, every group had sufficient access to lethal enough drone technology to make your life miserable, and that there was no sure defense against it, you might be incentivized not to do it in the first place.

Sarah Kreps: Yeah, maybe. But I think that what we saw — and I wrote about this in the 2010s, about these international organizations that were prohibiting the transfer of the kind of U.S.-style armed drones, and actually that worked fairly well. But what happens, I think, in these kinds of agreements is the unintended consequence is it basically pushes development below the threshold that was prohibited. And so everyone’s like, well, if there’s this MTCR — the Missile Technology Control Regime — that’s saying we can’t produce this large type of drone, we’ll just kind of produce and manufacture and make ubiquitous this smaller drone that is also very capacious. And so that’s exactly what happened to the industry.

Zachary Karabell: Yeah, I guess I’m projecting some sort of near science-fiction future, which is probably going to be the reality in 10 years, where there are just so many drones between delivery drones and agricultural drones. I just finished this book about corn as a technology we eat, and drones becoming fertilizer drones, and planting drones, and seeding drones. And you have delivery drones throughout China. You have delivery drones throughout the world. You have transportation drones. Like, if everybody’s got a drone, and all those can be weaponized, and you can’t defend against them…

I mean, maybe it’s a similar argument to Second Amendment purists in the United States, right? The idea of, well, if everybody has a gun, nobody will commit crimes because you know that the person you’re potentially going after has a gun. There’s no real evidence that it works that way, but the argument’s out there.

I mean, I hear you. You’re saying basically no, that it won’t work that way. But we’re still kind of at the early stages of just massive, ubiquitous drone use.

Sarah Kreps: Yeah, I don’t know. On the other hand, and this is where, in my book, I just arrived — because for the longest time, whatever I was looking at, drones, nuclear weapons, AI, I was such a doomer. And then I looked at all of the doomsday prognostications, including my own, and realized actually none of them had come to pass. And I started thinking, well, what happened? Why not?

And I think, coming back to your question of, we’re going to live in this world of ubiquitous drones — you know, when I was working on this, I guess more than 10 years ago, everything was about, we’re going to have delivery drones, and how is the FAA going to regulate airspace? And we’re going to have these collisions and this and that. And we’re not, I don’t think, anywhere close to that, much as what we’ve read about with flying cars for years and autonomous vehicles, Waymos and things like that. Now they are taking off, but nothing ends up being as fast as we think. And again, this is kind of what I get at in the book, is that there are two things that happen. One is I think that these doomsday prognostications basically mobilize a lot of resistance, so it stalls that acceleration. But another reason is often that it turns out there are much more efficient ways to do things.

For example, I wrote a lot in 2013 and 2014 about how we were going to see drones just going around and assassinating people. And I quoted the late Dianne Feinstein, and in a hearing she said, these would be the perfect assassination weapon. She meant in a civilian setting. I can’t think of a scenario where drones have been used in that way, and I think it’s because, just crudely, there are much more efficient ways to kill people than using a drone and strapping on some explosive and trying to kill one or two people.

So I think both of those things are at work, and I think it comes back to this thing of, a lot could have gone wrong, but there are a lot of reasons why that’s not what happens.

Zachary Karabell: That’s funny. I was going to ask you the, how does one sleep at night question, when you’re focusing on all the worst-case scenarios. And obviously there’s a whole series of people — people who are focused on counterterrorism, biological weapons, nuclear proliferation — who spend their time both gaming out worst-case scenarios and trying to plan contingencies for them, and also monitoring traffic in real time. I’ve always wondered: Did you have trouble sleeping when you were most in that? I mean that both metaphorically and literally. Like, how does one go about one’s life as if there is a tomorrow worth living for when you spend all of your time focusing on all the possibilities that tomorrow is either not going to happen or going to happen very badly?

Sarah Kreps: That’s a great question. I was in the military and then in the intelligence side of the military around 9/11 and the Iraq War, and so I thought about and wrote about this a lot — that you need to think about all of the scenarios that could lead to an attack.

Because I don’t know, probably a lot of listeners were not around or don’t remember that, but after 9/11 there was — and for years — this question of what would be the next way that there could be a terrorist attack, and that you really had to get imaginative about what those scenarios are to be able to guard against them. And so that was my intellectual world for many years.

You know, I think the reason you can sleep is it almost becomes — and this isn’t good or bad, I don’t think, it’s probably bad — that you become almost clinical about it. This is a problem you’re trying to solve. And it becomes an intellectual pursuit about solving that problem.

But I wonder if it was so many years of, again, waiting for the worst to happen and it not happening that led me to a much more optimistic conclusion, which is, you can think about the infinite ways things can go wrong, but those don’t seem to happen, and there must be a reason why.

And maybe it’s that imagination and the thinking and the planning and almost, in some ways, the scare tactics that cause us to put our guards up, and that that’s sort of a salutary process of ensuring that we don’t end up in those existential crises.

Zachary Karabell: Yeah. I mean, I’ve wondered this post-9/11. Why weren’t there more low-level attacks? Why weren’t there a bunch of people with machine guns in malls?

If you wanted to think of something that would have been hugely disruptive and presumably relatively uncomplicated — like it didn’t require logistics — and if the people doing it were not eager to live, then even fewer obstacles to doing it. And yet none of that happened, right?

And it just led to this question that I feel has never really been answered, which is, we do, as you just said, spend an immense amount of time focusing on all the what-ifs in a negative scenario, and we plan accordingly. And maybe that planning, as you said, is an ingredient in them not happening. But of course we don’t really have a good cognitive explanation of why things don’t happen, right? We don’t really understand why if something bad happened once, it doesn’t happen again. Because we don’t, I think, really understand — it’s much harder to chart the human instinct not to do bad stuff than it is to examine the human potential to do bad stuff.

I mean, did you ever sit around and have these conversations of why aren’t people doing all these things that we think they could do, that we’re planning they might do, and yet they haven’t done?

Sarah Kreps: Yeah. I had that conversation with my dissertation advisor, where I was just so worried about some sociopathic person using a drone to kill a bunch of people. And he said, they have much better ways to do it. They don’t need a drone to do that.

But I do think another ingredient, which we don’t tend to — as a society or media or intellectuals — articulate much, is that maybe there just aren’t a lot of nefariously minded people out there trying to do bad things. You know, the better angels of our nature. I found that kind of compelling, and I know it’s controversial, but this argument that we’ve gotten more “civilized” over centuries. And I’ve been reading a lot about European history, and it really is so striking that for centuries Europeans just killed each other, in the millions.

Zachary Karabell: Regularly. 

Sarah Kreps: All the time. And it’s kind of remarkable that that just doesn’t happen anymore.

Zachary Karabell: Yeah. 

Sarah Kreps: And that’s great.

Zachary Karabell: You’re totally right. This sort of waking up and going, wait a minute. We’ve done a lot of really bad shit as a species to each other at regular bases for a long time. And it has not led to these doom points nearly commensurate with the fear of them.

And here we are again, where AI is kind of the catch-all in the moment, and then when you attach it to certain physical technologies like drones, you have this supercharged sense of, okay, now we’ve finally gone and done it. We almost went and done it with nuclear weapons, but now we’ve finally gone and done it. And you can’t prove a future negative, right?

Sarah Kreps: Yes. Yeah.

Zachary Karabell: There is no argument that settles the argument about the future.

Sarah Kreps: Yeah, exactly. I think this is the thing. I was talking to a colleague who wrote a book about financial crises, Dan Drezner. It was called The System Worked. And basically these mechanisms that were put in place actually did a pretty good job to prevent another financial crisis, but no one likes that argument because if you’re an optimist, people can just say, well, the worst hasn’t happened yet, so you don’t know that you’re right. And so I think that there is a risk.

The one thing that I wonder — and I was in D.C. last week, and someone was talking about the guy who coined the expression “cyber 9/11,” kind of this catastrophizing language about how we were going to have a cyber 9/11. Well, years passed, and of course it didn’t happen, and he said his regret was that he felt like he sensationalized it unnecessarily. And I think it does raise this question. Was it the sensationalizing that actually is the reason why it didn’t happen, or were the ingredients just not there because what does that even mean? And I think it’s really hard to tease out which of these things is really operative.

Zachary Karabell: Yeah. I mean, I love Dan, and I did a piece for The Washington Post a couple of months ago about why the financial system was surprisingly stable even with this massive oil shock and fertilizer and supply shock in the Gulf. Now, I wasn’t saying stable meaning stuff doesn’t go down. I was saying stable as in, this was not a 2008 moment where there was a legit fear for a period of time that the entire system was going to fracture. It’s like a power grid. It’s one thing to say it’s being strained. It’s different if it actually goes down completely. And the problem, as you just talked about, is it gets the immediate blowback of, yeah, yeah, right.

And nobody wants to be that person saying — like the famous economist Irving Fisher in, I think, October of 1929 — that the stock market has reached a permanent plateau of prosperity or something like that. You don’t want to be that person who proclaims, you know what? We’re good, like two weeks before we’re clearly not. And that should not be a reason not to do it, but there’s always a human professional reputational instinct of, well, I really don’t want to be that guy who people look back on and say…

And weirdly enough, there’s some incentive to be the guy, like The Big Short people. You’d rather be the guy — and I’m just using it because it’s often guys — you’d rather be the guy in their mother’s basement in their underwear forecasting the end of days who turns out to be right than be the person like Dan, or in this case you or I, who’s saying, you know, probably not, and be wrong.

Sarah Kreps: Yeah. There’s this famous book in international relations by a guy — I mean, this was the early 1900s — Norman Angell. And he said that the economic interdependence between countries was getting so tight that neither would have any incentive to go to war because economically that would just be disastrous. And then a couple years later, World War I happens, and these two countries that were so inextricably linked economically are now killing each other in trench warfare. So yeah, you don’t want to be that guy either because a century later we’re still talking about how dumb that prediction was.

Zachary Karabell: So who’s more pessimistic, you or your students these days? Because you teach at Cornell. You have a lot of undergraduate students and graduate students.

Sarah Kreps: Oh, that’s a great question. More pessimistic or more optimistic?

Zachary Karabell: Well, I mean, it’s basically the same question, right?

Sarah Kreps: Well, I don’t know. The question is so good because now I’m wondering whether they are just mirrors of what they think is the right answer. But since I tend to be somewhat optimistic, that’s sort of what I seem to get back from them. But no, I guess there are people…

I had a student a few weeks ago who was saying that the student did not want to use AI for anything because she was worried that the AI was going to steal her idea, and then her Ph.D. dissertation would now have been scooped.

Zachary Karabell: Huh.

Sarah Kreps: Which was a very pessimistic take. So yeah, I guess I think it’s sort of like a distribution, and so there are people at the tails that think of things that probably don’t worry me at all.

Zachary Karabell: So when you think about those questions — we’ve talked about it a little with Anthropic, and you just referenced it, about the fears of, I guess, the sentient AI. Although people who are really talking the most Armageddon view of AI talk about a sentient AI whose sentience is not understandable to humans. It’s not human intelligence writ large. It’s a whole other thing.

I think people who are really in the most doomer camps talk about it as, we could no more understand the real artificial general intelligence of the AI that’s forming than ants could understand us. So it’s not just like they’re a really smart human. It’s like they’re a whole other thing. Which I at least find an interesting thought experiment, right? Like, what would that be? What would that look like? Would it be indifferent? Would it have its own dictates that are totally distinct from any programming dictates that humans have created?

How do you think about these things in terms of governance? Is governance overrated? Is it just this kind of — as we talked about a bit between Anthropic and the Defense Department — is it just this sort of post hoc human attempt to go, ooh, wait a minute. Didn’t quite mean to do that?

Sarah Kreps: Yeah, yeah. So I think these debates are so interesting, again, because they are somewhat recurrent, but also it seems to me that the proponents of certain positions are not always doing themselves a service. And I’ll give you an example of Dario, who in 2023 was talking about the end of days that would happen two years later, in 2025. And then in 2025, when he was testifying or speaking, he said, oh, no, no, no, it’s 2027. And what I think that does for the debate is take what I — and I think you started here with the unknown and uncertainty — it suggests that there is so much uncertainty with new technologies that you can’t even…

And people talk about the — it sounds arcane — but the difference between risk and uncertainty. Risk is something where there is a probability distribution that you can map, and I think the challenge with new technologies is there’s so much uncertainty you can’t even map this with a distribution. But I think there’s a fallacy in these individuals, the CEOs speaking as if there is a way to map this, and that they can speak with certainty about, well, two years from now. When you’re moving the goalposts, it’s not a helpful way to frame the conversation, and it’s not a helpful way to mobilize attention. And I think in some ways that has been a problem with the climate change debate, is that if you catastrophize an outcome and talk about all the things that can go wrong — and it comes back to this kind of type one, type two error. Would you rather be predicting the worst and it doesn’t happen, or predicting that it’s going to work and then something doesn’t work out?

But I think the challenge is, in terms of governance, you need to build coalitions of reasonable and serious people, and if your catastrophized predictions do not come to pass in the timeframe that you have stipulated, then you’re just not going to be able to get the coalition together to do something meaningful in terms of guardrails.

So I think that that has happened in the climate change space to some degree, and I think there’s a risk of that happening in the AI space, is that if you’re predicting doom and gloom and now this timeframe, this time horizon, comes and goes and we’re not anywhere close to that, then I think people start to kind of move on and think that that threat is not very serious.

And so I think that’s, for me, the important thing is, a sober take in trying to ask the right questions, not just talk about these paperclip scenarios of the AI being programmed and somehow now taking over the world because it’s trying to optimize paperclips.

So I think those are the risks and why you don’t get meaningful governance — not just because it’s really difficult to do in terms of verification and compliance compared to nuclear weapons, but also because the debate hasn’t developed credibly around what is the credible and verifiable risk.

Zachary Karabell: I think you’re totally right on about that. And this thing of false certainty being rewarded more than honest uncertainty, which is threaded throughout society. It’s threaded throughout academic advancement. A scientific paper is likely to get attention because it makes a conclusive statement about an uncertain set of data rather than a nuanced statement about an uncertain set of data. A warning about future problems is likely to get more attention if expressed in stark binary terms than if expressed as a series of probabilities. 

All these things mitigate against the humility of uncertainty and privilege the arrogance of certainty about future outcomes that are by definition unknown. We should all be humble about the future in the sense of, unless you can show me a crystal ball, a quantum computer, or some mechanism of future prediction that is near certain, then we should all be humble about what we think is going to happen and be aware of the fact that none of us know and we’re all trying to make our best guess. 

And you’ve articulated that beautifully in the sense that there are also risks to that certainty when it’s presented because it crowds out other potentials and it has behavioral effects, right? If we’re so certain about impending climate doom, either you do nothing about it because you can’t do anything about it if it’s certain, or when those things don’t come to pass, people who would otherwise be willing to do something that probably would be ameliorative and mitigative are like, well, you were not right about your future negative scenario here.

So what kind of things do you think are realistic? And I guess the other question is, you have the agency of being informed about something and to articulate a vision of, here’s what good AI governance would look like, here’s how we should understand the role of technology, here’s what we should be worried about with drones, and here’s what we shouldn’t. 

You know, you’re one person with agency and ideas, which is incredibly valuable. Some of the challenge is when you start empowering groups — whether it’s governments or industry coalitions or others — to determine the guardrails, those get determined, but they don’t necessarily work.

So how do you square the, you’ve got good ideas, but you’re also highly aware of how limited or how problematic it is when even good ideas get implemented as law or policy or you name it?

Sarah Kreps: So I think one of the venues I so distinctly remember in the AI governance space that I was part of that made me realize not the futility, but just the challenge of addressing this question, was this big tech company that was doing some responsible AI stuff. They had 30 experts in the room, and we went around and each said what we thought was the most important issue. And there were 30 different issues, from deepfakes, to bias and discrimination, to labor market displacement. And what it made me conclude is that AI is such a broad and generalized technology that part of the challenge was that these are all 30-plus different problems, and that we were so at the beginning, the fledgling stages of even understanding this technology, that we almost needed to fracture and take it as subcommittees. Okay, this is the group that needs to focus on deepfakes. And now you’re going to focus on bias and discrimination. And these are all very different things. 

So I think — and again, taking the long view from having looked at a longer history of technological innovation — that we are still so … even though AI has been around since the ’50s technically, in terms of the consumer and ubiquity aspect of AI, it’s basically like three years old. And so I think there was a sense that we needed to fix this right away without even understanding what we mean by AI and what the problems were, and that it was so heterogeneous depending on the issue.

And I think, I write about this in my book, about the European Union AI Act, that I think now largely looks like it was too early, and there seemed to be a virtue to taking a very preemptive, precautionary approach. But meanwhile these worst fears have not come to pass, but the European Union has fallen so far behind on AI, and AI has become a huge economic engine. By some statistic I read about last year, it was accounting for 90 percent of the United States GDP growth.

So that’s, I think, the dilemma. Not that there’s one right or wrong approach to this, but that it’s trade-offs, and that different people and different countries are going to weight these things differently.

Zachary Karabell: So final thoughts. If we’re having this conversation in — I was going to say 10 years, but it feels like five years is already the outer envelope of reasonable speculation given where these things go. But, are we having a radically different conversation? Are we having the same conversation with some different details?

The assumption is all this is going to move incredibly fast, and you and I both have lived through the birth of the smartphone, the efflorescence of internet web technologies in the ’90s, and all that is definitely rapid and quick.

Although, you know, I grew up in New York City. I lived for most of my life in a relatively Leave It to Beaver, Mayberry neighborhood. I don’t mean the neighborhood was like that, I mean the square footage was like that. I lived in a very small section of Manhattan for most of my life.

You know, most of the buildings, some of them are different. Human beings walking down the streets have different clothes. The cars look a little different. But the essential reality of human nature has not completely shifted. So I wonder if we’re still overdoing — yes, these technologies shape things and change things, but I don’t know that we’re going to be as radically other as we seem to think. I don’t know. Where do you go with all these thoughts?

Sarah Kreps: Yeah. No, I don’t disagree at all. I think that a lot of these technologies have kind of an S curve, hockey stick, whatever, where there is a steep part.

I don’t want to be falsely certain or, like I want to be humble with my predictions here, to heed your admonition. But I look at what’s happened since the end of, what was that, 2022 or 2023 when ChatGPT first came out.

Zachary Karabell: ’23, I think, yeah.

Sarah Kreps: And I think there have been huge bursts of uptake and improvement. I just have a hard time believing that we are still in the very steep part of this S curve. So my prediction would be that five years from now, even 10 years from now, our conversation would be recognizable in ways that would not have been true, let’s say, 10 years back. I really do think there was a before and after this commercial AI. I even feel like agents are not the game changer that the uptake of AI and Claude and these different tools were in that 2022, ’23, ’24 timeframe. So I mean, I guess we’ll see. We’ll have to meet back up and see what we’re talking about.

Zachary Karabell: And see what it is. You know, it’s funny, thinking about that made me wonder too. It was November of ’22, so almost ’23, that Chat came out with its first model.

All these things have clearly changed a lot of our lives between the ubiquity of the web, the smartphone as a compendium of human knowledge and a ubiquity of human connection at our fingertips with immediacy for almost everybody in the world, everywhere in the world. That has clearly shifted an aspect of human existence — our awareness of each other, our ability to be aware of each other across great distances in the moment instantaneously. You know, COVID as the first human global crisis experienced simultaneously with an awareness of how everyone else was dealing with it simultaneously. It’s obviously too soon to figure out what it all means, actually. We’re living in the midst of it.

My first trip to India, I was complaining to somebody that I had done no writing, and I was in my journaling phase, and this very odd but sort of famous wise guy who I had ended up sitting next to on the plane said, “You can’t write about India while you’re there. It’s like writing about a dream while you’re having it.” I just remember that very poetically.

And so trying to make sense of all this in our present is probably a fool’s errand. We won’t know till we know.

Sarah Kreps: Yeah, but I think that’s where actually going back and looking at historical patterns can be really helpful. And there are a lot that I don’t cover in my book, but I think about something like the iPhone compared to a non-smartphone. That’s the sort of before and after.

And yeah, it improves dramatically, but now everyone laments, like, is the 17 that different from the 16? Was it that different from the 15? And even with nuclear weapons, I don’t want to dismiss the qualitative, quantitative difference between a hydrogen bomb and an atomic bomb, but I would still submit that the biggest change, the step-function change, was between having a nuclear weapon and the world before that. So I think a lot of technologies have that flavor to them.

And I think that’s true with drones as well, that the conversation — and again, I’ve been working on this for many years — even though these new Iranian low-cost loitering drones are sort of different, they’re very recognizable to what the U.S. was using in the early 2000s. And the biggest difference was weaponizing the Predator, not these iterations that came after.

Zachary Karabell: I want to thank you for a really cool, wide-ranging, unusual conversation. I very much appreciate your own journey of getting to a point of investigating your own assumptions of doom, which does not mean that you’ve stopped looking at all the challenges and problems. It just means that the perspective shifts, which is what I’m trying to do with these conversations. So I certainly appreciate that.

Sarah Kreps: Well, you’re doing it really well, and I appreciate being part of that conversation.

Zachary Karabell: So I want to thank you all for listening. As always, I know you all have a choice of what to do with your time, and choosing to do this is an honor and a privilege, and I hope you will keep coming back. Tell your friends, tell your family, tell your pets. They too can enjoy it when you are not.

Always interested in feedback. Go to theprogressnetwork.org, and there is a tab there.

You can email me directly through The Edgy Optimist site on Substack. Please sign up for that newsletter as well.

Thank you to the team at Kaleidoscope for producing this and to my team at The Progress Network for also making this possible.

And we’ll talk to you next week.