#7: Katja Grace on the Future of AI and Insights From AI Researchers
Where AI is heading and what thousands of AI researchers think about it, including analysis of likely the largest-ever survey of AI researchers
Katja Grace, Lead Researcher and Co-Founder of AI Impacts, joined the podcast to discuss where AI is heading and what AI researchers think about it, including analysis of likely the largest-ever survey of AI researchers.
Available on YouTube, Apple Podcasts, Spotify, or any other podcast platform.
Our music is by Micah Rubin (Producer) and John Lisi (Composer).
Highlights
Why AI May or May Not Pose an Extinction-Level Risk
Katja: The important thing that it seems like AI might involve creating soon is something like new agents that are not humans, that have their own goals and direct themselves in the world. If we do that, it really matters what those other entities want and what they can do. I think very plausibly they can do a lot. […] The experience of playing chess against an AI these days is very much one of being soundly thrashed. Or you can play computer games against AIs and they're pretty agentic at getting the thing they want in a computer game. But a computer game or chess is very small compared to the real world. But with more progress, I think you can imagine a situation where you're playing real life against an AI that's well honed and has some particular goals. You just have no chance. And so I think if you had a bunch of agents like that in the world, then it really matters what they want. And I don't know that we're on a course for them to necessarily want things that are good in the long run. […]
And then why might everything be fine? I think the biggest question for me, as I was saying, is probably the “what do they want?” It might be that it's not very hard to have them want things that are relevantly close to what humans want. I think it's pretty unclear how close is close enough. […]
Is there any reason to think that any particular AI system will manage to take control of the world? I think not necessarily. It seems like if you have relatively continuous progress and there are a bunch of different AI systems at any point and they're being used by different people to do things, and it's not like almost any of them are getting to spend all of their thinking cycles on how to take over the world, usually they're being used to do some other thing that they need to actually do in order for people to run them. So that's a different class of reason to not be so worried. […]
I've said we're going to create these new agents. I feel like in general, maybe we don't have a really good model of what we're talking about there. […] You might imagine that you could just make a creature that would kind of do what you want and then sit there and not try and take over the world.
How Much to Defer to AI Researchers' Opinions
Katja: I think the AI researchers are more right about these things than the general public who aren't thinking about them at all. I still expect them to be, like, not very right about them… for it to be quite a noisy signal. Because even predicting when your own project is going to be done is notoriously difficult. And predicting when your field's projects are going to be done when you're not even involved with most of them is going to be quite a bit trickier than that. And predicting what the social consequences are is clearly not what they're trained for. But still, they do have more information about it than many people, so I think if you're looking for who to listen to, they're a reasonable bet. […]
I think if you're pretty ignorant, you might be like, “oh, AI seems pretty risky and maybe it'll cause huge destruction.” I think it's kind of natural then to be like, “ah, you don't know anything about this, probably all of these experts working on it are doing so because it's reasonable and safe, and surely they know what they're doing, and they wouldn't do this it was dangerous.” So I still think it's very interesting to hear from those people “no, it seems like it's pretty dangerous, we don't know what we're doing.” Even if you don't think that they are great judges of it. It's kind of like if someone's fixing the brakes in your car and you're like, “so are you very confident that this is going to be fine?” And they're like, “no, I actually don't know how it's going to go. I'd say 5% this is going to crash soon.” That's informative even if you don't think they're a great forecaster. It's undermining a kind of faith you might have otherwise had.
Relevant Links
The range of human intelligence (AI Impacts)
AI Is Not an Arms Race (Katja Grace)
Incentivized technologies not pursued (AI Impacts)
Survey of 2,778 AI authors: six parts in pictures (Katja Grace)
World Spirit Sock Puppet (Katja’s blog)
Transcript
This transcript was generated safely by AI with human oversight. It may contain errors.
(Cold Open) Katja Grace | 00:00.798
So are you very confident that this is going to be fine? And they're like, no, I actually don't know how it's going to go. I would say 5% this is going to crash soon.
Jakub Kraus | 00:18.773
Welcome to the Center for AI Policy podcast, where we zoom into the strategic landscape of AI and unpack its implications for U.S. policy. I'm your host, Jakub Kraus, and today's guest is Katja Grace. Katja is co-founder and lead researcher at AI Impacts, and we discuss her perspective on AI, as well as results from likely the largest ever survey of AI researchers, which AI Impacts conducted last year. I hope you enjoy. Katja, thanks for coming on the show.
Katja Grace | 00:57.178
Pleasure to be here. Thanks for having me.
Jakub Kraus | 01:00.385
What motivated you to start working on AI?
Katja Grace | 01:05.003
I was originally just trying to do whatever would cause the most good in the world, broadly. This was actually long before the EA movement existed, so I was also not that much in touch with anyone else thinking along similar lines. So initially I was trying to save up my money to send it to the developing world. And then I got interested in sustainability and climate and stuff in undergrad. And then I came across people thinking about AI on the internet, I guess in 2007 or so. I wrote asking “well, I don't know if this is important, but if you convince me that it's important, then I will help.” Snd so as a result I got to be talking to such people, and I guess I started working on it before I was convinced that it was directly the most impactful thing. But it seemed at least important to figure out whether or not it was a big deal, or in particular was like a sort of extinction risk. Well, I think even without it being an extinction risk, there are likely things in the vicinity that are a very big deal for humanity. But I dropped out halfway through my PhD in philosophy. I joined the Machine Intelligence Research Institute initially as a more normal employee to do the things they wanted me to do. But after a year of that, I started running AI Impacts, which I guess had previously been a kind of in my spare time summer project type thing.
Jakub Kraus | 03:05.183
And how has AI impacts evolved from then to where it is now?
Katja Grace | 03:12.620
The thing we were doing before AI Impacts was… we called it the structured case project. We were trying to make structured arguments for important things that people believed or cared about. There's some claim at the top, and then there are other claims underneath it that support it. And so it's sort of a structured argument type thing. And I think AI Impacts kind of evolved from that. But it's actually really hard to make a structured argument very cleanly and well that includes everything you want it to have. It was sort of like maybe instead of an extremely tight and logical thing we could have sort of clusters of information on pages and still perhaps you're trying to like support some high-level claim.
I guess AI impacts was then aiming to have various high-level claims to do with AI, for instance about AI timelines or something. But then underneath some tentative conclusion about AI timelines, you might have a page about how much compute a human brain is equivalent to in some sense. And under that, you might have several different ways of estimating how much compute a human brain is worth in some sense. And then parallel to that, you might have something about how fast computer hardware is getting cheaper, and that would have more subpoints under it. The idea was this kind of tree-structured argument thing, trying to really work out various important claims to do with what's happening with AI, in the hope that if we lay out such reasoning well publicly that it will help other people think about it. You know, I guess on the broad theory that if more people are right about what's happening, people are mostly good and trying to not cause the world to be destroyed and stuff and will also make better choices.
And I think with getting other people involved, it was sort of more natural to move in the direction of it being more normal-ish research projects. And so it's a bit more in between now, but still broadly directed at what is going to happen with the future of AI.
Jakub Kraus | 05:35.282
And when you were looking through these smaller pieces that seem important to consider—like you mentioned how much computational resources you might need to run a human brain—were there any of these investigations that had results which surprised you or that you found counterintuitive?
Katja Grace | 06:02.368
I think yes, but often they're like... relatively far down the tree. I think it was harder than I expected to make progress on things at the top of the tree. So, I don't know, for instance, we did a big project on discontinuous progress in technologies. And we looked for examples of technologies that people thought had seen big discontinuous jumps in some metric and then looked into them. I think I was surprised by various of those, but that's not immediately relevant to AI risk. I don't know that there were big updates for AI risk for me, really. But yeah, it's possibly somewhat hard for me to be that surprised, because I generally don't have super strong views. I'm sort of like, well, I don't really know what's going to happen, let's try and narrow it a bit.
I guess another one like that is, how long does it take to cross the human range of intelligence? I think people often say that the distance between an idiot and Einstein is actually very small. And it just seems large because we're in the middle of it, as though that's kind of fact. And so we looked into how long it took to go from idiot to Einstein, metaphorically, in various more narrow tasks. Because presumably if it's very narrow in general, you might think that, for instance, in chess, going from an amateur chess player to a grandmaster would be a flash. But that's not true in chess or most places we looked.
Jakub Kraus | 07:48.530
And you mentioned that you feel uncertain about a lot of things. Is there parts of this AI risk forecasting and arguments that you do feel quite confident in?
Katja Grace | 08:08.162
Yeah, probably. I think I would be very surprised if it was somehow impossible to have AI that was substantially better than any given human at any given task. That would surprise me a lot.
Jakub Kraus | 08:19.626
Which is a pretty bold claim to some people. So you're saying that just about everything a human can do, a machine can do one day.
Katja Grace | 08:40.895
Yeah. I guess it may be except sort of corner case type things where it matters a lot that you're a human. For instance, if you're doing something for another human and the other human really cares that you're a human.
I think it's less clear that there'll be one machine that can do anything that any human can do better than any human. I would guess yes, ultimately, but I think that's not as obvious.
Jakub Kraus | 09:13.967
And why is that less obvious?
Katja Grace | 09:18.267
It might be that in order to have all of those capabilities, you actually need to dedicate something like a human brain worth of learning or something to each of the things, such that you couldn't actually be good at all of them without having quite a lot of brains, or at least more than one brain worth of...
Jakub Kraus | 09:37.247
So basically how it might take someone 10 years or 10,000 hours to become really good at violin and another to become really good at computer programming, and so we might need to spend equivalent computations to have a machine be good at each of those as well?
Katja Grace | 10:01.547
Something like that, but not so much that you might need to spend that much time equivalent, but maybe at the end of the practice, your brain is specifically good at violin, and you couldn't actually use the same brain that is so perfected for violin at all of the other things at once that you somehow need. Like, maybe a human brain can't actually fit all of the skills in it once fully learned. I don't expect this to…
Jakub Kraus | 10:37.474
It would be bad news for violinists who want to become programmers, I guess.
Katja Grace | 10:42.915
Yeah, though I think, in fact, we've seen enough with humans to know that you can learn at least two skills. But can you learn all of the skills with a human brain's worth of compute? I mean, my intuition is that yes, you can probably just have very much more efficient use of something like a human brain. But that's much more speculative. That's just my intuition. And so if we're talking about things I'm actually confident about, we're definitely out outside of that range.
Jakub Kraus | 11:26.307
What's a really important claim about AI that you feel very uncertain about? And why might it be true or why might it be false?
Katja Grace | 11:40.369
I mean, I think that the most important one that I'm most uncertain about is whether by default AI is likely to destroy humanity sometime soon if there aren't huge efforts to avoid that.
The reason why that might be true is… I guess the important thing that it seems like AI might involve creating soon is something like new agents that are not humans, that have their own goals and direct themselves in the world. If we do that, it really matters what those other entities want and what they can do. I think very plausibly they can do a lot. They're much more competent than humans at various things and at least competent enough to beat us in any kind of contest. I guess I think of like, I don't know, the experience of playing chess against an AI these days is very much one of being soundly thrashed. Or you can play computer games against AIs and they're pretty agentic at getting the thing they want in a computer game. But a computer game or chess is very small compared to the real world. But with more progress, I think you can imagine a situation where you're playing real life against an AI that's well honed and has some particular goals. You just have no chance. And so I think if you had a bunch of agents like that in the world, then it really matters what they want. And I don't know that we're on a course for them to necessarily want things that are good in the long run.
So I guess in summary, we're probably going to create new agents. That seems very likely to me. It's a bit unclear how soon, but people are really trying to do that. It doesn't seem like it should be that hard. Those new agents will probably be quite capable, at least sometime soon. I think that also seems quite likely. And then the main question is, what do they want? And that seems like quite an open question to me. So that's my basic case for we should be very worried for potential extinction risk.
And then why might everything be fine? I think the biggest question for me, as I was saying, is probably the “what do they want?” It might be that it's not very hard to have them want things that are relevantly close to what humans want. I think it's pretty unclear how close is close enough. I think there's arguments out there that even if you slightly disagree about what is good, that it's worth competing really hard for you to have control over the entire future rather than someone else. Because the stakes are so high that even if an AI had the power to take the world away from human control, just to get a very slightly different utopia than the one humans would have imagined, that it would be worth it, so it would do it, or something like that. I guess I don't super trust that. It seems like at that point, also, you should be worried about any human versus any other human, and you're just kind of already in a situation where as soon as anyone gets a lot of power, then they should want to take total control if they possibly can.
And I think that there's a different class of reason things might be okay. If you just have this general concern that if anyone can take power, they should, then is there any reason to think that any particular AI system will manage to take control of the world? I think not necessarily. It seems like if you have relatively continuous progress and there are a bunch of different AI systems at any point and they're being used by different people to do things, and it's not like almost any of them are getting to spend all of their thinking cycles on how to take over the world, usually they're being used to do some other thing that they need to actually do in order for people to run them. So that's a different class of reason to not be so worried.
Maybe there's another class of reason for me that's kind of like, I don't have a great sense of what being an agent is. I've said we're going to create these new agents. I feel like in general, maybe we don't have a really good model of what we're talking about there. I guess I think people who are worried about extinction risk very often think about the kind of economically rational agents that have a utility function, and then they make whatever choices will maximize that utility function. And I don't think that's a great model of humans, though it's sort of a good model in ways. I think there are arguments about why creatures that are more capable would become more agent-like, and maybe thus be more bloody-mindedly likely to destroy the world to get something they want. I'm not super sold on those, and it seems quite plausible you could make different kinds of entities that behave in different ways. I guess you might imagine that you could just make a creature that would kind of do what you want and then sit there and not try and take over the world. It seems like the kind of thought that leads to worrying a lot about extinction risk says ah, if it's smart enough, it's just very hard to make a thing like that. And I guess I wonder if that's a lack of imagination rather than a thing about the world.
Jakub Kraus | 18:35.531
Yeah, it does seem like we're seeing these large language models like GPT-3 and 4, and they seem to be getting smarter across a lot of different tasks. You can make them an agent version where you have them act autonomously, or you can just have the normal prompt it, get its response, and then it sits and waits for you to give it another prompt. So maybe that would be a vision for this kind of future you're describing.
Katja Grace | 19:09.843
Like maybe you can make the thing that just sits there and responds to your prompt.
Jakub Kraus | 19:14.832
An oracle sort of AI, yeah.
Katja Grace | 19:18.046
Yeah, it seems right. I bet there's been a lot more thinking about whether that's possible that I haven't read all of, but to my knowledge, it seems like an open question.
Jakub Kraus | 19:31.216
Great. Now, let's move into some policy topics. You have been thinking about this for a long time, but it's only really in the last year that the US started paying a bunch of attention to AI in Washington, DC. So do you think there are any aspects of the current policy discourse on AI that might be getting neglected?
Katja Grace | 20:00.442
Yeah, I admit that I'm probably not totally in the loop regarding what the policy discourse at present is. It seemed to me in the past, at least in talking about this with people, that they pretty quickly jumped to the conclusion that we're in an arms race. I guess “we” can mean various things, and the arms race can be various different arms races. But that there's an arms race between America and China, there's an arms race between the different AI companies. And I guess that seems to me… I hope people are thinking about it more carefully at the moment, and I haven't checked. But at least casually it seems like a kind of slip of thought, where it is true that the various parties are racing each other. But I think an arms race is a more specific strategic situation where it's actually in the interests of each party to race—maybe at great risk to everyone, but it's a kind of tragic situation where still, from your own personal perspective, you should race.
I think that's very unclear in the current situation. Because I think it depends a lot whether going faster is very risky to you and everyone. For instance, if when you “win,” it’s very likely you die, then that's a different payoff matrix than if you get to take it all if you win. So I think the payoffs at the end depend a lot on the risk that everyone dies, or whatever they cared about in the future gets destroyed, but also things like if the whole time any kind of efforts at safety get shared between everyone, for instance. And I don't know what the risk of destroying everything is or should be treated as, or how much safety progress is made each year and all that sort of thing. I’ve made a kind of low quality spreadsheet model of this, where it seemed to me like it could kind of go either way. You can potentially change the parameters of this, right? You could come to some sort of agreement to both not move ahead but also to be putting more effort into the thing that would save you, like the safety research, or something that would mean that whoever wins in five years probably actually gets to win.
Jakub Kraus | 23:07.586
And is your perspective that the main bad part about this arms race narrative is that it encourages people to go quickly at times when they should be a little more cautious?
Katja Grace | 23:20.592
Yeah I think so.
Jakub Kraus | 23:22.232
If you had these leading AI companies say, okay, well, we want to prioritize safety because we think we're pretty close to this really dangerous AI. And we're going to… maybe there's some collective agreement between the top ones to hold off. How long do you think that should be?
Katja Grace | 23:49.713
I think until. And all those sort of reasonable consensus that proceeding was very unlikely to destroy humanity. It's probably not like a time window so much as a let's get the technology safe and then proceed window.
Jakub Kraus | 24:16.059
Okay, yeah. Seems sensible. But the biggest backlash people give is that this is impossible—what would you say about that?
Katja Grace | 24:27.981
Yeah I'm not sure why they think that… I think there's one model around that's kind of like “Technology! You can't really slow it down, it just has a certain place it's going, and it just does that.” And a different model that’s like, “Well, that's not always true, but it's true if there's a lot of money involved.”
My impression is that if you actually look at the world, technology has slowed down a lot all over the place, including extremely valuable and lucrative technology. For instance, the whole field of medicine is very important to lots of people. There's a lot of money in it. But the FDA does manage to cause things to go more slowly sometimes. It seems like there are whole industries of quite interesting and probably valuable in ways research that we just sort of don't do, like genetic engineering of humans. But it's not like there was a massive well-resourced push to prevent it to my knowledge. I think there are quite a lot of places that research has slowed down because of even relatively small risks. As we have a list of more of them on the AI Impact Wiki if you want to see it.
Jakub Kraus | 25:57.955
Okay, nice. Next, I wanted to turn to this 2023 survey you did with 2,778 AI researchers who published recently peer-reviewed research in one of six top AI venues. And you also did similar surveys in 2016 and 2022. So the first part of all these surveys asked when 32 different tasks would become feasible for AI systems to do. How have researchers' opinions on that changed over time in those surveys from 2016 to 2022 to 2023?
Katja Grace | 26:45.958
I should know the answer between 2016 and 2022, but I actually don't off the top of my head. I know that the overall high level machine intelligence number stayed pretty similar between those years.
But between 2022 and 2023, most of these things got closer. Like, not just a year closer because a year passed; the actual date that people predicted moved up. On average, it was only like a year closer. Like over one year, it became two years closer. But some of them had quite a big jump. Like I think writing a good novel got quite a bit closer.
Jakub Kraus | 27:34.757
Yeah, that part stuck out to me. Some of them are write a high school history essay that would get an A and no plagiarism flags. And are some of these tasks not things that AI can already do?
Katja Grace | 27:53.979
Yeah, I think they probably are. We just asked all of the questions the same as previously, rather than trying to judge which ones had actually happened. Partly because judging which ones have actually happened is surprisingly tricky often.
My colleague tried to figure out if the Starcraft one had been achieved a little while back, when Starcraft was in the news. And I guess it's sort of complicated by like, does it have to use a video to look at the screen. I guess the questions that people are predicting about are when the thing will be feasible in the sense that a lab could do it soon, so it doesn't have to be that the thing described has literally been done; it's whether it's easy from where we are. And so that's a sort of harder question to answer. Though, yeah, if it's been decisively done, it seems like we should be able to tell.
Jakub Kraus | 29:04.673
And another one of the survey's questions was giving estimates of when AI will reach high-level machine intelligence, it was called, which is when unaided machines can accomplish every task better and more cheaply than human workers. And then you also used a different framing as the full automation of labor, where researchers gave much longer estimates, I think decades longer. What's going on there?
Katja Grace | 29:38.529
I actually don't know what's going on. Yeah, there are various hypotheses. I guess more straightforwardly... In the full automation of labor one, we actually asked them a series of questions leading up to it. So we asked them about some particular occupations and what they thought would be a very late occupation to be automatable. And when they thought that would happen. So perhaps having a sequence of questions where you have to think through the thing might cause you to get a different answer. Which would suggest that that one's the more accurate answer, I think, and that we should be paying attention to that.
A different possibility is the high-level machine intelligence one also says to assume that there won't be any kind of, I forget the wording, but like disruption to scientific progress, which I think is just a mistake on our part to have that in one type of question and not in the other. For whatever reason, we made that choice in 2016 and are just trying to have identical questions. I can't remember why we did it in 2016. My guess is that we shouldn't have, just because it makes them harder to compare. My guess is that that's not causing a difference. It would be kind of wild if that many people thought that there would in fact be a huge disruption to scientific progress. But that is a difference between them.
I think that there's some chance that when people think of occupations being automatable, they just have a different… Even if it means something very similar for all the occupations to be automatable and all the tasks, they bring to mind a different thing. And in particular for the occupations one, it sort of brings to mind actually the world having changed a whole bunch and no one having occupations and AI doing all kinds of physical tasks. Even though literally the two questions are both asking about physical tasks as well, and they're both not asking about the things actually being implemented, just being possible.
Jakub Kraus | 32:25.276
Another noticeable effect was between 2022 and 2023, people's timelines shortened significantly. I can't remember the exact numbers, but it might have been decades for the full automation of labor and…
Katja Grace | 32:37.141
Yeah, I think it was at least a decade even for the HLMI…
Jakub Kraus | 32:44.942
So is that just ChatGPT?
Katja Grace | 32:47.606
I don't really know. I think that would be my guess. Though, heading into running the survey, people around me were often kind of like “oh yeah, this has been a wild year, it'll be good to see the very different results that we get” or something. And I think I was kind of like maybe we will, but maybe they'll just be the same as always. And I think during the year , the feeling around was that people's timelines were shorter, but I think the feeling around was also “oh wow the world is scared about extinction risk now, this has reached the mainstream as a concern.” And I think it's interesting that the extinction risk numbers are pretty similar from 2022 to 2023. Like, if you told me that one of them had changed and the other one hadn't, I don’t know if I could have predicted which one it was.
It may be that the extinction risk numbers didn't change very much because they were sort of already pretty high and it was more like the general public views had not caught up with the views that the AI researchers already had. Like they're already at like 5% median extinction risk or something. But then I'm sort of like, well, weren't the AI researchers also kind of already up with what was happening in AI? But maybe not. Or I feel like ChatGPT… people at least say that it was more striking to the general public, it's not like it was a huge innovation, it was more like this made the kind of GPT stuff that we're already seeing usable to more people. So it's sort of surprising if that updated the AI researchers a lot. I could imagine the sort of change in the tone of the discussion in the world making a thing feel more real if it wasn't previously or something, but then it's sort of weird if the extinction risk numbers didn't change.
Jakub Kraus | 35:01.450
Yeah it's interesting that the extremely good outcome also didn't change very much. So people gave their chances of the effects of AI being extremely good (for example rapid growth in human flourishing), on balance good, more or less neutral, on balance bad, and extremely bad (e.g. human extinction) as the consequences of “high-level machine intelligence” that can do any task that humans can better and more cheaply than human workers. It seems to me strange that you could imagine such an extremely capable machine being closer, but then not think the effects will be very extreme. I would expect people to say, oh, now I can see sort of what this looks like.
Katja Grace | 35:56.548
Yeah, right. As in just like, as it's more kind of viscerally close to you, you expect your image of what's going to happen as a result to change somehow. That seems right.
Jakub Kraus | 36:10.573
The piece about do AI researchers track what's happening with language models before ChatGPT comes out or not… my guess is a lot of them don't. At least they didn't go and play with GPT-3, for example, if they worked in audio modeling.
Katja Grace | 36:31.543
Yeah.
Jakub Kraus | 36:31.920
Yeah. And this leads me into another question, which is how much should we be deferring to the researchers'opinions on topics like the speed of AI progress and the risks of AI? Because a lot of people might not have the time to actually feel like they can give these questions justice, so they might reasonably say, well, I'll just turn to what the AI researchers say. So what is important to be careful about there?
Katja Grace | 37:06.323
Yeah, I mean, I think the AI researchers are more right about these things than the general public who aren't thinking about them at all. I still expect them to be, like, not very right about them… for it to be quite a noisy signal. Because even predicting when your own project is going to be done is notoriously difficult. And predicting when your field's projects are going to be done when you're not even involved with most of them is going to be quite a bit trickier than that. And predicting what the social consequences are is clearly not what they're trained for. But still, they do have more information about it than many people, so I think if you're looking for who to listen to, they're a reasonable bet.
I do think that there are people who are thinking more about those particular questions and are probably better experts on those questions. But I think it's not very legible who they are. If you don't know anything about this topic, it's easier to tell that an AI researcher is at least familiar with AI than to tell which of the other people claiming to think well about it actually are reliable.
I think if you're pretty ignorant, you might be like, “oh, AI seems pretty risky and maybe it'll cause huge destruction.” I think it's kind of natural then to be like, “ah, you don't know anything about this, probably all of these experts working on it are doing so because it's reasonable and safe, and surely they know what they're doing, and they wouldn't do this it was dangerous.” So I still think it's very interesting to hear from those people “no, it seems like it's pretty dangerous, we don't know what we're doing.” Even if you don't think that they are great judges of it. It's kind of like if someone's fixing the brakes in your car and you're like, “so are you very confident that this is going to be fine?” And they're like, “no, I actually don't know how it's going to go. I'd say 5% this is going to crash soon.” That's informative even if you don't think they're a great forecaster. It's undermining a kind of faith you might have otherwise had.
Jakub Kraus | 39:49.415
And turning to the survey itself, how much should people be deferring to its results? So are there important limitations to keep in mind? Especially, I think I've seen people say, well, not everyone responded to this, or only certain kinds of people responded to this.
Katja Grace | 40:10.352
Yeah, I think the concerns about response bias are not very concerning, as I think we made a lot of effort to avoid response bias compared to what you would usually expect in a survey. And we also tried to measure it, and I think we didn't see much reason to expect that any of the interesting conclusions are not what a broader set of people would have thought.
I guess like different things you can look at how differently the different demographic groups responded. Like, what difference was there in their response rates, and also what difference was there in their actual responses. So you know, if you thought that no one over the age of 40 responded and that people over the age of 40 are much more likely to think everything's fine, then you might be like “ah this is a problem.” So looking at things like this isn't going to catch all of the problems—like it could be that the people who aren't concerned about AI risk don't respond and they just look the same demographically as everyone else.
But looking at different demographics… people in industry were substantially less likely to respond than people in academia. But I don't know that they answered very differently. If I recall, people who did their undergraduate studies in Asia were maybe like 80 something percent as likely to respond as other people. So they're somewhat undercounted. And they thought that AI would be markedly sooner on average. From my recollection of different things and intuitive sense of how big they are, rather than having done a lot of calculations about this, maybe that was the most notable combination of a substantial group being undercounted and having a substantially different view. But my guess is that the effect of that would be that the timeline should be slightly shorter. And my guess is that it still wouldn't be that much shorter. It wouldn't change the high level results we're talking about.
I think though that the main concern that people have is more that people are kind of ideological about this, or that there are a bunch of people who are think that AI is a risk and are then going to answer it and then other people don't answer it or something like that. I think there's some evidence about that sort of thing. Like, I think we asked people how much they had thought about both timelines and risk, basically. And both of those are sort of not very common. I think at least 90% of people didn't answer the highest amount of thinking about it. So I think, you know, the bulk of people here are not ideologues. Assuming that an ideologue would at least claim to have thought about this a bunch.
Jakub Kraus | 43:38.869
Yeah, and importantly, then you could look at the median respondent answer, and you would cut out if it's just 10% who have thought a lot about it, whether that's people who think it's totally unlikely to be very bad or people who think it's extremely likely. The median respondent would not represent them, but the median respondent did give like five percent or ten percent to the extremely bad outcome and a little higher to the extremely good.
Katja Grace | 44:18.103
Yeah, that seems right. I mean, you might think for instance that just the 50% of less concerned people don't take it at all, and the 50% of more concerned people behave like normally or something, and so then what looks like the median is really the 75th percentile or something. But I guess to me the takeaway on the human extinction type thing is just that most of these people think there's a really non-negligible chance of this.
Jakub Kraus | 44:49.471
And before we wrap things up, was there anything more about the survey you wanted to talk about or your perspective on AI? Anything you wish I had asked?
Katja Grace | 45:02.030
I guess I think another interesting thing about the survey that maybe hasn't got that much airtime is that we also asked about other things that might be concerning… All these different scenarios. And I think I think I was a bit surprised by how concerned they were by all of them. So there's things like AI making it easy to spread false information like deepfakes, or manipulating large-scale opinion trends, or authoritarian rulers being able to use it… those were the top ones. But I think… setting extinction risk aside, this is really just a lot of substantial problems for society to deal with.
I think it's interesting to think about what it's like for them all to come at once. I think each of them on their own is substantial, and I guess I don't usually give them much attention because I'm more concerned about the extinction risk one, but it seems plausible to me that I should be more concerned about these other ones. Partly because they sort of interact with the extinction risk one as well as each other. I think it's probably much harder to coordinate avoiding some sort of catastrophic use of a technology if at the same time the technology is making it hard to tell what is true and hard for people to coordinate their opinions on something driven by what they care about and what is true rather than what would benefit whoever is paying to direct their opinions one way or another. And if there are authoritarian rulers in the mix there… AI-driven control. That all sounds a bit worrying and maybe it would be good to to think more about that
Jakub Kraus | 47:07.272
So in summary, they saw a variety of different threats as quite concerning. And especially if we're developing really general purpose AI, it might bring all these threats together at around the same time to cause problems.
Katja Grace | 47:30.420
Yeah, I guess I sort of expect all of these things to be continuous rather than, like, occurring one day. So it's sort of not that surprising if they're at the same time. Probably the kind of thing that makes it easier to manipulate large-scale opinion trends is not that different from the kind of thing that makes it easier to disseminate false information.
Jakub Kraus | 47:58.600
And if listeners want to learn more about your work or follow you or find some cool links, where would you recommend they go?
Katja Grace | 48:10.038
Well, AIimpacts.org for AI-related things. If they just want my thoughts on all kinds of things, they could go to worldspiritsockpuppet.com.
Jakub Kraus | 48:21.598
Great. That's the end of the episode. Katja, thank you for coming on.
Katja Grace | 48:28.218
Thank you for having me.
Jakub Kraus | 48:31.931
Thanks for listening to the show. You can check out the Center for AI Policy Podcast Substack for a transcript, links, and more. And if you have any feedback for the show, I'd love to hear from you. You can email me at jakub at AI policy dot us. Looking ahead, next episode will feature Tamay Besiroglu of Epoch AI to discuss the trends driving past and future AI progress. I hope to see you there.