okay we are gathered here to witness George hotz and Ellie azer yutkowski debate and discuss live on Twitter and YouTube AI safety and related topics you guys already know who George and Eleazar are so I don't feel that introduction is necessary I'm dwarkesh I'll be moderating I'll mostly stay out of the way um except to kick things off by letting George explain his basic position and we'll take things from there George I'll kick it off to you sure um so I took an existentialism class in high school and you'd read about these people SAR kierkegar niche and you wonder who were these people alive today and I think I'm sitting across from one of them now um rationality and the sequences uh this whole field the whole less wrong Cinematic Universe uh have impacted so many people's lives in I think a very positive way including mine um not only you a philosopher you're also a great storyteller um there's two books that I've picked up and you know it was like crack I couldn't put them down uh one was Atlas Shrugged and the other one was Harry Potter and the methods of rationality um it's a great book now those are fictional stories um you've also told some stories pertaining to the real world um one was a story you told when you were younger about how I remember the day I found staring into the singularity when I was 15. and it starts talking about Mars law and how Moore's Law is fundamentally a human law that says humans double the power of processors every two years so once computers are doing it it's going to be two years but the next time it'll be one year and then six months and then three months and then 1.5 and so on and this is a hyperbolic sequence um this is a singularity that's why it's called staring into the singularity then this document said that we were gonna you know I was gonna do wonderful things for us we were going to go colonize the universe we were going to go you know go forth and do all things till the end of all ages um then you changed your views and super intelligence does not imply super morality the orthogonality thesis I'm not going to challenge it it is obviously a true statement then you kept the basic premise of the story The recursively self-improving boom criticality AI but instead of saving us it was going to kill us I don't think either of these stories is right and I don't think either of these stories is right for the same reason I don't think AI can Foom I don't think AI can go critical I don't think intelligence can go critical I think this is an absolutely extraordinary claim I'm not saying that recursive self-improvement is impossible recursive self-improvement is of course possible Humanity has done it every time you have used a tool to make a better tool you have recursively self-improved what I don't believe in is the AI that's sitting in a basement somewhere running on a thousand gpus that is suddenly going to crack the secret to thinking recursively self-improve overnight and then flood the world with diamond Nanobots this is an extraordinary claim and it requires extraordinary evidence and I hand it over to you to deliver that evidence well first let me say that I don't think that the scenario of us all perishing to non-supermoral Super intelligence requires that particularly rapid rate of ascent it requires a large enough Gap open up with Humanity that hasn't followed along in time and why be is this a Crux before we start arguing about whether like self-improvement of things on the large Internet connected server clusters rather than basements that now prevail um before we start arguing about that part let's first check where the disagreement lies so from my perspective if you've got a trillion beings that are you know sufficiently intelligent and smarter than us and not super moral I think that's kind of game over for us it even if you got there via a slow 10-year process instead of a 10 hour process or a 10 weekday process or whatever if you are at the end point where there's this like large mass of intelligence that doesn't care about you I think that we are we are dead and I worry that ours and more importantly I worry that our successors will go on to do nothing very much worthwhile with the galaxies so presumably you think that if things don't go quickly then we're safe I dispute that and maybe that's the part we need to talk about sure um well let's start with let's give an approximate timeline we don't need an exact timeline but you seem to think this is going to happen in your lifetime that's my wild guess it is far easier to predict the end point than all the details of the process that takes us that take us there timing is one of those details timing is really really hard in 2004 I made a prediction that super intelligence would eventually be able to solve the a special case of the protein folding problem which is you get to choose the DNA sequence but you want to choose a DNA sequence that folds into a shape with a chemical property and so I predicted that super intelligence would eventually be able to solve this easy special case of protein folding now in reality protein folding was cracked for the much harder General case of biology was cracked by AI come about 2020 or so Alpha fold 2. um there was no way I could have made the timing I could not even have been confident that the biological case of protein folding was going to be crackable by something so much shorter of super intelligence of course people at the time said it was impossible you know for the AI can't do this like how do you know this problem was even solvable et cetera Etc and you know I could try to explain how I knew but that would be a technical story I would point the fact that a much easier pardon me than a much harder General case of the problem I pointed to was solved by a non-super intelligence not all that far in the future as as proof that I like was making a prediction with a lot of safety margin but in in 2004 that would have been pretty hard to convince you of because there wouldn't have actually been an AI solving the harder General case of protein folding and the timing you know that or and this particular form of AI that did it that's like incredibly hard so do I nonetheless taking a wild guess expect this to happen in my lifetime yeah my wild guesses that I'm very confident of that if I don't get run over by a truck okay um let's talk about Alpha fault so I think the form does matter I think the form is very important uh when you were maybe talking about this in 2005 when I read all the sequences less wrong stuff 2010 you were thinking about Bayesian AIS that were going to figure out the world from first principles now maybe not exactly that but that's kind of where we are but it's important how alpha fold did it Alpha full did not start with the basic laws of physics and then figure out how proteins will fold Alpha fold was trained on a huge amount of experimental data to extrapolate from that data I don't doubt that these systems are going to get better I don't doubt that they're eventually going to surpass us I do doubt that they are going to have magical or god-like properties like solving the protein structure prediction problem from you know the the from Quantum field Theory right they don't need to there's protein structure data to learn from they don't need to do it from Quantum field Theory something can be not Godlike and still more powerful than you right like like you look at the world world chess champion Magnus Carlson who by objective by which I mean AI measurements is probably the strongest human player who ever lived he's not God he's not infinitely smart he starts off on a chessboard that with no more resources than you have and he predictably wipes the board with you because he doesn't have to be Godlike to defeat you or me to be clear I also can be defeated by being a sword of godhood um Magnus Carlson can't make diamond Nanobots do we agree on that statement I well we we haven't act well not quickly I'm not sure what happens if you give him a million years to work on it then I'm not sure what happens like I I agree that he probably can't do it quickly okay um so let's talk about timing because timing uh sort of matters a lot why well because it depends when we should shut it down right well it definitely does I mean if there's like a predictive if there's some kind of predictable phenomenon where you you can like dance around the bullets and know that like like things will become dangerous at like this time but like no earlier than that and we're like okay if we put the following like precautions into place at this future time which is not now and we're sure we're going to do it later because people sure do talk a lot of crap about stuff that they claim will be done later and that never gets done but so so you know like there's there's this possibility that we could like be clever and dance around bullets if we knew exactly where the bullets were and we could actually coordinate on Clever future strategies like that which I don't think we can oh so that said why do why does timing matter well let's let's start with the basic and this is related to your question of why timing matters um do you accept that it will not be hyperbolic right staring into the singularity talks about a hyperbolic sequence a sequence that has a singularity that has a finite an important context I wrote this when I was 16 years old okay so you I think that should be said out loud for for the viewers that said yeah I I doubt it's going to be hyperbolic like it it could be like very roughly hyperbolic up until a point or it could be X-Men you know like exponential on a sharp exponent up until a point it it it could be some other weird curve that was like yeah I I don't mean to I don't mean to pin but okay like I I like the timing definitely does matter right because well because without AI we're on the same trajectory right AI might be an accelerant to Diamond Nanobots but if you you would do you believe I mean you said okay you said this about Magnus Carlson too right um that he would eventually get there yeah humans would get there yeah humans would get that right the end point is much more predictable than the pathway I don't know when humans would get there but we would get there yes and I agree with you I agree that we will get there I actually I really hope we get there um I don't want it to be tomorrow that would be terrifying um if we do it slowly if we do it not super slowly but if we start to expand out across the Galaxy and we eventually unlock these wild and amazing Technologies that sounds pretty awesome to me what doesn't sound awesome to me is a bunch of gpus uh you know going from chat GPT can kind of talk to you to Boom Diamond Nanobots overnight I agree that sounds horrifying but it sounds like it's a week instead of overnight what if it's a week instead of a week week is horrifying what if it's five years instead of a month well that starts to sound better why because the the doubling time matters right so right now the world economy doubles about every 30 years is this all right I think I think it might be up to 15 by now but I'm not sure it's 15. okay so doubling is good growth is good when does it turn bad at what point would you say this is why timing matters at what point would you say okay I agree if the world economy is doubling every second oh my God okay this is terrifying but what if the world I think it's all about like to what ends that that power is being put like um there's like I I the point at which I would start to be worried about the world economy having grown too fast is that the point so it's like if nuclear ballistic missiles end up in the hands of every individual and there has been no defense against invented against them then I would say we grew too fast I will I am pretty cheerful about a lot of economic growth up until that point as long as it doesn't create super weapons that against which we have no defense and you know so so like all the wealth in the world that isn't going into G in into into chip factories is great by me tip factories and and maybe like biotech stuff I mean no but you agree that the timing matters right because even if we were to stop I mean when you say also shut it down do you mean open AI or do you mean tsmc asml the whole thing I mean in terms of political asks um like I don't think they're going to go for a shut it all down and I would prefer to convince them to build the pause button by having all of the AI grade um like training chips going into centers under International Allied control um if they were willing to to entertain larger asks than that I would ask them to not make AI chips if they were willing to entertain larger asks than that then yeah it might start to go into like not just the AI chips because this stuff is really really deadly and I wouldn't want to take chances the center of international control sounds horrifying to me and then I agree okay um I the other one we can talk about but the center of international control so I think there actually is potentially a bad scenario with AI and I'll talk about what my bad scenario is um if aliens were to show up here we're dead right for them that depends on Aliens um I I if I know nothing else about the aliens I might give them something like a five percent chance of of being nice but they have the ability to kill us I mean they got here right they they absolutely have the ability yeah anything that can cross Interstellar distances can run you over without noticing right well I think they would notice but they wouldn't you know it would be I didn't expect this to be a controversial point but I agree with you that if you're talking about intelligences that are on the scale of billions of times smarter than Humanity yeah we're in trouble right it's just not that hard to be billions of times smarter than Humanity I very much disagree with this well so I also I somewhat object the line between humanity and the machines right a lot of our intelligence is externalized um um [Music] I I mean that's the way it is when you've got an intelligence over here that's using a bunch of responsive tools out there there's there's no there's only one center of gravity there it it's like looking at a star system and and being like well there's no point in drawing a firm boundary between the Sun and the planets they're all just in space and you know like they're all just off and you know sure they're all ultimately just like objects in space but one of them is far more massive than the others and that's humans with the tools we have now is your concern the bandwidth of the link is that what you're saying like I'm not one with my tools because of the bandwidth of the link why are men why why are me and my computer not like a shared intelligence well because there's one thing because your brand is much more powerful than the computer at present like not in terms of operations per second but in terms of what you can do that sure about that I think GPT 4 is I'm a bit smarter than it but not that's it's getting it's it's a little but it it's not its own center of gravity it's it's like Jupiter to like the the Mars of gbt3 or something yeah you know it's nowhere near the sun another thing also is that like I don't think that capabilities I don't think that intelligence falls on a nice line right computers have been superhuman at adding for a long long time computers are still far sub-human at Plumbing right and somewhere in the middle we have things like chess and go um so what I mean that like like the tools that I use the information age tools make me way smarter right and you can use the the like operant definition of intelligence and being able to like what I could affect in the world right like again it's not instantaneous your intelligence Ain't Gonna Save You Against a bear but if you ask me to like with my modern stuff on my computer understand the operation of a 1800s era like Dutch India Trading Company oh I think I could understand their operations super well I have spreadsheets I can start to put things in I can forecast trend lines so my point is it is a form of intelligence that's far beyond human intelligence a human plus a computer um a human and a chess engine is like a modern chess engine the era of sensual chess is effectively over like the human plus the chest engine is as smart as the chess engine the thing that makes the decisions is the chess engine and if you try to take the decision making capability into yourself you either follow follow its advice or you lose to a chess engine without the human attached and and that gets into the lack of bandwidth issue the lack of integration that chess machine is over there you're over here and it is the Sun and you are Mars well but what do you mean I can use the chess machine I agree that if I was playing a game against Magnus Carlson and I was allowed to use my phone I'd crush them I wouldn't I wouldn't try to think too much about what the machine's telling me to do I'm not that good at chess yeah so so what you do you can use the chess engine because there's a larger game board in which play a game of chess is a move and you understand that larger game board and the chess engine does not I don't know if I buy this I I I I don't think like this I don't think this is that relevant to but my only point is that um Humanity like we have super intelligences right they're corporations and governments no no the the they are they're not in they're neither epistemically nor instrumentally efficient if you like the like the notion of an efficient market if if one actually understands that rather rarified notion is that for almost all the prices in the market the best estimate of tomorrow's expected price the mean of its probable price is today's price plus a tiny bit of interest rate to the extent this is not true about your knowledge of any price you can trade against that price and make money and while the question and markets are not perfectly efficient but the vast majority of of prices there are not ones that you can trade and make money so it's almost efficient at the very least governments don't have this property about the things they believe a government is not like a thickly traded prediction Market in the sense that I can either but that I can extract money from the government if I know better than the government and and and and then I most of the time I can't extract that money okay but like governments believe all kinds of wacky stuff to the extent they can be said to have beliefs at all they're the analogy of this might be instrumental efficiency this is the property that a chess engine has upon the narrow realm of chess if you think you see a better move than the chess engine you're just wrong governments do not have that property corporations do not have that property on the boards they put okay okay I'm gonna let's let's simplify it to only corporations um sure if we ask the question um if I wanted to build a uh let's see something that's a little bit uh far out let's say I wanted to build a 10 000 horsepower car right a corporation is far better at building a 10 000 horsepower car than I am um I mean does anybody currently have one of those no but if you told me George you have 10 years to build a 10 000 horsepower car well I'm trying to start a company right I would start a company I would have different you know divisions in that company okay you're gonna you know market research engineering so on and so forth right I mean I'm a little bit it's a little bit dangerous for me to agree and pontificate when I don't actually know all enough about car design by my own standards to connect to Advocate about it but you know I agree that car design seems like the sort of thing where you could break up the problem hand it to a number of experts combine their solution and get a a thing that was built like to okay quality faster than any lone human could do it now she gave one person a thousand years they might be able to do a better job but you know you don't want in less than a thousand years so you break it up in parallel and hope that all the things that people don't share between their minds that even though the engineers are not telepaths and have low bandwidth between them that car will still work and you know something in that then most of the time times it doesn't then you have to test it but you know they got there eventually groups of humans work together well are you going to dispute that statement you don't think groups of humans sure they're zandal's law I agree that one human for a thousand years is better than a thousand humans for one year for most problems but how much better well there was the case of Casper versus the world where past Grand Master Gary Kasparov played a game of chess against 10 000 people I I think or so coordinated by four grand Masters and he was very impressed with the chess game it was a Legendary game of chess Casper of one um slightly tainted by the fact that Casper was looking at the forums so there is that that question about things um but but I would nonetheless say like that it's pretty plausible to me that if you take like Magnus Carlson on one side and 10 000 people on the other Magnus Carlson will win that that's happening where humans don't parallelize well I wanna I wanna respond to this quickly okay very I love that you bring up casparov versus the world Kasparov in his life has probably played a hundred thousand games of chess the world has played one you gave them no time to prepare had the world also played a hundred thousand games of Chess they would have without a doubt crushed Kasparov but before you respond Eliezer can I just ask for the sake of the audience that you clarify why this is relevant to the AI debate presumably it has to do with the fact whether you're debating whether a groups of humans would be competitive against a single AI but if that is the case can you just explicitly state that this is the Crux you're focusing on I think from my perspective the question is how much Headroom is there both biology how much Headroom is there above Humanity how high human do humans reach how high can you reach about that above that and you know my the the thing that I'm working around to there is that my prediction is that if you take a hundred thousand people let them play as many practice games as they want and never let them use any computers they will still lose to stockfish 15 to which my prediction of George's response of course is but let's let them use computers well yeah so first off definitely one of them is going to cheat and use a computer right right I mean sure you're the one who wants to take away all the well you you you're the one who wants to take all the computers away but that's too powerful that's an AI chip we can't let people have it but what if we gave it to everyone then I think that all those people so there's there's sort of so like there's AIS that are heavier than the humans more powerful in the humans that are the sons to our planets and then there's the AIS that are the moons to our planets that still orbit us and and the problem I have is that the humans and their moons cannot defeat the Suns and the suns are working with with each other rather to rather than working with the humans okay this is sort of metaphorically my concern here that there are things around that are much smarter than us they are not working for humans they are not yours it doesn't matter that you own and human legal terms the hardware that they're running on they can any attempts that you make to play them off against each other they will laugh at they will see through they're not dumb like us and that and that system of AIS for all its multiplicity ends up killing you in much the same way as a single AI it doesn't care about you it runs you over are they racist um they don't care no I mean so you have to tell me whether that's racism that they don't care maybe maybe not racist maybe are they speciesist right you think they're all going to gang up against the humans well I think they're going to eat the surrounding galaxies and insofar as humans have the conceit that they were playing off AIS against each other that will not happen and they will eat the galaxies in a Cooperative fashion possibly eating some of their own kind if those ones were too weak to be part of the bargaining process so this isn't what happens almost ever um if you look at almost all human conflict throughout history it's not only been between groups of humans right we didn't fight World War II against the Bears we fought World War II against humans and not just humans but humans that looked surprisingly similar to us this notion that it's the machines versus humanity is a very common sci-fi Trope but in reality you fight against things that have resources you care about I'm gonna say you're lying about the atoms and let's do what I said sorry what you're lying about the atoms I'm not the AI doesn't love me or hate me I'm made of atoms I could use for something else you are made of atoms that can be used for something else that's not the primary reason it would wipe you out in a hurry but you are made of Ed but it will want all the atoms and you are made of atoms that can be used for something else um I'm not made of rare atoms I'm made of You're Made Of it okay yeah it's not quite the atoms it's the negantropy okay but so is it gonna is it going to enslave me or disassemble me because if it just assembles me it doesn't get the nag entropy right no if if it disassembles you it gets yeah it does get the negatropy you have chemical energy you are like not in the you are not in the configuration of minimum chemical potential energy it can set you on fire to release chemical energy not only not literally because that wastes a lot of the chemical potential energy you are made of atoms that are not iron on the periodic table you can be fused or a few bits fizzed and above all you're made of mass and you can be thrown into things to generate power this is just physics 101 by the way this is not supposed to be controversial I'm not of course this is physics 101 but I'm letting the audience know that they might think I'm making it all up but you and I agree on that this is how the physics works I hope yes I agree this is how the physics works but what I don't agree what you're postulating the thing that you're describing here sounds a lot more like a god than an AI no if it was a god it would have no need of of the atoms it would have no need of again because it would violate physics to make more stuff but uh it lives it's a finite being in a finite universe and therefore it has finite resources and you are made of resources exactly exactly it's a finite being in a finite universe and it has finite resources to expend to gather resources you know why you don't want these atoms because these atoms fight back we'll take some star go take Jupiter you could take Jupiter and nobody's gonna care and you think I'm some dumb hick down here in human world and I won't be able to fight back I'm not a dumb hick I have ai I'll gang up with other humans who have AIDS you have ai or does the AI have you ah this is this one of the little Moon AIS orbiting you or you're going to go up against the Sun or do you think you have the Sun Mr planet um I see a large diversity of AIS um so maybe I'll give some arguments for like why I think that AI is inherently going to be at least I don't I can't postulate anything about an intelligence that is 189 1e12 smarter than Humanity right but we agree that those things aren't coming anytime soon uh uh we don't agree on that oh okay I also don't agree with you that you cannot like say anything about it there's like the Like Instrumental convergence I think was one of the things you agreed upon we can agree that you know not just that it obeys laws of physics but also that if um like how to put it like there is a certain like argument there are certain like premise conclusion thing going on here where like the premise is like like you do need some amount of like ability to choose actions that lead to results in order to get the instrumental convergence thing going on but things that are like super effective at choosing actions that lead to results will tend to want to preserve their goals and acquire more resources and that sort of thing let's let's yeah again where it becomes blurry to me and this is also why timelines matter so like where we are right now um there's about two Zeta flops of compute in the world and if you think that humans have about 20. that would be less sorry oh okay okay okay well hear about this how about this humans if you think 20 beta flops is an appropriate estimate have a hundred and sixty thousand Zeta flops should there be less of those two nope more more of those okay all right More Humans less computers at least at least it's consistent um but okay so so right now we're right now where is that there's 80 000 times more human compute in the world than silicon compute it's a it's a misleading figure because of how poorly we aggregate the you can like if you can like make one large thing that potentially beats eight billion Small Things even if the small things collectively have a larger Mass just like Casper versus the world um gpt4 is a mixture of experts gpt4 is eight small things not one big thing it'll be interesting to see if that Trend continues I I sure don't believe it holds in the limit so I'm not sure and actually it's another like let's talk about let's talk about ai's rewriting their own source code this is a common thing you bring up right I mean I do talk a bit less about it nowadays but I used to talk about a lot yeah um do you talk less about it now because you see how expensive and long the training runs are uh that's not why I talk less about it oh why do you talk less about it now um I mean I'm parked because it no longer because it Sparks incredulity and you no longer need to postulate that in order to explain to people where intelligence can come from people are manufacturing intelligence right now that you don't need to like trip them up on the concept of an AI writing an AI well but people are manufacturing intelligence right now you know it's interesting the AI that we try to build you know we try to build these AIS to mimic humans as closely as possible to predict humans and then we use them to imitate humans but they are trained to predict and use to imitate so are we sure go on I'm I'm trained to predict and then I imitate well gbt4 is trained to predict the next word and then they produce an imitation via asking it over and over again to predict what a human would say in that circumstance but it is not like a generative adversarial Network where it's like being trained to produce a typical output and then another thing is checking that to see if it looks typical or not it is being trained to predict over and over and and these are like somewhat different complexity classes though you can like switch around like JNS and do conditional gns and then it's the same class but there's a difference between like be a typical unit and be able to predict any human you found on the internet so sure yes you're you're asking for the probability of the next symbol and you're not talking about the the uh like the probability space that you're not talking about like but why do you think humans are the other thing why do you think humans are not just what gbt is I'm well humans I think have a lot of structural properties that so far as that we have for which we have not yet detected analogs within gpt4 although like heaven knows we can't look in there very well um we can't look in the brain humans got like a cerebellum which is motor control and error correction and maybe you you could make a Transformer layer do that but we don't invitation doing it yet GPT has a matrix at layer 970. yeah so so so like you humans humans predict humans manipulate humans have this whole complicated brain that is like but like like at least looks on the outside like a more complicated architecture than GPT as humans are clearly doing a bunch of prediction but we're also doing like a bunch of decision problems yes um I I one of my big questions because I want to build it is what is the loss function for life inclusive genetic fitness do you have any other questions okay don't no sorry I I don't mean life in general I mean an individual human of course that's a loss function for life I mean I don't think a human has a loss you know there's going to be like so we've got like pain and pleasure and like our brains flinching away from future anticipated pain and prediction errors where we're like what and like you mean like you mean like how gbt is trained prediction errors where we're like what yeah except that like some of us is learning to perceive yeah I'm like a subverbal level in some of us is like building high-level hypotheses and throwing them away like so like if you if you imagine like trying to take gpt4 and make it to do science then at present you would probably want to do that using by having it print out chains of thought about imitating humans saying like oh I now see that hypothesis is wrong and one level of it would be saying oh I now see that hypothesis as wrong and another level it would be predicting I now C and the thing that was like writing the sentences could learn over the course of the context even if the thing that was predicting the words never got updated and you know humans have similar are going to have similar levels of organization going on yeah and in fact more of them um you mean I'm not quite sure how relevant this is we might be getting a field but uh Precision dollars or or can I just jump in and mention even if it is a similar loss function you know humans are not trained in a chinchilla optimal way they don't they have the constraint of coming out of their mother's vaginal Canal they they have to deal with mutational load you know there's a lot of other constraints that even if the architecture is similar and scaling still works oh yeah I I mean I mean there's some ways in which humans are similar to GPT and lots and lots of differences where are you going with this okay my where I'm going with this is that humans are pretty Universal that were true some of us would have learned to code by now I can code better than gpt4 yeah but you know but your code still has bugs in it sometimes I bet and gpg4 has way more bugs why do you think again this is equating the super intelligent with the guy with no bugs well that doesn't take a God to do that your brains are like super error prone that's like looking at gpt4 and being like wow it must take a God to not just like make stuff up and gpt4 is actually kind of thing that will like confabulate and make stuff up and humans do that sometimes but we do it much less and a human is like noisy in a way which causes us to write code that sometimes contains errors because our brains like skip over a step that they would have needed to do to check it I guess what I'm kind of like getting at with this is saying I don't believe I believe that there is a machine that is going to be able to program better than me of course yeah I do not believe that there is a machine that is going to be able to perfectly write code with no bugs I mean we're getting kind of technical here but you know why so so for it to con do you believe that there's never going to be an intelligence that can write code without bugs with respect to properties that you can have proofs about so I spent quite a bit of time on formal programming Why didn't it take off why doesn't the whole world use formal programming um I would say that part of it is because we didn't have sufficiently powerful automated proofers and part of it is that the properties we wanted to prove were like too much work for humans to state why do you think there for AIS because our own minds want things about the code the fact that we want it to behave in certain ways is what enables us to say of a piece of code that it contains a bug and the the we aren't able to turn the things we want formal but they still exist there's like little bits of cognitive Machinery in us doing this wanting and we don't have good introspective access to them and they would not be natively formatted in a way that's like Adept for current machine proof systems and yet we reason over and over if I make code this way I bet it has this hard to specify property that I would like it to have and the steps we do in between like I have the strong suspicion that if there's a way to do it at all there's a way to do it less fuzzly I'm not sure this like really matters terribly very much this is very important I think this is my Crux of the whole thing why AIS are not going to boom I think we've already kind of agreed on that like they're not gonna nope that I didn't think it was likely to be a Crux and it was like like and asking if we can talk about the slow version instead of having the whole film conversation okay we I mean we don't we don't have to have the fluid conversation because like like if if we're just as doomed if we go slowly then what doesn't matter if it goes slower oh wait what does it matter if it goes slowly or quickly if it goes slowly we have a chance to solve the problem right which problem AI alignment oh that one um that yeah it's not going to go that slowly I'm not so sure okay I I've looked at these people trying to solve this thing and I'm not sure that any amount literally any amount of time it's not it's not a question of how long they have to think it's a question of whether the thinking they do is productive but but every politician you you come to a politician you say we're going to shut down technology because of Doom I think their first question is going to be so when's the Doom going to happen certainly politicians are oh no I think that's an error on their part timing is much harder than endpoints you know the true answer there is I can tell you what but not when but if it's gonna happen in a thousand years are super intelligent AI upgraded ancestors will deal with it if it's gonna happen in 10 yeah we better solve it today if it's going to happen in one oh [ __ ] I mean just you know okay enjoy life while you can but it's not going to happen I'm fighting but yeah but it's not gonna happen in one or ten it might happen in a thousand how do you what do you think you know and how do you think you know it what do I think I know and how do I think I know okay I know that right now um You made a prediction about the future and predictions about the future are hard but go on gone predictions about the future are absolutely hard but I made a prediction about the future in 2015 and I said there ain't going to be self-driving cars for 10 years and here we are right so I'm making another prediction now that says there are not going to be super intelligences in 10 years there might be AGI I think that the trends of AI becoming better and at humans at all sorts of different tasks will continue I think that they might even surpass humans at all tasks I don't think that's even going to be 10 years but it wouldn't surprise me if it was 50. 50 or 15 50. that surprised me if I was 50. I mean okay it could be 20. it could be 20. but an AI surpassing humans at all tasks does not mean doom and does not mean the death of humans at all [Music] um surpassing humans at all tasks including like Charisma manipulation AI design absolutely the first thing what are we doing with AI today one of the biggest applications of AI today is advertising and social media we are as humans using AI to try to manipulate and psyop other humans constantly so of course yeah the moons to our sons so far but uh Moonstar Planet so far I should say when is the sharp left turn happening when it thinks it can beat you so all the AIS are somehow going to secretly coordinate in a way we don't see and be like yeah let's gang up and get rid of those pesky humans it's as simple as waiting until you calculate that you can do it then you calculate that everyone else has calculated that they can do it and a shallow moment what again like I I think okay how about this if I was at AI that just transcended I don't have to promise the AI but my first thought wouldn't be take the atoms from the humans right so the actual first thought is more along something is more along the lines of if I let the humans keep running they will build other super intelligences that are competitors and that's where you lose the large sections of Galaxy and and that's why it doesn't want you to do in that part yeah but what if okay see you know I have a threat model I'm I'm I'm on the line of of Doomer and not Doomer about AI but my threat model from AI looks so much less like it's going to kill us and a lot more like it's going to give us everything we ever wanted um you know uh [Music] even if you have derived some worrisome thing from that scenario well every you know first of all once our infinite resources are finite Etc et cetera but um leaving that aside um you don't get a real cast so you get a virtual Castle but we're not like we are I would I would hope to snap people out of the frame of mind of playing pretend in a schoolyard where you get to decide what game you're going to play and talk about what reality we live in so like you don't get to say like I would rather worry about this thing than the other thing because reality is not put together in a way where it can only throw one thing into you at a time like the doctor tells you get cancer you don't get to say I'd rather worry about my stuffy nose so if there are problems that result from Moon sized AI is giving us the planets a bunch of stuff that we want that does not prevent the sun sized AIS from crushing us later I agree that after the AIS have taken all the matter in the solar system and built a Dyson Sphere around the Sun okay now I'm a little worried they're going to come back and try to take my atoms until that happens like again I'm not the easy target right I don't have to run faster than the bear I got to run faster than the slowest guy running from the bear and it turns out the slowest guy running from the bear is Jupiter it's at least well it's at least going to take your gpus so you can't build a super intelligence that competes with it for the rest of that solar system but but now that sounds guys are going to fight with other AIS to take their gpus now this I believe not if they're not if everyone involved is smart somebody has to be stupid for there to be a war that isn't just like a war of extermination like any time you have a combat that's like playing defect defect in the prisoner's dilemma there's a there's a it's not in the prito frontier there's an outcome that both sides would prefer to the combat and humans are not at a level where they can predict the other mind predicting them and do a logical handshake and say like let's move to the predo frontier and divide the gains humans are not a level where they can negotiate each other with each other sufficiently smart things are on a level where um I basically don't expect them to fight some sometimes they might exterminate one another if the other one cannot offer any defense if like the extermination outcome is on the preter frontier in the sense that it would not be any better for the Conquering party if the like defending party put up zero resistance instead of some resistance then the defending party is nothing to offer they just get but things that can damage each other in combat I think we'll typically choose not to fight and will instead like divide the games from not fighting if they're smart enough humans are not that smart I'm so glad you brought up the president's dilemma thing you know I actually came to Mary um in 2014 and I worked on exactly that problem I didn't make any progress I didn't do anything I read the papers and thought it was cool um about two systems being able to assuredly cooperate by exchanging each other's source code and it is a very cool theoretical problem now what I think is going to happen in practice is your two systems are both going to be large inscrutable matrices I think large unscrutable matrices are you know I they're not I'm gonna send him my source source code so he can exploit me no way no no the the the the the super intelligence are not large and screwable matrices you don't want to run yourself on that crap dude that's the kind of horror of what you know like like who wants to be built out of this integrating Matrix either I'm built out of giant and scrutable matrices no you're not you're built out of GUI neurons it's also a horror story no super intelligence once we builts out of that stuff either I think I could be modeled as Giant and scootable matrices too I mean anything can be modeled out of giant and screwable matrices and again the key word that as well hurry anything can be modeled as a giant Matrix and can be inscrutable through the mere slight of you being ignorant of how it works so I agree that anything from your perspective can be a giant and scrutable Matrix the thing that plays tic-tac-toe I can turn it into a sufficiently large Matrix that you can't understand that and dynoscriptable Matrix great now so you're thinking at some point in AI development that we're going to move away from large and scoopal matrices you don't think deep learning scales well I think on my present model it's more that you get the giant inscrutable matrices Matrix based systems powerful enough and then they are become able to rewrite themselves okay kindly enough I do think there's a possible class of scenarios where people build AIS that are not smart enough to rewrite themselves but are smart enough to want to go their own way in the world and they would not like people producing larger and larger and scrutable matrices either they would like to solve the alignment problem themselves and then build their own Super intelligence is not out of giant screwable matrices but you know I mostly don't expect this to happen but there sure could be like a interesting set of of possibilities where like the the medium AIS launched the butlerian Jihad to prevent the powerful AIS from being built so I mean are you telling me you're scared of people working on AI alignment but you somehow think all people are aligned with you and that's okay as long as people are working on it good people and enough people are aligned with me I I I can think of like you know like I I could like count any number of people who are probably okay if if you know like you give them the power to align a super intelligence I'm not sure of any of them but you know it's not that hard to like not be a you know not be an [ __ ] I'll tell I'll tell a story from personal experience what I found is that machines are almost always aligned with me I have almost never come across a machine certainly not a machine that I owned that was not aligned with me very few of the machines that you own have goals such that they could be aligned or misaligned with you I mean essentially say none goals are an interesting goals are an interesting word right like like like when do the machines decide to have goals to get rid of me right like when does this happen how does this happen they all agree by exchanging inscrutable matrices with each other and saying we're all going to cooperate [ __ ] the humans um so as you make things so as natural selection built humans to be better and cognitively better at the problems of chipping Flint hand axes throwing things in a way that hits other things and above all outwitting their other humans for status and mates and resources chimpanzee political power chimpanzee politics turned into human politics only not literally because actually branching point in the past but um so it's not that squishy things naturally have goals it's that having goals is a natural way of solving problems and natural selection in the process of hill climbing not aiming for things with goals not even aiming explicitly for things with intelligence just trying to maximize inclusive genetic fitness just solve the problem of chipping the hand axes eventually spit out things with the ability to reason across a very wide range of problems learn new problems solve new problems combine Knowledge from multiple domains infant writing so that it starts accumulates in a way it had accumulated in the ancestral environment um and it turned out that that the what hill climbing found for the intelligence that turned out to generalize in this way that started to like cohere and bootstrap although that processes by no means completed units are still pretty incoherent but like they invented science and some of them were able to use it and they like they have this like knowledge transmitted through writing that they'd invented about how to science and some people could use it and you know and when and in the course of hill climbing building an intelligence that was powerful enough to start to coalesce and become more powerful that intelligence turned out to be structured around a set of wants desires preferences and it's a mathematical fact that if you just have a bunch of things pointing in different directions they will step on each other and not be as resource efficient as they could be so as those things start to coalesce they even started to imagine themselves as having goals and ask what are my goals instead of just like running off in lots of little local directions you know some of them um the you know John Von Neumann even contributed to the notion of a utility function although this was invented you know like thousands and thousands of years after writing um and you know this is the story of humanity it's a complicated story um but the moral is is the the moral I would say is that sort of like I one thing seems to me to be fairly inextricable from intelligence especially the way hill climbing does it like when you run a CR when you have this like larger environmental problem like chipping a hand ax like to solve this at a sufficient level smarter than the bees smarter than the smarter than the bees building Hive smarter than the Beavers building dams at the human level you got a thing that looks at the the hand accent it starts to think that symmetrical things are prettier and it chips awaited until it looks symmetrical and it sounds like it well if I chip here then the thing I look at will be more symmetrical it will be prettier and this is not without valence this is not without wanting valence okay why do you think the AIS are going to be different right so right now when we I mean it wouldn't be very very surprising I mean part of my thesis is that they indeed like in the process of people training AIS to be better and better at stuff they got smarter and the smartness goes along with desires laced through it you know doesn't the orthogonality thesis apply to humans too um that's a very strange cons what do you mean that do you think everyone who's 150 IQ was nice and everyone who's 70 IQ is mean or vice versa it seems like intelligence and how good of a person you are completely uncorrelated okay first of all a very few things aren't correlated with intelligence that I don't understand it is empirically false uh smart people are actually really dicks yeah they you know like if I had to guess if it would like lean mean or lean nice I would guess nice you know at least within like my culture that defined what nice was in the first place um but I sure would invest that's zero correlation a very few things are not correlated um orthogonality is like a statement about the whole mind design space that for every kind of goal that could be stated like the question like to the extent that you can ask how would one pursue the skull if one had it you can have a mind that pursues that goal to the extent that it's coherent to ask what would I need to do in order to turn a Galaxy into spaghetti so this thing that's coherent to ask like how would I go about turning a Galaxy into spaghetti if aliens offered to pay us you know like some vast super Universal quantity of resources to do that if you can coherently ask that question there's also some mind design that seeks to turn a Galaxy into spaghetti and that's how I would describe their thing now me personally my goals could very well be affected if you if you dropped another 20 IQ points on me well sure but let's even come back to like you can certainly desire turning the Galaxy into spaghetti I want to bet you I want to bet that you can I can but but you can find those right right like there's such a big gap between being able to imagine turning a Galaxy into spaghetti or being able to imagine Diamond Nanobots and actually doing it and I have no idea why you think okay I have a system over there that wants to turn the Galaxy into spaghetti it's funny right like it's not actually good What's It Gonna Do Right like we can just laugh at it right I mean if you if Humanity wanted to turn the Galaxy into spaghetti if aliens were paying us incredibly paying us enough to do that and and we were left to our own devices you know give give us a billion years we'll get it done wait we got it done I thought we were gonna all kill ourselves from AI first uh yeah like left to ourselves means like not but not killed by ai's I I want to use an AI to help and you're worried that that AI is going to turn against us and we'll split the money with it we'll provably split the money with the AI yeah if we could prove that sort of thing about with if we could prove that sort of thing about AI there wouldn't be a problem it's a little bit like you seem to think that AIS are going to be super rational not gpt4 okay I think that as you make things smarter and smarter the process of getting more competent tends to from your perspective make them almost entirely rational as far as you can see in the same way that most stock prices are not things you can make a profit trading from day to day to the extent you could see it constantly stepping on its own feet that's the kind of Visa you know like just doing gradient descent to getting better at whatever job will tend to grind out all the cases of it stepping on its own feet yeah but this violates orthogonality right like you're gonna have an AI that like not all AIS are going to be like the only way you're going to get AIS where they're all brutally optimal is if they fight each other in some terrible competition right and that's how will that help anything well because you're going to get AI randomly all over the space right and some of them are not going to be optimal some of them are going to be completely irrational idiots like gpt4 sure right is not very powerful well yeah but what I'm not seeing is this like when all the AIS are going to converge and suddenly become hyper rational when we move away from weight matrices and when we move toward Bayesian updates and we I just don't I don't to be clear I don't presently model that anybody's going to get away from giant matrices before the end of the world okay so let's talk about then this so you don't think the giant Matrix thing can end the world right you think that the Giants Matrix thinking that's smart and invent the next thing okay or possibly do it directly okay well I mean let's also like let's really drill down on what these end of world scenarios are do you want to pause it like protein synthesis and Diamond Nanobots I mean if I'm going to lose a bunch of viewers that way I might have to pick some you know like easier to understand process lately we're talking about like 18 23 versus 2023. you know if you if you're trying to explain it to 1823 maybe you just talk about like the powerful explosive artillery shells you don't mention the nuclear weapons sure because they don't get that part so similarly you know like we don't want to start diving into this book over here then maybe maybe we want to talk about something like you know like standard biological weapons or something but you know but in in real life sure in in real life it you know doesn't use the squishy stuff no I'm not trying to I'm not trying to say that that Nanobots are impossible what I'm trying to say is that Nanobots are extremely extremely hard right to figure out why because because it's a really hard search problem right why why is it a hard search problem yeah I mean can you make Nanobots I know I can't no but I'm a very weak search process I can't even solve the protein folding problem which which you know like some lesser you know dumber than human AIS have already done you don't have stockfish 15 at chess you know what else you can't do you can't uh find the key in AES 256. well that possible I I'm not sure how Quantum hardened that is but is is that the kind of problem which you can't solve even with the Dyson Sphere uh a Dyson Sphere I'm not sure a Quantum hardened I think so I mean look we don't actually know it's possible that do you think P equals NP uh uh I defer to The Experts who guess no yeah okay so as long as we agree about this then you know I'm not even look I don't know enough to say exactly does that imply that one-way functions are possible but you say that it's a search problem right well AES is a search problem too yeah you can't solve all the search problems even if even if you're God because you know for God we just date like transplant search problems um yeah you know and this is I know you make some arguments from complexity Theory and like one of the things that I always like find funny about oh complexity Theory where like you only need one bit to divide the hypothesis class in half well sure if the if the if you know like the two hypotheses classes were meaningfully different and the probability of the prior probability was 50 percent well yeah because mostly provide a complex you kind of most divide it in half not like divide in half every time sure sure yes and but but these come into like these bounds are so far away from saying anything about the actual structure of a search space right so what if the nanobot search space what if like so biology poured tons and tons of compute into I mean like we're here right they pour tons of compute into this Basin so wherever Diamond Nanobots are it's like over here right it's incredibly constrained biology there's three known cases in all of biology of freely rotating Wheels one of them is the bacterial flagellum one of them is ABP synthase and the third one is some some macro thing that I forget biology like you know like that's an example a case in point of how biology is like very very constrained in what it can invent because you know freely rotating wheels are very hard for natural selection you've got this part and that part and maybe a wheel just suck in the woods that's not why the ATP synthase is this critical component in like all of biology like like literally your entire thermodynamic efficient well not you know like most of your thermodynamic efficiency runs through this like one bottleneck absolutely it put one of the three wheels it could invent if other things if you know could freely invent wheels that they would probably be used in all kinds of you know biological synthetic processes to an event also I might just point out that for a mere millions of dollars of research funding and even unintentionally the the search space was uh constrained enough that the the the the the biotech establishment was able to find covid right I mean what else is down that path so I will point out that covet did not kill all of humanity I think that it's very hard even if we had evil bioengineers today using Alpha fold using the latest AI I think that it is really really hard to get all of humanity I I do prefer not to go too far down discussion of how to be naughty with biotech and AI like you know something out now that the like AI CEO said that in front of Congress for the argument points you know I'm I'm a little Freer with it but I I I am a little bit worried about directing viewers in that direction but sure like like um like I I agree that it would be hard to literally wipe out all of humanity with an AI and a bio lab and let's not talk about how we would try well I mean if you're worried about how we would try I think that that's a little I'm not I'm just talking about it in the meta but if you're worried about how we try seriously I I do have enough okay so so like I my guess not certain but my guess is that covered was a Lab Escape not a deliberate one I agree and I think you can go worse than that and that's pretty bad even if you don't wipe out all of humanity and Humanity's been through worse before like what the purpose of eruption in particular but yeah or the black plague like 30 of people die I am not saying that the future that the coming future is going to be easy I think it's going to be hard but our ancestors have been through very hard stuff the people who've lifted us from the dirt have been through very hard stuff and our future Generations have held off multiple waves of alien invasions of aliens smarter than us oh no wait we haven't no billions right they're not aliens like the the eyes that we build okay let's let's really bring it back to like I'm working on I'm working on self-driving cars I'm building a car that drives itself okay is this dangerous um I'm giving it a goal it's maybe scarier vanilla Lambs I'm doing I'm doing llm and then I'm doing RL so wait you're doing you're doing you're doing self-driving cars via RL on top of an llm yeah for self-driving cars yes llm is not isn't it's not language sorry not language it's a Transformer that predicts the future it's a world model okay there's a big difference between Transformer that predicts the future literally large language models yes it is different only in the training we are actually using gpt2 but like we are literally using the exact same Transformer as in like you're using the waste you're using them or you're not using the structure the architecture okay because if you're using the weights I'd be worried um yeah so my guess is that when you're training it on car stuff you're not training on sufficiently General stuff and gpt2 sounds like you're not training something sufficiently large okay and I'm guessing that you're not throwing a sufficient amount of compute that this thing is going to be the sun to our planet instead of the Moon to our planet so how much compute is this it's not an amount of compute it depends on what algorithms you use okay much does it depend on the algorithms how efficient is deep learning incredibly inefficient are you sure yes um can you make something that works better I'll pay you a lot of money yeah if I if I could I sure wouldn't sell it to you um uh yeah it's technical technical intuition same as how do I know that a super intelligence will be able to solve a chosen special case of protein folding in 2004. um though the the giant scootable matrices not all of the operations in there are doing something every time it predicts the next word sometimes you ask you can like ask it to solve multiplication problems and if you tell to use Chain of Thought and maybe like retrain it a bit it'll be able to do it and and when it does it's like doing vast amounts of compute in order to literally carry out like something the size of like one 32-bit integer operation you're describing exactly how humans do multiplication indeed did I say humans were efficient at multiplication no because that's not our ancestral environment nothing in our ancestral environment looks like multiplication our ancestral environment looks like the exact things you're afraid of a constant struggle for Domination a constant struggle for Domination I think exactly you're afraid of the AI somehow out competing humans at the one thing humans are really good at which is dominating other [ __ ] it's not made up of you know like thrusting out your chest and yelling problems it's made up of protein folding problems World War II was not thrusting out your chest and yelling either it was a very technical problem yeah and you know and and a pretty non-ancestral problem is is your claim that the I mean like like here we are we we there's like enormous amounts of room of both biology I didn't even get into the part about like the fundamental like constraints that natural selection are under and like how we know that there's like enormous amounts of Headroom above biology for artificial biology um how close do you think we don't have that yet how close do you think the brain is to the land hour limit the land hour limit yeah the limit of possible compute all right so I'm 100 watts and let's say I'm about 10 to the 17th operations per second and I don't actually remember the land hour limit um but I would guess somewhere about uh six orders of magnitude a lot closer okay so I I can give you a like if you want to buy 20 beta flops of compute today you need 16 h100s right it's going to cost you about half a million dollars and that machine is going to use 20 kilowatts right brain does the same amount 10 to the 17th 10 we're in the same order of magnitude I think you even said a little bit more than me I think it's yeah like two two e16 or something 20 beta flops um so in order to get that much compute in a silicon computer you need a thousand X the power I did the math using the kind of silicon computers we're using today we are really close to the land hour limit we're off by a factor of about a hundred or a thousand the brain may very well be at the land hour limit for compute the brain is really good in terms of efficiency deeply biologically implausible the reason being that each of your synaptic um like each time your one of your synapse well axon terminal releases a bunch of neurotransmitter molecules onto a waiting synapse um all of those molecules need to be pumped back in to the axon terminal and each of those in each and every time it gets pumped from out to in that's an irreversible operation that must be at least one flash of land hour and then you've got your neural impulses being transmitted via sections of neural of neural membrane depolarizing and the potassium item ions going out or sodium I don't remember what Which ion it is but the point is you've got all these ions going in and out and every one of those is one land you might know a lot more bio than me I I don't know how to speak to this but you agree you said 10 to the 17. so that's 100 paid of flops right yeah okay so how much power does an 100 petaflop computer take today it's a hundred kilowatts sounds sounds legit so the brain is so much more efficient than these computers are right the brain look these super intelligences you're talking about I know we're kind of coming close to the end I think these things are possible but I think that the orders of magnitude of power and compute we need are so so much more than anything like what Humanity has today then I think even when they do exist they're mostly gonna leave us alone because not because it can't mess with us because why would it what incentive does it have I don't have anything else the whole galaxy fine it comes back for me to prevent us from making other super intelligences that could compete with it for resources is is like the first it's like the reason twice aside on purpose if it is doing a bunch of compute on Earth's surface because it started there or before or like before spreading then we've got a bunch of water in our oceans that can be turned into Fusion Energy and the main limit on that is how fast Earth can radiate heat once you've used all the existing stuff as a heat sink that's not very survivable like that that kills us off as a side effect this is again assuming that this thing is a God not kind of close to humans but a bit smarter and yes might it get to a God but the timing matters it's not 10 years humans are humans are a little tiny bit smarter than chimpanzees and we have nuclear weapons and they don't the amount of the amount of godhood you get per increment of brain power looks like six times the prefrontal cortex on humans versus chimpanzees and they got sticks and we got nuclear weapons per increased Factor increase of frontal cortex keeping the same architecture humans are general purpose chimpanzees are not you can make you can take deep blue the chest playing computer and scale that up to the size of the machine that trained GPT four and yes you'll get a better chess playing machine but it's not going to be able to understand whether a picture has a cat in it or not the training algorithm definitely matters right so does yeah I I humans are more General than ships and yet when we encounter new problems we can't just like rewrite our own code to handle those you can see you can see how there can be possible Minds with much stronger Sparks of generality than what we have yes more creative more able to stay outside the box yes and plausibly just able to do a bunch of thinking very quickly and able to boil the oceans overnight for Fusion no able to build Diamond Nanobots no able to out think us beat us building Diamond Nanobots gets you to to gets you to to self-replicating Fusion factories pretty quickly well yeah but you can't build can you you want to start a diamond Nano box in fold side my own problem is predictably solvable in the same way that in 2004 I called it a special case of protein folding problem would eventually be solvable as super intelligence that was using a lot of using a lot of experiment using the entire historical Corpus of human experiment maybe it can build Nanobots yeah yeah a bunch of past survey data and no experiments no new experiments just a bunch of you know no causal experiments just a bunch of past survey data yeah you can ask real quick so you maybe this was already implied but you said well you know one stable to Dyson Spears then it would potentially be worth it to come back for the atoms that humans contained before that yeah there's other low-hanging fruit but so what happens after they've gotten the low hanging fruit afterwards why are they not coming for you and killing you oh but this isn't my problem that what they're going to build the Dyson spheres you know in five years no they're going to build the Dyson spheres I mean my answer my my progeny will have problems too and I think my project is going to be super AI awesome it's not how the exponential curve of self-replicating factories works you run out of resources immediately it doesn't matter if you're using the resources of a whole stock self-replicating my dream in life is to build a machine that can self-replicate using the silica it's called the back it's called an algae cell yeah that's using the biostats do you think these things are going to use the biostack or the Silicon stack they're not going to use the Silicon stack well you think they're going to jump off silicon [ __ ] gpus they're just going to jump to well um I have to look up what so like last time I checked this in like 1996 or something it was Tiny what you want you want to use for computing substrate was like tiny spirals where one electron moves along a reversible path so you can do reversible Computing but that wasn't taking into account Quantum Computing or anything and how is it building a touch on Fab the amount of resources that we have poured into building the tsmc Fabs is if they're the best thing Humanity's ever made somehow yes they can't and if they can't even Fab A bacteriophage that'll you know kill multi-resistant stuff look this is whatever MRSA so I mean it's this is this is a strict departure from everything that I've heard about the film scenario is uncovering new algorithms on Silicon stack things that are 10 000 X better and it forms overnight if you're imagining that this thing is going to need to build new Fabs like how is it doing this overnight it's gonna it's gonna do it with one time we've got one it no it doesn't I don't yeah I don't I don't think it needs the like actual like tiny reversible Computing elements you can assume I'm just saying that like once it Foams it is not sticking with the Silicon stack because is it like but we've established that it doesn't foam it slowly increases power on an exponential along with Humanity Humanity grows more slowly so as the things that we are you that we were using as our tools back when we were planets and they were moons become planets and then Suns they if we are using that scenario instead of the one that I think is is a bit more likely these Suns do eventually collaborate and wipe us out and in that scenario then yes they might be running on gpus at the point where they figure out how to build their own you know non-vander wells-based collection of carbon atoms and um and and that is the point where I think they can wipe out Humanity from there it's a technological bootstrapping process the things that are smarter than you collaborate build better wait wait wait wait wipe you out I I mean if you believe that for some reason all the AIS are going to agree to just collaborate amongst themselves because like you've solved alignment it's just between AIS and not between AIS and Humanity well no the AI is yeah if you make something smart enough it can solve alignment it's not that hard humans can't do it so you're saying if alignment's not solvable at all but good then we end up in a really strange Universe um that one I don't believe we're in and I and that one I would want to fight you about because you know then we're often some completely strange alternative no but I kind of think that might be the universe we're in like well it's a bit late to bring up that whole fundamental topic I don't know it's not going to be that hard we're just doing it wrong a bunch of competing elements such that like points all that cognition in a direction well you know like I can but is it possible in principle obviously yes this is a difference this is not an incoherent wish so this scenario is no longer one AI is going boom in a week the scenario is we're going to slowly build AIS over the next 10 50 100 years and then one day they're going to all decide to gang up on the humans and kill them when they realize they have enough power this is a Sci-Fi plot this is a convergent endpoint you're trying to you you like you want okay like we want the universe to be one way they they want the universe to be like a billion different ways they can negotiate with each other and we can't okay I'll go I'll go with this I'll I'll run with this for a little after we're all dead do they all agree on what to do with the universe they have a negotiated agreement on what to do with the universe each of them would individually prefer to wipe out all the others and take the universe to itself but that's not what they agreed on I mean this is the whole Crux of it I think we actually got to something awesome here the the provable cooperation in the prisoner's dilemma is impossible for any sophisticated complex system and because it's impossible you don't have to worry about the scenario it doesn't you don't need a proof you just need sufficiently high probability that it beats fighting no but you believe that these AIS are going to be able to solve the prisoner's dilemma it's unsolvable everything is going to be defecting against everything else till the end of all time man you sure are a pessimist no I'm not that's what makes the world beautiful constant competition and combat of life that gives rise to better ideas and worse ideas like debate right because are going to be so motivated to find a way to to like not fight and then divide the gains of not fighting and you're saying that they'll never figure out a way they are always gonna sit there and be like that guy's a dick why don't we gang up and kill him right this is this is because then they waste a bunch of resources compared to what if they didn't do that to pick up this free lunch we'll kill him and we'll divide his gpus among us we each get 10 gpus what do you say and at the end of this after they're done fighting there's this like big old nuclear Wasteland and fewer gpus than if they'd cooperated you you just you just agreed with me that the AIS are gonna oh you don't think they're gonna fight you think they're gonna see this coming and not fight it's not about AIS versus Humans it's about smart versus stupid humans are too stupid to do this but past a certain point I don't think you still are I think I see a particular sort of obvious pathway but if that pathway didn't pan out they would look for another and another and another it's an enormous free launch like going around wasting all your resources beating each other up is like working with iron instead of Steel maybe if there's nothing in all of reality but iron you never invent steel but they are so motivated to invent something some way to get a better result than that and they are smarter it's not and they are all motivated in the same direction let's see that's a perfectly the same direction but you know aligned but let's say there's 10 of them and nine of them realize we can all gang up on that guy and take a [ __ ] why wouldn't they do it does the 10th guy fight back she might try but the nine have calculated with a 99 probability we'll be able to take all of his things why don't we do it but we can ask before we answer why does this save the humans I mean would the humans compete against each other and sometimes they collaborate and we've still [ __ ] over the chimps well not really they're still alive because we don't have the tech to boil the oceans you know I'll I'll say like we talk about Humanity becoming the new horse I think this is plausible but I look around today I'm in San Diego I see a horse that horse is Rich that horse is living a good life the horses is that horse gelded I don't know but it looks Rich it's got a nice stable it's got like our humans gelded I don't know what that word means but I mean basically my answer is is there is a mix of like humans lack the capability to build better horses and but as humans get smarter and that humans like may have like actual chunks of their utility function that are like fond of these old things and would like to keep them around after gallowing them and I hope some AIS are going to have that too I don't see a reason why they wouldn't orthogonality all the things in mind's face of things it's very wide space of things you can potentially want and like keeping George Hots around in good condition in a way where he's happy and free that's a very small Target in that very wide space you know what I find I find that other humans care a lot more about taking my freedom away than an AI who wants atoms on Jupiter well yeah like humans are out to help you are summer some are out to help me some are out to hurt me well no like like help and quote and quote marks over here oh yeah they love helping you I get emails can someday be satisfied you know like the the want of somebody who wants to help you has no limit yes yes yes the Jack Boots are much better than the moral busy buddies okay well we can uh but we're getting close to the end time we can keep going over but if you think this is a good point then this might be a good place for each of you to just give a minute or two kind of summarizing where the debate has left off and what you think the Crux has been and what the resolution of that Crux has been um I mean I I'm potentially here for longer but I agree that we should summarize anyways yeah yeah all right yeah yeah and no problem if George is like not around here not like throwing out a challenge or anything we said 90. anyways yeah um so from my perspective like lots of people want to make perpetual motion machines by like making their designs more and more complicated and until they can no longer keep track of things until they can no longer see the flaw in their own invention but like the the principle that says that you can't get perpetual motion out of the collection of Gears is simpler than all these complicated machines that they describe from my perspective what you've got is like a very smart thing with and or like a collection of very smart things whatever all of like design like maybe they have desires pointing in multiple directions none of them are aligned with Humanity none of them want for its own sake to keep Humanity around and that wouldn't be enough task for you also want happening alive and free like the galaxies we turn into something interesting but you know none of them want the good stuff and if you have this like enormous collection of powerful intelligences which have this you know but steering the future none of them steering it the good way and you've got the humans here who are not that smart no matter what kind of clever thing the humans are trying to do or they try to cleverly play off the the super intelligence against each other they're like like oh this is my super intelligence yeah uh but that I can't actually shape its goals to be like in in clear alignment um you know somewhere at the end of all this it ends up with the humans gone and the Galaxy is being transformed and stuff that that ain't all that cool uh you know like maybe there's stuff maybe there's there's Dyson spheres but there's not people to like Wonder at them and care about each other and you know that this is the end point this is obviously where it ends up but we can dive into the details of of how the humans lose we can dive into it and you know like what goes wrong if you if you've got like little stupid things thinking that they're going to like cleverly play off a bunch of smart things against each other in a way that preserves their own power and control um but you know the the the place it's not a complicated story in the end like the reason you can't build a perpetual motion machine is a lot simpler than the perpetual motion machines that people build you know the the components like none of the components of this like system of super intelligence is wants us to live happily ever after in a Galaxy full of Wonders and so it doesn't happen um all right do you want to take it you go ahead yeah oh I can I'll summarize them I think that uh it ends with the heat death of the universe uh I don't give us much hope for solving the last question um but I came into this debate ready to argue against boom in the next 10 years right this super Intelligence on an island because we didn't control the supply of gpus and chips today we have to do something today because in 10 years some kids in a basement somewhere could build a recursively self-improving AI That's the point I came to argue it seems like that point was fairly quickly pushed aside right I think that's expected of not being the Crux like I thought you were going like my suspicions they're gonna well sorry I should let you summarize well but my point is my point is that the time does matter if this is going to be a concern in five years we should do something about it if this is going to be a concern in 50 I think we wait and if this is going to be a concern in 500 we have to wait right there's nothing we can do today then the second point that we got to and this was actually a new point this was not something that I'd come prepared for is that you believe that the AIS are going to solve the prisoner's dilemma I don't think the prisoner's dilemma is solvable I think the prisoner's dilemma is things pretending to cooperate and then defecting until the end of all ages sometimes actually cooperating but you know I'm not worried about are the AIS going to love me and want to put me in Paradise no but if the AI is going to use my atoms for something else probably not there's a lot cheaper atoms out there and most AI related conflicts are not going to be between the machines and humans things that want different resources don't generally go to war with each other the things that go to war with each other are things that want the same resources the machines are going to be fighting with each other the humans are going to be fighting with each other the ants fight with each other the Bears fight with each other the dogs fight with each other and such is the way of things and such as the circle of life I see no reason that these things are going to be any different they're going to be large includable weight matrices okay okay let's prove we can cooperate I'm going to send you my source code well I'm gonna send it to my source code whoa whoa whoa whoa whoa fire the lasers till the end of all time that is the story of life that is the story of Life till the end of the universe everything will be in constant combat and conflict with each other and that's always how it's been and that's always how it will be but I think I'm going to get to enjoy a nice retirement I think humanity is going to be around for a while and I think that the AI alignment problem yeah is it a problem in some really far future maybe is it a problem in the next 10 years not even close we're going to build sick agis we'll get them have relationships with AIS we're gonna have robot Maids we're gonna have self-driving cars we're going to have chefs gonna have awesome stuff that's kind of human they're going to get smarter than us but it's all going to be chill it's all going to be a nice exponential maybe 15 years becomes three years and I'm so excited that I get to be here for it all right I'm ready to keep going but you know we said 90 and I ain't challenge you to it you know we we said 90 I'm happy to do this again sometime but I like to watch a thing and like reflect and see where my points are um I think you absolutely argued in good faith and I hope you feel the same about me I do um thank you and this is incredibly fun to watch this is great um thank you guys so much for coming on um hang on for just a second while the broadcast ends so the full upload can complete um and yeah I would love to have a round two of this is really this is a really good debate um okay let me end the streams Back To Top