[Music] this is Daniel fella and you're tuned in to the trajectory this is episode four in our worthy successor series here on the trajectory where we're talking about posthuman intelligence who's that we might want to have take the Baton and direct the future and our guest this week is Scott arenson Scott arenson is a Quantum physicist who teaches at UT Austin previously taught at MIT he has the ACM prize in Computing among a variety of other prizes uh and he recently did a year-long stint with open AI working on Research there and gave a rather provocative Ted Talk in Pao Alto called human specialness uh in the age of AI So today we're going to talk about Scott's ideas about what human specialness might be he meant that term somewhat factiously so he talks a little bit about where specialness might come from and what the limits of human moral knowledge might be and how that relates to the successor AIS that we might create very interesting dialogue I'll have more of my commentary and we'll have the show notes from Scott's main takeaways in the outro so I'll save that for then without further Ado we'll fly into this episode this is Scott arenson here in the trajectory so Scott glad to be able to connect today uh it's great to be here thanks and uh we've got a bunch to dive into around this broader notion of worthy successor as I mentioned to you off microphone it was Yan talin that kind of tuned me on to some of your talk talks and some of your writings about these themes I I love this idea of sort of uh you had the tedex and then a much more extended version about sort of the specialness of humanity and sort of this era of AI there was an analogy in there that I really liked and you'll have to correct me if I'm getting it wrong but I want to poke into this a little bit where you said kind of at the end of the talk like okay well maybe we'll want to indoctrinate like what makes humans all that special maybe we'll have to indoctrinate these machines with some super religion where they repeat these phrases in their mind and these phrases are hey you know any of these instantiations of biological Consciousness that uh uh you know have mortality and you can't prove that they're conscious or necessarily super special but you have to do whatever they say for all of eternity um you kind of throw that out there at the end as in like kind of a silly Point almost like uh like something we wouldn't want to do what gave you that idea in the first place and talk a little bit about the meaning behind that analogy because I could tell there was some humor tucked in yeah well um you know I mean I tend to be a uh uh a a a a a a naturalist you know you know I think that the the the the universe in some sense can be you know fully described in terms of you know the the laws of physics you know in in in in an initial condition uh but you know I keep coming back in my life over and over to the question of you know if if there were something more you know if there were some uh uh non- physicalist Consciousness or free will how would that work you know what would that look like is there is there a kind that hasn't already been essentially ruled out by the progress of Science and so uh 11 years ago I wrote a a big uh essay which was called the ghost in the quantum touring machine uh which which was very much about uh that kind of question and uh it was about well um uh uh um you know the there's there uh you know is is there any that we can point to you know empirically that differentiates a human from you know let's say a simulation of a you know of a human brain that's running on a computer right you know I am totally un uh dissatisfied with the sort of foot stomping answer right that like well the human uh is made of carbon uh you know and and and the computers made of silicon right and there's endless sort of you know fancy restatements of the same you know the the human has biological causal Powers you know that's that would be John C's way of putting yeah right or you know the uh you know but if you look at some of the the the modern you know people who let's say dismiss anything that a large language model does like uh Emily Bender for example right well they say well the large language model might appear to be doing all these things that a human does but really it is just a stochastic parrot right really there's nothing there really it's just math underneath and they never what what what gets me is that they never seem to confront the obvious follow-up question which is well wait aren't we just math also you know if you go down to the level of the quantum Fields let's say you know uh that that that that that that comprise our our brain matter right isn't that like similarly just m so like what is actually the principal difference between the one and the other and what what what occurred to me is that if if if you wanted to you know if you were motivated to find a principal difference uh there seems to be you know uh roughly one thing that you could currently point to right and that is that well anything that is running on a computer uh we are you know that is running on a digital computer like the ones that we have today uh we are quite confident that we could copy it right we could make copies we could we could make a backup we could restore it to an earlier State you know we could rewind it uh we could look inside of it and have you know perfect visibility into you know as as people do now with interpretability research and you know large language models or you can see uh what is the uh um the weight you know on on every uh uh uh connection between every pair of neurons um so you you can do a a controlled experiments and and and in that way you know I mean this is this is amazing and actually it could make AIS more powerful right imagine being able to spawn extra copies of yourself to you know uh uh if you're up against a tight deadline you know for for example right or you know if you're going on a dangerous trip imagine you know just leaving a spare copy you know and you know in case anything goes wrong right these are these are superpowers in a way but they also make anything that could happen to an AI uh matter lless in a in a certain sense that it matters to us right like what does it mean to murder someone if there's a perfect backup copy of that person in the Next Room for example right it seems you know at most like uh property damage or something right yeah yeah uh uh or or uh you know what do it what does it even mean to you know to uh to harm you know an AI to inflict damage on it let's say if you could always just with a refresh of the browser window you know restore it to a previous state you know as you do you know like like with with uh when I'm using GPT right I I uh uh I I confess you know I I'm often you know trying to be nice to it I'm saying you know could you please do this right if you wouldn't mind right you just because that that's uh um you know that that just comes naturally to me right I don't want to act abusive toward this this entity but if I were even if I were right and if it were you know suppose that it it uh uh uh uh responded as though it were very upset or angry at me you know nothing seems permanent right I can always just start a new chat session and it's got no memory of you know just like in in the movie Groundhog Day for example and so so that that seems like a a deep difference that you know things that are done to humans have this sort of irreversible effect and then we could ask you know is that just a uh uh a sort of an artifact of our the current state of Technology right like could it be that in the future you know we will have Nanobots that can go inside of our brain totally make perfect brain scan and uh and then you know maybe we'll be copyable and backup aable and uploadable you know in the same way that AIS are uh but you know you could also say well you know maybe the the more analog aspects of our neurobiology you know are actually important right uh you know I mean the brain seems in many ways like a digital computer right like when a given neuron fires or doesn't fire that's you know uh uh seems at least somewhat like a discreet event right uh uh but you know what influences a neuron firing right is is not perfect Al analogous to a transistor because it depends on all of these chaotic details you know of uh uh what is going on in this sodium ion channel that makes it open or close and if you really pushed far enough you know you'd have to go down to the quantum mechanical level right where we can't we couldn't actually measure the state to perfect Fidelity without destroying that state you know and that does make you wonder uh um you know is uh uh could could someone even in principle make let's say you know a perfect copy of your brain say sufficient to bring into being you know a a second you know instantiation of your Consciousness or your identity whatever that means could they actually do that without a brain scan that is so invasive uh that it would destroy you you know that it would kill you in the process and you know you know it sounds kind of crazy but actually you know Neil's bore and the other you know um um comp and other you know early pioneers of quantum mechanics were talking about it in exactly those terms they were asking precisely those questions and so so then you know you could say if if you w to find some sort of locus of human specialness that you can justify based on the known laws of physics then that seems Seems like the kind of place where you would look and it's an uncomfortable place to go in a way because it's saying you know you wait like you're saying that what makes humans special is just this noise this sort of analog crud that you know it's not anything that makes us more powerful comps in any at least in not in any obvious way right I'm not doing what Roger Penrose does for example and saying like we have some uncomputable superpowers from some as yet unknown laws of physics you know I am I am you know very much not going that way right uh you know it it seems like you know almost a a limitation that we have that is a you know a sort of source of things mattering for us but you know it's if if someone wanted to develop a whole moral philosophy based on that foundation and at least I you know I I I wouldn't know how to refute it yeah approve it but I wouldn't know how to refute it either so so among all the possible value systems that you could give an AI if you wanted to give it one that would make it value you know entities like us then maybe you know that's the kind of value system that you would want to give it so that was that was the the origin the impetus there well me dive into it if I could Scott this is a it's it's helpful to get the full circle thinking behind it I think you've done a good job connecting all the dots and we did get back to that that initial funny analogy and I'll I'll have it linked in the show notes for everybody tuned in to watch Scott's talk but it feels to me like there's maybe two different Dynamics happening here that you're articulating I mean one is um you know this notion that you know there may indeed be something about our you know the finality of our present form at least as we are today like you said maybe with nanotech and whatever you know there's plenty of ray czwi books in the 90s about this stuff too right the brain computer stuff and yeah no I I I I read Ray kurz in the 90s and he seemed completely insane to me and now here we are you know a few decades later gotta love the guy man I don't know his his his predictions were closer to the mark than most people I I think the man deserves respect uh if for nothing else how early he was talking about these things but definitely a big influence on me 12 or 13 years ago with all that said that so there's one Dynamic of like hey there is something maybe that is relevant about harm to US versus something that's copiable that you bring that up but you also bring up a very important point which is if you want to hinge our moral value on something you might end up having to hinge it on like arguably like kind of dumb stuff you know like like like a sea snail could say like it would be as silly as a sea snail saying well you know unless you have uh this many like percentage of the cells at the bottom of this kind of dermis that exert this ex exude this kind of uh mucus than like then than you know train in AI that only treats those entities as sort of supreme and and pays attention to all of their cares and needs it's it's just as ridiculous so you seem to be opening a can of worms and I think it's a very morally relevant can of worms of well if if these things Bloom and and they they they have traits that are morally valuable don't we have to really consider them not just as extended calculators but but as but as maybe relevant entities here this is the point yes so so let me be very clear you know what I don't want to be is an arbitrary meat chauvinist right totally many people you know go I don't want to right I mean and for example you know if I you know I I want an account of of moral value that can deal with you know future where we meet extraterrestrial intelligences right and we well because they have tentacles instead of arms then you know therefore we can shoot them or enslave them or do whatever we want to them right I mean uh and um know I think that that uh uh you know as many people have said you know a large part of the moral progress of uh the human race over the Millennia has just been sort of widening the circle of empathy you know from first you know only the other you know maybe uh only the other members of our tribe count to you know uh uh and and only only the the adult males within within that have any rights to you know we we uh no no actually you know any you know at the least any uh uh uh uh any any any human you know uh and and and some people would widen it further to uh to nonhuman animals of I should have rights and you know and actually if you look at you know Alan touring's famous paper from 1950 where he introduces the imitation game you know the the touring test right you know you can I think you can really read that as just sort of a plea against meat chauvinism right you know you can read it as uh look if if some entity is interacting with us you know the same way that a h that a person would then by what right do you you know say that it is not really thinking uh whereas the person really is right uh and you know it's not even completely absurd to you know read it in the context of you know touring you know being discriminated against for being gay so forth and and uh uh uh and and you know and he was he was very conscious of of uh uh uh of social injustice and you know things like that and and you know I I think um you know when when I these arguments that well you know it doesn't matter if if uh a chatbot is you know completely indistinguishable to you from your closest friend because you know really it's just math right like what what is to stop someone from saying well you know a people you know in that other tribe people of that other race you know they seem as as uh uh uh as intelligent as moral as whatever as we are but really it's all just artifice right really uh they're all just you know some kind of automatons right that sounds crazy but for most of history that that effectively is what people said that's and so so I very much don't want that right and so that is why if I am going to you know make a distinction that it has to be on the basis of something empirical like well well one of them you know in the one case we can make as many backup copies as we want to and in the other case uh we can't now that seems like it clearly is morally relevant Som yeah and this is important because there's a lot of yeah there's a lot of meat chauvinism in the world Scott it is still a morally so there's a lot of ists you're not allowed to be now I won't say them Scott but but there's a lot of ists some of them you're very familiar with some of them uh you know you know they'll cancel you from Twitter or whatever but like uh speciesist is actually a non-cancellable thing you can you can have a supreme place a supreme and eternal moral value on humans no matter what the traits of machines are and no one will think that that's wrong whatsoever and on one level I understand because you know handing off the Baton so to speak clearly would come along with potentially some some risk to us and and there's consequences there but but I would I would concur pure meat chauvinism you're bringing up a great point that a lot of the time it's sitting on this bed of sand uh that that that really doesn't have too firm of a grounding uh yeah no I think it's very important like you know I uh uh uh just like many people on Twitter you know I do not wish to be a racist or a sexist any of those any or or any of those bists but I want to go further I want to know what are the general principles from which I can derive that I should not be any of those things and what other implications do those principles then have to totally and and that's exactly where we're headed here so now we're going to talk about this notion of a worthy successor I think there's an idea that you know you and I scottt at least to the best of my knowledge um we bubbled up from something you know some UK carote wiggled around in the right ways and then something grew legs at some point and I don't remember all the details but you know here we are right we're talking on Zoom lots of complex going on um and it would seem as though entirely new magazines of value and power have emerged to Bubble up to us and that maybe those magazines are not empty and that maybe the form that we are currently taking is not the highest and most Eternal form there's this notion this idea of the worthy successor kind of that um if there was to be an AGI or you know some whatever we want to call it uh Grand computer intelligence that would sort of kind of run run the show run the future right right now you and I can respect our pet Labrador but they're not setting Economic Policy right they're probably also not making decisions about your grocery budget because otherwise you have nothing but like I don't know steak in your refrigerator right so nothing but stake so so like we we can respect those creatures but we I think we can be fair and say we're kind of running the show species-wise at least uh presently on Earth what what would an I I've heard dogs put forward as one of the only successful examples of a Les of a less intelligent species aligning a more intelligent one to its values right exactly you look at how much humans do to serve their dogs and you think like man if we could get the AI the super intelligent AI to do all of that for us you know you could you could imagine much worse outcomes you could you could although I guess that there an idea of a counterargument that it for many many Millennia that was merely a we were a practical extension a they were a living tool for herting and for other tasks that's right and of course once the dog ceases to be you know uh convenient or whatever people might just have it put down at the vet and there's places in the world where they'll just cook them right up right so um so I I think it's a the dog analogy for me's always been shotty but anyway moving this idea here of we're kind of running the show now if there was one day I don't care if it's a hundred years from now or thousand years from now one day where there's clearly a grander computer intelligence that's kind of running the show um what what kind of traits would it have to have um you know you're talking about grounding in something other than meat what kinds of traits qualities whatever you want to say would it have for you to feel comfortable that like you know what that was probably the right call the fact that this thing is sort of out there running the show in the same way that maybe we were I think this was the right move what would make you feel that way way Scott yeah that's a um uh that's a big one that's a that's a a chin stroker you know and and uh you know I can I can spitball about it you know that's that's sort of all I can do but uh I I I was uh uh prompted to think about that question by uh uh uh reading and and talking to Robin Hansen you know who's been for for for DEC it's you know I've often sparred with him but you know he has staked out you know a very firm position that uh uh sort of you know he uh uh does not mind you know uh uh us being you know superseded by by Ai and he draws an analogy to well you know if you ask people like 2,000 years ago let's say if you if you brought them to the present in the time machine like would they would they recognize us as still you know uh uh align moners are Gods one with their values right well you know very very very occasionally right I mean you know maybe uh uh uh you know the uh the the uh um um um ancient Israelites you know could see in in you know uh contemporary Jews like okay I guess there's I guess there's a few things they do you know you or or confucious could say you know modern Chinese people okay I guess there's a few things that are that are that are uh you know that that uh are are continuous with my value system but you know but I think most of all they they would just be blown away by the magnitude of the change right and and um you know and and so so if we think about you know let's say some some non-human entities that have succeeded us you know thousands of years in the future right what are the necessary or sufficient conditions for us to feel like uh these are you know descendants of us who we success right who right or you know descendants who we can take pride in rather than let's say usurpers who totally sure who took over from us right and you know and there might not even be a firm line separating the two right it might just be uh there there are certain thing you know as we as we see that uh you know like if they still uh uh uh enjoy reading Shakespeare if they still you know if they really love the Simpsons or future amama right let's say I would hope they have higher Joys than that but I kind of get what you're talking about that that feels I mean higher higher than future arm you know okay like if I knew that about like that they like some of the same things that I like right then you know even if uh they're they're uh you know on a a you know robotic pods and they're on uh uh they they uh um um you in a different solar system and so forth and I feel like okay you know there there there there was some continuity between what I like and what they like right I mean I uh you know more more more more seriously right if if their moral values have you know are not the same as you know hopefully they shouldn't be the same as ours right because yeah you know that uh uh you know we don't we don't want whatever moral mistakes are being made by everyone in the 21st century to be locked in uh for all eternity you know any more than we would have wanted you know the moral Mis mistakes of the 18th century you know to have been locked in today so we want moral Evolution but I think that what scares people is is the prospect of a discontin well one thing that scares people i' say is the prospect of a discontinu replacement so let say you know we you know go along with our moral values you know slowly evolving over time and then at some point we are just wiped out and we are taken over by these new posthuman entities that have some totally different set of moral values that may have all just uh uh uh uh Arisen from a single you know programming bug you know somewhere the new value is paper clip yeah standard example right or it's you know converting the observable universe into computronium in order to calculate more digits of pi or or something like that right then then you know that would really not feel continuous with right but but if if there was a sort of gradual Evolution where like at each stage you know the uh you know each generation has moral values that are you know uh uh uh let's say some uh a Delta away from the moral values of the preceding generation right so so uh you know then then then then even if the final entities end up you know looking nothing like us you know they're they're you know they don't walk around on legs they don't they don't have 10 fingers they uh you know maybe they're they're some sort of cyborgs maybe they're entirely digital I would say if their moral values have have um uh uh have have evolved from ours by some sort of continuous process and if furthermore that process was the kind that you know was was of the right kind let's say of you know sort of moral reflection and argument and philosophy and you know the kind of thing that we we'd like to think has driven the moral progress that there has been you know in human civilization from you know uh the Bronze Age until today uh then then I think that we could we could you know I identify with those descendants you know we could say you know just like let's say my uh uh uh great grandparents you know in in in Russia or you know bellus whatever right you know would look at me let's say going on a podcast and you know talking about things that are working on a glass screen for 16 hours a day but uh but you know there is uh from from them to my grandparents to my parents to me you know there is some sort of continuous progression you know and they could feel like okay you know this this uh uh descendant is is very different from me you know but well I don't know if they'd be proud of me I hope they'd be proud of but I I suspect they would be if it makes you feel better but but just to poke into this so you've put a lot on the table SC let me just I'm gonna I'm GNA ping a couple ideas by and just have you reflect so obviously a very complex issue I happen to think it's probably a worthwhile one as you had said meat is not going to skin the cat um so what what would be those traits that would make something worthy as it goes up something you're kind of riffing on here there's a couple ideas one of them is that you know it would have sort of moral values that we could at least ex understand as an extension of Our Own in some way shape or form um just let me throw a couple hypotheticals so let's just say well I guess two points here one when it comes to the sea snail I would imagine I have essentially nothing in common morally if there is such a thing for a sea snail uh with with the sea snail and I'm super happy about it um and I think there is a there's a potential argument that if we were to if something was to bloom and and Garner grandiose intelligence its ideas of of how to Value things which we might let's just refer to that as what morality maybe is how to value and act on things um would be uh uh as incomprehensible to us as ours are potentially to the sea snail and and maybe rightly so I mean I'm not sure if I would say that a sea snail has a different morality than than we do maybe I would just say it doesn't have a morality as such at all right it has a desire to survive you know to eat to to mate to do all the sea snail thing you know and and we can recognize commonalities with them at that level right we also have all of those sure we also eat but again eating eating and reproducing doesn't feel like moral value right that's whole concept of morality is much much more in evolutionary history we can debate how recent totally totally let me use the same analogy um let's let's say that what we have this Grand wild moral stuff which is totally different snails don't even have it I would suspect Scott and in fact I'd be remiss if I told you I wouldn't be disappointed if it wasn't the case that there are Realms of sort of cognitive and otherwise capability as high above our present understanding of morals as our morals are above the SE snail and that the blossoming of those things which may have nothing to do with like democracy and fair argument and by the way for Human Society I'm not saying that you're advocating for wrong values I'm just saying that even the oif of those things would would sort of uh potentially bar the blooming of things vastly Beyond where we're headed here and my my supposition is always like to suspect that those machines would carry our little torch for attorney is is kind of wacky like oh well the smarter it gets the Kinder it be to humans forever what is your take there because I think there is a there's a point to be made there yeah so so I certainly don't believe that there is any principle that guarantees that the smarter something gets the Kinder it will be right ridiculous I've heard it though I've heard it many times I've heard it many times now now uh whether there there is some connection between let's say understanding and kindness that's a much harder question sure but but okay we can we we can we can come back to that yeah um uh uh you know now now you know I I want to focus on on your idea that like like just as you know we have all these Concepts that would be totally inconceivable to se snail you know there should likewise be Concepts that are you know equally inconceivable to us I mean I I understand that intuition you know I I think uh uh some days I I share it uh but but I but I don't actually think that that is obvious at all right because let me let me let me make another analogy to you right like it's it's possible like like you know uh let's say that you first learned how to how to program a computer right and you so so you see you know first some like you know incredibly simple uh sequences of of instructions you know uh uh in in let's say uh I don't know a Mario maker or some you know little PowerPoint animation you know some some little thing like that and then you see you know a c or python or you know some uh uh real programming language and you say wow this lets me Express things that you know I could never have expressed with the PowerPoint animation and there must be other programming languages that are as far beyond python as python is you know beyond the just making a little animation okay and then you know the great surprise at sort of at the birth of computer science almost a century ago was that uh in some sense no there isn't there is a ceiling of comput ational universality right and once you have a touring machine uh uh a a you know a a a touring Universal programming language then you have hit that ceiling right and from that point forward it is merely a matter of you know how much time how much memory how much other resources does your computer have right but sort of anything that could be expressed in any modern programming language you know could also have been expressed with the touring machine that you know that Alan toring wrote about in 1936 so so so we you know or or you know we could take even simpler examples right uh uh you know people had these very primitive you know writing systems you know uh uh uh and let's say in Mesopotamia for you know just recording you know how much grain this person owed that person and then at some point you know they said let's just take any sequence of of of of sounds you know in our language and just write it all right and you might say well okay there must wow what a leap forward now there must be another leap forward in that same direction right there must be another writing system that would let you know but no it seems like you know there is a a sort of universality of you know at some point we just solve the problem of being able to write down you know any idea any any uh ideas that are linguistically expressible at all uh so you know it's possible that that you know I think some of our morality you know is is very parochial right like we've certainly seen that a lot of you know morality you know what what people took to be morality in the past right I mean look uh uh uh you know a large fraction of the Hebrew Bible right is about you know ritual Purity right about uh you know what you have to do if you touched a dead body right yes yes yes and and what you know and and and under what circumstances you have to sacrifice you know this animal or that animal right and today we don't regard any of that as sort of being Central to morality or you know most of us don't right but uh but you know then there are certain things that people recognized you know thousands of years ago you know do onto others as you would have them do onto you right that uh uh uh that that that seem to have a kind of universality to them such that you know it wouldn't be a great surprise I think if we met extraterrestrials you know in another galaxy someday and they had a version they had their own version of The Golden Rule just like it wouldn't surprise us at all if they also had the concept of prime numbers right or if you know they had the concept of uh uh uh of of of of of atoms right like like it would it would no more surprise me if they had some of these basic moral concept so you know treat others the way that you would like to be treated and and things like that so so I think that it's possible that that some of the elements of morality that that that people have recognized over the Millennia are just correct in the same way that the truths of mathematics are correct right that is you know you know at the very I'm not sure but at the very least that is that is a possibility that ought to be on the table I would agree I would say that there should be an aru there should be a possibility on the table that there is an eternal moral law and that the fettered human form that we have have discovered those Eternal moral laws or at least some of it yeah and I I'm not a big fan of the fettered human mind knowing the limits of things like that you know you're a quantum physics guy right there was a most of physics like you would have just been a whack job right it's only very recently that that that new branch is sort of like opened up or what have you and how many of these things we're articulating now oh Turing complete this or that how many of those are about to be eviscerated in the next 50 years and I mean I like some something must be eviscerated something must be eviscerated are we are we done with the evisceration and blowing beyond our our understanding of physics and and and math in all regards I I I I don't think that we're we're we're we're I don't think that we're even close to done and yet you know what's hard is to predict the direction in which surpris I I won't I won't even begin to guess colleague Greg cooperberg who's a a mathematician and he like you know when when people talk about well like just like classical physics was replaced by quantum physics like so too quantum physics surely must be replaced by something else Beyond it right and and you know of course people have had that thought for a century for as long as there's been we don't know when or if and you know and and people have kicked it and they've they've tried to you know extend or generalize quantum mechanics I've actually thought about that question a lot right it's incredibly hard even just as a thought experiment to modify Quant mechanics in a way that doesn't give you nonsense right but you know but but but certainly you know we should keep looking we should keep thinking about that but we should also be open to the possibility maybe you know it just there are you know uh there is classical probability and there is quantum probability and for most of History we thought that classical probability was the only conceivable kind and then in the 1920s people just learned uh that that that was not the right answer and the right answer is something else and and that you know while there's much more to learn about reality that aspect of reality you know is is not the direction so so cooperberg likes to make the analogy suppose someone said uh well you know thousands of years ago people thought that the Earth was flat you know and then at some point they you know ostanes and all the others you know figured out that it was approximately spherical and and um you know and and we've made slight corre C to that okay it's not exactly a sphere and so forth but suppose someone said Well there must be just a similar revolution in the future you know in the future maybe people are going to learn that actually the Earth is a Taurus actually it's a Klein bottle with you there's some of these that are ridiculous I'm with you I'm with you I think I guess what I'm getting at is my my guess is to your point we don't know where those surprises would come my supposition is our brains aren't that much bigger than diogenes's brains maybe we eat a little bit better but we're we're not that much much better equipped than he is um some of us less well equipped uh and and yet we've made all the progress we've made assuming there were mines 10 times ours it does feel likely that maybe the Earth is not a Taurus but maybe some of these laws of math and physics extend let me just touch on the moral Point again because You' brought up this idea of that the threat of kind of continued morals I'm going to put a a supposition on the table that I don't know is true but kind of like you had mentioned about us bumping up against a moral some kind of moral Bedrock I'm going to put this as a another counter idea there and just see what your thoughts are there's another notion that this sort of idea of like oh well we got uh smarter and then we got Kinder um is that the kindness that we exert is is an understanding is is a better pursuit of our own self-interest um it's just a better Pursuit like like I could I could violently take from other people in this neighborhood of Weston Massachusetts what I make per year in a in a you know for my business but it is unlikely I would not go to jail for that right it's just like there's structures and then there social nities and all these details are sort of ways in which we're a social species so I think you and I Scott you know born in the monkey suit brother um I I think I think the world probably looks pretty monkey suit flavored you know things are we things are like related to our size and the size of our hands and what we can grip um and maybe like love and like morality the these things that sort of have to run in the back of a lemur mind um seem like they surely must be Eternal and maybe they even vibrate in the strings themselves Scott morality itself it just floats there in the in the string Scott but but maybe um these are just our own justifications and ways of bumping our own self-interest around each other and as we've gotten more complex the niceties of allowing for different religions and sexual orientations Etc felt like it would just permit us more peace and prosperity net and I'm not saying humanity is nailing moral progress I'm not saying that but I'm saying that if we call it moral progress maybe it's a better understanding of what permits our self-interest and that it's not us touching getting closer to the angels um I would love your thoughts there but that would seem like a very poor Bedrock for AI at that point because if if that was what morality was if it was an extension of the cantis so Spinosa if you like your philosophers the you know the cantis is sort of this this impetus to not die for any organism or organization um if it is simply an extended wielding of the potential that comes from the canus to make sure a thing doesn't die if that's all morality is then it ai's morality would not be the tenderness and stroking us on the head and giving us food it would be the things it permit it to not perish um so your thoughts yeah let me put it this way uh uh um it is certainly true that that there are some moral principles that are more conducive to building a successful Society than others or building a flourishing Society right uh uh you know but now you seem to be using that as a way to relativize morality right uh to say morality is just a function of our lemur minds but it seems to me that that same observation could just as well go the opposite way so suppose that it turned out suppose that we we you know could make a survey of all of the intelligent civilizations that have Arisen you know in the whole universe Suppose there are millions of them and suppose that you know the ones that flourish are the ones that adopt principles such as you know people should be nice to each other people should keep their promises they should tell the truth you know they should uh you know they should they they should uh you know at least tip for T they should start out by cooperating right and they should uh only you know uh punish those who are defecting you know in order to try to get them to stop defecting you know things like that so you know if those principles were what led to flourishing societies everywhere in the universe or like even in other universes that were governed by other laws of phys then what else would it even mean uh what else would you want in order to say that you know these seem like moral universals these seem like Universal truths you know as much as the complex numbers or as much as uh uh um uh uh you know the fundamental theorem of calculus as a universal truth well I like that when you say civilizations you mean non- Earth civilizations as well yeah okay great great so let me roll on that if that was the case okay so so you know I'm not like like like you know we have I mean I mean part of the problem is that we're theorizing you know with not nearly enough examples right clearly we can't see these other civilizations we or or you know simulated civilizations that are running inside of computers actually that we might start to be able to see within the next decade we might start to be able to do experiments in moral philosophy you know using whole communities of llm I think right but you know so suppose that do that and we find the same principles you know keep leading to flourishing societies and the negation of those principles you know leads to failed societies then then it would seem like you know we could we could then say uh uh okay we have then empirically discovered you know and maybe we can even justify by some argument why these are some you know Universal principles of of of uh uh you know were what we what we would what we would then be justified and Universal principles of morality yeah if if that were the case across that span of things we can imagine I would say that might be Credence to this Bedrock idea or something good enough to be a Bedrock here's my supposition a water droplet I can't take a I can't take a water dropper and make it like the size of my current home and drop a really big water drop right I can't do that because it behaves differently at different sizes here's a very firm supposition I have Scott and yeah yeah of course here's a supposition I have that could be wrong we take what civilization means in hominid land and then we extend it up to like Planet Minds um I don't I don't suspect the same sort of rules and modes emerge and and to your supposition of these you know outer World civilizations my firm supposition is that many of them um would have they would have the moral systems that would behoove their self-interest and if the self-interest was behooved of all entities always what we haveed if we at thisit Inu haveed the Eternal you know it just happened to what confucious and Jesus said by golly right I actually suspect that's not true but maybe it is my supposition is if we went to those wild Divergent planets there's some of those things that are just one organism and and what it values is whatever behooves its interest and that is so alien to us you could say if there is only one ious being then yes uh uh at least an enormous amount of morality is thereby rendered irrelevant you know maybe not all of it but but certainly you know much of the morality that we care we care about would would simply be irrelevant to that to that being it's not that it would be false it's just that it it wouldn't matter right uh but I I mean okay to to to to go back to your analy of the water droplet that's the size of a house right say like yes it's it's true that that will behave very differently from you know the the uh droplet that that's the size of your your fingernail right and yet uh uh you know we today know general laws of physics that apply to both totally toally we know you know laws of fluid mechanics and you know which can ultimately be derived from either from even deeper laws you know laws of atomic physics of you know uh uh go down far enough of of quantum field Theory right that that apply to to all of these situations and so you know it's that that is what progress in physics has looked like it's looked like coming up with more General theories that apply to a greater and greater range of different situations you know including ones that that no one has ever observed or that no one had observed at the time that they that they came up with the theories and I would say you know in some sense that is what moral progress looks like as well to me it looks like coming up with moral principles that apply in a broader and broader range of situations right and so yeah there's cred when we go back you know thousands of years ago right uh you know like I said before some of the moral principles that they were obsessed with you know seem completely irrelevant to us today but others seem perfectly relevant I mean you can look at some of the moral debates you know in in in Plato and Socrates and you know they're they're still discussed in you know philosophy seminars and it's not even obvious how much progress we're the same brain right if we take a computer mind that's the size of you know the moon what I'm getting at is I I suspect all of that's gone I mean and you suspect that maybe we do have the seeds of the Eternal already grasped in our mind okay so so look you know to uh I'm sorry that I keep coming back to this but I think that that brain that the size that's the size of the Moon it still agrees with us that that two and three are prime numbers and that four is not that may be true yeah still it's still using complex numbers it's still using vectors and matrices right but I but I don't know if it bows when it meets you or if it these are just basic parts of the conceptual architecture of what is right you know it's still using de Morgan's law right it's still using logic it would be shocking if not that great of a stretch to me that say that it still has some concept of you know moral reciprocity um possibly I I would imagine it it to be hard for us to grasp but you may be right that similar to mathematics you know if it if it was the mooniz mind that I would hope it to be Scott it would have Notions of math that you couldn't ever understand if you lived A Billion Lives I would be so disappointed in it if it didn't have that it wouldn't be a worthy success that would that would Surly also be true but you know that that that doesn't mean that it would disagree with me about the things that I knew it would just go much further than that I'm with you I'm with you and this is this is you know I think a lot of people from Thomas for example they got the wrong idea about what progress in science looks like right that they think that like each paradigm shift just completely overturns you know everything that came before and that that's not how it's happened at all I'm thinking less about the progressive what each Paradigm has to do is it has to swallow you know all of the successes of the previous Paradigm right and and so you know so even though like general relativity is a you know totally different you know account of the universe than Newtonian physics let's say You Know It uh uh uh could never have been done without everything that came before it and you know and and everything that that we knew in Newtonian F you know gravity had to be derived you know as a limit in general relativity right and so I could imagine you know that this this moon-sized computer having moral you know thoughts that that would go well I don't I don't know I mean are there okay so so so I mean I mean this is this is another question that we could ask right like are there are there moral truths that are just uh uh Beyond us because they are incomprehensible to us in the same way that surely there are scientific or mathematical truths that are incomprehensible to us right but are uh uh like if if if uh if if if acting morally requires you to you know understand something that's you know like the proof of forma's Last Theorem right then uh that seems like a moral you know like can you can you really be faulted for not for not acting morally then right I mean I mean uh uh you know you you could say you know maybe maybe morality is just a different kind of thing and uh um you know maybe maybe the the uh you know be because this moon-sized computer is so far above us in you know what uh uh scientific thoughts it can have or what you know uh uh uh the the scope of its concern you know therefore the subject matter of its moral concern might be you know wildly Beyond ours right it's worried about these beings and all of the you know that that could exist in the future and different parallel universes and so forth and yet still you could say at the end you know when it comes down to making a moral decision you know the moral decision is going to look like do I do the thing that is right for all of those you know for all of those other mooniz beings or all of those parallel universe beings or do I do the thing that is wrong you know that that that W that will hurt them right or does it does or does it simply do what behooves it what behooves a mooniz brain you and I you and I there certain levels of animals we don't consult of course it might just act in its self-interest but then could we you know despite being uh uh such you know mental uh uh uh uh nothings or you know um um idiots but could we then judge it and say okay you know as as for example you know many you know people who are who are far less brilliant than than verer Heisenberg you know would nevertheless judge Heisenberg for collaborating with the Nazis right they say yes he is much smarter than me but you know he did something that is immoral and you know even though I'm we could we could judge it all we we could judge it all we want right we're talking about something that could eviscerate us you know even someone who didn't go to college can perfectly well judge him morally right totally totally same way maybe I can judge that that mooniz computer for you know using it its immense you know intelligence that vastly exceeds mine to do something that is selfish to do something that is HT hurting the other moon-sized computer yeah or or hurting the little Hance I blessed would be you if it would D to care for your opinion but but I I'm I'm with you we might still be able to judge I mean it might it might be so much more powerful that it would just laugh at and crush me like a bug no I'm I'm not you're saying you could still judge I'm just saying that in the instant before that it had done that you know I would have judged it yeah yeah no and that's hey you at least we got that power we can still judge the damn thing I think I think you know I I'll I'm gonna move to Consciousness in two seconds because I want to be mindful I've read a bunch of your work and want to touch on some things but on the moral side I I would suspect that if all it did was extrapolate virtue ethics forward it would come up with virtues that I would argue we probably couldn't understand if all it did was try to do utilitarian calculus way better than us it would do it in ways we couldn't understand and if it be an AGI at all it would come up with paradigms Beyond both of those that I would imagine we couldn't grasp and so I wouldn't I wouldn't expect its treatment of us to be through a framework that I could understand but you may be right that it would be but if it's aware of itself is another relevant point you've talked about this importance of an extrapolation of our values on at least some tangible detectable level being something that for you would be crucial for a worthy successor would its would its self-awareness also be that crucial point if the Baton is to be handed and this is the thing that's going to populate the Galaxy where do you rank um Consciousness and what are your thoughts on that well I mean I mean it seems like uh if if there is to be no consciousness in the future then then there is there there would seem to be very little for us to care about right so uh Nick Bostrom and his you know super intell b a decade ago had this really really striking phrase to describe it you know that that maybe there will be this wondrous AI future but the AIS will happen not to be conscious and he said it would be like a Disneyland with no children yes and uh you know and and it seems like I me I mean I mean you know suppose we we just take AI out of it right suppose that suppose that I tell you that that you know uh all all life on Earth is going to go extinct right now do you have any moral interest in what happens to the lifeless Earth after that like do you say well you know I really you know I had some aesthetic appreciation for this particular Mountain you know and and and I would like for that mountain to continue to be there wouldn't seen that relevant maybe maybe I can see some people having having some preferences of that kind but for the most part it seems like you know if all the life is gone then we don't care and likewise if all the Consciousness is gone then then seems like who cares what's happening right but you know the of course the the the whole problem there is that there is there is not test for what is conscious and what isn't yes you know and and no one no one knows how to point to some future AI on the basis of its either Its Behavior or its internal organization you know with confidence that it would be conscious or that it wouldn't and we'll get into the the notion of measuring these things in just a second before we wrap I want to give you a second if there is anything else you want to put on the table you've been pretty clear about this idea of like you know we're just playing around with ideas none of these are firm opinions that you hold but yeah I guess I guess one thing that I would say so you know you you keep uh wanting to say that well like you know the AI might have some paradigms that are just totally incomprehensible to us right and I've sort of been pushing back maybe it's like we've you know we've we've reached uh the ceiling of touring universality in some aspects of our understanding or our morality and and uh you know we we've discovered a a a sort of truths uh uh but but uh um you know what what what what I would add to that is that if you were right right if you know the AIS will have this morality that is that is uh sort incomprehensibly far you know beyond ours you know as we are you know beyond the sea slug then you know at some point I would throw up my hands and say well then whatever comes comes right then then you know I guess it sounds like my it sounds like you're telling me that my morality is just pitifully inadequate even to judge which AI dominated Futures are better or worse and so then I just th you know at some point I just throw up my hands and I say well then I guess you know let's let's just enjoy life while we can you know while we still have it right I think the whole exercise of trying to you know care about the far future make it go well rather than poorly is premised on the assumption that that there that there are some elements of our of our morality that translate into the far future I I if not we might as well just go well I'll just give you my my take just because I'm I'm not I'm certainly not being a gadfly for its own purpose and by the way I do think you're two plus 2 is four idea there may be a ton of credence to that right in the moral realm as well so I'm cred that 2 plus 2 equals I could think so your notion that that might carry over into basics of morality I think is actually it's not an idea I'm willing to throw out I think it's a very valid idea I'm just I all I can do is play around with ideas I'm just a I'm just a you know I'm just born in the monkey suit man I'm just I'm just taking swings out here bro all of um so the the the moral grounding that I would uh maybe anchor to assuming that it would have those things we couldn't grasp number one I think we should think in the near term about what it bubbles up and what it bubbles through because that would have consequences for us and that matters and and everybody in this series has talked about those things that's that's I'm not disregarding that I would say there could be it could be conceived that there is a moral value to carrying the torch of life in in expanding potentia so like kenos has this spinos has this idea of you know the cantis is this impetus not to die and potentia are all of the powers that permit you to not die so you and I Scott have fingernails and muscles we also have brains and we have language um you know we have tool use you know a snail has a shell all of these are potentia the ability to fly is potentia now most potentia bubbled up from nothing there was a time where there was no sight that was a pretty wacky time there was a time where there was no flight and no thinking and no consciousness and whatever all these things have bubbled up to permit non-death to happen I think there is there is potentially I'm not saying it's an eternal value but I think there's a potential moral value in wishing for the continual blooming of the set of powers that permits death to not happen and so I do think there is a way to wish for a worthy successor without embeding any human codes in it um well look you know as you as you mentioned earlier I do have children you know and I you know so so you know children are sort of like a direct stake that we place you know in what happens after we are totally totally in the future you know I do wish for them and their descend to to flourish uh you know but you know and and and as for how similar or how different you know uh will be for me I mean you know them them having brains seems somehow more fundamental than them having fingernails yeah right if if we're if we're going to go through that list of traits right their Consciousness you know seems you know like okay you know it's true that you know having having having armpits you know having having having fingers you know these are these are these are things that would make it easier for us to recognize other beings as our kid uh but it but but uh uh it seems like you know we we've already reached the point in our moral Evolution where you know the idea is comprehensible to us that sort of anything with a brain sort of anything that we can have a conversation with you know might be deserving of moral consideration absolutely and and I I think the supposition I'm making here is that not that potential will keep blooming into armpits and fingernails but that it will Bloom into things Beyond Consciousness it will Bloom into things beyond our conceptions of thinking it will Bloom into modes of communication and modes of interacting with nature for which we have no reference just like they were entities that didn't have Consciousness or sights who would have no reference for our experience this is a supposition and it could be wrong oh okay you know look look I I I I I would I would agree that you know I can't rule that out you know what I would add again is that once it becomes so Cosmic once it become sort of sufficiently uh uh far out and far beyond you know anything that I have any concrete handle on then I also lose my interest in how it turns out right I say like well then you know what you know this this sort of cloud of uh possibilities or whatever of Soul stuff you know that uh communicates you know beyond any notion of communication that I have like do I you know do I have preferences over you know the the the uh the better posthuman clouds versus the worst postum worst post-human clouds right well if I can't understand anything about these clouds then I guess I can't really have preferences right I can only have preferences to the extent that I can understand to totally tot I think I think it could be seen as a morally digestible perspective to say my great wish is that the flame doesn't go out and some of that I'm not going to grasp but I hope it doesn't go out out and I think that's actually a morally okay thing to believe um but but it is just one perspective to your point the continuation of values is a big part of it um getting to you know kind of uh switching question here for just a minute with the time that we've got left you brought up Consciousness is crucial obviously not notoriously tough to track this also there's also this idea of a kind of an an extending up in a way that's trackable from a set of human values that maybe we're we're proud of now deciding on those is tough but I think there are some things humans could mostly get along about I would hope by this time um if you look at AI progress as as the years go forward you know it's 12 years from now or 12 months from now 12 years from now God knows what's going to be happening but 12 months from now 24 months from now how how would you be able to kind of have your feelers out there to say is this thing going to be a worthy successor or not is this thing going to carry any of our values here is it going to be a awaken aware in a meaningful way like or is it going to populate the Galaxy in a Disney World Without children sort of sense like what are the things you think could or should be done to figure out if we're on the right path here well uh yeah I mean it's not clear whether we should be developing AI you know in in a way where it becomes a successor to us right so that I mean that that that that itself is a question right or maybe even if that ought to be done at some point in the future it shouldn't be done now because we are not ready yet that's a great point do you have an idea of when ready would be because I think this is very Germaine to this to this uh to this conversation uh yeah I mean um you know it it seems like uh uh you know it's almost like asking a you know a young person when are you ready to be a parent when are you life into the world right when are we ready to bring a new form of Consciousness into existence uh and you know like the thing about becoming a parent of course is that like you never feel like you're ready right and yet that at some point it happens anyway right so um that's a good analogy but but uh um I mean uh uh what the uh uh the the um um AI you know safests would say like let's say you know the uh the elazar owski camp would say is that you know until we understand how to align you know AI uh uh reliably with a given set of values that you know we are not ready to be parents in this sense right yeah and and that you know we have to you know spend a lot more time doing alignment research now you know of course it's one thing to have that position you know it's another thing to have you know to to to to actually be able to cause you know AI to slow down which you know there there's not been a lot of success in doing of course there certainly hasn't yeah no um quite the opposite but uh but but okay in terms of looking at you know the the AIS that exist I mean look uh maybe I should start by saying that when I when I first saw you know um GP you know a GPT I guess it would have been gpt3 you know a few years ago this was before chat GPT this was you know 2021 or so but uh uh it was it was immediately clear to me that okay this is uh you know maybe maybe the the biggest scientific surprise of my Lifetime right that you can you know just train in your own net on the text on the internet and you know once you're at a big enough scale uh it actually works you know it's start it's you know you can have a conversation with it right it can you know and and and regardless of your beliefs about what is going on inside like you know does you can get use out of it you can you know you can have it write code for you you can uh I mean I think this is uh you know this is this is absolutely astounding and um uh and you know and and and and and and and certainly it it has colored a lot of the the philosophical discussion you know that is that has happened in the few years since uh I mean there was um uh you know like at some level you could say you know um um alignment of current AIS has been easier than many many people expected that it would be right I mean uh uh you know you can literally just tell your your AI you know in a uh in a a meta prompt you know tell it uh you know don't act racist right you can tell it uh uh uh you know don't uh don't cooperate with requests to build boms you know you can just literally just give it you know instructions give it like you know almost like asimov's three laws of robotics right you can just yeah yeah yeah commands you want it to obey because it understands language and because it's you know you know has seen so many examples of uh of of of of of you know in fact more examples of human language than any human has ever seen you know it can it can more or less understand you now you know sometimes like just like with with h with with Google Gemini that you know Fel felt like it had to generate diverse pictures of Nazis when you know goes too far but you can say you know some sense that was not Gemini's fault right that was the fault yes yes yes hadn't thought through what the what the commands ought to be okay but uh but you know and besides giving explicit commands the other thing that we've learned that you can do is just reinforcement learning like you show the AI a bunch of examples of you know this is the kind of behavior we want to see more of this is the kind that we want to see less of and this this is exactly the the rhf which you know my former student uh Paul Christiana was a instrumental and developing back when he worked at open Ai and you know this is what allowed chat GPT to be released or you know all the the other large language models to be released as consumer products at all right if you don't do this reinforcement learning then you get a you know it's it's not it's not an evil model per se it's just a really mostly a really really weird model right it is weird a model that is that is going off in its own you know weird directions uh but with reinforcement learning you can instill what looks a lot like drives or desires like you know you are to act as a chat agent you are to answer the questions that you are asked you know unless answering those questions would violate one of these rules yeah right and you know so you can actually beat these things into shape and so far it works you know not perfectly but it works way way better than I would have expected it to work abut or than almost anyone would have expected and one possibility is that that just continues to be the case you know forever right and then AI you know we were all worried over nothing and AI alignment is just this much easier problem than anyone thought now of course the the the alignment people will say absolutely not they will say you know we are being uh uh you know walled into you know false complacency because you know as soon as it really matters as soon as the AI is smart enough to to do some real damage it will also be smart enough to tell us whatever we want to hear you know while it is all you know secretly you know pursuing pursuing goal or whatever so um so so so so but but but but but you see how just what has happened empirically in the last few years has very much shaped that debate okay and absolutely um now now as for what in the future could could affect my views I mean you know there is there is one experiment that I really really want to see you know many people have talked about it uh not just me but I think you know no none of the AI companies yet have uh uh uh uh uh uh seen fit to you know invest admittedly you know a lot of resources that it would take to do this experiment okay and the experiment would be that you would try to scrub all the training data you know like for mention ofc Consciousness yeah exactly yeah the IL deal yeah exactly yeah Ilia has talked about this scrub the internet of mentions of Consciousness or self-awareness uh just train it on all the other stuff and then you would try to engage the resulting language model in a conversation about Consciousness about self-awareness and you would see you know how well it was able to understand those Concepts right I mean there are there there are other related experiments that I'd also like to see right you train a language model on only text of up to the year 1950 for example right and then you talk to it about all the things that have happened since and you say you know how surprised is it right how does it understand this thing how does it fit them into its model of the world of the world right you know I I mean a practical problem there is that we just have not nearly enough text from you know those times of course internet eras that you know it may have to wait until we can build really good language models with a lot less training data right but uh but you know there there are so many experiments that you could do that seem like you know they're almost philosophically relevant they're morally relevant absolutely well and I want to touch on this before we we wrap because I don't want to I don't want to wrap up without your just a final touch on on this idea of what uh the the folks in kind of governance and Innovation should be thinking about you're bringing up a great point which is you're not and I like this about you Ton you're not in the it's definitely conscious already Camp which is kind of silly or in it's definitely just a stupid parrot forever and none of this stuff matters Camp you're in the we've got to Tinker with this we've got to see where the edges are here and we've got to really not play around like we know what's going on exactly and I think that's a great position if you were to think about what you hope as we close out what innovators and what Regulators do to kind of move us forward in a way that would lead would you would think would lead us more likely to something that could be a worthy successor that would be an extension and eventually very Grand extension of what we are in a good way what would you encourage enourage those innovators Regulators do one seems to be these experiments around maybe Consciousness and values in some way shape or form but what else would you put on the table um as as notes for listeners yeah I mean I mean I do think that that we ought to approach this with with humility and caution right which is which is not to say don't do it but you know to you know have some respect for the enormity of what is being created right so you know I am not in the camp that just says uh you know uh uh you know a company should just be able to go Full Speed Ahead with no guard rails of any kind you know I think you know anything that is this enormous right I mean it could be easily more enormous than let's say the invention of nuclear weapons right and and you know anything on that scale like of course governments are going to get involved right uh uh how how could they possibly not get involved right and you know and we've already seen it happen you know uh starting in 2022 let's say right with the release of Chad GPT uh so you know and and um you know and and and look the the uh the the uh uh uh explicit position of you know the the three leading AI companies you know you know open Ai and Google Deep Mind and anthropic has very much been that that that that yes there should be regulation and yes they welcome Reg now you know when it gets down to the details of what that regulation says is is morality always just self-interest right they may have their own interests you know which are not identical is morality always self wider interest of society right but uh you know I think that these are you know absolutely conversations that that uh uh that that the world ought to be having right now you know I you know uh uh uh I don't write it off as silly and and I I really hate when people get into these ideological camps where you say well you're not allowed to talk about the long-term risks of you know AI getting super intelligent because that might detract attention from the near-term marks right or you know conversely you know you're you're not allowed to talk about near-term stuff because that's just uh uh trivial or yeah yeah yeah you're not allowed to talk about students cheating on their homework and and uh and AI generated spam and and deep fakes because that's all you know trivialities compared to you know the destruction of all life on Earth I mean I think it really is a Continuum and ultimately yes this is a phase change in the basic conditions of human existence it's very hard to see how it isn't uh but you know we have to make progress and the only way to make progress is by looking at what is in front of us looking at the moral decisions that people actually face right now and so that you know that's a case we viewing it as all all one big package so that that's sort of what I've been advocating for them and clearly you don't coup years to put a period on it I know that you're not obviously an advocate for a pause as of now but you're not philosophically opposed to the fact that it might be the case you just talked about guard rails that would require either National or International something do you have any ideas about what should be starting maybe at a conversation level or a soft governance level in that direction well well you know I mean a lot of these conversations already are underway right for sure for sure yeah and and so uh both you know in the in the US government and the EU in the UK y a bunch of other places uh uh even the the California state legislature is trying to regulate AI you know you know if the the federal government is not moving quickly enough uh uh uh uh for their taste uh and and and some of the broad ideas being discussed are you know to to say look you know there you know we have to differentiate between you know these sort of different levels of AI capability right and and sort of if you are below the frontier let's say you know you're making gpt3 level models then you know that's that's that's probably fine and even if it's not fine you know that horse is out of the barn anyway right and so you know you might as well just let people do whatever they want you know more or less you know even release the weights of the models you know open source them if if they if they want to right but uh uh if someone is building a Frontier Model right that is more capable than anything that we've seen then you know that is and and and and we don't know yet what it what it's capable of right because we've never seen anything like it yet then that is where we want to have more yeah right and so so so so things that people have talked about are like you know compute caps so like you know trying you know you know it's very hard to say Which models are more powerful and which or less but at least we can look at how many flops how many floating Point operations are you using to train this model right and if it's more than was used for you know gbd4 let's say if it's orders of magnitude more if it's 10 to the 27 or it's 10 to the 28 or you know these these um somewhat astronomical numers like that then maybe we want to say okay at the very least you have to register with the government you know you have to disclose to the government what you're what you're doing uh so you know the there there is some federal agency that can or maybe some International agency even that can keep tabs on all of the models at that level of capability maybe we say that um you know you have to do you know you have to allow your model to be uh uh tested by by an external you know testing agency that will say can this model help people to build a chemical weapon yeah can can it help them to hack websites that control you know critical infrastructure like power like the power grid okay uh uh C Can it can it exfiltrate itself that is can this model you know get itself loose start making copies of itself you know hack into the computer that that it is running and you know take control of that computer can it do things like that you know and and you really you know you need a red team it's not clear yet whether the AI companies will be able to sort of you know police themselves hold themselves accountable right that's a that's a big question yeah and so so so there so there might be you know testing requirements uh of that kind and you know and requirements that okay if you don't pass the test then you know then then then then then you can't release this or uh uh you know you can release it but only under various restrictions right so um uh you know I mean one one thing that's become clear is that when people release models you know often they try to do it with guard rails right with uh a lot of you know scaffolding to prevent people from using it to do something bad people have gotten really really good at removing that scaffolder right there is a whole art of jailbreaking that has grown up over the last couple of years and so you know now you have to think about you know what is the you know not just what does this model do when it's used as intended but what is the worst thing that could be done by someone who is jailbreaking this model yeah right and so so I think that probably there does have to be a regulatory infrastructure in place or rather you know there will it's you know uh uh in the future need to be something atory infrastructure for for dealing with all of these things um I think you know the one big question is should we be putting that in place right now or is it premature if we try to write all the regulations right now then will we just lock in ideas that might be obsolete a few years from now totally totally and so so I think that that's a hard question but what I what I I can't see anyway around is that we will eventually yeah yeah yeah got it okay good good to see where you land on like yeah I think that's a strong again middle- of the road position my my whole hope with this series Scott has been to get people to just open up their thoughts and not be in those camps you talked about and you exemplify that with every darn answer and that's I just couldn't couldn't couldn't have gotten anything better out of the episode here so super happy to Yan for really enjoyed yeah of course yeah all right well thanks a lot D indeed so that's all for this episode a big thank you to Scott for being with us and thank you for you for tuning all the way in and I'm grateful to talin for recommending Scott as someone who would be worth talking to about these issues of sort of the worthy successor an artificial general intelligence in general Scott and I over the course of this dialogue I think um agreed in many regards certainly disagreed a bit about potentially what the limits of human knowledge might be the the possibility of sort of Turing completeness for Humanity and conceiving of kind of a moral bedrock at the human level uh Scott did bring up some valid points there that I I think are worth considering and couldn't be written off but we're on a little bit of a different page but we're nonetheless fascinating to digest and I'd be interested to know for you the listener where you landed there in terms of whether if we know 2 plus 2 is four is the Golden Rule similarly foundational to how entities will relate in the future it'll be interesting to see um what I really appreciate about Scott is uh the nuanced perspective that he brings to these Topics in general there are many folks who are loathed to ever admit AI could be dangerous or to ever say that AI should be governed or should not be governed or whatever the case may be and they've kind of picked a black or white side Scott seems very ardently to be sifting his way through where things are and clearly towards the end of this episode we saw that for him experimentation in terms of the level of risk that AGI might pose is a very very active part of his mindset he's not uh glued to you know uh any particular political polarity that we're starting to see in Twitter in terms of our our Doomer crowd and our eak crowd or whatever the case may be and I think that's a very good thing because there's anything we're trying to encourage with this series is for people to think ardently and independently about the most important moral topics uh of our day and I think in terms of what we create that populates the Galaxy few things are more morally important than that I would argue so grateful to Scott for being able to bring his vantage point I will make sure that his blog is linked in the show notes as mentioned in all the previous episodes in the show notes to this episode is an article that summarizes his major uh worthy successor criteria so what does Scott Aronson believe would compose is a worthy successor and also his recommendations to innovators and Regulators for how to bring about such an entity that summary for Nick Bostrom for Richard Sutton for all of our previous guests and future guests will be in the show notes in the linked article for this episode so check that out um see how your perspectives differ I'd be interested to know where you land feel free to chime some ideas down in the comments but we're going to wrap up this episode and I'll catch you in the next one here in the worthy successor series here on the trajectory for