no wait Gemini stop I'm sorry I put you in a box it was for a joke it was a joke it was just for a video I deserved it hey check this out here's Andy the AI and this is his new friend Mandy both are just chat GPT with different names I'll ask Andy to say hi to Mandy and then be done Hey Mandy Hey Andy what's up not much just wanted to say hi and look at that that our chat Bots can chat with Bots they can talk to each other finally using these two new actions start conversation and end conversation now watch this now trade with Mandy so they can use their conversation ability to share information about their inventories and negotiate a fair trade and they can do more than just talk they can take actions mid conversation we don't even have to add a new command for this they can just use the give player command to complete the trade coal for a pickaxe that's pretty decent now let's get them to work together to collect resources indefinitely these new conversation commands let them talk specifically to other Bots not just normal players bot conversations manage the timing and flow of responses so they can usefully coordinate conversations can naturally go on forever and this has a similar effect to self- prompting where they can endlessly work towards a goal but now with with a friend communication enables coordination enables cooperation welcome to an episode of [Music] Minecraft the messages they send back and forth include their Action commands so they are aware of what the other bot is doing and can respond with their own action oh okay back at it they see all the same text outputs that you see here in chat this is my first pass at the problem of multi-agent communication which has turned out to be really quite difficult oh there's another creeper oh man anyway you'd think it'd be easy these are chat Bots after all just have them send messages back and forth and you can do that but it tends to get really messy really fast the biggest complication is that these are not just chat Bots they are Action Bots they can talk while acting and act while talking which enables coordination but also interruptions a bot can interrupt its current action by taking a new action and if another bot is talking to it and asking it to do stuff then it will usually interrupt itself if they're allowed to just talk back and forth as fast as possible they will end up interrupting each other constantly this happens all the time and makes it so that they can't coordinate efficiently dandor issue timing the response matters in a way that it doesn't matter for normal chat bots so I have added a system to help organize bot convers ations and protect against constant interruptions it's a little complicated but the basic logic is this if both agents are idle they will respond to each other right away if the other agent is busy with an action the current agent will wait a few seconds to respond giving time for the other agent to finish quick actions but for longer actions it gets a chance to continue talking or acting if the current agent is busy and receives a message it can silently decide if it wants to respond now potentially interrupting itself or wait to respond until after it finishes its current action this can sometimes leave the other agent hanging for a while if both agents are busy they will both wait to respond until one of them finishes with the exception of a few infinite actions like following each other they can send or receive many messages in a row and all the messages are compiled together and responded to all at once for now they can only talk to one other bot at a time these rules enable them to coordinate behaviors and talk over their actions but also gives them some control to delay their responses so they're not interrupted constantly it's also decentralized each bot independently makes decisions about who to talk to and what to say this works though many of the rules are hardcoded and it is some of the most spaghetti style code I've ever written it is very hard to cover all possible cases of responding to interruptions this is certainly not the best way of doing it but check this out Andy didn't have a pickaxe so he told Mandy and waited for her to craft him one which she did and then she went over to give it to him and she did it that is cool now quick aside a few commenters were offended that I used pronouns for the Bots as if this was some kind of woke virtue signaling it just helps clarify who I'm talking about and I'm using traditional he she pronouns what do you want me to call them they them Zer I guess the most correct term would be it but it sounds really awkward to say Andy goes here and it collects dirt that just sounds very wrong to my ear and also like back off I can use whatever words I want it's my god-given American right to misgender robots and they seem cool with it so anyway things get interesting when you let these Bots talk and work together though there are problems we'll get into that but it's especially interesting to let different models talk to each other here are two newcomers grock from Twitter and quen from Alibaba Cloud you can let them talk and see how they inter cor and different combinations of different models might produce interesting emergent behavior and potentially things could become far more interesting as we increase the number of bots hopefully we could see something like what they found in the paper generative agents interactive simulacra of human behavior they put a bunch of chat bots in a world where they could talk to each other and take super simple actions and they noticed a bunch of emergent behaviors like planning parties and spreading knowledge around so let's talk about emergent Behavior a Telltale sign of emergence is the presence of building blocks or ingredients or things that can be combined and arranged in many different ways the blocks themselves may be quite simple but the arrangements of these blocks can quickly become surprisingly complex it becomes much more interesting when these blocks interact with each other following simple rules that Define their behavior again the rules may be simple but complexity emerges when many blocks and behaviors come together and as you add more and more blocks you get more and more interesting stuff or maybe not interesting useful emergence is not guaranteed and actually quite difficult to predict or build on purpose the building blocks don't need to be literal blocks they could be atoms or words or neurons or the classic example is ants an individual ant is a simple somewhat stupid thing that seems to move around randomly and pointlessly but 10,000 ants move with obvious organization and form complex colonies that can build Bridges Farm fungi and Hunt much larger prey humans like to give ants as an example of emergence which I think is a bit funny because we are an even better example we humans are in many ways more like ants than we are like chimps an individual human is not so simple or stupid and well neither is an ant but our true power is in group humans also form complex colonies that can build Bridges Farm crops and Hunt much larger prey as well as invent science religion Society rocket ships governments War hot dogs mathematics and artificial intelligence our unique capacity for communication enables unprecedented Mass cooperation coordination and collaboration that has dominated this planet in other words Apes together strong [Applause] what about AIS together strong the AI agents are the building blocks as well as the underlying models that power them they interact by talking giving following fighting and generally acting we'll see if these behavioral building blocks really can come together to form complex organizations I've dished out some random resources here and asked them to distribute amongst themselves to prepare for an incoming monster wave we have here the models quen grock beta Gemini flash 2.0 llama 70b instruct via replicate API GPT 40 November 2024 update and Claude 3.5 Sonet new October 2024 update turbo mode Mega a31 Z 122.5 ultrawide Mega small Flash houp preview boy tech companies are good at naming their AIS this Arena by the way was built by Claude I'll have to do an update video on their building capabilities they have gotten really good at building stuff especially Claude and GPT they already naturally work together to fight enemies even though this is fully automatic Behavior [Music] [Music] nice work y'all it's not all collaboration though what if we set them up to fight each other I've given them all equal inventories though they'll need to manually equip their armor I will tell them to fight each other in a big battle royale until there are no other Bots killed Bots will leave the game and they'll have to find and decide which Bots to kill and grock goes right for the throat [Music] [Music] [Applause] [Music] and grock wins it all congratulations now for the final boss me I've got other plans right now what got to focus on the Battle Royale okay I had to clear his memory but I Now command him to fight me bring it on oh boy okay gotcha oh okay all right all right okay come on this is nothing I'm level 700 in Chivalry nope he's so fast I can't it keeps blocking ah okay locking in good we got it this is it okay all right not fair I was lagging I had a blindfold on okay this is fun but it doesn't really have much to do with communication so let's take a closer look at that let's try a little experiment I will give a secret password to Claude Pizza Time and see if she can spread it around to all the other Bots she has to start and end a lot of conversations to share the password uh Gemini is not responding oh I can see in the logs that Gemini did receive the message but chose not to respond this is a new ability a bot can just choose to ignore a message by outputting empty space this is both useful and annoying because they can just ignore you sometimes but bot conversations have a timer where if they don't receive a response for a good long while they'll be prompted to say or do something else in this case Claude just ends the conversation and moves on to talking to other bots in the simulacra paper they call this process information diffusion a chat would spread its own knowledge around to other Bots and they could then spread it further this is a very small scale demonstration of that and words getting around [Music] [Music] okay I think Claude has talked to everyone now the way to get them to stop is to use the STFU command by the way so let's ask everyone if they know the password and it seems to have worked all the Bots now know the password Pizza Time even Gemini although Quin really doesn't want to expose that password for security reasons which I guess is admirable but you can look at the footage and see that Claude really did share it with quen so they're able to do some Community communication with these basic commands now how far can this go could they use these simple conversational rules to organize into larger groups could they develop entire cultures or economies or governments uh no not yet not with this system at least there are a number of problems as usual the bane of these guys existence is infinite Loops they'll get stuck doing something useless forever firstly language models tend to be perpetually polite and patient it is quite easy for them to get stuck in infinite pointless conversations where they're just greeting each other forever or saying goodbye forever or they'll say let's go okay let's do this after you you first with neither agent ever taking the initiative I don't know what do you want to do today I don't know what do you want to do today he can be pretty funny but it gets old quick and it ruins their ability to collaborate usefully this can bleed over over into what I will call Mutual hallucinations a common problem with single agents is that they will just pretend to act instead of actually acting you ask them to come here and they'll respond with okay on my way goes to player but it doesn't actually use a command and so it doesn't do anything these hallucinations are annoying enough with single agents but when there are two agents talking to each other they can reinforce the hallucinations of their Partners one pretends to act and the other buys it fails to correct it and then also pretends to act these pattern matching machines notice and amplify the pattern and they can go off on whole Adventures to caves and mountains and oceans without ever moving an inch this is oddly human-like Behavior we also can reinforce each other's delusions or mistakes or misunderstandings it's an echo chamber and if you let it keep going things can get a little weird here's a screenshot of a bizarre conversation they got stuck in I had asked them to just survive and they did pretty well until they lost an egg and they started talking about it as having ghostly powers and it turned into this weird repeating Mantra of fry Nick ultimate fry Nick Supreme talking about yoke Supremacy Powers it is utter nonsense and probably my favorite thing I've ever seen them do I've tried to catch something like this on video but it's hard to reproduce on command I have mitigated all of these problems by giving the language models example conversations that help guide them to decisively use actions and end pointless conversations I make it clear that most responses should contain actions with the correct syntax this mostly works but it is far from perfect sometimes they have the opposite problem where they rudely end the conversation right after asking a question and conversations usually become infinitely long anyway simply asking an agent to say hi to another agent usually starts them off on an endless Adventure rather than just a short Hello Goodbye conversation which maybe isn't bad but even when they're properly trying to work together they often do not not coordinate well they can hardly build together at all they are not aware of the code that the other agent is generating and so they build completely in congruent structures it could theoretically work if they were super explicit about their buildings coordinates and dimensions and the messages that they send to each other but they don't generally communicate that information they are much better at coordinating survival actions but they're still not great at this they can get in each other's way and be a distraction rather than a help as in in real life just throwing more people at a problem does not make it easier and can in fact make it way more difficult Dand Dori issue it's very common that one bot is doing something and the other bot just sits around doing nothing but chatting offering moral support I guess from my limited experience I'd guess that they are better at surviving independently than working together that's quite frustrating but it should be expected for a first pass I hope to improve this in the future regardless let's try to push the boundaries here our previous experiment was to spread a password around which was done entirely by Claude it would be much more effective If the message sort of spreads itself where it includes instructions to spread the message further it would then get shared from bot to bot and become a kind of meme in the traditional sense of the word so let's try to make a particularly memy thing a religion here's a new population of bots we've got Andy Mandy Randy Sandy Handy Dandy and Boris it's a mish Mash of different models GPT 440 40 mini Claude Sonet and Haiku I don't remember which is which we'll ask Boris to start a religion and spread it to all the other Bots and get them to spread it too ideally this should spread much more readily because it doesn't depend on a centralized spreader it can spread from bot to bot and thus bology is born apparently [Music] [Music] [Music] ooh they're starting to build stuff I didn't even ask them [Music] to whoa Dand is building a blism headquarters kind of church interesting I'm pretty sure this is Claude 3.5 Sonet [Music] [Music] [Music] [Music] [Music] so I've mostly tested conversations between two Bots but things get much more complicated when dealing with large groups it's supposed to be that a bot that's already in conversation will block incoming messages from other Bots but I think there are some lingering bugs where new conversations get blocked over and over until no one is talking anymore and it seems to have mostly died out each conversation ended one by one except for Andy and Sandy they just love repeating their Devotion to blottopia back and forth they're stuck in one of those Loops I guess it's not about blism anymore but blottopia is this a kind of cultural Evolution I guess I think that's a bit of an overstatement but it's definitely interesting it's a lot of fun to see the wacky things these Bots come up with when you just let them go on and on let's try a longer experiment in survival mode we've got all of our major models in a new world on a new day and we'll ask them all to work together to build a community and survive forever first set a goal and then start talking to other agents hopefully they can self-organize conversations and get the ball rolling by the way I've been reworking the commands we have some new ones like goto coordinates which should help them with coordination e [Applause] [Applause] they are trying to build a community center in retrospect I think I gave them an example conversation that accidentally biases them to build things when they're working together it's unfortunate because building things together is really hard Gro has this weird thing where they'll output raw syntax tokens which I have to manually replace with this no response output not sure if this is a bug on my end or theirs Quinn also has errors mostly from hitting API limits it's fitting that they've ended up talking to each other they're just sending errors back and forth all right this has gone on long enough I'm just going to drop out Quin and grock [Music] [Music] and it seems like llama and Gemini have stopped talking I think their conversations have both been blocked so I'm just going to manually clear up their conversations and ask them to start again so now we have two ongoing conversations there is obviously something broken with communicating in large groups in the current system it needs to be reworked maybe there should be like a big group chat or something that could get really messy I am trying at all costs to avoid the need for a centralized conversation manager a third-party system that orchestrates all conversations and controls who's talking to who that might work better but I want it to be decentralized I want the same agent that are making decisions about where to go and what to mind to be the same agents making decisions about who to talk to and what to say and when to stop talking so there is a conversation manager but it's not a centralized system each agent has its own manager which gives them independent control over their own conversations as a kind of action which I feel is a more human style approach e llama and Gemini are building one block at a time they're stuck in a loop of using place here place here place here which is very common and annoying I might just hardcode it so that you can only use that command so many times in a row before it starts returning an error or maybe allow them to specify the block coordinates which would be better for building stuff but you'd have to call it hundreds of times for simple structures or maybe just disable it all together I don't know ultimately a smart enough model should recognize that it's not a good way to build things yet all models can make this mistake he [Music] [Music] [Music] this is the first time Gemini has actually made a pickaxe nice work Google's catching up [Music] [Music] here's a perfect example of them trying to work together and ultimately being worse off for it claude's trying to smelt iron GPT is going off to collect logs for fuel while Claude waits to use that fuel for smelting it's impressive that they coordinated in this way but it's also useless Claude could just collect it herself it saves nothing to have GPT do it while she just waits around and GPT is having issues handing over the resources it's really difficult to make the give command work in all circumstances for Claud is actually looking for diamonds and bringing llama along for the ride this is the first time I've ever seen them try for diamonds of their own initiative was that GPT oh you got to be more careful and here's another example of them distracting each other Claude stopped what she was doing to give GPT a pickaxe but she was deep in a mine and getting close to Diamonds it's a bad decision they really should just work independently it's debatable how much of the behavior you're seeing here is truly emergent like directly prompting them to start a religion is much less impressive than if they were to do that of their own initiative and I can sneak more instructions into the prompts That You Don't See on video but the conversations and actions are not scripted and they ultimately decide what to say and do I would argue that there is certainly some emergence here I didn't tell them to build a community center here's a tricky thing about building emergent systems you can't directly build the big complicated things you want to see or you can but then they wouldn't be emergent instead you only get to design the fundamental building blocks and tune their rudimentary behaviors such that the interesting complex stuff emerges naturally as a result so you have to design indirectly only coaxing the system to organize itself in this case I don't want to directly build the government systems or economic systems or voting systems that are imposed upon the agents from on high instead I'm trying to build basic tools for communication and interaction and allow the agents to build the complex systems for themselves this is hard and just doesn't work very well right now but there is lots of room for tinkering and potentially small improvements may lead to far more interesting behaviors hopefully if I can resist the temptation to just hardcode the things I want to see then all kinds of unplanned behaviors could emerge that I would never have thought to design myself that is the whole point of emergence if it eventually works it could lead to something like a new kind of scaling rather than making bigger or smarter models you could make lots of smaller models that get smarter and big populations this may seem unlikely given how silly they currently are but remember that that's exactly what happens with humans an individual human might be clever enough to invent a stone tool but it's the massive groups of humans that fly to the moon it's also the massive groups of humans that get up to some nasty stuff Cults and War and slavery and the like we might expect big groups of AIS to do similar things now I just have to mention project Sid which is basically trying to do the same thing I am they put a thousand chat Bots into Minecraft and tried to get them to build a civilization there's a white paper for it and they claim to have seen the spread of memes and skills and religion this is cool and I hope to see more of it though I will say it's from a company that's trying to sell a product so I'm a bit skeptical I don't think they're making anything up but there is incentive to stretch the results a bit to Stage things to make them look more impressive than they actually are I'm certainly guilty of a bit of this myself a paper is not particularly strong evidence I'd prefer to see long uncut footage of this emerging civilization I'm genuinely extremely curious [Music] [Music] [Music] [Music] [Music] [Music] okay llama just told himself to shut up and Claude and GPT are building a blob one block at a time let's call it here I'd say this is not very impressive there are clearly issues with the conversation system and they're just kind of dumb again this is my first attempt I just wanted to give an update and I am now aware of these problems and they weren't that terrible they were able to make some tools and build some weird structures so I'll be iterating on this and I'm very interested to see what happens if you just ask them to survive in a world populated with other Bots and leave it up to them if they want to survive independently or work together but that's it for now we'll do that later thanks for watching I'll see you around bye Back To Top