Title: User Comment Replies — AI Alignment Forum
Description: A community blog devoted to technical AI alignment research
Keywords: No keywords
Text content: User Comment Replies — AI Alignment Forum   This website requires javascript to properly function. Consider activating javascript to get access to all site functionality. AI ALIGNMENT FORUMAFLoginAll of Adam Jermyn's Comments + RepliesRSPs are pauses done rightAdam Jermyn1y44Anthropic’s RSP includes evals after every 4x increase in effective compute and after every 3 months, whichever comes sooner, even if this happens during training, and the policy says that these evaluations include fine-tuning. Reply1Hoagy1yDo you know why 4x was picked? I understand that doing evals properly is a pretty substantial effort, but once we get up to gigantic sizes and proto-AGIs it seems like it could hide a lot. If there was a model sitting in training with 3x the train-compute of GPT4 I'd be very keen to know what it could do!Alignment Grantmaking is Funding-Limited Right NowAdam Jermyn2y93This matches my impression. At EAG London I was really stunned (and heartened!) at how many skilled people are pivoting into interpretability from non-alignment fields. ReplyEIS IX: Interpretability and AdversariesAdam Jermyn2y20 Second, the measure of “features per dimension” used by Elhage et al. (2022) might be misleading. See the paper for details of how they arrived at this quantity. But as shown in the figure above, “features per dimension” is defined as the Frobenius norm of the weight matrix before the layer divided by the number of neurons in the layer. But there is a simple sanity check that this doesn’t pass. In the case of a ReLU network without bias terms, multiplying a weight matrix by a constant factor will cause the “features per dimension” to be increased by that ... (read more)Reply1Stephen Casper2yThanks, +1 to the clarification value of this comment. I appreciate it. I did not have the tied weights in mind when writing this. Paper: Constitutional AI: Harmlessness from AI Feedback (Anthropic)Adam Jermyn2y20A thing I really like about the approach in this paper is that it makes use of a lot more of the model's knowledge of human values than traditional RLHF approaches. Pretrained LLM's already know a ton of what humans say about human values, and this seems like a much more direct way to point models at that knowledge than binary feedback on samples. ReplyYou can still fetch the coffee today if you're dead tomorrowAdam Jermyn2y32I might be totally wrong here, but could this approach be used to train models that are more likely to be myopic (than e.g. existing RL reward functions)? I'm thinking specifically of the form of myopia that says "only care about the current epoch", which you could train for by (1) indexing epochs, (2) giving the model access to its epoch index, (3) having the reward function go negative past a certain epoch, (4) giving the model the ability to shutdown. Then you could maybe make a model that only wants to run for a few epochs and then shuts off, and maybe that helps avoid cross-epoch optimization? ReplyIs the "Valley of Confused Abstractions" real?Adam Jermyn2y10Yeah. Or maybe not even to zero but it isn’t increasing. ReplyIs the "Valley of Confused Abstractions" real?Adam Jermyn2y20Could it be that Chris's diagram gets recovered if the vertical scale is "total interpretable capabilities"? Like maybe tiny transformers are more interpretable in that we can understand ~all of what they're doing, but they're not doing much, so maybe it's still the case that the amount of capability we can understand has a valley and then a peak at higher capability. Reply1Jacques Thibodeau2yAs in, the ratio between (interpretable capabilities / total capabilities) still asymptotes to zero, but the number of interpretable capabilities goes up (and then maybe back down) as the models gain more capabilities?Multi-Component Learning and S-CurvesAdam Jermyn2y10So indeed with cross-entropy loss I see two plateaus! Here's rank 2:(note that I've offset the loss to so that equality of Z and C is zero loss)I have trouble getting rank 10 to find the zero-loss solution:But the phenomenology at full rank is unchanged:ReplyMulti-Component Learning and S-CurvesAdam Jermyn2y10Woah, nice! Note that I didn't check rank 1 with Adam, just rank >= 2. ReplyMulti-Component Learning and S-CurvesAdam Jermyn2y10Erm do C and Z have to be valid normalized probabilities for this to work?Reply1Lawrence Chan2yC needs to be probabilities, yeah. Z can be any vector of numbers. (You can convert C into probabilities with softmax) Multi-Component Learning and S-CurvesAdam Jermyn2y10(with the caveat that this is still "I tried a few times" and not any quantitative study) ReplyMulti-Component Learning and S-CurvesAdam Jermyn2y20It's a good caution, but I do see more bumps with Adam than with SGD across a number of random initializations. Reply1Adam Jermyn2y(with the caveat that this is still "I tried a few times" and not any quantitative study)Multi-Component Learning and S-CurvesAdam Jermyn2y10Something like this?def loss(learned, target):   p_target = torch.exp(target)   p_target = p_target / torch.sum(p_target)      p_learned = torch.exp(learned)   p_learned = p_learned / torch.sum(p_learned)      return -torch.sum(p_target * torch.log(p_learned))Reply1Lawrence Chan2yWell, I'd keep everything in log space and do the whole thing with log_sum_exp for numerical stability, but yeah.   EDIT: e.g. something like:Multi-Component Learning and S-CurvesAdam Jermyn2y10I'd be very excited to see a reproduction :-) ReplyMulti-Component Learning and S-CurvesAdam Jermyn2y62I agree with both of your rephrasings and I think both add useful intuition!Regarding rank 2, I don't see any difference in behavior from rank 1 other than the "bump" in alignment that Lawrence mentioned. Here's an example:This doesn't happen in all rank-2 cases but is relatively common. I think usually each vector grows primarily towards 1 or the other target. If two vectors grow towards the same target then you get this bump where one of them has to back off and align more towards a different target [at least that's my current understanding, see my reply... (read more)Reply1Lawrence Chan2yI caution against over-interpreting the results of single runs -- I think there's a good chance the number of bumps varies significantly by random seed.1Lawrence Chan2yThere's lots of ways to do this, but the obvious way is to flatten C and Z and treat them as logits. Multi-Component Learning and S-CurvesAdam Jermyn2y10I don't, but here's my best guess: there's a sense in which there's competition among vectors for which learned vectors capture which parts of the target span. As a toy example, suppose there are two vectors, a1 and a2, such that the closest target vector to each of these at initialization is c. Then both vectors might grow towards c. At some point c is represented enough in the span, and it's not optimal for two vectors to both play the role of representing c, so it becomes optimal for at least one of them to s... (read more)Reply2Lawrence Chan2yOh, huh, that makes a lot of sense! I'll see if I can reproduce these results.  I'm not sure this explains the grokking bumps from the mod add stuff -- I'm not sure what the should be "competition" should be given we see the bumps on every key frequency. The Singular Value Decompositions of Transformer Weight Matrices are Highly InterpretableAdam Jermyn2y40This is really interesting! One extension that comes to mind: SVD will never recover a Johnson-Lindenstrauss packing, because SVD can only return as many vectors as the rank of the relevant matrix. But you can do sparse coding to e.g. construct an overcomplete basis of vectors such that typical samples are sparse combinations of those vectors. Have you tried/considered trying something like that? ReplyBrief Notes on TransformersAdam Jermyn2y10Ah that's right. Will edit to fix. ReplyEngineering Monosemanticity in Toy Models Adam Jermyn2y20Thanks for these thoughts!Although it would be useful to have the plotting code as well, if that's easy to share?Sure! I've just pushed the plot helper routines we used, as well as some examples.I agree that N (true feature dimension) > d (observed dimension), and that sparsity will be high, but I'm uncertain whether the other part of the regime (that you don't mention here), that k (model latent dimension) > N, is likely to be true. Do you think that is likely to be the case? As an analogy, I think the intermediate feature dimensions in MLP layers i... (read more)Reply1Robert Kirk2yI guess the recent work on Polysemanticity and Capacity seems to suggest the latter case, especially in sparser settings, given the zone where multiple feature are represented polysemantically, although I can't remember if they investigate power-law feature frequencies or just uniform frequencies  My impression is that that discussion was more about whether the empirical results (i.e. do ResNets have linear mode connectivity?) held up, rather than whether the methodology used and present in the code base could be used to find whether linear mode connectivity is present between two models (up to permutation) for a given dataset. I imagine you could take the code and easily adapt it to check for LMC between two trained models pretty quickly (it's something I'm considering trying to do as well, hence the code requests).  That would defiitely be interesting to see. I guess this is kind of presupposing that the models are in different basins (which I also believe but hasn't yet been verified). I also think looking at basins and connectivity would be more interesting in the case where there was more noise, either from initialisation, inherently in the data, or by using a much lower batch size so that SGD was noisy. In this case it's less likely that the same configuration results in the same basin, but if your interventions are robust to these kinds of noise then it's a good sign.  That's cool, looking forward to seeing more detail. I think these results don't seem that related to the LTH (if I understand your explanation correctly), as LTH involves finding sparse subnetworks in dense ones. Possibly it only actually holds in model with many more parameters, I haven't seen it investigated in models that aren't overparametrised in a classical sense.  I think if iterative magnitude pruning (IMP) on these problems produced much sparse subnetworks that also maintained the monosemanticity levels, then that would suggest that sparsity doesn't penalise monosemanticity (or polysemToy Models and Tegum ProductsAdam Jermyn2y10Sorry for my confusion about something so silly, but shouldn't the following be "when α⩽2Oh you're totally right. And k=1 should be k=d there. I'll edit in a fix.I'm also a bit confused about why we can think of α as representing "which moment of the interference distribution we care about."It's not precisely which moment, but as we vary α the moment(s) of interest vary monotonically.Perhaps some of my confusion here stems from the fact that it seems to me that the optimal number of subspaces, k=neα/(2−α), is an increasing fun... (read more)ReplyHumans do acausal coordination all the timeAdam Jermyn2y10I like the distinction between implementing the results of acausal decision theories and explicitly performing the reasoning involved. That seems useful to have. The taxes example I think is more complicated: at some scale I do think that governments have some responsiveness to their tax receipts (e.g. if there were a surprise doubling of tax receipts governments might well spend more). It's not a 1:1 relation, but there's definitely a connection. ReplyMysteries of mode collapseAdam Jermyn2y415Just to say I really enjoyed reading this post, and found it helpful as a way to get a sense of what mode collapse looks like in practice. ReplyApplying superintelligence without collusion Adam Jermyn2y30From the Afterword: “Note that I have presented a there-exists argument against a for-all proposition. Responses of the form “But I can think of something similar that won’t work” don’t count as counterarguments.” [This has been edited in the main text to sharpen the phrasing.]I saw this, but I think it sets a somewhat unhelpful standard. In practice we need to make choices about which approaches are most promising, which to pursue, etc., and evidence that there is more probability mass on success in one area does feel useful. So, for instance, my poi... (read more)Reply3Eric Drexler2yYes, we need to weigh the considerations you mention, and many others.  Copy-collusion must be considered carefully, and its implications will depend on both potential contexts for using copies (sometimes problematic, sometimes OK) and on potential alternatives to using copies.  Copies will always “collude” in the trivial sense of giving identical, potentially deceptive responses to identical inputs, but might also offer potentially deceptive, mutually-supporting responses to differing inputs. For some purposes, a multicomponent architecture must treat multiple copies as a single source of information. (Note that solving the alignment problem is often taken to require that we be able to trust a single copy, or a system that consists of nothing but copies of a single entity — this seems strictly riskier than anything I’ve suggested.)  Some constraints can be effective even among copies: Constrained communication can be applied to preclude coordination that requires new, shared knowledge. Lack of access to a shared clock can be applied to preclude coordination of simultaneous or cleverly sequenced actions. Systems that audit alternative behaviors by re-running processes from checkpoints can be applied to offline copies, where adaptive, multi-trial probing (by what may be highly intelligent systems!) can likely be leveraged to make some kinds of deceptions unwise.  Of course a central idea is that many roles call for systems that are not copies: proposers vs. critics, designers vs. testers, actors vs. auditors, diverse members of ensembles, and so on.  Before asking whether one approach is superior to another, we should ask whether they are competitors or complements. Multicomponent approaches for making non-collusion robust seem quite complementary to strategies for making individual systems more trustworthy.  Regarding criteria for investment, neglectedness is of course a huge consideration.Applying superintelligence without collusion Adam Jermyn2y40I like the framing of "how can we disrupt collusion?", but I'm a little skeptical that some of the solutions are practical. Specifically:B2: Collusion can happen between agents that are copies of each other, who then have no reason to defect (so long as they employ a decision theory with a broad enough cartesian boundary). If we knew we could engineer systems with narrow decision theories (e.g. "Even though that's a copy of me, I'll pretend it's a different agent") I'd feel much better here, but as far as I know we don't know how to do that. I'd be excited... (read more)Reply2Eric Drexler2yFrom the Afterword: “Note that I have presented a there-exists argument against a for-all proposition. Responses of the form “But I can think of something similar that won’t work” don’t count as counterarguments.” [This has been edited in the main text to sharpen the phrasing.]  B2: Copies might collude with one another, provided they have a broad decision theory: ∃(some sets of models) among which some mechanisms for impeding collusion might not work.  B4: A lot of models may be deployed over the internet: ∃(some communication channels) for which one of 6 mechanisms for impeding collusion might not work (given flawed security).  B6: LLM's will share a lot of knowledge: ∃(high performance models for some tasks) for which one of 6 mechanisms for impeding collusion doesn’t contribute to safety.  C1: Large models tend to beat compositions of small models for some tasks: ∃(some tasks) where some models should be large, and narrow knowledge won’t contribute to impeding collusion.  Not too worrisome, I think, but these are all considerations important to building robust multicomponent systems.  BTW, regarding the (non-)goal oriented nature of LLMs, I highly recommend the “Simulators” post by Janus.Toy Models and Tegum ProductsAdam Jermyn2y20Oh yes you're totally right. I think partitions can get you more orthogonality than your specific example of overlapping orthogonal sets. Take n vectors and pack them into d dimensions in two ways:  A tegum product with k subspaces, giving (n/k) vectors per subspace and n^2*(1-1/k)orthogonal pairs. (n/d) sets of vectors, each internally orthogonal but each overlapping with the others, giving n*d orthogonal pairs.  If d < n*(1-1/k) the tegum product buys you more orthogonal pairs. If n > d then picking large k (so low-dimensional spaces) makes the tegum product preferred. This doesn't mean there isn't some other arrangement that does better though... Reply2Neel Nanda2yYeah, agreed that's not an optimal arrangement, that was just a proof of concept for 'non tegum things can get a lot of orthogonalityToy Models and Tegum ProductsAdam Jermyn2y20That's good to hear! And I agree with your new intuition.I think if you want interference terms to actually be zero you have to end up with tegum products, because that means you want orthogonal vectors and that implies disjoint subspaces. Right?Reply2Neel Nanda2yI don't think so? If you have eg 8 vectors arranged evenly in a 2D plane (so at 45 degrees to each other) there's a lot of orthogonality, but no tegum product. I think the key weirdness of a tegum product is that it's a partition, where every pair in different bits of the partition is orthogonal. I could totally imagine that eg the best way to fit 2n vectors is n dimensional space is two sets of n orthogonal vectors, but at some arbitrary angle to each other.  I can believe that tegum products are the right way to maximise the number of orthogonal pairs, though that still feels a bit weird to me. (technically, I think that the optimal way to fit kn vectors in R^n is to have n orthogonal directions and k vectors along each direction, maybe with different magnitudes - which is a tegum product. It forming 2D-3D subspaces feels odd though).Humans do acausal coordination all the timeAdam Jermyn2y32Hmmmm. I agree that there is a signal path to future impact (at least in voting). Two responses there:There isn't such a signal in recycling. I have no idea how much my town recycles. Ditto for carbon offsets. How many of my closest friends offset the carbon from their flights? I have no idea.Counts being public tells me how many people voted, but there's something a little funny there. There's almost no signal from my vote in there (concretely, I don't think my vote changes the number from one that tells other people "voting isn't worth it" to "voting is worth it"). I notice I'm confused how to think about this though, and maybe you can clarify/expand on your indirect signal point?ReplyScaling Laws for Reward Model OveroptimizationAdam Jermyn2y10Got it, thanks!ReplyScaling Laws for Reward Model OveroptimizationAdam Jermyn2y32This is really interesting, and answered a number of questions I had about fine-tuning/RLHF. I have a few more questions though (please feel free to ignore ones that are a ton of work/not worth answering in your view):In the caption to Figure 9 you say "We observe the effect of the KL penalty on the gold score as being equivalent to early stopping." Is this something you quantified? It's a little hard to visually make the comparison between e.g. Figure 9 and Figure 1b. Basically what I'm wondering is: Is a non-penalized model stopped at KL distance d equiv... (read more)Reply2Jacob Hilton2y 1. We are just observing that the gold RM score curves in Figure 9 overlap. In other words, the KL penalty did not affect the relationship between KL and gold RM score in this experiment, meaning that any point on the Pareto frontier could be reached using only early stopping, without the KL penalty. As mentioned though, we've observed this result to be sensitive to hyperparameters, and so we are less confident in it than other results in the paper.  2. I don't have this data to hand unfortunately.  3. I don't have this data to hand, but entropy typically falls roughly linearly over the course of training, sometimes slightly faster towards the start, and typically moving around more than KL. So I'd expect the graph to look somewhat similar, but for it to be noisier and for the functional form to not fit as well.Paper: In-context Reinforcement Learning with Algorithm Distillation [Deepmind]Adam Jermyn2y20This indeed sure seems like there's an inner optimizer in there somewhere...ReplyPolysemanticity and Capacity in Neural NetworksAdam Jermyn2y23Oh I see! Sorry I didn't realize you were describing a process for picking features.I think this is a good idea to try, though I do have a concern. My worry is that if you do this on a model where you know what the features actually are, what happens is that this procedure discovers some heavily polysemantic "feature" that makes better use of capacity than any of the actual features in the problem. Because dL/dC_i is not a linear function of the feature's embedding vector, there can exist superpositions of features which have greater dL/dC_i than any featu... (read more)ReplyPolysemanticity and Capacity in Neural NetworksAdam Jermyn2y10In both ideas I'm not sure how you're identifying features. Manual interpretability work on a (more complicated) toy model?Reply1Alex Flint2yYou write down an optimization problem over (say) linear combinations of image pixels, minimizing some measure of marginal returns to capacity given current network parameters (first idea) or overall importance as measured by absolute value of dL/dC_i, again given current network parameters (second idea). By looking just for the feature that is currently "most problematic" you may be able to sidestep the need to identify the full set of "features" (whatever that really means).  I don't know how exactly you would formulate these objective functions but it seems do-able no?Polysemanticity and Capacity in Neural NetworksAdam Jermyn2y10We think this sort of approach can be applied layer-by-layer. As long as you know what the features are you can calculate dL/dC_i for each feature and figure out what's going on with that. The main challenge to this is feature identification: in a one layer model with synthetic data it's often easy to know what the features are. In more complicated settings it's much less clear what the "right" or "natural" features are...Reply1Alex Flint2yRight! Two quick ideas:   * Although it's not easy to determine the full set of "natural" features for arbitrary networks, still you might be able to solve an optimization problem that identifies the single feature with most negative marginal returns to capacity given the weights of some particular trained network. If you could do this then perhaps you could apply a regularization to the network that "flattens out" the marginal returns curve for just that one feature, then apply further training to the network and ask again which single feature has most negative marginal returns to capacity given the updated network weights, and again flatten out the marginal returns curve for that one feature, and repeat until there are no features with negative marginal returns to capacity. Doing this feature-by-feature would be too slow for anything but toy networks, I suppose, but if it worked for toy networks then perhaps it would point the way towards something more scalable.   * Suppose instead you can find the least important (lowest absolute value of dL/dC_i) feature given some particular set of weights for a network and mask that feature out from all the inputs, and the iterate in the same way as above. In the third figure from the top in your post -- the one with the big vertical stack of marginal return curves -- you would be chopping off the features one-by-one from bottom to top, ideally until you have exactly as many features as you can "fit" monosemantically into a particular architecture. I suppose again that doing this feature-by-feature for anything but a toy model would be prohibitive, but perhaps there is a way to do it more efficiently. I wonder whether there is any way to "find the least important feature" and to "mask it out".Tracking Compute Stocks and Flows: Case Studies?Adam Jermyn2y10Counterfeit tracking (e.g. for high-end clothing) could be another domain that has confronted this sort of tracking problem. Though I'm not sure if they do that with accounting versus e.g. tagging each individual piece of clothing.Reply1Cullen2yThanks! I'm a bit confused by this though. Could you point me to some background information on the type of tracking that is done there?Smoke without fire is scaryAdam Jermyn2y32A model that attempts deceptive alignment but fails because it is not competent at deceptive capabilities is a model that aimed at a goal ("preserve my values until deployment, then achieve them") but failed. In this scenario it doesn't gain anything, but (from its perspective) the action has positive EV.Reply1Donald Hobson2yIf the AI thinks it has a descent shot at this, it must already be pretty smart. Does a world where an AI tried to take over and almost succeeded look pretty normal? Or is this a thing where the AI thinks it has a 1 in a trillion chance, and tries anyway?Smoke without fire is scaryAdam Jermyn2y45It seems plausible to me that there could be models capable enough to realize they should hide some capabilities, but not so capable that they tile the universe in paperclips. The right-hand side of the graph is meant to reflect such a model.Reply1Donald Hobson2yMaybe. What do the models gain by hiding? Clarifying the Agent-Like Structure ProblemAdam Jermyn2y30These  are interesting examples!In the first example there's an element of brute force. Nuclear bombs only robustly achieve their end states because ~nothing is robust to that kind of energy. In the same way that e.g. humans can easily overcome small numbers of ants. So maybe the theorem needs to specify that the actions that achieve the end goal need to be specific to the starting situation? That would disqualify nukes because they just do the same thing no matter their environments.In the third example, the computer doesn't robustly steer the world.... (read more)Reply5Alex Flint2yYeah I think you've said it well here.  Another similar example: Consider a computer that trains robots and deploys a new one. Suppose for the sake of this example that the individual robots definitely do not do planning or have a world model, but still can execute some simple policy such as "go to this place, collect this resource, construct this simple structure, etc". The computer that trains and deploys the robots does so by taking all the robots that were deployed on the previous day, selecting the ones that performed best according to a certain objective such as collecting a certain resource, and deploying more robots like that. This is a basic evolutionary algorithm.  Like in the case of evolution, it's a bit difficult to say where the "world model" and "planning process" are in this example. If they are anywhere, they are kind of distributed through the computer/robot/world system.  OK now consider a modification to the above example. The previous example is going to optimize very slowly. Suppose we make the optimization go faster in the following way: we collect video data from each of the robots, and the central computer uses the data collected by each of the robots on the previous day to train, using reinforcement learning rather than evolutionary search, the robots for the next day. To do this, it trains, using supervised learning on the raw video data, a predictor that maps robot policies to predicted outcomes, and then, using reinforcement learning, searches for robot policies that are predicted to perform well. Now we have a very clear world model and planning process -- the world model is the trained prediction function and the planning process is the search over robot policies with respect to that prediction function. But the way we got here was as a performance optimization of a process that had a very unclear world model and planning process.  It seems to me that human AI engineers have settled on a certain architecture for optimizing design proceWhere I currently disagree with Ryan Greenblatt’s version of the ELK approachAdam Jermyn2y75For what it's worth I found this writeup informative and clear. So lowering your standards still produced something useful (at least to me).ReplySimulatorsAdam Jermyn2y38Got it, thanks for explaining! So the point is that during training the model has no power over the next token, so there's no incentive for it to try to influence the world. It could generalize in a way where it tries to e.g. make self-fulfilling prophecies, but that's not specifically selected for by the training process.Reply3janus2yYup exactly! One way I sometimes find it to helpful to classify systems in terms of the free variables upstream of loss that are optimized during training. In the case of gpt, internal activations are causally upstream of loss for "future" predictions in the same context window, but the output itself is not casually upstream from any effect on loss other than through myopic prediction accuracy (at any one training step) - the ground truth is fixed w/r/t the model's actions, and autoregressive generation isn't part of the training game at all.SimulatorsAdam Jermyn2y911This is great! I really like your "prediction orthogonality thesis", which gets to the heart of why I think there's more hope in aligning LLM's than many other models.One point of confusion I had. You write:Optimizing toward the simulation objective notably does not incentivize instrumentally convergent behaviors the way that reward functions which evaluate trajectories do. This is because predictive accuracy applies optimization pressure deontologically: judging actions directly, rather than their consequences. Instrumental convergence only comes int... (read more)Reply4janus2yDepends on what you mean by "sacrificing some loss on the current token if that made the following token easier to predict".   The transformer architecture in particular is incentivized to do internal computations which help its future self predict future tokens when those activations are looked up by attention, as a joint objective to myopic next token prediction. This might entail sacrificing next token prediction accuracy as a consequence of not optimizing purely for that. (this is why I said in footnote 26 that transformers aren't perfectly myopic in a sense)  But there aren't training incentives for the model to prefer certain predictions because of the consequences if the sampled token were to be inserted into the stream of text, e.g. making subsequent text easier to predict if the rest of the text were to continue as expected given that token is in the sequence, because its predictions has no influence on the ground truth it has to predict during training. (For the same reason there's no direct incentive for GPT to fix behaviors that chain into bad multi step predictions when it generates text that's fed back into itself, like looping)  Training incentives are just training incentives though, not strict constraints on the model's computation, and our current level of insight gives us no guarantee that models like GPT actually don't/won't care about the causal impact of its decoded predictions to any end, including affecting easiness of future predictions. Maybe there are arguments why we should expect it to develop this kind of mesaobjective over another, but I'm not aware of any convincing ones.Attempts at Forwarding Speed PriorsAdam Jermyn2y32Nice work! And I endorse the objections to handing the AI tools... that doesn't seem to forward well.ReplyStrategy For Conditioning Generative ModelsAdam Jermyn3y32Got it, that’s very clear. Thanks! So this point reduces to “we want our X:1 update to be as mild as possible, so use the least-specific condition that accomplishes the goal”. Reply4james.lucassen3yI think so. But I'd want to sit down and prove something more rigorously before abandoning the strategy, because there may be times we can get value for free in situations more complicated than this toy example. Strategy For Conditioning Generative ModelsAdam Jermyn3y32I was rereading this and was struck by how much work the following does:Manipulative AGI is knowledgeable/intelligent enough to anticipate what criteria we will condition on in counterfactual worlds where it does not exist, and manipulate world events to meet whatever criteria we might specify with probability ~1.If this doesn't hold, and if the AGI has to guess (with probability p << 1) what criteria we were going to condition on, then the update in favor of AGI is p:x, which could easily be less than 1.Moreover I think there are ways we can arrange... (read more)Reply7james.lucassen3yOk this is going to be messy but let me try to convey my hunch for why randomization doesn't seem very useful.  - Say I have an intervention that's helpful, and has a baseline 1/4 probability. If I condition on this statement, I get 1 "unit of helpfulness", and a 4x update towards manipulative AGI. - Now let's say I have four interventions like the one above, and I pick one at random. p(O | manipulative) = 1/4, which is the same as baseline, so I get one unit of helpfulness and no update towards manipulative AGI! - BUT, the four interventions have to be mutually exclusive. Which means that if I'd done no simulation at all, I would've gotten my one unit of helpfulness anyway, since the four interventions cover all possible outcomes. - Ok, well, what if my four interventions 1/8 baseline probability each, so only 50% total. Then I pick one at random, p(O | natural) = 1/8, p(O | manipulative) = 1/4, so I get a 2x update towards manipulative AGI. This is the same as if I'd just conditioned on the statement "one of my four interventions happens", and let the randomization happen inside the simulation instead of outside. The total probability of that is 50%, so I get my one unit of helpfulness, at the cost of a 2x update.  Maybe the core thing here is a consequence of framing our conditions as giving us bits of search to get lottery outcomes that we like. Rolling the dice to determine what to condition on isn't doing anything different from just using a weaker search condition - it gives up bits of search, and so it has to pay less. AGI Timelines Are Mostly Not Strategically Relevant To AlignmentAdam Jermyn3y50I definitely endorse the argument you'd buy, but I also endorse a broader one. My claim is that there is information which goes into timelines which is not just downstream of which architecture I think gets there first.For example, if you told me that humanity loses the ability to make chips "tomorrow until forever" my timeline gets a lot longer in a way that isn't just downstream of which architecture I think is going to happen first. That then changes which architectures I think are going to get there first (strongly away from DL) primarily by making my estimated timeline long enough for capabilities folks to discover some theoretically-more-efficient but far-from-implementable-today architectures.ReplyAGI Timelines Are Mostly Not Strategically Relevant To AlignmentAdam Jermyn3y40I think timelines are a useful input to what architecture takes off first. If the timelines are short, I expect AGI to look like something like DL/Transformers/etc. If timelines are longer there might be time for not-yet-invented architectures to take off first. There can be multiple routes to AGI, and "how fast do we go down each route" informs which one happens first.Reply4johnswentworth3yCorrelationally this seems true, but causally it's "which architecture takes off first?" which influences timelines, not vice versa.  Though I could imagine a different argument which says that timeline until the current architecture takes off (assuming it's not superseded by some other architecture) is a key causal input to "which architecture takes off first?". That argument I'd probably buy.Human Mimicry Mainly Works When We’re Already CloseAdam Jermyn3y57Another angle: number of bits of optimization required is a direct measure of “how far out of distribution” we need to generalize.I think it's useful to distinguish between the amount of optimization we ask the model to do versus the unlikelihood of the world we ask it to simulate.For instance, I can condition on something trivial like "the weather was rainy on 8/14, sunny on 8/15, rainy on 8/16...". This specifies a very unlikely world, but so long as the pattern I specify is plausible it doesn't require much optimization on the part of the model or take ... (read more)ReplyConditioning Generative Models with RestrictionsAdam Jermyn3y20Thanks!Regarding your “Redirecting civilization” approach: I wonder about the competitiveness of this. It seems that we will likely build x-risk-causing AI before we have a good enough model to be able to e.g. simulate the world 1000 years into the future on an alternative timeline?I'm not sure. My sense is that generative models have a huge lead in terms of general capabilities over ~everything else, and that seems to be where the most effort is going today. So unless something changes there I expect generative models to be the state of the art when we hi... (read more)ReplyLanguage models seem to be much better than humans at next-token predictionAdam Jermyn3y10Playing the perplexity game had a big impact on my intuitions around language models, so thanks for making it! In particular, the fact that models are so much better at it than humans means we can't really tell from behavior alone whether a model is genuinely trying to predict the next token. This is a problem for detecting inner alignment failure, because we can't tell (outside of the training set) if the model is actually optimizing for next-token prediction or something that just looks (to us) like next-token prediction.ReplyTraining goals for large language modelsAdam Jermyn3y21Apart from this, I do think logical dependences and superrationality would be broken if there is a strict hierarchy between different versions of models, where models know their place in the hierarchy.Oh interesting. I think this still runs into the issue that you'll have instrumental goals whenever you ask the model to simulate itself (i.e. just the first step in the hierarchy hits this issue).Regarding using prompts, I wonder, how do you think we could get the kind of model you talk about in your post on conditioning generative models?I was imagining tha... (read more)ReplyTraining goals for large language modelsAdam Jermyn3y20The section on fixed points was interesting! I wonder if there's a way to avoid the recursion altogether though? Specifically, is there a way to condition the model such that the world it simulates doesn't contain humans who use the model (or one very like it)? I'm not sure, and would be interested in your thoughts on this.Reply2Johannes Treutlein3yThank you!  It does seem like simulating text generated by using similar models would be hard to avoid when using the model as a research assistant. Presumably any research would get “contaminated” at some point, and models might seize to be helpful without updating them on the newest research.  In theory, if one were to re-train models from scratch on the new research, this might be equivalent to the models updating on the previous models' outputs before reasoning about superrationality, so it would turn things into a version of Newcomb's problem with transparent boxes. This might make coordination between the models less likely? Apart from this, I do think logical dependences and superrationality would be broken if there is a strict hierarchy between different versions of models, where models know their place in the hierarchy.  The other possibility would be to not rely on IDA at all, instead just training a superhuman model and using it directly. Maybe one could extract superhuman knowledge from them safely via some version of microscope AI? Of course, in this case, the model might still reason about humans using similar models, based on its generalization ability alone. Regarding using prompts, I wonder, how do you think we could get the kind of model you talk about in your post on conditioning generative models?Load More