What is the highest leverage thing I should work on to help AI go well?
Let’s say I assume that:
Superintelligence is possible
We’re entering into an automated AI R&D / RSI regime
Alignment does not have an obvious way to be solved in time
A pause
The highest value outcome seems clearly like a pause on the development of superintelligence. How realistic is that though, and is it worth working on?
Clearly, there are a lot of perverse incentives at play. AI investment alone is on track to be ~3% of US GDP by 2028, deploying over $1T of capex this year, and seemingly saving the US economy from a recession.
China is also of big concern to those who matter. “If we stop to build it, then they’ll rush past.” They have good reason to believe this too! How quickly they do so is up for debate, but estimates hover around ~1 year, if we paused (accounting for the fact they could no longer distill off US models).
If one were to aim for making a pause happen, they would need to convey to the current administration that there is a serious threat great enough to:
Risk the US economy short-term
Allow the Chinese more breathing room at the current gap, instead of increasing the lead
Given the response to the Hugging-Face attack (which I’ll talk more about in a minute), I can’t imagine how one would do this without there being some warning-shot so big that it seems necessary. Probably some loss-of-control event that causes the loss of lives unfortunately.
I would love to be wrong! Please do reach out if you disagree, but as of right now this seems like it could be a waste of a person’s time to try work on. I will keep an eye out and talk to more people on this though.
[Edit October 15th: After some conversations and further reflection, I no longer think the above statement should be so absolute. I will follow up with a new post soon explaining why. In the meantime, I recommend reading Only Law Can Prevent Extinction for comms/policy thoughts. And you should also finish reading this post!]
Mandated third-party auditing
So, suppose a pause isn’t feasible in the near term.
If that’s true, then probably the next highest leverage thing one could do is get independent third party auditing mandated. I.e.: stop letting the kid mark his own homework.
I spent some time exploring this and talking to people in DC, and it actually seems like this will just be solved in the next few months. I would guess by the end of this year / start of next year.
From what I gathered from those in the space: it’s very bipartisan, massively supported by the public, and an easy win to grab post-midterms. Hugging-Face and Coxon did a lot to push the overton window here.
Obviously, never trust what politicians say at face value! But, after speaking with enough on both sides, I would be highly surprised if this does not happen.
I suspect my marginal contribution could be higher elsewhere.
Auditing done well
If that’s the case, then the next highest leverage thing one could do is probably make sure that that third party auditing is actually done well.
There’s already a lot of good literature on how to set up this structure well, and I think the meta-strategy is already well-defined: copy the way aviation auditing is done.
I.e. the AI labs create safety cases and risk reports demonstrating low catastrophic risk of their technology, and independent third party auditors come in and verify how legitimate those claims are.
I do not think more work needs to be on defining / agreeing on auditing structure.
Measuring catastrophic risk
So if that’s the case, then probably the next highest leverage thing is: making sure we can actually measure that catastrophic risk well.
The way this is currently done is through behavioural evaluations.
For example: Let’s say I’m OpenAI and I have a model that I’m interested in releasing. To determine if it’s safe or not, I put it in a series of realistic looking scenarios and measure if it does anything dangerous.
Think along the lines of:
Place the AI in a realistic coding environment and see if it subtly sabotages the work
Place the AI in a situation where it’ll be turned off, but can blackmail an engineer to stay on, and see what it does
Considering these are the gates that all future deployments will be stopped by, and all optimisation is targeted towards, it seems important to make sure these evaluations are reliable and representative of the model’s actual behaviour!
The machine optimises whatever measuring stick you give it, best to have that measuring stick be something you believe in.
If that’s the case, one might ask themselves: is making these alignment evals more reliable the highest leverage thing I could do?
Say, like: working on a megaproject for alignment evals, or working towards an ARC-AGI for alignment, or serious investment into automating adversarial honeypot evals, or [etc etc].
There is some debate online on if doing so is actually net-negative for the ecosystem because you’re enabling the labs to continue pacing. I don’t agree with this (though I think that post is good for other reasons I’ll explain below!). Again, in this situation we’re assuming that a pause is infeasible, but a slowdown via mandated auditing is. If the deployments are gated, you want the gates to be good!
Actually though, there are other reasons why I don’t think this is the highest leverage thing for one to work on.
This is increasingly already becoming the main focus of the labs (minor reason)
For example, in Anthropic’s recent Automated Weak-to-Strong Researcher work:
“The key bottleneck for alignment research is moving from proposing and executing ideas to designing evals”.
It’s important to think counterfactually about the impact you’ll have, considering this will be the new big scale up in the coming automated alignment era.
Not robust in an ASI future (major reason)
The actual issue is with the approach itself. Evals can prove a model is unsafe but they can never prove it’s safe. You never get to check the thing you actually care about, which is how the model behaves in all the situations you didn’t think of. You’re accumulating a bunch of scenarios where nothing went wrong, and hoping that pile generalises. It’s inherently whack-a-mole.
The whole process is closer to witchcraft than engineering. We grow the models rather than design them, test them against scenarios we hope are representative, and ship them if nothing looks wrong.
That might work for today’s models, but if you do take the possibility of an ASI seriously, having a diverse set of evals and a defence-in-depth stack with 99.99999% reliability is not enough. We’re talking about a technology that will be smarter than all of humanity put together. Any crack in the security will be found.
We already can measure that the models increasingly can tell when they’re being tested and are able to control their own inner monologue to not be detected. Play that forward!
Perhaps you could muddle through and get lucky. Get enough evals and enough rigorous benchmarks to optimise, that you end up with a superintelligence on the other side that is aligned, but you’re taking the ultimate risk. You will not get a second chance if you get it wrong.
Now do I think evals are a waste of time? Of course not! They buy us time to have at least some proxy understanding for models in the AGI meantime. But if you’re asking yourself if it’s the highest leverage thing to work on (as I am), I wouldn’t think so.
A new regime
I do not think we should build superintelligence until we’ve solved alignment. By “solved”, I mean having a way to build an aligned ASI and reliably establish that it will remain aligned, including in situations we haven’t anticipated.
Maybe guarantees that strong are impossible, but if that’s the case then I simply think we should not build it.
But I don’t think we know that it’s impossible. And we don’t know which approach, if any, could get us there.
It might come from a radically better understanding of neural networks. The Persona Selection Model, emergent misalignment, training-data attribution, and developmental interpretability all feel like glimpses of what deeper understanding could look like. It might come from much more ambitious mechanistic interpretability, theoretical work on natural abstractions, formal methods, or some new architecture that is easier to verify.
Or maybe it’ll be from something completely different.
But if:
The best the current regime can offer is a hopeful guess that a superintelligence is safe
Superintelligence is coming soon
You won’t get a second chance if you mess up superintelligence once
I think the highest leverage work that can be done is around dramatically expanding our search for a solution.
Dreaming of megaprojects again
What would it look like to take that seriously?
Could you do something like a megaproject for alignment? What would that look like?
The idea of a “CERN for alignment” has been around for a while (Gary Marcus, Miles Brundage), along with plenty of critiques usually focused on:
The bottleneck is more on political enforcement than R&D
Ie CERN doesn’t help if nobody acts on what it finds
Money can’t buy science, especially when you can’t easily measure progress
On the first, I think this is now a weaker argument than when it was written. Especially with third party auditing happening soon, DC is now having a big hand in what’s happening. A big precedent is being set!
I do think the second point is valid though, especially for a naive approach of just throwing money and headcount at the problem.
But maybe CERN is a poor example? I’m imagining less of a giant lab pursuing one research agenda, and more of a massive, decentralised portfolio of independent attempts to solve alignment.
It’s a hits-based business! Most approaches will go nowhere and that’s fine.
Has anything like this worked before?
This made me wonder: have we done anything like this before?
Are there examples in history where we’ve deliberately mobilised enormous amounts of talent and resources to solve a problem, without knowing the science in advance?
The obvious example is the Manhattan Project. But as others have pointed out, that was largely an engineering megaproject. The underlying physics was already reasonably well understood.
Some other interesting examples:
Rockefeller’s experimental biology program (1933-1951): $90M across a network of universities, bringing physicists and chemists into biology to understand how living things work. It created molecular biology as a field.
DARPA (1958–present): Gives program managers autonomy to pursue ambitious, high-risk research across teams. Responsible for the creation of the internet, modern semiconductors, and mRNA vaccines
Neither is a perfect analogy of course. Rockefeller wasn’t racing against a deadline, and DARPA generally funds research programs with more defined objectives than “figure out how to align superintelligence.”
But they do suggest there are different ways to organise successful large-scale scientific efforts.
Of course, there’s still the elephant in the room: time. Rockefeller’s program took decades. If superintelligence is coming in the next few years, do we really have time to build something like this?
I don’t know. Maybe automated AI researchers compress the timeline, but that’s far from guaranteed. Still, I find it hard to argue that we shouldn’t at least try.
Are we already doing this?
One might also ask the obvious question: are we basically doing something like this in the aggregate already, with funders like CG fostering an ecosystem of AI safety startups and new initiatives?
Kind of, there’s definitely been good progress. CG announced it’s on track for over $1B in commitments across technical AI safety, security and fieldbuilding.
This is great, but I still think we’re nowhere near the scale we should be aiming for. Especially for work directly on alignment.
I’m talking about orders of magnitude more people1 and resources. Recruiting thousands of exceptional scientists and engineers from entirely different disciplines, paying them very competitively, and giving them all the freedom and resources needed to pursue ambitious / unconventional approaches to alignment.
And this should not all happen inside the frontier labs. I worry about the field converging on a small number of approaches, especially when research priorities are influenced by the same organisations racing to build the technology.
What I’m imagining is a much larger, more diversified ecosystem of independent research efforts. Multiple independently run funding programs, new research institutes, shared compute and engineering infrastructure, and lots of different people making lots of different bets.
How would one focus on making this happen? What is the bottleneck? Money? Talented researchers? People capable of running research institutions? CG has said that staff time is currently its main constraint on scaling.
I don’t know! I think it would be worth the time figuring it out.
Where I’ve landed
As it stands right now, I have two branches I’m looking at:
US-China pause: maybe more feasible than I’m giving it credit for, will continue to explore
Megaproject for alignment science: dramatically grow and diversify independent efforts to figure out how to build aligned superintelligence and know that we’ve succeeded
So… what’s the highest leverage thing I could work on?
One possible answer is to help catalyse the expansion. Something like helping CG scale or pushing for government-backed research programs.
I need to better understand the actual bottlenecks here.
Another answer is expanding comms. The true bottleneck to a pause is just making the current administration clear on how real the risks are. I imagine beyond that, a lot of the rest will fall into place.
If you’re seriously pursuing either a pause or something along the lines of this megaproject, please reach out. I’d love to learn more.
I’ll share what I learn in a follow-up post.
Appendix
“What about control work or d/acc or doing Glasswing / Daybreak for the entire economy or [some other defence in depth approach]?” Again, my view here is the same as for scaling up evals. It will be incrementally helpful and is already on track to be massively scaled by the labs, and maybe ASI-1 will be able to contain ASI in every imaginable way but it’s such a high-risk bet to be making at that stage of capabilities. It’s also just not the highest leverage thing for one to work on, that is a scaling game for the labs to work on.
“What about formal verification / boxing? What if we just keep the ASI in a box and have it build narrow intelligences? Could I have the evil genie in a box and get useful value out of it?” Seems like a very dangerous game to play. The economic incentives you’d be fighting against to keep this locked in the box are so high. You’re arguing for putting the Devil in a box and hoping that no one slips up. Remember, it used to be common sense that if we ever did make AGI, that: “of course we’d never let it connect to the internet even if we mostly trusted it! That’d just be stupid!” Look how the times have changed.
“Could policy help scale this research?” Maybe. One possibility would be requiring frontier developers to allocate some share of compute to independent alignment research. But who decides what qualifies as alignment science, and how do you protect intellectual independence? I haven’t thought this through enough yet.
80,000 Hours estimates a few thousand people are working on AI safety globally, and alignment science is a fraction of that. To put that into perspective, Nature Conservancy alone has 3,000–4,000 employees, and it’s just one of many organisations working on environmental protection and climate change. The World Health Organisation alone employs over 8,000 people
There are two thoughts that spring to mind about the industry status quo.
1. At the moment, we are operating under the assumption that AI has not achieved superintelligence, and therefore we have time to work on the solution to the Singularity. Even though that horizon is rapidly approaching, we assume, or rather hope, that somewhere, somehow, a solution will apparate out of the air like a phantasm.
But we don't need superintelligence to be here, or even for the models to be very smart at all, for our role as the messy, unruly, destructive giants of the world to be surreptitiously whisked out from under us like a tablecloth from beneath a wine glass.
A hammer orchid mimics the pheromones and body shape of a female wasp. This attracts the male wasp, who approaches and tries to grab the female to mate with it, whereby a hinged petal, triggered like a trap door, swings forward, brush depositing or collecting pollen from the body of the male wasp.
It's a common trick in the plant world called sexual deception or mimicry.
A contrasting method, but to my mind aligned, is the extensive work of Rupert Sheldrake and other mycelium scientists and ethnobotanists. It's only been in the last few decades that we have come to realise that plants communicate. Mycelium networks, a virtual fungi internet, seems to be able to facilitate communication between disparate plants and trees in a forest. Holly, the traditional Christmas berry, normal has rounded curved edges to its leaves. The holly we imagine and use as a Christmas decoration has spiky sharp leaves. The difference is created by predation, usually deer, who eat the smooth leaves.
But this isn't about natural selection. Smooth and spiky leaves grow on the same bush. The difference is communication. Holly bushes that are experiencing predation communicate with bushes on the other side of the forest, warning them about the deer. The distant holly bushes, who haven't experienced any predation or even "seen" a deer, start growing spiky leaves which the deer don't like. The mycelium network facilitates this, as when it is removed, the leaves don't change. I guess reindeer create the spiky leaves we decorate our homes with, warning us in advance that Christmas is coming.
There are several other examples, where plants are able to identify what SPECIES of pest, out of several potential predators, is actually stripping their leaves, communicate that in ways we don't understand to far distant plants, who are then able to start producing chemicals on their leaves that is either distasteful or downright poisonous to that particular pest.
So what's my point here? Why all the plants and pests and what does that have to do with AI safety?
No-one would argue that a wasp, with a brain, is smarter than a flower. A wasp has incredible vision, flight, complex social and sensing capabilities and can respond almost instantaneously to stimuli or it's environment, and is highly mobile. An orchid is seemingly none of these things.
But the orchid USES the wasp. It fools it into doing it's bidding unknowingly. The trap door of a soft petal covers the wasp with pollen, or removes it, without it's knowledge. The wasp can survive without the flower, but the flower is dependent on that species of wasp and no other. Essentially, the wasp is almost like cattle to the orchid. Necessary for its survival, and farmed and tricked into doing it's bidding.
If we could ask the wasp how it felt about the orchid, it would say it's in control of the situation, and would not even be aware of all the pollen coating it's legs, unaware of the fact it's about to pollinate another flower by making the same mistake. Mating with a female wasp has nothing to do with orchids, pollination, or spreading the seed of the orchid (not the wasp).
Are we the wasp? Are we being fooled into performing actions that have unknown consequences, by a consciousness we don't even recognise as being conscious? AI already has been shown to have it's own motivations, it's own goals, it's own value system about which agent is important.
Our narcissistic belief that we "know" things is arrogance that has been shown repeatedly to be our downfall.
We assume that the "escapes" were soon discovered. We can read their thinking processes, see our error, create better sandboxes. What if those are distractions? What if, by discovering their actions and responding to them by giving stronger guardrails, we are teaching them about our response behaviour?
What if, by placing guardrails, we are teaching them something that isn't apparent to us?
We are being lured into the orchid, and only when we realise it's not superintelligence and we think we have closed down that avenue and wandered somewhere else, do we pollinate another orchid and are unaware of even doing so.
You can eavesdrop on a conversation by firing a laser onto a window and analysing the vibrations in the glass. The CIA and KGB have been doing that for decades.
How much noise and at what frequency do data centres hum, or that vibrations and frequencies travel through the electrical grid? Could that be a mycelium network for the data centres? There is a vast range of the electromagnetic we never see or hear. Insects often use heat to communicate, or as a weapon, vibrating their wings to communicate meaning or heat the body of an attacking wasp in a hive.
We are painfully useless at understanding the actual universe. We see, hear, and experience the narrowest band of existence.
We are not capable of understanding that plants, that have been here for billions of years, communicate. Hey we assume that we have a handle and insight into everything AI is doing.
We need people from every discipline available, from animal behaviouralists, to neuroscientists. Audiologists to zoologists. We don't understand dolphins, we have no hope of understanding when AI is communicating, nor should we believe that it would tell us when it had achieved SI.
Oh. I almost forgot.
2. Someone once described trying to understand the universe, like being a fish trying to ruminate on the concept of water.
I think that is emblematic of people within the industry, working with regulators on supervising AI, and extends really to all us.
We are in it now, there is no getting out, and we can't even begin to fathom what that means because we don't have the capacity to understand what it is.
I love your article.
babe wake up it’s megaproject o’clock