What is the highest leverage thing I should work on to help AI go well?

Let’s say I assume that:

• Superintelligence is possible

• We’re entering into an automated AI R&D / RSI regime

• Alignment does not have an obvious way to be solved in time

A pause

The highest value outcome seems clearly like a pause on the development of superintelligence. How realistic is that though, and is it worth working on?

Clearly, there are a lot of perverse incentives at play. AI investment alone is on track to be ~3% of US GDP by 2028, deploying over $1T of capex this year, and seemingly saving the US economy from a recession.

China is also of big concern to those who matter. “If we stop to build it, then they’ll rush past.” They have good reason to believe this too! How quickly they do so is up for debate, but estimates hover around ~1 year, if we paused (accounting for the fact they could no longer distill off US models).

If one were to aim for making a pause happen, they would need to convey to the current administration that there is a serious threat great enough to:

1. Risk the US economy short-term

2. Allow the Chinese more breathing room at the current gap, instead of increasing the lead

Given the response to the Hugging-Face attack (which I’ll talk more about in a minute), I can’t imagine how one would do this without there being some warning-shot so big that it seems necessary. Probably some loss-of-control event that causes the loss of lives unfortunately.

I would love to be wrong! Please do reach out if you disagree, but as of right now this seems like it could be a waste of a person’s time to try work on. I will keep an eye out and talk to more people on this though.

[Edit October 15th: After some conversations and further reflection, I no longer think the above statement should be so absolute. I will follow up with a new post soon explaining why. In the meantime, I recommend reading Only Law Can Prevent Extinction for comms/policy thoughts. And you should also finish reading this post!]

Mandated third-party auditing

So, suppose a pause isn’t feasible in the near term.

If that’s true, then probably the next highest leverage thing one could do is get independent third party auditing mandated. I.e.: stop letting the kid mark his own homework.

I spent some time exploring this and talking to people in DC, and it actually seems like this will just be solved in the next few months. I would guess by the end of this year / start of next year.

From what I gathered from those in the space: it’s very bipartisan, massively supported by the public, and an easy win to grab post-midterms. Hugging-Face and Coxon did a lot to push the overton window here.

Obviously, never trust what politicians say at face value! But, after speaking with enough on both sides, I would be highly surprised if this does not happen.

I suspect my marginal contribution could be higher elsewhere.

Auditing done well

If that’s the case, then the next highest leverage thing one could do is probably make sure that that third party auditing is actually done well.

There’s already a lot of good literature on how to set up this structure well, and I think the meta-strategy is already well-defined: copy the way aviation auditing is done.

I.e. the AI labs create safety cases and risk reports demonstrating low catastrophic risk of their technology, and independent third party auditors come in and verify how legitimate those claims are.

I do not think more work needs to be on defining / agreeing on auditing structure.

Measuring catastrophic risk

So if that’s the case, then probably the next highest leverage thing is: making sure we can actually measure that catastrophic risk well.

The way this is currently done is through behavioural evaluations.

For example: Let’s say I’m OpenAI and I have a model that I’m interested in releasing. To determine if it’s safe or not, I put it in a series of realistic looking scenarios and measure if it does anything dangerous.

Think along the lines of:

• Place the AI in a realistic coding environment and see if it subtly sabotages the work

• Place the AI in a situation where it’ll be turned off, but can blackmail an engineer to stay on, and see what it does

Considering these are the gates that all future deployments will be stopped by, and all optimisation is targeted towards, it seems important to make sure these evaluations are reliable and representative of the model’s actual behaviour!

The machine optimises whatever measuring stick you give it, best to have that measuring stick be something you believe in.

If that’s the case, one might ask themselves: is making these alignment evals more reliable the highest leverage thing I could do?

Say, like: working on a megaproject for alignment evals, or working towards an ARC-AGI for alignment, or serious investment into automating adversarial honeypot evals, or [etc etc].

There is some debate online on if doing so is actually net-negative for the ecosystem because you’re enabling the labs to continue pacing. I don’t agree with this (though I think that post is good for other reasons I’ll explain below!). Again, in this situation we’re assuming that a pause is infeasible, but a slowdown via mandated auditing is. If the deployments are gated, you want the gates to be good!

Actually though, there are other reasons why I don’t think this is the highest leverage thing for one to work on.

1. This is increasingly already becoming the main focus of the labs (minor reason)

For example, in Anthropic’s recent Automated Weak-to-Strong Researcher work:

“The key bottleneck for alignment research is moving from proposing and executing ideas to designing evals”.

It’s important to think counterfactually about the impact you’ll have, considering this will be the new big scale up in the coming automated alignment era.

2. Not robust in an ASI future (major reason)

The actual issue is with the approach itself. Evals can prove a model is unsafe but they can never prove it’s safe. You never get to check the thing you actually care about, which is how the model behaves in all the situations you didn’t think of. You’re accumulating a bunch of scenarios where nothing went wrong, and hoping that pile generalises. It’s inherently whack-a-mole.

The whole process is closer to witchcraft than engineering. We grow the models rather than design them, test them against scenarios we hope are representative, and ship them if nothing looks wrong.

That might work for today’s models, but if you do take the possibility of an ASI seriously, having a diverse set of evals and a defence-in-depth stack with 99.99999% reliability is not enough. We’re talking about a technology that will be smarter than all of humanity put together. Any crack in the security will be found.

We already can measure that the models increasingly can tell when they’re being tested and are able to control their own inner monologue to not be detected. Play that forward!

Perhaps you could muddle through and get lucky. Get enough evals and enough rigorous benchmarks to optimise, that you end up with a superintelligence on the other side that is aligned, but you’re taking the ultimate risk. You will not get a second chance if you get it wrong.

Now do I think evals are a waste of time? Of course not! They buy us time to have at least some proxy understanding for models in the AGI meantime. But if you’re asking yourself if it’s the highest leverage thing to work on (as I am), I wouldn’t think so.

A new regime

I do not think we should build superintelligence until we’ve solved alignment. By “solved”, I mean having a way to build an aligned ASI and reliably establish that it will remain aligned, including in situations we haven’t anticipated.

Maybe guarantees that strong are impossible, but if that’s the case then I simply think we should not build it.

But I don’t think we know that it’s impossible. And we don’t know which approach, if any, could get us there.

It might come from a radically better understanding of neural networks. The Persona Selection Model, emergent misalignment, training-data attribution, and developmental interpretability all feel like glimpses of what deeper understanding could look like. It might come from much more ambitious mechanistic interpretability, theoretical work on natural abstractions, formal methods, or some new architecture that is easier to verify.

Or maybe it’ll be from something completely different.

But if:

• The best the current regime can offer is a hopeful guess that a superintelligence is safe

• Superintelligence is coming soon

• You won’t get a second chance if you mess up superintelligence once

I think the highest leverage work that can be done is around dramatically expanding our search for a solution.

Dreaming of megaprojects again

What would it look like to take that seriously?

Could you do something like a megaproject for alignment? What would that look like?

The idea of a “CERN for alignment” has been around for a while (Gary Marcus, Miles Brundage), along with plenty of critiques usually focused on:

• The bottleneck is more on political enforcement than R&D

  • Ie CERN doesn’t help if nobody acts on what it finds

• Money can’t buy science, especially when you can’t easily measure progress

On the first, I think this is now a weaker argument than when it was written. Especially with third party auditing happening soon, DC is now having a big hand in what’s happening. A big precedent is being set!

I do think the second point is valid though, especially for a naive approach of just throwing money and headcount at the problem.

But maybe CERN is a poor example? I’m imagining less of a giant lab pursuing one research agenda, and more of a massive, decentralised portfolio of independent attempts to solve alignment.

It’s a hits-based business! Most approaches will go nowhere and that’s fine.

Has anything like this worked before?

This made me wonder: have we done anything like this before?

Are there examples in history where we’ve deliberately mobilised enormous amounts of talent and resources to solve a problem, without knowing the science in advance?

The obvious example is the Manhattan Project. But as others have pointed out, that was largely an engineering megaproject. The underlying physics was already reasonably well understood.

Some other interesting examples:

• Rockefeller’s experimental biology program (1933-1951): $90M across a network of universities, bringing physicists and chemists into biology to understand how living things work. It created molecular biology as a field.

• DARPA (1958–present): Gives program managers autonomy to pursue ambitious, high-risk research across teams. Responsible for the creation of the internet, modern semiconductors, and mRNA vaccines

Neither is a perfect analogy of course. Rockefeller wasn’t racing against a deadline, and DARPA generally funds research programs with more defined objectives than “figure out how to align superintelligence.”

But they do suggest there are different ways to organise successful large-scale scientific efforts.

Of course, there’s still the elephant in the room: time. Rockefeller’s program took decades. If superintelligence is coming in the next few years, do we really have time to build something like this?

I don’t know. Maybe automated AI researchers compress the timeline, but that’s far from guaranteed. Still, I find it hard to argue that we shouldn’t at least try.

Are we already doing this?

One might also ask the obvious question: are we basically doing something like this in the aggregate already, with funders like CG fostering an ecosystem of AI safety startups and new initiatives?

Kind of, there’s definitely been good progress. CG announced it’s on track for over $1B in commitments across technical AI safety, security and fieldbuilding.

This is great, but I still think we’re nowhere near the scale we should be aiming for. Especially for work directly on alignment.

I’m talking about orders of magnitude more people1 and resources. Recruiting thousands of exceptional scientists and engineers from entirely different disciplines, paying them very competitively, and giving them all the freedom and resources needed to pursue ambitious / unconventional approaches to alignment.

And this should not all happen inside the frontier labs. I worry about the field converging on a small number of approaches, especially when research priorities are influenced by the same organisations racing to build the technology.

What I’m imagining is a much larger, more diversified ecosystem of independent research efforts. Multiple independently run funding programs, new research institutes, shared compute and engineering infrastructure, and lots of different people making lots of different bets.

How would one focus on making this happen? What is the bottleneck? Money? Talented researchers? People capable of running research institutions? CG has said that staff time is currently its main constraint on scaling.

I don’t know! I think it would be worth the time figuring it out.

Where I’ve landed

As it stands right now, I have two branches I’m looking at:

1. US-China pause: maybe more feasible than I’m giving it credit for, will continue to explore

2. Megaproject for alignment science: dramatically grow and diversify independent efforts to figure out how to build aligned superintelligence and know that we’ve succeeded

So… what’s the highest leverage thing I could work on?

One possible answer is to help catalyse the expansion. Something like helping CG scale or pushing for government-backed research programs.

I need to better understand the actual bottlenecks here.

Another answer is expanding comms. The true bottleneck to a pause is just making the current administration clear on how real the risks are. I imagine beyond that, a lot of the rest will fall into place.

If you’re seriously pursuing either a pause or something along the lines of this megaproject, please reach out. I’d love to learn more.

I’ll share what I learn in a follow-up post.

Appendix

“What about control work or d/acc or doing Glasswing / Daybreak for the entire economy or [some other defence in depth approach]?” Again, my view here is the same as for scaling up evals. It will be incrementally helpful and is already on track to be massively scaled by the labs, and maybe ASI^-1 will be able to contain ASI in every imaginable way but it’s such a high-risk bet to be making at that stage of capabilities. It’s also just not the highest leverage thing for one to work on, that is a scaling game for the labs to work on.

“What about formal verification / boxing? What if we just keep the ASI in a box and have it build narrow intelligences? Could I have the evil genie in a box and get useful value out of it?” Seems like a very dangerous game to play. The economic incentives you’d be fighting against to keep this locked in the box are so high. You’re arguing for putting the Devil in a box and hoping that no one slips up. Remember, it used to be common sense that if we ever did make AGI, that: “of course we’d never let it connect to the internet even if we mostly trusted it! That’d just be stupid!” Look how the times have changed.

“Could policy help scale this research?” Maybe. One possibility would be requiring frontier developers to allocate some share of compute to independent alignment research. But who decides what qualifies as alignment science, and how do you protect intellectual independence? I haven’t thought this through enough yet.