AI Safety and Governance

Summary

Artificial Intelligence (AI) technologies are advancing at a rapid rate and already outpacing humans in many areas including writing code, predicting protein folding, understanding humour, and passing the bar exam. AI has the potential to serve society and significantly improve living conditions. Like any other powerful technology however, it also comes with inherent risks, some of which are potentially catastrophic.

As AI technologies are likely to get even more powerful in the near future and integrate further into our lives, it becomes even more important to acknowledge and mitigate these risks. Many AI experts, academics, policymakers, and public figures are concerned about the most severe risks of AI, including extinction. By prioritising the development of safe and responsible AI practices, we can fully harness the potential of this technology for the benefit of humanity.

Engineers may be able to contribute effectively to reducing the risk from transformative AI by switching to technical AI safety research, bringing their technical experience to AI governance, or participating in advocacy and field-building.

This Article is a Stub Page

If you’re looking for a way to contribute to HI-Eng, can spare 3 hours a week, and would like to try out some generalist research by fleshing out this article, please contact us!

How to Use This Page

The first link in each section provides a general overview. You can read the following links to dive deeper.

Published: 12 Oct 2023 by Jessica Wen

On this page

Cause area overview

What is AI?

Viewing suggestion

Future capabilities and risks

Viewing suggestion

How can engineers do impactful work in AI Safety and Governance?

What are the bottlenecks?

Technical AI Safety

One of the most promising ways to reduce the risks from AI systems is to work on technical AI safety. Potentially the most high-impact work in this field focuses on making sure that AI systems are aligned, i.e. that they do what we want them to do, aren’t power-seeking, are robust against misuse, and do not cause accidental or intentional harm. Read more on 80,000 Hours’ career review of AI Safety technical research.

The types of work you can do in technical AI safety include (taken from AI Safety Fundamentals’ career guide in technical AI safety):

  • Research: Advancing research agendas aimed at developing algorithms and training techniques to create AI and ML systems that align with human intentions, offer improved controllability, or enhance transparency.

  • Engineering: Implementing alignment strategies within real systems necessitates the collaboration of ML engineers, software engineers, infrastructure engineers, and other technical professionals typically found in modern technology companies.

  • Support Roles: This includes activities like recruitment and organizational operations at rapidly growing entities focused on building aligned artificial intelligence. Note that our guide does not currently cover advice for these roles.

  • Other Opportunities: As the field expands, alignment roles are likely to resemble positions within the broader tech ecosystem, such as product development, design, and technical management. Additionally, there may be other opportunities not yet considered in this list.

AI Policy and Strategy

To reduce risks from powerful AI, technical work needs to be complemented by strategy work to answer more human-centred questions, such as how to avoid an arms race to develop powerful AI, how to distribute the benefits widely and fairly, etc. Some good introductory resources are below:

If you’re interested in the AI industry and cloud providers’ views on AI policy (they are very vocal about them!), here are some places to start:

There are many think tanks who are focusing on AI policy and strategy, and some produce excellent work. Greg Allen’s work from the Center for Strategic and International Studies, especially his writing on export controls, comes highly recommended by Chris Phenicie. To keep up to date with research from other think tanks, you can follow newsletters from places like:

If you’re interested in how these views and information get communicated to politicians and what approaches are successful, watching congressional hearings can be very useful. In particular, watch the “Oversight of AI” hearings that led to the “Blumenthal-Hawley” framework for governing AI. These hearings include:

Some other interesting hearings include:

If you are interested in understanding China’s progress and approach to AI, the following resources are useful:

  • ChinaTalk for a focus on China, which is increasingly about AI and occasionally has very technical guests

  • Subscribe to the newsletter ChinAI

If you’re more interested in some of the more immediate problems and solutions, AI Snakeoil is a great source of information on the capabilities and limitations of AI.

Compute Governance

Although compute governance is a sub-category of AI policy and governance, we think that it is a very promising way for engineers to use their technical knowledge to contribute to AI governance and make sure AI development goes well. As a result, we have separated it out into its own section.

For an overview of compute governance, the following resources are excellent:

Engineered for Impact: AI Governance from an engineering perspective

Featured
#007 - AI compute governance with Lennart Heim
#007 - AI compute governance with Lennart Heim

Paying attention to the state of the art in AI chips and data centres, and being able to explain these to laypeople, can put you in a good position to work on compute governance. Some places to keep up to date include:

  • The Open Compute Project is also a great resource and you can even join their calls.

    • A good series to sit in on is the security calls. Knowing a lot about hardware security seems to be impactful and potentially has great career prospects even if AI X-risk ends up being unimportant.

    • One project that spun out of this is Caliptra, which seems like a great project to keep tabs on and contribute to.

For some ideas of how to apply technical concepts to governance, the following reports and papers are very useful:

If you really want to dig into some government text with a technical angle, there’s the recent AI Executive Order as well as the export controls on chips and chip making equipment (though don’t be discouraged if these are hard to read or seem boring!)

Career moves

Risks, pitfalls, and things to keep in mind

Don’t work in certain positions unless you feel awesome about the lab being a force for good. This includes some technical work, like work that improves the efficiency of training very large models, whether via architectural improvements, optimiser improvements, improved reduced-precision training, or improved hardware. We’d also guess that roles in marketing, commercialisation, and fundraising tend to contribute to hype and acceleration, and so are somewhat likely to be harmful.

Think carefully, and take action if you need to. Take the time to think carefully about the work you’re doing, and how it’ll be disclosed outside the lab. For example, will publishing your research lead to harmful hype and acceleration? Who should have access to any models that you build? Be an employee who pays attention to the actions of the company you’re working for, and speaks up when you’re unhappy or uncomfortable.

Consult others. Don’t be a unilateralist. It’s worth discussing any role in advance with others. We can give you 1-1 advice, for free. If you know anyone working in the area who’s concerned about the risks, discuss your options with them. You may be able to meet people through our community, and our advisors can also help you make connections with people who can give you more nuanced and personalised advice.

Continue to engage with the broader safety community. To reduce the chance that your opinions or values will drift just because of the people you’re socialising with, try to find a way to spend time with people who more closely share your values. For example, if you’re a researcher or engineer, you may be able to spend some of your working time with a safety-focused research group.

Be ready to switch. Avoid being in a financial or psychological situation where it’s just going to be really hard for you to switch jobs into something more exclusively focused on doing good. Instead, constantly ask yourself whether you’d be able to make that switch, and whether you’re making decisions that could make it harder to do so in the future.

Learn more

Additional resources

Relevant organisations