AI safety reading list

If you're new to all of this, start here.

• BlueDot ImpactAI Safety Fundamentals ↗Free, structured courses on alignment and governance. The single best on-ramp; several of us went through it ourselves.
• Lens AcademyLens Academy ↗A free course on why aligning superintelligence is hard, with readings, an AI tutor and weekly group discussions. No application and no waitlist.
• 80,000 HoursRisks from advanced AI — problem profile ↗A careful, readable case for why this might be one of the most pressing problems — with honest treatment of the counterarguments.
• Anthropic, 2023Core views on AI safety ↗A frontier lab's own account of when and why AI could become dangerous, and what they think can be done about it.
• CAIS, 2023Statement on AI risk ↗One sentence, signed by many of the field's leading researchers and lab heads. Useful as a marker of how mainstream the concern has become.
• Field mapAISafety.com ↗A living map of the AI safety ecosystem — organisations, courses, funders, communities. Good for finding your next step.

The core research problem: how do you make capable AI systems reliably do what their designers and users intend?

• Amodei et al., 2016Concrete problems in AI safety ↗The paper that translated vague worries into concrete research problems — reward hacking, safe exploration, robustness to distribution shift. Still a useful vocabulary.
• Ngo, Chan & Mindermann, 2022The alignment problem from a deep learning perspective ↗Why standard training methods could produce systems that pursue goals their developers didn't intend. Probably the best single technical overview.
• Anthropic, ongoingTransformer Circuits ↗Mechanistic interpretability — actually opening up neural networks to see what's computed inside. Start with "Toy Models of Superposition."
• Bai et al., 2022Constitutional AI: harmlessness from AI feedback ↗How current frontier models are actually aligned in practice — training models against written principles rather than only human labels.
• ForumAlignment Forum ↗Where the field argues with itself in public. Uneven, but the best place to watch ideas get stress-tested.
• LessWrong, 2025Shallow review of technical AI safety, 2025 ↗A broad survey of where technical safety research stands right now — useful for seeing which problems are getting traction and which remain open.

The bigger-picture arguments — for and against treating advanced AI as a potential catastrophe.

• Carlsmith, 2022Is power-seeking AI an existential risk? ↗The most careful version of the x-risk argument, built as an explicit chain of premises with probabilities attached — so you can see exactly where you'd disagree.
• Hendrycks et al., 2023An overview of catastrophic AI risks ↗A taxonomy of how things could go badly — malicious use, racing dynamics, organisational failure, rogue systems — beyond just the classic misalignment story.
• Scott Alexander, 2023Why I am not (as much of) a doomer (as some people) ↗A thoughtful case for lower risk estimates, from inside the conversation. We try to keep the strongest versions of both sides on the table.
• Kokotajlo et al., 2025AI 2027 ↗A concrete, contested scenario forecast of the next few years. Read it less as prediction and more as a way to make your own expectations explicit.

Policy levers, international coordination, and what governments can do now.

• Centre for the Governance of AIGovAI research ↗The central academic hub for AI governance — compute governance, frontier-lab policy, international agreements.
• UNCPGA, 2025Governance of the Transition to AGI ↗A report for the UN General Assembly on governing the transition to AGI — urgent policy considerations as industry leaders anticipate AGI within this decade.
• Bengio et al., Science 2024Managing extreme AI risks amid rapid progress ↗A short consensus piece by senior researchers on what governments should do now. A good baseline for policy discussions.
• The Millennium ProjectGlobal governance of the transition to AGI ↗A multi-year study drawing on AGI leaders and experts from 65 countries, setting out regulatory options, global governance models and scenarios to 2035.

The domestic AI landscape — policy, community, and ecosystem.

• Govt. of IndiaIndiaAI ↗The national AI programme — useful for understanding where Indian state capacity and attention currently sit.
• India AI TrackerIndia AI Tracker ↗Tracks AI developments, policy moves and ecosystem growth across India — a useful pulse-check on the domestic landscape.
• Carnegie IndiaCarnegie India — technology research ↗Some of the most serious writing on AI policy from an Indian vantage point.
• SAFL, Delhi & BengaluruSecure AI Futures Lab ↗Research and capacity building on AI safety and governance for India and South Asia, including fellowships for people moving into the field.
• Electric SheepElectric Sheep ↗A welfare-focused AI safety community working to make AI go well for people, animals and the planet, through courses, hackathons and a global network of builders.
• Bangalore AI Safety & Alignment StudyBASIS ↗A Bangalore reading group for technically grounded discussion of AI safety and alignment papers, from interpretability to agent evaluations.
• TARA, Asia-PacificTARA — Technical Alignment Research Accelerator ↗A free, 14-week part-time programme that works through the ARENA curriculum in Saturday cohorts across Asia-Pacific, including Delhi and Bangalore. Ends with a three-week research project.
• EA IndiaEffective Altruism India ↗The Indian EA community hub — connects people working on high-impact causes including AI safety, and a good way to find others thinking about these problems locally.

For longer-form treatment. Links go to overviews; copies circulate within the group.

• Brian Christian, 2020The Alignment Problem ↗The best narrative introduction — how alignment research actually emerged, told through the people doing it.
• Stuart Russell, 2019Human Compatible ↗One of AI's founding textbook authors argues the field's standard objective is the problem, and proposes a redesign.
• Nick Bostrom, 2014Superintelligence ↗The book that put x-risk on the map. Dated in places, but many of the core concepts still frame the debate.

If the reading has you wanting to contribute, these are the doors people in our group have actually used or recommend.

• Research programMATS — ML Alignment Theory Scholars ↗A mentored research program that has become one of the main pipelines into full-time alignment work.
• Technical bootcampARENA ↗An intensive technical curriculum (interpretability, RL, evals) — also excellent for self-study; the materials are open.
• Hackathons & sprintsApart Research ↗Regular global research sprints and hackathons — a low-cost way to try doing safety research before committing to it.
• Careers80,000 Hours job board ↗The most complete listing of safety-relevant roles across research, engineering, policy and operations.