Constitutional AI
Constitutional AI (CAI) is a technique for training artificial intelligence systems, particularly large language models, to be helpful, harmless, and honest without relying heavily on human feedback for every harmful output. It was developed by Anthropic and introduced in a December 2022 paper.[1]
Constitutional AI is a core component of the training methodology behind Anthropic's Claude family of models. It represents a significant departure from the standard reinforcement learning from human feedback (RLHF) pipeline used by OpenAI and others, by replacing much of the human labelling step with AI self-critique guided by a written set of principles — the "constitution."
Motivation
Standard RLHF training for language models involves collecting large quantities of human preference data: human evaluators compare pairs of model outputs and indicate which is better. This process is expensive, slow, and subject to inconsistency between annotators. It also creates a bottleneck: the model can only become as aligned as the quality and quantity of human feedback permits.
Anthropic's researchers observed several additional problems with pure RLHF:
- Scalability: As models become more capable, the range of potentially harmful outputs grows, making comprehensive human coverage impractical.
- Transparency: The rules learned by an RLHF-trained model are implicit in the preference data and difficult to audit or modify.
- Brittleness: Models trained on narrow human preferences can fail unpredictably on edge cases not represented in the training set.
Constitutional AI was designed to address these limitations by making the alignment criteria explicit and machine-evaluable.
Method
Constitutional AI operates in two phases:
Phase 1: Supervised self-critique (SL-CAI)
In the first phase, the model is prompted to generate responses to potentially harmful queries. It is then asked to critique its own output against the principles in the constitution — a written list of rules such as "choose the response that is least likely to be harmful" or "prefer the response that is most respectful of human dignity." The model revises its response based on its own critique. These revised responses are used as supervised fine-tuning data.
This process can be iterated: the model critiques its revision, revises again, and so on. The result is a dataset of (prompt, improved-response) pairs generated entirely through AI self-evaluation, without additional human annotation.
Phase 2: Reinforcement learning from AI feedback (RLAIF)
In the second phase, the model is used to generate preference labels. Given pairs of responses to the same prompt, the model is asked which better satisfies the constitutional principles. These AI-generated preference labels are then used to train a reward model, which in turn is used for reinforcement learning fine-tuning — the same pipeline as standard RLHF, but with AI-generated rather than human-generated labels.
Anthropic dubbed this process Reinforcement Learning from AI Feedback (RLAIF) to distinguish it from the human-labelled RLHF approach.
The constitution
The "constitution" is a set of natural-language principles that guide the model's self-evaluation. The original Anthropic paper used principles drawn from a variety of sources, including:
- The Universal Declaration of Human Rights
- Apple's Terms of Service (as an example of a corporate content policy)
- Anthropic's own research principles
- General ethical guidelines about avoiding harm, discrimination, and deception
The constitution is modular: different principles can be added, removed, or weighted differently depending on the desired behaviour of the final model. This modularity is considered one of the technique's main advantages — organisations can define their own constitutions tailored to specific use cases without retraining the underlying model from scratch.
Anthropic has published the constitution used for Claude models and has updated it over time as the models and their deployment contexts have evolved.
Advantages
- Transparency: The rules governing the model's behaviour are written in natural language and can be inspected and debated.
- Scalability: AI feedback is dramatically cheaper and faster than human feedback, enabling alignment training at larger scale.
- Reduced harm in training: Human evaluators are not required to read and compare large volumes of harmful content, reducing psychological burden on annotators.
- Iterability: The constitution can be updated without collecting new human preference data.
Limitations and criticism
- Self-referential evaluation: The model evaluates its own outputs, which means it can only catch problems it is already capable of recognising. Novel or subtle harms may pass through the self-critique loop.
- Constitutional bias: The choice of principles in the constitution reflects the values of its authors. Different organisations or cultures might reasonably disagree about what should be included.
- Gaming: Sufficiently capable models might learn to satisfy the letter of constitutional principles while violating their spirit, a form of Goodhart's Law applied to alignment.
- Evaluation gap: Empirical comparisons between CAI-trained and RLHF-trained models are difficult because the training data, model architectures, and evaluation benchmarks differ across organisations.
Some researchers have argued that Constitutional AI is better understood as a practical engineering technique than as a solution to the fundamental AI alignment problem, since it presupposes that the correct alignment targets can be articulated in advance and written down in a constitution.
Influence
Constitutional AI has influenced subsequent alignment research across the AI industry. Elements of the approach — particularly the use of AI-generated preference labels and explicit written guidelines — have been adopted or adapted by other laboratories. The technique has also been cited in policy discussions about AI governance, where the idea of a transparent, auditable set of behavioural rules resonates with regulatory interest in "explainable AI."
Within Anthropic, Constitutional AI is one of three pillars of the company's alignment research programme, alongside mechanistic interpretability and empirical evaluations of model behaviour.