AI Safety

AI Safety is a research and engineering field focused on ensuring that AI systems are reliably beneficial — that they behave as intended, avoid harmful outputs, and remain aligned with human values as they become more capable.

Near-term safety (today's concern):

• Jailbreaking — bypassing safety guidelines through clever prompting
• Prompt injection — malicious instructions hidden in user content that redirect an agent
• Bias and fairness — models encoding historical prejudices from training data
• Misuse — generating disinformation, malware, or harmful content at scale

Long-term alignment (future concern):

• Specification gaming — model achieves the stated goal but not the intended one
• Reward hacking — optimizing proxy metrics while missing the real objective
• Scalable oversight — how do you evaluate outputs of a model smarter than you?
• Deceptive alignment — a model that behaves well during training but not deployment

Key approaches:

• RLHF (Reinforcement Learning from Human Feedback) — train on human preference data
• Constitutional AI (Anthropic) — self-critique against a written set of principles
• Interpretability — understanding what's happening inside the model's weights
• Red-teaming — adversarial testing to find failure modes before deployment

Leading labs: Anthropic, DeepMind's safety team, OpenAI's safety team, ARC (Alignment Research Center), MIRI.