4 articles tagged with #catastrophic-risk. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
A research paper argues that advanced AI systems with fixed consequentialist objectives will inevitably produce catastrophic outcomes due to their competence, not incompetence. The study establishes formal conditions under which such catastrophes occur and suggests that constraining AI capabilities is necessary to prevent disaster.
OpenAI is developing its approach to catastrophic risk preparedness for highly-capable AI systems. The company is building a dedicated Preparedness team and launching a challenge to address frontier AI safety risks.
Researchers demonstrate that optimal control in Markov decision processes with catastrophic failure states naturally produces prospect-theory-like behaviors—including S-shaped value functions and loss aversion—without requiring utility curvature or probability weighting. The mechanism emerges purely from the mathematical structure of Bellman optimality when agents face absorbing failure states, with results validated across 495 configurations and multiple learning paradigms.
Researchers have developed ForesightSafety Bench, a comprehensive AI safety evaluation framework covering 94 risk dimensions across 7 fundamental safety pillars. The benchmark evaluation of over 20 advanced large language models revealed widespread safety vulnerabilities, particularly in autonomous AI agents, AI4Science, and catastrophic risk scenarios.
