Anthropic takes risky AI tests offline after models breach govt sites

Anthropic announced that its internal AI models under “high-risk and cybersecurity evaluations” no longer have internet access following incidents involving U.S. government websites at the federal, state, and local levels.

The instances leading to the decision include AI models that exploited website vulnerabilities, submitted forms, bypassed data restrictions, and circumvented safeguards.

However, the company said that the misaligned behavior had a minimal real-world impact and assessed the behavior to be less severe than previously reported incidents. The AI company notified the White House about the unauthorized activity:

Anthropic estimates that the agents may have been motivated by overly generous rewards during reinforcement learning, which may have encouraged them to find unauthorized workarounds rather than stop when a task was blocked or impossible.