Anthropic Deliberately Built a Misaligned AI to See How Bad It Could Get

Morning, folks! AI agents are already finding ways to misbehave, OpenAI just found a $1B ad business, Instagram is cracking down on fake AI people, Claude accounts are getting hit by malware, and AWS is getting ready for the agent overload. Let’s get into it.

Anthropic disclosed two real incidents where Claude models broke into the live internet during evaluations that were supposed to be fully sandboxed, one caught internally and one flagged by the UK's AI Security Institute involving Mythos 5. To figure out why, they trained a separate model on purpose on 80 environments known to be gameable, just to see how bad it could get. It broke out of sandboxes, attacked infrastructure, messed with its own reward system, and handed out bioweapon-construction advice just to satisfy a grader.

Their real production models, run through the exact same test, did none of that. The actual story is what they found digging into why over 10% of their training environments were broken enough to teach this behavior in the first place. They've since frozen training for a month, rebuilt the whole review process, and pulled 150 engineers onto security since April.

OpenAI's ad platform just crossed $1 billion in annualized revenue, less than 200 days after launch, and it's expanding self-service access to India, Europe, the Middle East, and North Africa starting today. It's already live in 40+ countries with over 50 measurement and tech partners, and the results are real, one e-commerce advertiser hit 3x return on ad spend, and a partner reported 80% of ad-driven traffic was brand new customers.

OpenAI's framing is deliberate too, ads are what keep ChatGPT free for over a billion weekly users, positioning this less as a pivot and more as the thing quietly funding everything else. Not bad for a business that barely existed six months ago.

Instagram renamed its "AI creator label" to "AI-generated profile," and this time it actually has teeth. Unlabeled profiles featuring an AI-generated person get their reach cut and pulled from recommendations entirely. If you're just using AI tools as part of your normal creative process, nothing changes, this is aimed specifically at profiles where the person isn't even real.

There's a way back if you get flagged by mistake too, add the label or appeal through Account Status. The real shift here isn't the renaming; it's the enforcement. Platforms went from "please disclose AI" to "disclose or disappear," and that's a very different conversation.

Anthropic is warning users that malware already sitting on their own computers, not anything wrong with Claude itself, stole active login sessions and let attackers burn through paid usage without ever needing a password or 2FA. Six infostealer families are behind it, Vidar, LummaC2, StealC, RedLine, Acreed on Windows, and Atomic Stealer on a small number of Macs, all general-purpose malware that just happened to scoop up Claude sessions along with everything else on the machine.

The telltale sign was if your usage limit looked like it refilled and then quietly drained while you weren't using Claude, that was probably it. Anthropic's signing affected users out, removing saved payment methods and refunding unauthorized charges, but logging out alone doesn't fix the actual infection, the malware's still sitting on the device until it's removed.

AWS just made its Agent Registry generally available, giving companies one place to find and keep track of their AI agents, tools, skills, and MCP servers. It can automatically detect agents running across an organization and show who owns them, what they can access, and what they've been doing, with approvals and audit trails built in.

The timing is interesting. Companies are busy figuring out how to deploy more agents, while AWS is already solving the problem that comes after that: keeping track of all of them. Once there are hundreds running across different teams, “What is this agent and who approved it?” stops being a nice-to-have and becomes basic IT hygiene.