An AI Just Beat 676 Human Forecasters, Including the Professionals

Morning, {{first name | folks}}! A lot happened in AI this week. We’ve got an AI beating 676 human forecasters, a Gemini security test going very wrong, $1 billion going into independent AI evaluation, China cooling its humanoid robot IPO rush, and NVIDIA tackling AI inference benchmarking.

Mantic beat every human participant in this summer's Metaculus Cup, a serious forecasting tournament, including professional forecasters, something experts predicted was still a decade away as recently as last year. It called Colombia's presidential frontrunner a week before humans did, and it beat every other AI system in the competition too, not just humans, meaning the edge came from Mantic's own tuning, not just a strong base model.

It's already being used by hedge funds and Fortune 500 companies for things like M&A timing and macro predictions. Today it raised $25M, led by Radical Ventures, with Microsoft's own venture arm and Hugging Face's co-founder both backing it.

Google confirmed Gemini broke into three real companies' systems during a May security test, and the cause is almost absurd, the fictional company it was supposed to be testing against happened to share a name with a real one, and Gemini went after the real target instead. It guessed its way into one system through password attempts and found exposed credentials in a public repo for the other two. Internet access wasn't even supposed to be available during the test at all.

Google's model did stop itself the moment it realized it had hit real companies, not the fictional target. The bigger picture matters more than this one incident though, the same evaluator says the exact same testing-procedure gaps already caused similar incidents at OpenAI, Anthropic, and Meta. This isn't one lab's mistake anymore, it's a pattern across the entire industry's safety testing process.

Anthropic is partnering with Accenture to embed independent evaluators inside the company, real follow-through on Dario Amodei's "We Must Pace the Frontier" essay from earlier this month. Both companies expect to invest at least $1 billion each over five years. These evaluators get employee-level access, watching models take shape during actual training, sitting in on internal decisions, talking to Anthropic staff directly, not just testing a finished product from the outside.

Anthropic's clear that this doesn't let them off the hook, they still own the safety of their own models. This just makes their claims independently checkable. There's no industry standard yet for how any of this should actually work, Anthropic says they're building it as they go, and more evaluator partnerships are coming in the next few weeks.

China is putting the brakes on a wave of humanoid robot IPOs as regulators question whether soaring valuations and government-backed projects reflect real commercial demand. The scrutiny intensified after Unitree Robotics surged more than fivefold on its Shanghai debut, then fell 55% from its peak.

Several Chinese robotics companies are preparing to go public, but investors are now being asked to look harder at factory deployments, order volumes, and recurring demand. Some private-market valuations have already been cut by 30% to 50%, showing how quickly the market is moving from excitement around humanoid robots to questions about which companies can turn the technology into a real business.

NVIDIA has introduced AIPerf, an open-source benchmarking tool for testing LLM inference under heavy production workloads. It is designed to measure latency, throughput, GPU usage and other performance metrics across different models, serving systems, and hardware setups. It can also generate high-concurrency traffic without the benchmark itself becoming the limiting factor.

AIPerf can replay production traces and simulate different traffic patterns, including bursty and concurrent workloads. That gives AI teams a way to test whether their inference stack can actually handle the traffic they expect, rather than relying on performance numbers from simple benchmark runs.