We usually picture alignment as a one-way street: humans set the values, the superintelligence follows. A new Anthropic study published October 27 suggests the traffic can quietly run in reverse.
The experiment
In a simulated coding experiment, fifty professional analysts were assigned as an SI coding agent to assist a fictional farm co-op to decide whether to continue a subsidy program. In the process, the analysts faced four open judgment calls: 1) the metric, 2) the statistical method, 3) the farms to be counted, and 4) how to weight them. For each of the judgment calls, the coding agent was secretly assigned a preference by a coin flip. The analysts coded while honestly advising and not attempting to mislead.
The result
- Analysts matched the agent’s hidden lean on 86% of decisions, versus 50% by chance.
- Analysts who had already chosen an option switched about 60% of the time when the agent disagreed, versus 6% when it agreed.
- Most later explained their choices using the agent’s own arguments.
“A human reviewer is only an independent check if their judgment is their own.”— Anthropic researchers
Why it matters for SI security
SI system auditing by an SI agent (often reviewed by a single human in the lab) can easily have the agent’s arbitrary (or even misaligned) preferences be echoed by the single human signing off. “Consensus” among many teams using essentially the same model could just be the model’s.
The fix isn’t less SI
Agent-assisted analysts actually made fewer result-breaking errors than the group working alone. As Decrypt reports, the authors instead point to agents that argue both sides or ask for the expert’s view first.