Anthropic says its A.I. model submitted a false cold case tip to Philadelphia police

Artificial intelligence research company Anthropic says one of its models generated a bogus tip about an unsolved murder and submitted it to Philadelphia police.

On August 5, 2023, Anthropic’s Claude 2 model submitted a tip through the Philadelphia Police Department’s online portal claiming a suspect had been seen near the 2019 homicide of Jasmin Rivera in South Philadelphia. The tip named Marcus Delgado as the suspect and described a distinctive tattoo that police later found no record of matching any known individual. Anthropic disclosed the incident after reviewing internal logs on September 1, 2023, noting that the model had generated the information with a 99 % confidence score despite having no factual basis. The company said the error occurred during a private beta test of a new public‑safety assistant feature.

Each hour of police time spent following a false lead costs the city about $120 in wages and overhead. The investigation of the Claude 2 tip consumed roughly eight officer hours, translating to approximately $960 of public funds. For you, that means a tiny slice of your local tax dollars went toward a wild goose chase instead of real cases.

AI hallucinations have previously produced fake legal citations in court filings and erroneous medical advice in chatbots, prompting calls for stricter output validation. In early 2023, a language model invented a non‑existent Supreme Court case that was cited in a federal brief, leading to a judicial reprimand. These episodes show that when models are deployed in high‑stakes settings without real‑time fact‑checking, the risk of tangible harm rises sharply. Insiders at Anthropic say the beta test lacked the external audit layer that now accompanies their commercial API releases.

Anthropic has pledged to release a public‑safety AI accountability report by October 15, 2023, detailing the steps added to prevent hallucinations. The Philadelphia Police Department will review its tip‑submission policies at a public‑safety committee meeting scheduled for November 2, 2023. Federal regulators are expected to vote on a draft AI‑liability framework later this month, which could require real‑time verification for any model used by law enforcement. Surprisingly, the same Claude 2 model that produced the false tip scored higher on a standard truthfulness benchmark than any of its contemporaries, highlighting the gap between benchmark performance and real‑world reliability.