This Is Flock’s AI Search Tool for Cops

The code WIRED collected holds Flock’s user interface and shows both the network calls it makes and the answers it expects in return. It shows that departments can, for now, configure the tool for some searches without a reason or a case number linked to a real investigation, and can leave off the automated system that flags suspicious use; Flock says the controls will be mandatory by year’s end.

Search activity is recorded in audit logs, Flock says, and required search restrictions apply across its search tools. The company did not say whether anything on its servers would stop a search submitted without a reason or a case number.

Fired, Suspended, Resigned

Police officers have already used Flock’s software to track people for reasons unrelated to police work. At least 50 officers in the US have recently been charged with, or accused of, misusing license plate readers, according to The Washington Post. Forty-six of those cases involve Flock. In over half, the alleged target was a wife, a girlfriend, an ex-partner, an ex-partner’s new boyfriend, or a woman the officer wanted to meet.

Details of how police databases can be abused appear repeatedly in the reports of officer misconduct: A Milwaukee officer searched for a woman he was dating 124 times and her former partner 55 times, and logged each search as an “investigation.” A Georgia police chief ran his ex-girlfriend and her daughter 600 times, was charged, and took his own life five months later. A Kansas chief, who was never charged, ran his ex 164 times and her boyfriend 64 times, and lost his badge. A North Carolina officer ran her boyfriend’s ex-wife 31 times and filed 29 of them as traffic infractions.

Police abusing investigative tools is not new. A 2016 Associated Press investigation found officers running database searches on exes and people they found attractive, with police employees at state agencies and dozens of the nation’s largest departments fired, suspended, or forced to resign more than 325 times between 2013 and 2015. WIRED reported in 2023 that Immigration and Customs Enforcement employees and contractors had been investigated at least 414 times since 2016 for misusing sensitive government databases, and found hundreds of more allegations against Customs and Border Protection employees this year, among them officers who queried databases to track down women they wanted to date, monitor family members, and follow colleagues’ phones.

Two notification systems in Flock’s software are aimed at human beings. The watchlist based on police search queries is one. It runs on an officer’s own written description: They draw a shape on a city map, the alert runs against every camera inside it, and anyone the model decides is a match triggers a notification. The frequency of alerts can be adjusted.

The other, dubbed a People Detection Alert, takes no description at all. Here an officer draws a box inside a single camera’s field of view—a doorway, a stretch of pavement—and is notified when a person is inside it. One person is enough to trigger it. The software will not accept an alert set below 75 percent certainty that what the camera is looking at is a person.

Three Verdicts

Flock described its moderation system in April, saying it was designed to ensure First Amendment protections are “baked in.” The code WIRED reconstructed shows an AI model on Flock’s servers scoring an officer’s description of a person against eight categories of sensitive content before it runs against stored footage, then returning one of three verdicts: allow, block, or warn. The model relies on a confidence score to reach that verdict. Because the model runs on Flock’s servers, the department never sees that score. All it gets is a canned warning or block.

Blocked categories of search include race or ethnicity, religion, nationality, “subjective or biased terms,” and “offensive content,” which the code describes as sexual content, innuendo, offensive words and phrases, and slurs. In vehicle mode, the tool blocks any search using person-related terms, gender, clothing, and behavior among them.

Religion and nationality are the only two categories written both ways, with wording for a block and wording for a warning that lets the officer proceed. Political, social, and cultural expression is written only as a warning.

Flock tells WIRED that officers cannot run searches using prohibited attributes including religion and nationality, and that an attempt to search prohibited terms “will be blocked.” Asked what circumstances produce a warning in those two categories instead, the company did not say.

When a T-shirt or a bumper sticker draws a warning because the text includes speech with “constitutional protections,” officers are told the search will be logged and that administrators will be notified. Officers then have to tick an acknowledgement box and leave a comment before they’re free to click “Continue With Search Anyway.”

Flock says the warning leaves room for legitimate work, offering as an example a victim who describes a suspect in a biker gang jacket bearing “a certain gang logo or emblem containing a flag or other insignia.” An officer who proceeds is reported to an administrator at their own department for review.

Kate Ruane, who directs the Center for Democracy and Technology’s free expression project, has spent years studying how automated moderation systems work. Political, social, and cultural expression is “an incredibly amorphous category,” she says, and one reason an officer’s search might contain language including it is to see who attended a protest. That’s already happened. The category worries Flock enough that the system returns a warning, she points out, “but you can still get the results if you just click through.”

“No content moderation done at scale is necessarily accurate,” she says. Analysis of moving video, she says, is harder still, and it goes wrong more often.

The categories that do get blocked can be worked around, Ruane says. The system blocks searches based on religion but not clothing, meaning in practice that an officer could get around the block by searching for the distinctive dress worn by some members of a particular faith. “Lots of people are having their images returned in response to these types of queries that would probably be upset if they knew about it.”

In a search last year an officer in California typed, “American flag.” The search drew a block when aimed at a person, then ran across 11,000 cameras when aimed at vehicles instead.

A warning only stops the officers who are not already determined to run the search, says Deepak Kumar, an assistant professor of computer science and engineering at the University of California San Diego. Kumar, who studies trust and safety systems, points to interfaces built to address online harassment, where motivated users sometimes did more harm after being warned, and to browser alerts meant to steer people away from malicious websites, where the effect depends on how warnings are presented.

“This kind of logging can be useful administratively, to see which officers or end-users bypass warnings often, but that value depends entirely on whether anyone administrates it,” Kumar says. “It functions less as a deterrent and more as a record.”

Reviewing search terms before a query runs will inevitably catch many attempts at misuse, but in isolation, such safeguards are known to fail without equal scrutiny of what the model sends back. “Best practice examines both inputs and outputs,” Kumar says, “which is why so many AI safety efforts now check both to prevent models from generating harmful outputs.”

Jay Stanley, a senior policy analyst at the ACLU’s Speech, Privacy, and Technology Project, agrees that the moderation system addresses a narrow slice of the problem. The filtering, he says, is “a speck on the back of a giant surveillance machine,” whose database can be probed by AI for “your-imagination-is-the-limit fishing expeditions.” Nobody outside Flock can say how its statistical models reach those calls.

Like Flock’s other recent reforms, he says, the moderation appears to be aimed at “bad apple cops” and leaves the structural dangers of mass surveillance untouched.

Local Rules

The searches described so far run against every person or vehicle Flock’s cameras have seen, back as far as an agency stores footage. Besides a written description, the tool takes a plate or a cropped photograph, returns cars present at locations an officer marks on a map, and infers associations from driving patterns, handing police who enter a plate number the vehicles seen beside it. How far a search reaches depends on the search mode, the agency’s permissions, and the agreements it holds with other organizations. A search can run on the agency’s own cameras, on cameras other organizations share, or, for some plate searches, across statewide and nationwide networks.

Flock says much of the responsibility for governing its technology sits with local agencies, which set their own rules for access and use. Don De Lucca, a former Miami Beach police chief and past president of the International Association of Chiefs of Police, says that is largely right. Vendors can recommend safeguards, he says, but departments have to set the rules before the technology goes into use. “It has to be tied to some kind of legal reason.”

A police commander who oversees special investigations at a Southern California police department, and requested anonymity because he’s not authorized to speak, says the possibility of abuse is not unique to Flock or surveillance technology. New technologies routinely create opportunities for both legitimate use and for misuse, he says. The real question is “whether the technology gets implemented with strong guardrails, active oversight, audit ability, and consequences for any misuse.”

As scrutiny of Flock intensified this spring, the company announced Audit Assistance, which surfaces patterns that may warrant review. Flock says more than a third of its customers enabled it in August. The code WIRED reviewed shows it currently off by default. It can reportedly flag repeated searches by one officer for the same plate, searches limited to other agencies’cameras, and for one plate searched under multiple case numbers.

Flock says account lockouts for abnormal activity will arrive alongside it. A South Carolina sheriff’s office turned Audit Assistance on earlier this month, and an internal-affairs officer identified more than 2,700 allegedly unauthorized searches the next day.