When the Algorithm Decides: Race, Risk, and AI in Canada's Justice System

Back in 2016, I started learning about algorithmic bias and how the data collected can disproportionately impact racialized groups. We learned about these errors from their outputs and the cases of harm they caused. For instance, Google’s image-recognition system failed to recognize Black people, and self-driving cars can pose a risk to darker-skinned pedestrians, especially at night. Reading about Joy Buolamwini’s experience as a Black computer scientist who needed a white mask for facial recognition software to recognize her, I learned how she went on to build the Algorithmic Justice League to challenge bias in decision-making software.

The AI field of facial recognition technology is growing, and it is leading to false arrests and non-consensual deepfakes. For instance, the US-based company Clearview AI has taken billions of photos from social media platforms to build its database without people’s consent. In some cases, law enforcement has continued to use facial recognition technology despite warnings that it violates Canadians’ privacy rights. The proliferation of this technology threatens our anonymity and human rights, as outlined by the Canadian Civil Liberties Association.

Since 2021, Black prisoners in Ontario jails have been subject to harsher living conditions because of an artificial intelligence tool that claims to predict a person’s behaviour. The Security Assessment for Evaluation Risk (SAFER) program collects information including arrests, charges, and disciplinary records. The program assigns each prisoner a score from 0 to 100 that determines whether they’ll be placed in minimum, medium, or maximum-security detention.

Desmond Cole writes, “critics of the program argue that the data that SAFER is fed is racially biased: they cite documented patterns of police and courts handing out more severe punishments to Black people because of anti-Black racism. SAFER then uses that data to make harsher risk assessments of Black people who are sent to jail.”

What frustrates me is that we already know these risk assessment tools, which determine “behaviour” and “recidivism” rates, have led to harsher outcomes for Black inmates. ProPublica compared the risk scores assigned to over 7,000 people arrested in Florida and found that only 20 percent of those predicted to commit violent crime went on to do so.

Not only do we know this from past examples, but the ministry also knows the risks. According to The Breach, the ministry wrote in internal documents that “Indigenous and racialized individuals face systemic discrimination in our justice system…As a result, assessments like SAFER would likely contribute to the overrepresentation of Indigenous inmates in maximum security.” Despite this awareness, the SAFER project has continued for the past five years.

For one, these concerns are not inmates’ alone; they are backed by evidence and deserve further scrutiny. Secondly, we need to be more critical of what technology is procured. We’ve witnessed how algorithmic bias manifests, yet we continue to repeat the same mistakes hoping for new results. Thirdly, if you believe this issue does not impact you because you aren’t incarcerated, think again because surveillance is expanding pervasively across the city and country. More to come on these expanding intersections.

Here is a summer reading list to complement this piece:

Unmasking AI: My Mission to Protect What Is Human in a World of Machines by Joy Buolamwini

Race After Technology by Ruha Benjamin

Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil

Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor by Virginia Eubanks

Predict and Surveil: Data, Discretion, and the Future of Policing by Sarah Brayn