Note from the Editor, Tricia Wang: The next author for the Co-designing with machines edition is Angèle Christin (@angelechristin), sociologist and Postdoctoral Fellow at the Data & Society Institute. In a riveting post that takes us inside the courtrooms of France and newsroom of the the US, Angèle compares how people deal with technologies of quantification in data-rich and data-poor environments. She shows how people in both contexts us similar strategies of resistance and manipulation of digital metrics in courtrooms and newsrooms. Her post is incredibly valuable as both courtrooms and newsrooms are new areas where algorithmic practices are being introduced, sometimes with appalling results, such as this Propublica article reveals. Angèle studies sectors and organizations where the rise of algorithms and ‘big data’ analytics transforms professional values, expertise, and work practices. She received her PhD in Sociology from Princeton University and the EHESS (Paris) in 2014.
I came to the question of machines from the study of numbers, more precisely the role that numbers play in organizations. Ten years ago, I wasn’t very interested in technology: I was a student in Paris, I barely had an email address, and what I wanted to study was criminal justice.
The fall of 2005 in France was marked by the events that came to be known as the “urban riots” (émeutes urbaines), a period of unrest among the young men and women living in city outskirts (banlieues). Their protests were triggered by the death by electrocution of two teenagers who had sought refuge in an electric substation while being chased by the police.
Over the next couple of months, cars were burning, the police were everywhere, and many young men of African and North-African descent were arrested, arraigned, and sentenced, usually to prison. Parisian courts relied heavily on an old penal procedure for misdemeanors, the comparutions immédiates (emergency hearings), which makes it possible to sentence defendants immediately after their arrest. The procedure was originally designed to control “dangerous” urban crowds in the second half of the 19th century.
During and after the urban riots, journalists and intellectuals denounced the revival of a bifurcated justice system, in which lower class and minority defendants were tried in a hurry, with meager resources for public defenders, insufficient procedural safeguards, and high rates of prison sentences. Crowds of friends and supporters congregated in the courts and attended the hearings, cheering the defendants and booing the judges. The police heavily guarded the courtrooms in order to prevent direct attacks on the magistrates.
In all of this, judges and prosecutors remained silent. No one knew what was really going on before or after the hearings. I decided to go behind the scenes to examine how prosecutors, judges, and lawyers worked on the cases and decided on the charges and sentences of the defendants. I was able to conduct a yearlong ethnographic study of several criminal courts, including one in Paris and one in a North-East banlieue.
Managing overflow in emergency hearings
My book Comparutions immédiates (Emergency Hearings) (Paris, La Découverte, 2008) presents the main findings of this inquiry. During my interviews and observations, I realized that judges and prosecutors were often critical of the emergency hearings procedure, but also felt that they did not have much of a choice.
Every day, urban courts received hundreds of files from the police. Every case needed to be processed. The main question, for most prosecutors and judges, was what to do with all these files, which overwhelmed the courts. On the prosecutorial side, district attorneys could sometimes drop charges or ask police officers for more information about cases, thus delaying the moment to make a decision about the case.
Photo by Melanie Richter
But at the end of the day, the whole system was clogged up: alternative procedures (such as pleading guilty, a novelty in France at the time) and non-custodial sentences (such as community service or counseling) were already full. The only procedure where there was space, they argued, was the comparutions immédiates, an omnibus hearing for misdemeanors. It was also generally characterized as an “efficient” procedure: in emergency hearings, judges could sentence more than twenty cases in an afternoon.
The only problem – not a minor one, even to the judges and prosecutors – was that emergency hearings tended to send defendants to jail to a much higher rate than other penal procedures. This was for several reasons. When defendants were brought to the hearings, the notice was often too short for their families to bring documentation proving their employment, residence, or family status. The average quality of indigent defense was also low: lawyers, who were paid by the case, did not have enough time to prepare a thorough defense. Last, the defendants were exhausted and traumatized when they were presented to the judges: they usually had not been able to sleep or wash since their arrest a couple of days before.
As a result, judges tended to perceive defendant in emergency hearings as more “risky” and disaffiliated than if they were coming from home with all their documents several months after the events.
Most judges in comparutions immédiates resented that situation. They argued that they received too many cases every day from the police and the prosecutor’s office. They explained that that they could not do their work as conscientiously as they would have liked, because they were unable to afford to spend more than thirty minutes per case – otherwise they would have to postpone the hearings, thus encroaching on the next day’s schedule. Marie, a judge in comparutions immédiates, told me: “We have so many cases every day that we have to go quickly to make sure that we’ll end at a reasonable hour… The first couple of months I was here, my hearings ended around 3am each time… (laughs). Now I’m faster, but it’s not easy. With the lack of time, the stressful conditions, I’m more likely to forget things, to make mistakes, especially late at night. Sometimes I’m scared, I look at the number of cases for the day and I’m like, ‘how am I going to manage?’ Time matters… I just can’t sentence people to jail in twenty minutes.” Judges in comparutions immédiates frequently complained that prison was de facto their main sentencing option: it was the only place where there was space for the twenty-five defendants or so they saw on any given day.
As I did more interviews and fieldwork, I realized that everybody in the courthouses kept talking about numbers: how many cases were processed every day, what percentage of defendants had criminal records, how fast each judge was, what was the percentage of prison sentences for each procedure and court, how many charges were abandoned on average, how many cells were available, etc.
Such numbers were at the center of heated controversies within the courts. Judges, prosecutors, lawyers, and police officers argued about charts, excel spreadsheets, paper clippings, and blackboards with numbers on them. Numbers were hotly debated, contested, and manipulated by everybody involved – except, of course, by the defendants, who were largely silent in the process.
Instead of unilaterally blaming individual judges and prosecutors for the failures of the criminal justice system, I showed in the book how limited resources and managerial constraints shaped the (unfair) outcomes of criminal procedures. In the process, I started to pay close attention to metrics and their unintended consequences in organizations.
Analytics and the ordering of web newsrooms
At first sight, nothing could be more different from courtrooms than web newsrooms:
News organizations are for-profit companies, whereas courts are public institutions. Journalism is a semi-open profession based on a loose set of skills, whereas legal professions have a strict monopoly on their area of expertise through legal education. Web journalists do most of their work online, whereas judges and prosecutors are still operating in a largely analog environment.
Photo by Matthew Simantov
Yet when I started doing fieldwork in web newsrooms for my dissertation, I was struck by the many similarities between newsrooms and courtrooms.
Like judges and prosecutors, journalists and editors constantly try to manage the overflow of information they receive every day and need to make decisions about which news to prioritize.
Like judges and prosecutors, journalists feel that they cannot do their job properly because they lack time. That said, their constraints are kind of opposite: whereas judges and prosecutors have to process and sentence every single case sent to them, web editors and journalists instead have to create news articles and publish relentlessly to “feed” their news sites.
More importantly, metrics are also ubiquitous in web newsrooms. As I have described elsewhere, traffic numbers became central actors in the daily functioning of most newsrooms. Today’s editors and staff writers cannot escape the pressure of web analytics. They follow the online popularity of their articles in real-time, allocate the placement of articles on the homepage depending on their click-through rates, conduct A/B testing to decide which headlines are the most popular, receive detailed optimization recommendations about how to promote their pieces, and are increasingly compensated based on the traffic they attract to the sites.
Of course, metrics for digital publishers are much more sophisticated than the ones I observed in French courthouses. They draw on many more observations and are largely automated, with software programs such as Chartbeat, Social Flow, Parse.ly, and others. Whereas the judges and prosecutors in France had to build the metrics themselves by filling spreadsheets manually, journalists are usually not involved in the construction of web analytics.
Similarities between courtrooms and newsrooms
I also found similar processes of resistance against technologies of quantification during my fieldwork in newsrooms and courtrooms.
One strategy developed by both journalists and magistrates is to draw on their professional ethos to criticize the legitimacy of metrics and refuse to engage with them. They argue that “good justice” or “good journalism” cannot be measured and evaluated in quantitative ways. They also explain that doing their job properly often takes time, which leads to poor performance in terms of number of cases or number of clicks.
Second, judges and journalists engage in “gaming” strategies: they actively try to manipulate metrics in order to get “good” numbers without changing the way they do their daily work. In criminal justice, it often means reclassifying cases by changing the charges and offense levels so that they end up on someone else’s desk. For journalists, this may entail writing a short “clickbait” article as a way to “buy” time to write a long investigative piece. As Agnès, a web journalist working in a Parisian newsroom, explained, “it’s almost like a natural law, at least for me… (laughs). After 10, 12 pieces that didn’t work, I give myself a break and I write a short piece that I know is going to be a hit! It’s not that I sell my soul to the devil, no, no, I just… I reset the scale…”
Third, judges and journalists are usually highly conscious of the arbitrary nature of metrics. Some of them thus engage in political strategies to convince their hierarchies to change the metrics used in the organization. For journalists, this means arguing in favor of “quality metrics” such as “time engaged” instead of page views. For French judges and prosecutors, this meant asking their hierarchy and the Ministry of Justice to change the goals of the penal policy, an annual document setting the relative importance of different penal procedures and sentencing options in criminal courts.
Overall, however, I realized that the question of metrics was more prominent for web journalists than for judges and prosecutors. This has to do with the profession under consideration – journalism being a for-profit activity in a competitive market – but also with the technology itself: Big Data analytics are much more sophisticated, personalized, and ubiquitous in web newsrooms than excel spreadsheets were in the French context.
Predictive algorithms and the new analytics of criminal justice
But courts are changing: many Big Data initiatives are currently under way in the U.S. criminal justice system.
Of particular interest is the increasing reliance on predictive algorithms, also called “risk-assessment tools,” in criminal courts. Based on a small number of variables about the criminal history (number of offenses, violent offenses, failures to appear in court, history of drug abuse, etc.) and socio-demographic characteristics (age, gender, employment status, residence, etc.) of defendants, these models draw on actuarial techniques developed in the insurance sector to assess an offender’s risk of recidivism, from “low” to “high.” In many U.S. jurisdictions, judges and prosecutors now rely on these predictive algorithms in the various stages of pretrial, sentencing, probation, and parole.
The idea behind the introduction of risk-assessment tools is to rationalize the decision-making process and help judges and prosecutors draw on “smart statistics” and engage in “data-driven” sentencing. Based on my fieldwork in French courts, this seems like an interesting development: the judges and prosecutors I spent time with usually did not have enough time to put together detailed files about the defendants or read the files as thoroughly as they wished. Consequently, most of their decisions were made in a hurry, sometimes with misleading information about the defendants.
Having more data probably would not hurt in these cases, right?
Except that the increased reliance on risk-assessment tools also raises important questions about the fairness of the statistical models, the values and theories of punishment embedded in them, and the unintended consequences they may have on sentencing practices in courts.
For instance, as discussed earlier, the introduction of metrics in web newsrooms and French courts led to many strategies of resistance among journalists, judges, and prosecutors. Will risk-assessment tools come with similar side effects when they are deployed within courts? This is what I am beginning to study in my new project on the reception of analytics and predictive algorithms in U.S. courts.
Like many other offline institutions, courts are becoming increasingly mediated by digital and data-driven technologies. As the different posts in this special issue of Ethnography Matters on “Designing with Machines” show, these processes of mediation provide many exciting opportunities for future ethnographic research on machines, humans, data, and their interaction in the digital age.
This article is part the Co-designing with Machines Edition. Read other articles in this edition.
Like what you’re reading? Ethnography Matters is a volunteer run site with no advertising. We’re happy to keep it that way, but we need your help. We don’t need your donations, we just want you to spread the word. Our reward for running this site is watching people spread our articles. Join us on twitter, tweet about articles you like, share them with your colleagues, or become a contributor. Also join our Slack to have deeper discussions about the readings and/or to connect with others who use applied ethnography; you don’t have to have any training in ethnography to join, we welcome all human-centric people from designers to engineers.
