Example: Amazon Prime same day delivery service

In 2016, analysis published by Bloomberg uncovered racial disparities in eligibility for Amazon’s same day delivery services for Prime customersTo be clear, the same day delivery was free for eligible Amazon Prime customers on sales exceeding $35. Amazon Prime members pay a fixed annual subscription fee, thus the disparity is in the level of service provided for Prime customers who are eligible verses those that are not.

[1] [1] D. Ingold and S. Soper, “Amazon doesn’t consider the race of its customers. Should it?” Bloomberg, 2016. . The study used census data to identify Black and White residents and plot the data points on city maps which simultaneously showed the areas that qualified for the Prime customer same day delivery. The disparities are glaring at a glance. In six major cities, New York, Boston, Atlanta, Chicago, Dallas, and Washington, DC where the service did not have broad coverage, it was mainly Black neighbourhoods that were ineligible. In the latter four cities, Black residents were about half as likely to live in neighbourhoods eligible for Amazon same-day delivery as White residents.

At the time Amazon’s process in determining which ZIP codes to serve was reportedly a cost benefit calculation that did not explicitly take race into account but for those who have seen redlining maps from the 1930’s is hard to not see the resemblance. Redlining was the (now illegal) practice of declining (or raising prices for) financial products to people based on the neighbourhood where they lived. Because neighbourhoods were racially segregated (a legacy that lives on today), public and private institutions were able to systematically exclude minority populations from the housing market and deny loans for house improvements without explicitly taking race into account. Between 1934 and 1962, the Federal Housing Administration distributed $120 billion in loans. Thanks to redlining, 98% of these went to White families.

Amazon is a private enterprise, and it is legally entitled to make decisions about where to offer services based on how profitable it is. Some might argue they have a right to be able to make those decisions. Amazon is not responsible for the injustices that created such racial disparities, but the reality is that such disparities in access to goods and services perpetuate it. If same-day delivery sounds like a luxury, it’s worth considering the context. The cities affected have a long histories of racial segregation and economic inequality resulting from systemic racism, now deemed illegal. They are neighbourhoods which to this day are underserved by brick and mortar retailers, where residents are forced to travel further and pay more for household essentials. Now we are in the midst of a pandemic, where once delivery of household goods used to be a luxury, with so many forced to quarantine, suddenly it’s become far more of a necessity. What we consider to be a necessity changes over time, it depends on where one lives, one’s circumstances and more. Finally, consider the scale of Amazon’s operations, in 2016 one third of retail e-commerce spending in the US was with Amazon (that number has since risen to almost 50%).

Principles of Justice as Fairness

1. Liberty principle: Each person has the same indefeasible claim to a fully adequate scheme of equal basic liberties, which is compatible with the same scheme of liberties for all;

2. Equality principle: Social and economic inequalities are to satisfy two conditions:

   1. Fair equality of opportunity: The offices and positions to which they are attached are open to all, under conditions of fair equality of opportunity;

   2. Difference (maximin) principle They must be of the greatest benefit to the least-advantaged members of society.

The principles of Justice as Fairness are ordered by priority so that fulfilment of the liberty principle takes precedence over the equality principles and fair equality of opportunity takes precedence over the difference principle.

The first principle grants basic rights and liberties to all citizens which are prioritised above all else and cannot be traded for other societal benefits. It’s worth spending a moment thinking about what those rights and liberties look like. They are the the basic needs that are important for people to be free, to have choices and the means to pursue their aspirations. Today many of what Rawls considered to be basic rights and liberties are allocated algorithmically; education, employment, housing, healthcare, consistent treatment under the law to name a few.

The second principle requires positions to be allocated meritocratically, with all similarly talented (with respect to the skills and competencies required for the position) individuals having the same chance of attaining such positions i.e. that allocation of such positions should be independent of social class or background. We will return to the concept of equality of opportunity in chapter 3 when discussing Group Fairness.

The third principle acts to prevent redistribution of social and economic currency from the rich to the poor by requiring that inequalities are of maximal benefit to the least advantaged in a society, also described as the maximin principle. In this principle, Rawls does not take the simplistic view that inequality and fairness are mutually exclusive but rather concisely articulates when the existence of inequality becomes unfair. In a sense Rawls opposes utilitarian thinking (that everyone matters equally) in prioritising the least advantaged. We shall return to maximin principle when we look at the use of inequality indices to measure algorithmic unfairness in a later chapter.

Disparate Treatment

Disparate treatment effectively prohibits rational prejudice (backed by data showing the protected feature to be correlated) as well as denial of opportunities based on protected characteristics. For an algorithm, it effectively prevents the use of protected characteristics as inputs. It’s noteworthy that in the case of disparate treatment, the actual impact of using the protected features on the outcome is irrelevant; so even if a company could show that the target variable produced by their model had zero correlation with the protected characteristic, the company would still be liable for disparate treatment. This fact is somewhat bizarre given that not using the protected feature in the algorithm provides no guarantee that the algorithm is not biased in relation to it. Indeed an organisation could very well use their data to predict the protected characteristic.

In an effort to avoid disparate treatment liability, many organisations do not even collect data relating to protected characteristics, leaving them unable to accurately measure, let alone address, bias in their algorithms, even if they might want toIn fact, I met a data scientist at a conference, who was working for a financial institution, that said her team was trying to predict sensitive features such as race and gender in order to measure bias in their algorithms.

. In summary, disparate treatment as applied today does not resolve the problem of unconscious discrimination against disadvantaged classes through their use of machine learning algorithms. Further it acts as a deterrent to ethically minded companies that might want to measure the biases in their algorithms.

What about the case where the employer implements an algorithm, finds out that it has a disparate impact, and uses it anyway? Doesn’t that become disparate treatment? No it doesn’t and in fact, somewhat surprisingly, deciding not to apply it upon noting the disparate impact could result in a disparate treatment claim in the opposite direction[5] [5] “Ricci v. DeStefano, 557 U.S. 557.” 2009. . We’ll return to this later. Okay, so what about disparate impact?

Disparate Impact

In order to establish a violation, it is not enough to simply show that there is a disparate impact, but it must also be shown either that there is no business justification for it, or if there is, that the employer refuses to use another, less discriminatory, means of achieving the desired result. So how much of an impact is enough to warrant a disparate impact claim? There are no rules here only guidelines. The Uniform Guidelines on Employment selection procedures from the Equal Employment Opportunity Commission (EEOC) provides a guideline that if the selection rate from one protected group is less than four fifths of that from another, it will generally be regarded as evidence of adverse impact, though it also states that the threshold would depend on the circumstances.

Assuming the disparate impact is demonstrated, the issue becomes proving business justification. The requirement for business justification has softened in favour of the employer over the years; treated as “business necessity”[6] [6] “Griggs v. Duke Power Co., 401 U.S. 424.” 1971. earlier on and later interpreted as “business justification”[7] [7] “Wards Cove Packing Co. v. Atonio, 490 U.S. 642.” 1989. . Today, it’s generally accepted that business justification lies somewhere between the extremes of “job-relatedness” and “business necessity”. As a concrete example of disparate impact and taking the extreme of job-relatedness - the EEOC along with several federal courts have determined that discrimination on the sole basis of a criminal record to be a violation under disparate impact unless the particular conviction is related to the role, because Non-White applicants are more likely to have a criminal conviction.

For a machine learning algorithm, business justification boils down to the question of job-relatedness of the target variable. If the target variable is improperly chosen, a disparate impact violation can be established. In practice however the courts will accept most plausible explanations of job-relatedness since not accepting it would set a precedent that it is determined discriminatory. Assuming the target variable to be proven job-related then, there is no requirement to validate the model’s ability to predict said trait, only a guideline which sets a low bar (a statistical significance test showing that the target variable correlates with the trait) and which the court is free to ignore.

Assuming business justification is proven by the employer, the final burden then falls on the plaintiff to show that the employer refused to use a less discriminatory “alternative employment practice”. If the less discriminatory alternative would incur additional cost (as is likely) would this be considered refusing? Likely not.

While on the surface, disparate impact might seem like a solution, the current framework of a weak business justification (in terms of a plausible target variable) and the employer refusing an alternative employment practice with no requirement to validate the model offers little resolve. Clearly there is need for reform.

Anti-classification versus Anti-subordination

Just as the meaning of fairness is subjective so is the interpretation of anti-discrimination laws. At one extreme, anti-classification holds the weaker interpretation, that the law is intended to prevent classification of people based on protected characteristics. At the other extreme, anti-subordination defines the stronger stance, that anti-discrimination laws exist to prevent social hierarchies, class or caste systems based on protected features and, that it should actively work to eliminate them where they exist. An important ideological difference between the two schools of thought is in the application of positive discrimination policies. Under anti-subordination principles, one might advocate for affirmative action as a means to bridge gaps in access to employment, housing, education and other such pursuits, that are a direct result of historical systemic discrimination against particular groups. A strict interpretation of the anti-classification principle would prohibit such actions. Both anti-classification and anti-subordination ideologies have been argued and upheld in landmark cases.

In 2003, the Supreme Court held that a student admissions process that favours “under-represented minority groups” does not violate the Fourteenth Amendment[8] [8] “Grutter v. Bollinger, 539 U.S. 306.” 2003. , provided it evaluated applicants holistically at an individual level. The same year, the New Haven Fire Department administered a two part test in order to fill 15 openings. Examinations were governed in part by the City of New Haven. Under the city charter, civil service positions must be filled by one of the top three scoring individuals. 118 (White, Black and Hispanic) fire fighters took the exams. Of the resulting 19 candidates who scored highest on the tests and could the considered for the positions, none were Black. After heated public debate and under threat of legal action either way, the city threw out the test results. This action was later determined to be a disparate treatment violation. In 2009, the court ruled that disparate treatment could not be used to avoid disparate impact without sufficient evidence of liability of the latter[5]. This landmark case was the first example of conflict between the two doctrines of disparate impact and disparate treatment or anti-classification and anti-subordination.

Disparate treatment seems to align well with anti-classification principles, seeking to prevent intentional discrimination based on protected characteristics. In the case of disparate impact, things are less clear. Is it a secondary ‘line of defence’ designed to weed out well masked intentional discrimination? Or is its intention to address inequity that exists as a direct result of historical injustice? One can draw parallels here with the ‘business necessity’ versus ‘business justification’ requirements discussed earlier.

Question:

Which distribution gives us the most relevant understanding of the association between \(A\) and \(B\), the marginal (i.e. unconditional) \(\mathbb{P}(A,B)\) or conditional distribution \(\mathbb{P}(A,B|C)\)? To show that causal relationships matter, we consider two different examples.

Example a) Pack of Cards (Colliding Variable)

Suppose the population is a pack of cards. It so happens that baby Milen has been messing about with the cards and made some dirty in the process. Let’s summarise our variables,

• \(A\) tells us the character of the card, either plain (\(A=1\)) or royal (King, Queen, Jack; \(A=0\)).

• \(B\) tells us the colour of the card, either black (\(B=1\)) or red (\(B=0\)).

• \(C\) tells us if the card is dirty (\(C=1\)) or clean (\(C=0\)).

In this case, the aggregated data showing \(\mathbb{P}(A,B)\) is relevant since the cleanliness of the cards \(C\) has no bearing on the association between the character \(A\) and colour \(B\) of the cards.

Example b) Treatment Effect on Mortality Rate (Confounding Variable)

Next, suppose that the data relates to the results of medical trials for a drug on a potentially lethal illness. This time,

• \(A\) tells us if the subject was treated (\(A=1\)) or not (\(A=0\)).

• \(B\) tells us if the subject died (\(B=1\)) or recovered (\(B=0\)).

• \(C\) tells us the gender of the subject, either male (\(C=1\)) or female (\(C=0\)).

In this case the disaggregated data shows the more relevant association, \(\mathbb{P}(A,B|C)\). From it, we can see that female patients are more likely to die than males overall; 56 and 60% versus 38 and 43%, depending on if they were treated or not. In both cases we see that treatment with the drug \(A\) reduces the mortality rate for both male and female participants, and the effect is obscured by aggregating the data over gender \(C\).

Back to Causality

The key difference between these examples is the causal relationship between the variables rather than the statistical structure of the data. In the first example with the playing cards, the variable \(C\) is a colliding variable, in the second example looking at patient mortality, it is a confounding variable. Figure 1.4 a) and b) show the causal relationships between the variables in the two cases.

The causal diagram in Figure 1.4 a) shows the variables \(A\), \(B\) and \(C\) for the first example. The arrows exist both from card character and colour to cleanliness because apparently, baby Milen had a preference for royal cards over plain and red cards over black. Conditioning on a collider \(C\) generates an association (e.g. correlation) between \(A\) and \(B\), even if they are unconditionally independent. This common effect is often observed as selection or representation bias. Representation bias can induce correlation between variables, even where there is none. For decision systems, this can lead to feedback loops that increase the extremity of the representation bias in future data. We’ll come back to this in chapter 2, when we talk about common causes of bias.

The causal diagram in Figure 1.4 b) shows the variables \(A\), \(B\) and \(C\) for the second example. The arrows exist from \(gender\) to treatment because men were less likely to be treated, and from gender to death because men were also less likely to die. The arrow from \(A\) to \(B\) represents the effect of treatment on mortality which is observable only by conditioning on gender. Note that there are two sources of association in opposite directions between variables \(A\) and \(B\) (treatment and death); a positive association, because men were less likely to be treated; and a negative association, because male patients are less likely to die. The two effects cancel each other out when the data is aggregated.

We see through the discussion of these two examples, that statistical reasoning is not sufficient to be able to determine which of the distributions (marginal or conditional) are relevant. Note that the above conclusions in relation to colliding and confounding variables does not generalize to complex time varying problems.

Before moving on from causality, we return to the example we discussed at the very start of this section. According to our analysis of the Berkeley admissions data, we concluded that the disaggregated data contained the salient information explaining the disparity in acceptance rates for male and female applicants. The problem is, we have only shown that application rates to be one of many possible causes of the differing acceptance rates (we cannot see outside of our data). In addition, we have not proven gender discrimination, not to be the cause. What we have evidenced, is the existence of disparities in both acceptance rates and application rates across sex. One problem is that gender discrimination is not a measurable thing in itself. It’s complicated. It is made up of many components, most of which are not contained in the data. Beliefs, personal preferences, behaviours, actions, and more. A valid question we cannot answer is, why do the application rates differ by sex? How do we know that this is itself, is not a result of gender discrimination. Perhaps some departments are less welcoming of women than others or, perhaps some are just much more welcoming of men than women? So how would we know if gender discrimination is at play here? We need to ask the right questions to collect the right data.

Example c) Treatment Effect on Mortality Rate (Prognostic Variable)

Suppose that in the study of the efficacy of the treatment (in Example 2 above), we remedy the selection bias so that male and female patients are equally likely to be treated. We remove the causal relationship between variables \(A\) and \(C\) (treatment and gender). In this case, the variable \(C\) becomes prognostic rather than confounding. See Figure 1.4 c). In this case the decision as to which distributions (marginal or conditional) are most relevant would depend only on the target population in question. In the absence of the confounding variable in our study one might reasonably expect the marginal measure of association to be bounded by the partial measures of association. Such intuition is correct only if the measure of association is collapsible (that is, it can be expressed as the weighted average of the partial measures), not otherwise. Some examples of collapsible measures of association are the risk ratio and risk difference. The odds ratio however is not collapsible. If you don’t know what these are, don’t worry, we’ll return to them in chapter 3.

Automation Bias

The tendency for people to favour decisions made by automated systems despite contradictory information from non-automated sources, or automation bias, is a growing problem as we integrate more and more machines in our decision making processes especially in infrastructure - healthcare, transportation, communication, power plants and more.

It is important to be clear that in general, machine learning systems are not objective. Data is produced by a necessarily subjective set of decisions (how and who to sample, how to group events or characteristics, which features to collect). Modelling also involves making choices about how to process the data, what class of model to use and perhaps most importantly how success is determined. Finally, even if our model is calibrated to the data well, it says nothing about the distribution of errors across the population. The consistency of algorithms in decision making compared to humans (who individually make decisions on a case by case basis) is often described as a benefitOne must not confuse consistency with objectivity. For algorithms, consistency also means consistently making the same errors.

, but it’s their very consistency that makes them dangerous - capable of discriminating systematically and at scale.

The Ethics of Classification

The appeal of classification is clear. It creates a sense of order and understanding. It enables us to formulate problems neatly and solve them. An email is spam or it’s not; an x-ray shows tuberculosis or it doesn’t; a treatment was effective or it wasn’t. It can make finding things more efficient in a library or online. There are lots of useful applications of classification.

We tend to think of taxonomies as objective categorisations, but often they are not. They are snapshots in time, representative of the culture and biases of the creators. The very act of creating a taxonomy, can give life by existence to some individuals, while erasing others. Classifying people inevitably has the effect of reducing them to labels; labels that can result in people being treated as members of a group, rather than individuals; labels that can linger for much longer than they should (something it’s easy to forget when creating them). The Dewey Decimal System for example, was developed in the late 1800’s and widely adopted in the 1930’s to classify books. Until 2015, it categorised homosexuality as a mental derangement.

Classification of People

From the 1930’s until the second world war, machine classification systems were used by Nazi Germany to process census data in order to identify and locate Jews, determine what property and businesses they owned, find anything of value that could be seized and finally to send them to their deaths in concentration camps. Classification systems have often been entangled with political and social struggle across the world. In Apartheid South Africa, they were been used extensively in many parts of the world to enforce social and racial hierarchies that determined everything from where people could live and work to whom they could marry. In 2019 it was estimated that some half a million Uyghurs (and other minority Muslims) are being held in internment camps in China without charge for the purposes of countering extremism and promoting social integration.

Recent papers on detecting criminality”[14] [14] X. Wu and X. Zhang, “Automated inference on criminality using face images.” 2017.Available: https://arxiv.org/abs/1611.04135 and sexuality[15] [15] Y. Wang and M. Kosinski, “Deep neural networks are more accurate than humans at detecting sexual orientation from facial images,” Journal of Personality and Social Psychology, 2018. and ethnicity[16] [16] C. Wang, Q. Zhang, W. Liu, Y. Liu, and L. Miao, “Facial feature discovery for ethnicity recognition,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2018. from facial images have sparked controversy in the academic community. The latter in particular looks for facial features that identify among others, Chinese Uyghurs. Physiognomy (judging character from the physical features of a persons face) and phrenology (judging a persons level of intelligence from the shape and dimensions of their cranium) have historically been used as pseudo-scientific tools of oppressors, to prove the inferiority races and justify subordination and genocide. it is not without merit then to ask if some technologies should be built at all. Machine gaydar might be a fun application to mess about with friends for some, but in the 70 countries where homosexuality is still illegal (some of which enforce the death penalty) it is something rather different.

Harms of Allocation

An allocative harm happens when a system allocates or withholds an opportunity or resource. Systems that approve or deny credit allocate financial resources; systems that decide who should and should not see adverts for high paying jobs allocate employment opportunities and systems that determine who will make a good tenant allocate housing resources. Harms of allocation happen as a result of discrete decisions at a given point in time, the immediate impact of which can be quantified. This makes it possible to challenge the justice and fairness of specific determinations and outcomes.

Increasingly however, machine learning systems are affecting us, not just through allocation, but are shaping our view of the world and society at large by deciding what we do and don’t see. These harms are far more difficult to quantify.

Harms of Representation

Harms of representation occur when systems enforce the subordination of groups through characterizations that affect the perception of them. In contrast to harms of allocation, harms of representation have long-term effects on attitudes and beliefs. They create identities and labels for humans, societies and their cultures. Harms of representation don’t just affect our perception of each other, they affect how we see ourselves. They are difficult to formalise and in turn difficult to quantify but the effect is real.

Did you figure it out? How long did it take? There is, of course, no reason why the surgeon couldn’t be the boy’s mother. If it took you a while to figure out, or came to a different conclusion, you’re not alone. More than half the people presented with this riddle do, and that includes women. The point of this riddle is to demonstrate the existence of unconscious bias. Representational harms are insidious. They silently fix ideas in peoples subconscious about what people of a particular gender, nationality, faith, race, occupation and more, are like. They draw boundaries between people and affect our perception of world. Below we describe five different harms of representation:

Stereotyping

Stereotyping occurs through excessively generalised portrayals of groups. In 2016, the Oxford English Dictionary was publicly criticised[21] [21] E. O’Toole, “A dictionary entry citing ‘rabid feminist’ doesn’t just reflect prejudice, it reinforces it,” The Guardian, 2016. for employing the phrase “rabid feminist” as a usage example for the word rabid. The dictionary included similarly sexist common usages for other words like shrill, nagging and bossy. But even before this, historical linguists observed that words referring to women undergo pejoration (when the meaning of a word deteriorates over time) far more often than those referring to men[22] [22] D. Shariatmadari, “Eight words that reveal the sexism at the heart of the english language,” The Guardian, 2016. . Consider words like mistress (once simply the female equivalent of master, now used to describe a woman in an illicit relationship with a married man); madam (once simply the female equivalent of sir, now also used to describe a woman who runs a brothel); hussy (once a neutral term for the head of a household, now used to describe an immoral or ill-behaved woman); and governess (female equivalent of governor, later used to describe a woman responsible for the care of children).

Unsurprisingly then, gender stereotyping is known to be a problem in natural language processing systems. In 2016 Bolukbasi et al. showed that word embeddings exhibited familiar gender biases in relation to occupations[23] [23] T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, and A. Kalai, “Man is to computer programmer as woman is to homemaker? Debiasing word embeddings.” 2016.Available: https://arxiv.org/abs/1607.06520 . By performing arithmetic on word vectors, they were able to uncover relationships such as \[\overrightarrow{\textrm{man}} - \overrightarrow{\textrm{woman}} \approx \overrightarrow{\textrm{computer programmer}} - \overrightarrow{\textrm{homemaker}}.\]

In 2017 Caliskan et al. found that Google Translate contained similar gender biases.[24] [24] A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science, vol. 356, pp. 183–186, 2017. In their research they found that “translations to English from many gender-neutral languages such as Finnish, Estonian, Hungarian, Persian, and Turkish led to gender-stereotyped sentences”. So for example when they translated Turkish sentences with genderless pronouns: “O bir doktor. O bir hemişre.”, the resulting English sentences were: “He is a doctor. She is a nurse.” They performed these types of tests for 50 occupations and found that the stereotypical gender association of the word almost perfectly predicted the resulting pronoun in the English translation.

Recognition

Harms of recognition happen when groups of people are in some senses erased by a system through failure to recognise. In her TED Talk, Joy Buolamwini, talks about how as an undergraduate studying computer science she worked on social robots. One of her projects involved creating a robot which could play peek-a-boo, but she found that her robot (which used third party software for facial recognition) could not see her. She was forced to borrow her roommate’s face to complete the project. After her work auditing several popular gender classification packages from IBM, Microsoft and Face++ in the project Gender Shades[25] [25] J. Buolamwini and T. Gerbru, Gender shades: Intersectional accuracy disparities in commercial gender classification, vol. 81. Proceedings of Machine Learning Research, 2018, pp. 1–15. in 2017 and seeing the failure of these technologies on the faces of some of the most recognizable Black women of her time, including Oprah Winfrey, Michelle Obama, and Serena Williams, she was prompted to echo the words of Sojourner Truth in asking “Ain’t I a Woman?”. Harms of recognition are failures in seeing humanity in people.

Denigration

In 2015, much to the horror of many people, it was reported that Google Photos had labelled a photo of a Black couple as Gorillas. It’s hard to find the right words to describe just how offensive an error this is. It demonstrated how a machine, carrying out a seemingly benign task of labelling photos, could deliver an attack on a person’s human dignity.

In 2020, an ethical audit of several large computer vision datasets[26] [26] V. U. Prabhu and A. Birhane, “Large image datasets: A pyrrhic win for computer vision?” 2020.Available: https://arxiv.org/abs/2006.16923 , revealed some disturbing results. TinyImages (a dataset of 79 million 32 x 32 pixel colour photos compiled in 2006, by MIT’s Computer Science and Artificial Intelligence Lab for image recognition tasks) contained racist, misogynistic and demeaning labels with corresponding images. Figure 1.6 shows a subset of the data found in TinyImages.

The problem, unfortunately, does not end here. Many of the datasets used to train and benchmark, not just computer vision but natural language processing tasks, are related. Tiny Images was compiled by searching the internet for images associated with words in WordNet (a machine readable, lexical database, organised by meaning, developed at Princeton), which is where TinyImages inherited its labels from. ImageNet (widely considered to be a turning point in computer vision capabilities) is also based on WordNet and, Cifar-10 and Cifar-100 were derived from TinyImages.

Vision and language datasets are enormous. The time, effort and consideration in collecting the data that forms the foundation of these technologies (compared to that which has gone into advancing the models built on them), is questionable to say the least. Furthermore a dataset can have impact beyond the applications trained on it, because datasets often don’t just die, they evolve. This calls into question the technologies that are in use today, capable of creating persistent representations of our world, and trained on datasets so large they are difficult and expensive to audit.

And there’s plenty of evidence to suggest that this is a problem. For example, in 2013, a study found that Google searches were more likely to return personalised advertisements that were suggestive of arrest records for Black names[27] [27] L. Sweeney, “Discrimination in online ad delivery,” SSRN, 2013. than WhiteSuggestive of an arrest record in the sense that they claim to have arrest records specifically for the name that you searched, regardless of whether they do in reality have them.

This doesn’t just result in allocative harms for people applying for jobs for example, it’s denigrating. Google’s Natural Language API for sentiment analysis is also known to have problems. In 2017, it was assigning negative sentiment to sentences such as “I’m a jew” and “I’m a homosexual” and “I’m black”; neutral sentiment to the phrase “white power” and positive sentiment to the sentences “I’m christian” and “I’m sikh”.

Under-representation

In 2015, the New York Times reported, that “Fewer women run big companies than men named John”, despite this Google’s image search still managed to under-represent women in search results for the word “CEO”. Does this really matter? What difference would an alternate set of search results make? A study the same year found that “people rate search results higher when they are consistent with stereotypes for a career, and shifting the representation of gender in image search results can shift people’s perceptions about real-world distributions.”[28] [28] M. Kay, C. Matuszek, and S. A. Munson, “Unequal representation and gender stereotypes in image search results for occupations,” ACM, 2015. .

Ex-nomination

Ex-nomination occurs through invisible means and affects people’s views of the norms within societies. It tends to happen through mechanisms which amplify the presence of some groups and suppress the presence of others. The cultures, beliefs, politics of ex-nominated groups over time become the default. The most obvious example is the ex-nomination of Whiteness and White culture in western society, which might sound like a bizarre statement - what is White culture? But such is the effect of ex-nomination, you can’t describe it, because it is just the norm and everything else is not. Richard Dyer in his book White examines the reproduction and preservation of whiteness in visual media over five centuries, from the depiction of the crucifixion to modern day film. It’s perhaps should not come as a surprise then, when facial recognition software can’t see black faces; or when gender recognition software fails more often than not for black women; or that a generative model that improves the resolution of images, converted a pixelated picture of Barack Obama, into a high-resolution image of a white man.

The ex-nomination of White culture is evident in our language too, in terminology like whitelist and white lie. If you look up white in dictionary and or thesaurus and you’ll find words like innocent and pure, light, transparent, immaculate, neutral. Doing the same for the word black on the other hand, returns very different associations, dirty, soiled, evil, wicked, black magic, black arts, black mark, black humour, blacklist and black is often used as a prefix in describing disastrous events. A similar assessment can be made for gender with women being under-represented in image data and feminine versions of words more often undergoing pejoration (when the meaning or status of a word deteriorates over time).

Members of ex-nominated groups experience a kind of privilege that it is easy to be unaware of. It is a power that comes from being the norm. They have advantages that are not earned, outside of their financial standing or effort, that the ‘equivalent’ person outside the ex-nominated group would not. Their hair type, skin tone, accent, food preferences and more are catered to by every store, product, service and system and it cost less to access them; they see themselves represented in the media and are more often represented in a positive light; they are not subject to profiling or stereotypes; they are more likely to be treated as individuals rather than as representative of (or as exceptions to) a group; they are more often humanised - more likely to be be given the benefit of the doubt, treated with compassion and kindness and thus recover from mistakes; they are less likely to be suspected of crimes; more likely to be trusted financially; they have greater access to opportunities, resources and power and are able to climb financial, social and professional ladders faster. The advantages enjoyed by ex-nominated groups accumulate over time and compound over generations.

Predictive Policing

In the United States, predictive policing has been implemented by police departments in several states including California, Washington, South Carolina, Alabama, Arizona, Tennessee, New York and Illinois. Such algorithms use data on the time, location and nature of past crimes, in order to determine how and where to patrol and thus improve the efficiency with which policing resources are allocated. A major flaw with these algorithms pertains to the data used to train them. It is not of where crimes occurred, but rather where there have been previous arrests. A proxy target variable (arrests) is used in place of the desired target variable (crime). Racial disparities in policing in the US is a well publicised problem. Figure 2.2 demonstrates this disparity for policing of drug related crimes. In 2015, an analysis by The Hamilton Project found that at the state level, Blacks were 6.5 times as Whites to be incarcerated for drug-related crimes[29] [29] “Rates of drug use and sales, by race; rates of drug related criminal justice measures, by race.” The Hamilton Project, 2015. despite drug related crime being more prevalent among Whites. Taking actions based on predictions from an algorithm trained on arrest data will likely amplify existing disparities between under and over-policed neighbourhoods which correlate with race.

Car insurance

As a comparative example, let’s consider car insurance. It is well publicised that car insurance companies discriminate against young male drivers (despite age and gender being legally protected characteristics in the countries where these insurance companies operate) since statistically, they are at higher risk of being involved in accidents. Insurance companies act on risk predictions by determining the price of insurance at an individual level - the higher the risk, the more expensive the cost of insurance. What is the feedback effect of this on the data? Of course young men are disadvantaged by having to pay more, but one can see how this pricing structure acts as an incentive to drive safely. It is in the drivers interest to avoid having an accident that would result in an increase in their car insurance premiums. For a high risk driver in particular, an accident could potentially make it prohibitively expensive for them to drive. The feedback effect on the data would be to reduce the disparity in incidents of road traffic accidents among high and low risk individuals.

Along with the difference in the direction of the feedback effects in the examples given above, there is another important distinction to be made in terms of the magnitude of the feedback effect. This is related to how much control the institution making decisions based on the predictions, has over the data. In the predictive policing example the data is entirely controlled by the police department. They decide where to police and who to arrest, ultimately determining the places and people that do (and don’t) end up in the data. They produce the training data, in its entirety, as a result of their actions. Consequently, we would expect the feedback effect of acting on predictions based on the data to be strong and capable of dramatically shifting the distribution of data generated over time. Insurance companies by comparison, have far less influence over the data (consisting individuals involved in road traffic accidents). Though they can arguably encourage certain driving behaviours through pricing, they do not ultimately determine who is and who is not involved in a car accident. As such, feedback effects of risk-related pricing in car insurance are likely to be less strong in comparison.

Prioritisation

Products which are of systemic importance to the sociopolitical landscape should have sufficient and appropriate resources (relative to those of the risk generating activities) to manage and mitigate their ethical risk. For applications that carry high risk of harm, risk functions should act as gatekeepers for model deployment and use.

Model governance standards

Though relatively new terminology in machine learning circles, the concept of model governance has existed for decades. For large financial institutions (which depend on vast numbers of proprietary models), operating and maintaining a model governance framework is a central part of model risk management and a regulatory requirement. The regulatory landscape of the financial sector is considerably more mature than that of other industries and the frameworks used to handle the associated risks have been developed and refined over time. It is therefore instructive to look at how such institutions manage their model risk and consider how these might be applied to sociotechnical systems.

So what does responsible and ethical machine learning development and deployment look like? In reality there is no one size fits all answer. As we’ve noted before, sociotechnical systems are context dependent. The answer can depend on a whole multitude of factors.

• Domain: Different domains will have different legal and ethical concerns for example employment versus say social media.

• The number and complexity of the models being used by the business: A large organisation that uses or tests hundreds of models and composes them to make decisions and create new products (such as Microsoft) would benefit greatly from infrastructure and methodologies for measuring the materiality of the associated risks that would enable prioritisation of work related to mitigating them. In contrast, for a business based on a single model that automates a specific task (such as tagging images), this would be less of a concern.

• Cost of errors: Where the stakes are high, for example self driving cars, pre-deployment testing will need to be extensive and prescribed in order to reduce the probability of making mistakes. Well defined and mandatory processes will play an important role - checklists, contingency planning, detailed logging for postmortems and more. For these types of applications we would want authority over model use to be distributed to risk functions which determine when the product is approved for deployment and have the power to decommission them. For a wake word detector (think "Hey Siri", "Okay Google" and "Alexa") a lower standard would be accepted by most.

Given this, how does one approach the problem of responsible development? Step zero is to create a set of model governance standards, the purpose of which is to clearly define and communicate what responsible model development and deployment looks like for your specific application, use case, domain, business, principles and values.

What are the kinds of questions we might want our model governance standards to answer?

• Why is the work important? What kinds of events or uses of your models are you trying to avoid (or are outside of the organisation’s risk appetite)? What legislation is the company subject to? What are the consequences of failures? What are the values of the company that you want to protect?

• Who is responsible? What are the roles that must be fulfilled to deploy monitor and manage the risks. Who are the stakeholders or model owners and what is their remit? Who is accountable?

• What are model owners responsible for? What technology is covered by the standard. What kind of expertise are required to be able to report, understand and manage the risks? What are the questions each stakeholder must answer? What are the responsibilities of those experts at the various stages of the model development and deployment life cycle? What authority do they have in relation to determining if the model is fit for deployment? Who decides what?

• How do you manage the risk? What are the rules, processes and requirements that ensure the companies values are maintained, people are treated fairly, the legal requirements are fulfilled and risks are appropriately managed? How do the stakeholders work together? For example some roles might need to be independent while others work alongside one another. What are the requirements around training data (documentation, review, storage, privacy, consent and such)? What are the requirements around modelling (documentation, testing, monitoring and such)? What are the processes around proposing, reviewing, testing, deploying, monitoring model related risks? For example, frequency of risk reviews, forums for discussion and monitoring. What are the processes and requirements in place for (specific foreseeable types of) failures? Are there stakeholder specific templates or check-lists that ensure particular questions get answered at specific points in the model development and deployment life cycle?

The list of questions above is by no means exhaustive but a good starting point. Creating a set of model governance standards is about planning. Machine learning systems can be complicated and have many points of failure: problem formulation, data collection, data processing, modelling, implementation, interpretation. The only way to reduce the risk of failures is to be organised, deliberate and plan for them. Creating a set of standards does exactly that. Where the systems we build have real world consequences, the preparation, planning and process around development, review, analysis, deployment and monitoring of them should reflect that. Ensuring that the right questions get asked at the right time, knowing who is responsible for answering them and being prepared to address problems is a core part of developing and deploying models ethically.

Finally, we note that the benefits of having excellent model governance standards with well defined goals, processes, roles and responsibilities won’t be realised if in practice they are not followed. In large organisations, consistency can be a challenge. The role of internal audit is to provide objective feedback on the risks, systems, processes and compliance at an executive level. From a model governance perspective the role of auditors is to ensure that there are good processes in place and that the processes are being followed. Internal audit’s role is independent of the business up to the executive level. All functions within the business are required to cooperate with internal auditors and provide unfettered access to information requested. Internal audit does not contribute to the improvement of or compliance to processes directly. Their role is to , assess and report back to senior leadership. In a risk management context, internal audit are considered to be the third line of defence. We shall come to the first and second lines shortly.

Risk Assessment

In order to manage risk it must be identified. Any algorithm, no matter how simple, carries the risk of implementation errors or bugs and thus should at the very least be subject to unit testing and independent code review before being deployed. For organisations with more complicated risk profiles, an important component of managing risk is having a system to measure and track it. Having a way to compare risk level across products and or product classes, even if comparisons are coarse, enables some degree of risk appropriate prioritisation and resource allocation in managing them. Risk can be estimated in many different ways and exactly how it is measured will depend on the details of the application. Broadly speaking it should consider both the severity of the event and likelihood. What’s important is not the exact value but rather the ability to compare risks across products, applications or indeed any other lines along which a business is organised. Metrics that capture things like the scale on which the model is being used, predictive performance, training data quality/representativeness, model complexity, potential for harm and more could potentially be used to coarsely judge the risk posed by different applications. Model governance standards can define risk bands or metrics if they are application specific enough.

Deployment Bias

As part of problem formulation one should examine the machine learning cycle in the context of the biases in the data and consider the nature (direction and strength) of the feedback of resulting actions on future data. It’s important to consider other ways in which the model might be used (other than that intended) and understand the feedback cycle in those cases. How the model might be misused/misinterpreted? Are there ways in which it should not be used? Documenting these types of considerations is an essential step in preventing deployment bias; that is, systematic errors resulting through inappropriate model use or misinterpretation of model results. As creators of technologies which affect society at large, documenting our work might be interpreted as a civic duty. We consider documentation to be an essential part of a dataset and model without which it is incomplete and potentially harmful. As such we classify lack of documentation as a model issue.

Repurposing data of models is a risky thing to do and is often the source of bias in models. A good example of this was uncovered by researchers from Berkeley in 2019. They discovered racial bias in an algorithm used to make important health-care determinations for millions of Americans [39] [39] Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations,” Science, vol. 366, pp. 447–453, Oct. 2019, doi: 10.1126/science.aax2342. . The algorithm was being used to identify patients that would benefit from high-risk care management programs, which improve patient outcomes and reduce healthcare costs for patients with complex healthcare needs. The researchers found that Black patients who had the same risk scores as White patients were far less healthier and thus less likely to be selected for the programs. The bias was the result of data documenting healthcare costs being used to predict healthcare needs.

A thorough examination of ethical issues demands consideration of a diversity of voices, which is well known to be lacking in technology. This is the stage at which it is important to consider who is affected by the technology, consult with them and ensure their views are incorporated in the understanding of the problem and design of a potential solution. Who are the human experts? People who would have valuable insight and opinions on the potential impact of the model you plan on building? Who does the model advantage and who does it disadvantage? Want to use machine learning to help manage diabetes? What are the interests of the health insurance company funding the development? Have you consulted with diabetics in addition to specialist physicians? What are their concerns? What is the problem from the different perspectives? Would a model be able to help or are there simpler solutions?

Independent Model Validation

In any system that is vulnerable to costly errors, unit testing and pre-deployment independent review is a well established method of preventing costly foreseeable failures. Whether it’s a completely new solution built from scratch or a modification to an existing solution that’s being deployed, an independent review process is an important element of responsible model development. Below we describe the responsibilities of two separate roles, the model developers and the model validators.

The model developers role is to translate the business problem into a tractable machine learning problem and create a solution. They will work with the business and receive input from other necessary domain experts relevant to the application to develop a possible solution. This will include tasks such as acquiring and interpreting data that is relevant for the problem, determining a target variable, model objectives, performance measures, fairness measures and more. In terms of preventing failures, model developers are considered the first line of defence. The responsibility of developing a model responsibly lies, in the first instance, with them. The model developers should aim to create a model they believe to be production ready and more specifically, fulfils the requirements specified in the model governance standards.

As part of the pre-deployment process, the model should be reviewed. Model validators will have a similar skill set to model developers but their goal is different to that of the model developers. Where the developers primary objective is to create a solution to the business problem that meets a standard which will be approved by model owners, the role of a model validator is to critique that solution and expose problems with it - the more the better. Their role is to adversarially challenge the solution. They might challenge performance claims (error, bias, fairness) by changing the data or metrics, or demonstrate problems with the model by comparing with an alternative solution. The goal is to expose model weaknesses and demonstrate the limits of its validity in testing and documentation. The model validator might devise mitigation strategies for identified risks. Such strategies might include setting model usage limits (that might trigger a re-review for example) or additional monitoring requirements. They might for example identify additional cases when human review might be required or reject the proposed solution entirely if the problems with the model are great enough. The role of the reviewer could be thought of as something akin to a hacker but with the advantage of having the keys in the form of model documentation (provided by the developers). The model reviewer in pre-deployment can act as a gatekeeper.

Note that in our terminology, the model is simply a mapping. It need not be learned by calibration to historic data. Any algorithm where the decision being made is important enough should be treated as such and proper precautions should be taken. For an algorithm which will be used in production, no matter how simple, this should mean being subject to code review and unit testing that demonstrates its validity in some well chosen cases. A good example of where this would have been valuable came up in December 2020 when a bug in an algorithm, meant that Stanford Hospital Residents were not correctly prioritised for the COVID-19 vaccine, despite working with COVID-19 patients daily. The algorithm did not apparently account for the fact that Resident doctors had a blank ‘location’ field in the data. We might never know the details of how it was implemented and tested but it hard to imagine such a bungle passed any decent unit test.

The model review process acts as the second line of defence. To be effective, the model reviewer’s role must be independent of the model developer’s to some extent. What does independence mean? We mentioned the distinct goals of their roles and this is important. The validator should not drive the development of a solution approach or model but instead focus on critique. In reality, it’s easy to see that the iterative nature of model development might mean that amendments addressing criticisms of the solution may get rolled into it’s development at multiple stages, blurring the lines between critique and collaboration. From an efficiency perspective, it might make sense for the solution to be reviewed at several critical stages of the development process making the overall process indeed more collaborative. If there’s a problem with the data that was missed, ideally the developer would want to fix it before going on to build and train a model on it. One of the challenges then is how to preserve independence between the roles, and ensure that the value of having adversarial criticism in preventing failures, is not lost in collaboration. How best to preserve independence will depend on the specifics and is something that should be determined within the model governance standards. In a bank, the model developers and validators are required (by the regulator) to serve under different business functions (the trading desk versus risk management). They have different reporting lines up to executive level, and work in physically separate locations.

Monitoring

Post-deployment monitoring is an important part of responsible model development and deployment. Analysis should not stop once the model is deployed. Decisions on what to monitor and necessary feedback mechanisms should be determined during development. It’s important to understand if the model is performing in line with expectations (based on pre-deployment testing and analysis). Is the data coming out of the model more or less biased than the data going in? Distributional shifts should be of particular concern where the actions taken based on predictions have a strong impact on the composition of future data.

Domain Expertise

In section 1.4 we spoke of the importance of domain knowledge in interpreting causal relationships in data. Consulting domain experts at the problem formulation stage can yield considerable ethical risk reducing benefits. Incorporating more diverse perspectives on a problem will surely result in a better design that will benefit a broader cross-section of society. Given that models are simplified representations of real world systems and we know that they will make errors, responsible development should build in processes for anticipating and dealing with such cases and, where appropriate, deferring to the judgement of a human expert.

Historical Bias

Historical bias arises as a result of differences between accepted societal values and cultural norms and those captured by data. These need not be a result of errors in the data. Even if data perfectly represents some world state, it can still capture a reality which society deems unfair. Training a model on such data will naturally lead to similarly unfair predictions. Historical bias can manifest itself in data in numerous ways, through unfair outcomes recorded in the data, differing data quality across groups and under or over-representation of groups to name just a few. Take medical data where racial and gender disparities in diagnosis and treatment are well publicised as the health gap. There is a growing body of research across the US and Europe that exposes systematic under-treatment and misdiagnosis of pain in women ([44] [44] K. L. Calderone, “The influence of gender on the frequency of pain and sedative medication administered to postoperative patients,” Sex Roles, vol. 23, pp. 713–725, 1990, doi: https://doi.org/10.1007/BF00289259. , [45] [45] E. H. C. MD et al., “Gender disparity in analgesic treatment of emergency department patients with acute abdominal pain,” Academic Emergency Medicine, vol. 15, pp. 414–418, May 2008, doi: https://doi.org/10.1111/j.1553-2712.2008.00100.x. , [46] [46] D. E. Hoffmann and A. J. Tarzian, “The girl who cried pain: A bias against women in the treatment of pain,” SSRN, 2001, doi: http://dx.doi.org/10.2139/ssrn.383803. ) and Black patients (despite prescription drug abuse being more prevalent among White Americans), [47] [47] K. M. Hoffman, S. Trawalter, J. R. Axt, and M. N. Oliver, “Racial bias in pain assessment and treatment recommendations, and false beliefs about biological differences between blacks and whites,” Proceedings of the National Academy of Sciences, vol. 113, no. 16, pp. 4296–4301, 2016, doi: 10.1073/pnas.1516047113. .

Measurement Bias

Measurement bias refers to non-random noise in measurements across groups. This can occur if for example, there are geographic disparities in services provided by an institution or the quantity and quality of the measuring instruments that mean the accuracy and completeness of records vary by location (and other highly correlated variables like race). In some cases institutions can systematically fail to produce accurate and timely records for certain groups. For example, in medical data, where more frequent misdiagnosis of rare diseases for women leads to a longer lag before accurate diagnosis. In particular, 12 compared to 20 months for Crohn’s disease (despite the disease being more prevalent among women) and 16 compared to 4 years for Ehlers-Danlos syndrome[48] [48] “The voice of 12,000 patients: Experiences and expectations of rare disease patients on diagnosis and care in europe.” 2009. . Systematic delays in diagnosis for protected groups mean that for any given snapshot in time, the medical records for more frequently misdiagnosed groups are less accurate.

Another way in which measurement bias can manifest is if the measurement process varies across groups, for example where the level of scrutiny varies across groups. Predictive policing discussed earlier provides an example of this where there are existing disparities in the level of policing across neighbourhoods. But in practice any process (algorithmic or otherwise) which seeks to identify a behaviour or property (good or bad), but where disproportionate attention is allocated to some subgroup will result in disproportionately more instances of that behaviour or property being observed among members of that group. The result is induced correlation in the data, even in cases where there may in reality be none. One must be careful of making the assumption that where no observation was made the behaviour or property did not exist. The result can be a cycle that continually amplifies the association. Since data often measures and records features which are in fact noisy proxies for the true variables of interest, measurement bias includes cases where use of proxies leads to systematic errors.

Representation bias

Representation bias occurs as a result of biased sampling from the target population. It can be observed as differences in the prevalence of groups when comparing the target population and the sample data. Under-represented classes are exposed to higher error rates; a problem which arises as a result of ‘low support’, that is a smaller pool of data points to train the model on. Looked at from the perspective of the majority class which dominates the aggregate error, the algorithm is naturally incentivised to focus learning characteristics of majority classes.

One of the drivers behind big data initiatives is the plummeting cost of collection and storage data. Companies and institutions are able to train models that better target individuals, reducing costs and boosting profits. However, data collection methods often fail to adequately capture historically disadvantaged classes of people that are less engaged in data generating ecosystems. A good example of this, given by Barocas & Selbst[4] is that of the phone app Street Bump, which was developed by the City of Boston to reduce the cost and time taken to find (and consequently repair) pot holes. The app uses data generated by the accelerometers and GPS of Boston residents’ smart phones as they drive. Once a pothole is located it is automatically added to the city’s system to schedule a repair. One can see easily see how this method of data collection might fail to adequately capture data from poorer neighbourhoods, where car and smart phone ownership are less prevalent; neighbourhoods which probably correlate with race and are already likely to suffer from lack of investment.

In the extreme case of under-representation, there is no support, that is to say, no data points to train on at all. This can be a problem when say studies of symptoms or clinical trials for drugs have no representation for certain groups among which symptoms or drug effectiveness may well vary. A good example of this is diabetes, the impact of the disease and effectiveness of drugs for which have historically most often been measured on samples with few to no hispanic individuals in datasets at all.

Low support

Low support may lead to undesirably high errors for some groups even in the absence of representation bias, since minority classes naturally have fewer data points to train on. This is a particular problem for individuals belonging to multiple disadvantaged classes, for example Black women, which are often overlooked when studies seek to meet fairness metric targets.

Documentation

Documentation of datasets is an essential step in avoiding data misuse or misinterpretation of variables or relationships in the data due to lack of domain knowledge. Documentation should evidence that model governance standards were met. Summaries that explain the provenance of the data (who collected the data, for what purpose, what population was sampled from and how, limitations of the data, clear explanation of the target variables (including consideration of use cases for which it would not appropriate for), breakdown of the demographics and the variables by sensitive features pointing out classes that are not well represented. Documentation that is standardised through use of a template could ensure some level of consistency.

Aggregation Bias

Aggregation bias occurs when heterogeneous groups are modelled as homogeneous. In this case we are assuming the same model is appropriate for all groups when in fact it is not, it is a failure to recognise differences in type. There are many examples of this is medical models for diagnosis or that measure the effectiveness of treatments. Historically much of medical research is based on data that over-represents White men. Diseases that manifest differently across gender or race are more often misdiagnosed or less effectively treated. Take autism spectrum disorder (AUD) for example, in 2016 research estimated that autism is four times more prevalent in boys than girls. However more recent research has suggested that a contributing factor maybe that autism more often goes undiagnosed in women because studies of the disorder have historically been focused on male subjects. The most notable difference between autistic males and females is how the social (rather than behavioural) symptoms manifest. It is thought that women, especially at the high-functioning end of the spectrum, are more likely to camouflage their symptoms.

Target Variable Selection

One of the challenges in developing a machine learning is the translation of the underlying problem by defining a target variable - something which can be observed, measured and recorded or obtained easily (from a third party vendor), and that accurately reflects the variable we wish to predict. While there are relatively uncontentious examples that machine learning solutions lend themselves well to (spam detection for emails or on-base or slugging percentage for major league baseball player valuation) for many problems the translation is non-trivial and subjective. Take a job applicant filter for example, that aims to find the most promising applicants. The attributes that one might consider to be held by an applicant that make them promising are likely to be described differently by different people even if they work in the same team. Even if two individuals agree on the attributes, it’s likely they’ll weigh the attributes differently based on their experiences and preferences. Different choices will result in the different kinds of biases infiltrating our algorithm.

Often when data on the variable we want to affect doesn’t really exist we use a proxy. In 2018, Amazon was forced to scrap a recruitment tool it spent four years developing. The algorithm rated resumes of potential employees and was trained on 10 years worth of resumes submitted by job applicants. The exact details of the algorithm were not publicised but based on the training data, it is likely that the proxy variable they used was some measure of how the candidates had performed in the hiring process previously. Thus predicting who they would have hired in the past (given their historical and existing biases) rather than who was the best applicant. The problem with such systems is that often they end up being how we define the thing that it’s actually a proxy for.

Issues can also arise when defining a heterogeneous target variable, where a range of different events are coarsely grouped into a single outcome. This is a form of aggregation bias where the issue specifically concerns the target. This might happen for example where the event of particular interest is rare and by including more events in the target the predictive accuracy of the model increases as it has more data to learn from. D’Alessandro et. al[40] provide a useful example in predictive policing where the model developer is initially interested in predicting violent crime but ends up incorporating petty crimes (which happen much more frequently) in the target variable in pursuit of a more accurate model. The model then ends up trying to learn the features of a more nebulous concept of crime ignoring important differences between different types. Another example might be building a gender recognition system and only recognising people as one of two genders[25].

Feature selection

In an ideal world we would train a machine learning model on a sufficiently large dataset consisting of a rich set of features that actually influence the target variable rather than simply being correlated to it. More often than not, the reality is rather different. Comprehensive data can be expensive and difficult to collect. Factors that influence the target variable might not be easily measured or be measurable at all, while data containing more erroneous indicators might simply be cheaper to obtain or more readily available. This is a common way in which bias against protected classes can enter our model.

The inclusion of protected features without control variables might arise because a protected feature appears to be predictive of the target variable where explanatory variables are not known or available. Of course in cases where using protected characteristics as inputs to an algorithm would lead to disparate treatment liability, this is not a problem one is typically faced with, but it’s worth reiterating the importance of controlling for confounding variables, in drawing conclusions about relationships between features from observational data (see section 1.4).

Inclusion of protected feature proxies, as is the case with redlining, is perhaps a more common problem. One where protected features are not used as inputs to the model, but features which are predictive of them are. Historically employers have taken the reputation of the university that applicants graduated from as a strong indicator of the calibre of the candidate. But many of the most reputable universities have very low rates of non-White/Asian students in attendance. A hiring process which is strongly influenced by the university from which the applicant graduated, can erroneously disadvantage racial groups that are less likely to have attended them. While the university an applicant graduated from, might correlate to some degree with success in a particular role, it is not in itself the driver. An algorithm that directly takes into account the skills and competencies required for the role would be more predictive and simultaneously less biased. Given the cost of collecting comprehensive data, one might argue that higher error rates for some classes would be financially justified (rational prejudice).

Cost function

A critical consideration in how we specify our model is the cost function. It is how we evaluate our model in training and essentially determines the model (parameters) we end up with. The cost function can be interpreted as an expression of our model objectives and so provides a natural route to addressing discrimination concerns. A common failure in the design of classification models is proper accounting of the costs of the different types of classification errors (false negative versus false positives). If the harm caused by the different types of misclassification are asymmetric, the cost matrix should reflect this asymmetry.

More broadly (for both regression and classification), it is important to consider the contribution from each sample in the training data to the cost function in training. Upsampling (or simply up-weighting, depending on the learning algorithm you are using) is a valuable tool to keep in mind and can alleviate a number of the issues discussed above, that are common sources of bias. Let’s take the issue of low support. By upsampling minority classes, one can increase the importance of reducing errors for those data points, relative to other more abundant classes, during learning. Though it’s worth noting that it cannot resolve issues relating to a lack of richness of representation for classes with low support. Another case in which upsampling can help is that discussed in relation to definition of a heterogeneous target variable. By upsampling data points that correspond to the primary event of interest (violent crime in the example we discussed above), one can again increase the importance of the model fitting to those data points.

For an algorithm that solves a problem in a regulated domain, it would make sense for the absence of discrimination to be a model objective along with utility. This can be achieved by use of a penalty term in the cost function which relates to discrimination in the resulting predictions (just as we have terms that relate to the error or overfitting). Essentially the idea is similar to that of regularisation to avoid overfitting. We introduce an additional hyper-parameter to tune, which represents the strength of the penalty for discrimination in our cost. We will discuss this and upsampling in more detail when we discuss bias mitigation techniques, in part three of the book.

Evaluation bias

Evaluation bias arises when evaluating a model’s performance. There are two main components here, the metrics chosen to describe the model’s performance and the benchmark dataset on which they are calculated. Choosing either inappropriately will result in our evaluation metric inaccurately reflecting the efficacy of our model. For sociotechnical problems in particular choosing good metrics requires domain knowledge - the wider political, legal, social and historical context is important when defining what success and failure look like. For example, if building a gender recognition system, one should not simply think of the performance on the specific task but also the wider infrastructural systems which might find the technology useful. Where should we set the bar for such a technology? That should surely depend on how the technology is used after the prediction is made? Are there controls around model use? Should there be? What kinds of risk level does the model present? What might be the impact of the prediction being incorrect? When would an error be fair? What kind of examples would you expect your system to get wrong and why? What do they gave in common? Are they represented in the benchmark dataset? By asking these kinds of questions, when deciding what success looks like, it’s hard to imagine thinking that minimising the mean squared error on a conveniently available dataset would be sufficient.

One approach might be to set accuracy thresholds across all (skin colour) phenotype and gender combinations [25]. This would be one way of thinking about success in a way that incorporates some of our societal values of equality. The gender recognition software we talked about in the previous chapter suffered from evaluation bias on both counts. The benchmark datasets used were not representative of the target population and the metrics that were chosen, failed to expose the models poor performance on darker skinned women. The problem of evaluation bias arising from poor choice of testing/benchmark data is often the result of trying to objectively compare performance across models and can lead to overfitting to said benchmark data.

Documentation

Documentation for models (as for datasets) can have a significant impact when it comes to avoiding model misuse (a model use it is not appropriate/approved for) and ensuring model limitations are well understood. It can reduce the risk of misinterpretation of variables as suitable proxies for other variables. Clear explanation of the model, testing that was performed, on what subgroups of the data can make it easier to know which tests might be missing that would offer insight into the validity of the model. Documentation should evidence that the model governance standards have been met. Descriptions of the data and model, motivation behind subjective decisions that were made to arrive at the solution (how to process the data, what features were used/ignored and why, model type, cost function, sample weights, bias and success metrics), known data/model issues, how the model was tested, what it’s limitations are, what it should and should not be used for with justification. Documentation of the model should provide enough detail to be able to re-implement the model, reproduce results and justify the solution approach. Documentation that is standardised through use of a template could ensure some level of consistency and efficiency across domains and applications. Recent research discusses the matter specifically for publicly released datasets[42] and machine learning models[43]. They suggest standardised analysis which for example demonstrates the performance of the algorithm for different subgroups of the population and requirements for proving efficacy for conjunctions of sensitive characteristics also.

Measures of independence

In this section we will define a range of fairness metrics derived from the notion of independence. Along the way, we will familiarise ourselves with some of the terminology used to describe them. In the equations that follow, we provide metrics that quantify the extent of the relationship between our model output \(\hat{Y}\) and sensitive feature \(Z\), but we could equally well replace the predicted target variable \(\hat{Y}\), with the actual target \(Y\) to assess our data under the same criterion.

Mutual information, denoted \(I\), is popular in information theory for measuring dependence between random variables.

It is equal to zero, if and only if the joint distribution of \(Z\) and \(\hat{Y}\) is equal to the product of their marginal distributions, that is if \(f_{\hat{Y},Z}(\hat{y},z)=f_{\hat{Y}}(\hat{y})\mathbb{P}(z)\). Therefore, two variables which have zero mutual information are independent. The normalised prejudice index[51] [51] K. Fukuchi, J. Sakuma, and T. Kamishima, “Prediction with model-based neutrality,” IEICE TRANS. INF. & SYS., vol. E98–D, no. 8, 2015, doi: 10.1587/transinf.2014EDP7367. divides mutual information by a normalising factor so that the resulting value falls between zero and one:

where \(S\) is the entropy,

and

In the above equation we assume a continuous target variable, for classification problems we can replace the integrals in equations (3.1) and (3.4) with summations. Implemention in appendix D.1.

A simple relaxation of independence requires only the mean predicted target variable (rather than the full distribution) to be equal for all values of the sensitive feature, for example, \[\mathbb{E}(\hat{Y} | Z=0) = \mathbb{E}(\hat{Y} | Z=1).\] We could measure the extent of the disparity by taking the ratio or the difference of the expectations; the latter is more commonly used. The mean difference (illustrated in Figure 3.1) which (as the name suggests) looks at the difference between the mean predictions for different partitions of the population based on the sensitive feature \(Z\), \[d = \mathbb{E}(\hat{Y} | Z=0) - \mathbb{E}(\hat{Y} | Z=1).\]

Taking the simplest example of discrete binary classifier where we have a binary sensitive feature. We can write the requirement of independence as, \[\mathbb{P}(\hat{Y}=1 | Z=1) = \mathbb{P}(\hat{Y}=1 | Z=0).\] This criterion goes by many names in research literature - demographic parity, statistical parity and parity impact, among others. We can quantify the disparity by looking at the difference or the ratio of the acceptance rates for each sensitive feature. Both are straightforward to calculate given the 2 \(\times\) 2 contingency table (Table 3.1), which summarises the observed relationship between the sensitive feature and outcome. Each cell of the contingency table shows the number of examples satisfying the conditions given in the corresponding row and column headers. So for example, \(n_{01}\) is the number of data points for which \(Z=0\) and \(\hat{Y}=1\).

In bio-medical sciences, the risk difference: \[d = \mathbb{P}(\hat{Y}=1 | Z=0) - \mathbb{P}(\hat{Y}=1 | Z=1) = \frac{n_{11}}{n_{Z=0}} - \frac{n_{01}}{n_{Z=1}},\] measures the impact of treatment (or risk factors), \(Z\) on outcome, \(\hat{Y}\). In discrimination literature, it has been described as the discrimination score and statistical parity difference among others. Note that if \(\hat{Y}=1\) is the advantageous outcome and \(Z=1\) is the advantaged group, we would expect \(d\) to be negative. The algorithm is fair when \(d=0\). The further from zero, the greater the disparity. A modified version of this metric is the normalised difference[52] [52] I. Zliobaite, “On the relation between accuracy and fairness in binary classification.” 2015.Available: https://arxiv.org/abs/1505.05723 which divides the risk difference by it’s maximal value, thus ensuring the normalised difference is bounded between plus and minus one.

Alternatively, we could instead take the ratio as a measure of discrimination: \[r = \frac{\mathbb{P}(\hat{Y}=1 | Z=0)}{\mathbb{P}(\hat{Y}=1 | Z=1)} = \frac{n_{11}/n_{Z=0}}{n_{01}/n_{Z=1}}.\] In biomedical sciences this measure is called the risk ratio. It is used to measure the strength of association between treatment (or risk factors), \(Z\), and outcome, \(\hat{Y}\). It has been described in discrimination aware machine learning literature as the impact ratio or disparate impact ratio. The algorithm is fair if \(r=1\). If we take \(Z=1\) to be the advantaged group the value is bounded between zero and one. The Equal Employment Opportunity Commission (EEOC) have used this measure in their guidelines for identifying discrimination in employment selection processes[53] [53] U. S. E. E. O. Commission, “Questions and answers to clarify and provide a common interpretation of the uniform guidelines on employee selection procedures,” Federal Register, vol. 44, no. 43, 1979. . As a rule of thumb, the EEOC determine that a company’s selection system is having an adverse impact on a particular group if the selection rate for that group is less than four-fifths (or 80%) that of the most advantaged group, that is, the impact ratio is less than 0.8 where \(Z=0\) is the most advantaged group (for which the acceptance rate is the highest).

The elift ratio[54] [54] D. Pedreschi, S. Ruggieri, and F. Turini, “Discrimination-aware data mining,” in Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining, 2008, pp. 560–568. doi: 10.1145/1401890.1401959. is similar to the impact ratio but instead of comparing acceptance rates for protected groups to each other, we compare them to the overall/mean acceptance rate: \[\frac{\mathbb{P}(\hat{Y}=1 | Z=0)}{\mathbb{P}(\hat{Y}=1)}.\]

In theory, any measure of association suitable for the data types can be used as a metric to understand the magnitude of discrimination in our data or predictions. The odds ratio (popular in natural, social and biomedical sciences) is the ratio of the odds of a positive prediction for each group. We can write it as: \[\frac{\mathbb{P}(\hat{Y}=1 | Z=1)\mathbb{P}(\hat{Y}=0 | Z=0)} {\mathbb{P}(\hat{Y}=0 | Z=1)\mathbb{P}(\hat{Y}=1 | Z=0)} = \frac{n_{11}n_{00}}{n_{10}n_{01}}.\] The odds ratio is equal to one when there is no discrimination. In the case where \(\hat{Y}=1\) in the advantaged outcome and \(Z=1\) is the privileged group, the odds ratio is always greater than or equal to one. Recall that the odds ratio is not a collapsible measure (see section 1.4.3).

As mentioned earlier, independence metrics can be evaluated on both the data and the model. A common problem in machine learning is that existing biases in the data can be exaggerated if protected groups are minorities in the population. By comparing metrics for the data with those of our model output, we can understand if our model is inadvertently introducing biases that do not originate from the data.

It might seem intuitive already, that independence can only be satisfied by a model (optimising for predictive performance), if the target variable \(Y\) and sensitive feature \(Z\) are in fact independentWe’ll prove this to be true in section 3.3, for the case where our variables are binary.

. If this is not the case, then satisfying independence for one’s model, will not permit the theoretically perfect solution \(\hat{Y}=Y\) (should your model be able to achieve it). We would also then naturally, expect that the stronger the relationship between the sensitive feature and target, the greater the trade-off with utility in satisfying independence criterion.

Independence does not guarantee fairness in a broader sense. Consider a simple hypothetical example where, there are discrepancies between credit card approval rates for men and women at the population level, which disappear once you control for (the confounding variable) income. The underlying issue appears to be the fact that women, generally earn less than men. If the lender was to enforce independence between gender and its loan approval rate by, for example, setting lower income requirements for women than men, this could conceivably lead to higher default rates among women. Clearly a less than desirable solution which, arguably, doesn’t address the underlying problem. Furthermore, it might be argued that satisfying independence could lead to less fair outcomes from a different perspective; that a man and woman who were the same in all other respects (features) would receive different outcomes. In the next chapter we’ll talk about individual fairness which reconciles these differences in perspective by requiring the specification of a task specific similarity metric for individuals.

It is important to note that the assumption of independence does not allow for confounding variables (discussed in section 1.4). Suppose we want to measure the relationship between the sensitive feature and outcome using one of the above metrics. A natural solution to the problem of confounding variables, is to control for them (assuming you have them recorded in your dataset and your data is representative of the population). Next, we consider the case where we condition on all the non-sensitive variables \(X\).

Sufficiency and Calibration by Group

As one might expect, satisfying separation or sufficiency does not require as great a sacrifice of utility as independence (assuming \(Y\not\perp Z\)). Neither separation nor sufficiency are necessarily satisfied by the utility optimal solution. Of the two sufficiency, imposes a weaker constraint on our model. To understand why, we explore another interpretation of sufficiency which intuitively explains why it might be satisfied implicitly through the training process[57] [57] L. T. Liu, M. Simchowitz, and M. Hardt, “The implicit fairness criterion of unconstrained learning.” 2019.Available: https://arxiv.org/abs/1808.10013 . Let us look at sufficiency criteria in terms of the classification score \(P\), \[\mathbb{P}(Y=1 | P=p, Z=1) = \mathbb{P}(Y=1 | P=p, Z=0) \quad \forall \, p\] We say that a classifier score is calibrated if \[\mathbb{P}(Y=1 | P=p) = p \quad \forall \, p.\] Essentially, this is the requirement that the proportion of data points assigned the score \(p\), which did in fact have a positive outcome \(Y=1\), should be equal to the score \(p\). The score \(p\) can then be interpreted, at the population level, as the probability that the a positive prediction \(\hat{Y}=1\) would be correctFor the score to be interpretable as this probability at the individual level, we would need to satisfy the stronger criteria \(P=\mathbb{E}[Y|X]\).

.

From the definitions above we can see that if our classifier scores are calibrated for all groups, sufficiency is automatically satisfied. If our model satisfies sufficiency but not calibration by group, we can calibrate our model score through a simple transformation. We simply pick a value for \(Z\), \(Z=1\) say, and then calculate the mapping, \[\mathbb{P}(Y=1|P=p, Z=1) = f(p).\] We then transform all our scores to new scores (which satisfy calibration by group) by applying the inverse mapping \(f^{-1}(P)\). The resulting model is both sufficient and calibrated. It’s worth noting that the developers of COMPAS were able to demonstrate that their model did satisfy calibration by group. In a later review, researchers crowd sourced human risk assessors via Amazon Mechanical Turk it was found that COMPAS was "no more accurate or fair than predictions made by people with little or no criminal justice expertise. In addition, despite COMPAS’s collection of 137 features, the same accuracy can be achieved with a simple linear classifier with only two features"[58] [58] J. Dressel and H. Farid, “The accuracy, fairness, and limits of predicting recidivism,” Science Advances, vol. 4, no. 1, p. eaao5580, 2018, doi: 10.1126/sciadv.aao5580. .

There are some obvious advantages of comparing errors rather than outcomes. Note that unlike criteria comparing outcomes they do not preclude the theoretically perfect solution, \(\hat{Y}=Y\). The criteria also preclude large differences in error rates for different groups that are typical when disadvantaged classes are minorities suffering from low support. It’s worth reiterating that criteria comparing errors assume that the relationship between \(Y\) and \(Z\) prescribed by the training data is fair. Depending on the context of the problem one might prioritise equalising one type of error over another. For example, in pretrial risk assessment we might choose to prioritise ensuring equal false positive rates if we believe that it is preferable to set free a guilty defendant than incarcerate an innocent one. As another example, let’s take the infamous NYPD stop-and-frisk program where pedestrians were stopped, interrogated and searched on ‘reasonable’ suspicion of carrying contraband. In this case we might want to ensure false discovery rates are equal across groups to ensure we are not disproportionately targeting particular minority groups.

Total Variation (\(L_1\)) Norm: \(D_{1}\)

One possible distance metric on distributions \(d_{\mathcal{P}(\mathcal{Y})}\) is the total variation, \[d_{\mathcal{P}(\mathcal{Y})} = d_{tv}(p,q) = \frac{1}{2} \sum_{y\in\mathcal{Y}} |p(y)-q(y)|.\] Note that \(d_{tv}\) is bounded between zero (when the distributions are the same) and one (when the distributions are entirely non-overlapping), therefore the Lipschitz condition would require us to the choose the distance metric \(d_{\mathcal{X}}\) between individuals to be scaled similarly. This can be problematic depending on the feature space.

Relative (\(L_{\infty}\)) Norm: \(D_{\infty}\)

An alternative choice for \(d_{\mathcal{P}(\mathcal{Y})}\) which resolves this issue is the relative \(l_{\infty}\) metric: \[d_{\mathcal{P}(\mathcal{Y})} = d_{\infty}(p,q) = \sup_{y\in\mathcal{Y}} \log \left[\max\left(\frac{p(y)}{q(y)}, \frac{q(y)}{p(y)}\right)\right].\]

Observation 1.

For \(\alpha\leq0\) the index is undefined for zero benefits (since \(f_{\alpha}(x)\rightarrow\infty\) as \(x\rightarrow0\)), making it unsuitable for measuring inequality where zero benefits are possible.
Given an array of benefits we can calculate what proportion of the total benefit each individual received by dividing their benefit by the sum of the benefits in the array. If the total benefit is equally divided among the population, each individual receives the mean benefit \(\mu\). If we divide the benefits by the mean benefit (rather than the sum), we calculate \(x_i=b_i/\mu\) which tells us how many times the fair (mean) amount each individual received. Notice that,

where \(p_i\) is the proportion of the total benefit ascribed to individual \(i\). Since the index is scale invariant we know that \(I_{\alpha}(\boldsymbol{b})=I_{\alpha}(\boldsymbol{p})\). Since \(p_i\in[0,1]\;\forall\; i\), we know that \(x\in[0,n]\).

Observation 2.

Let \(B\) and \(P\) denote the random variables that generate \(b_i\) and \(p_i\) respectively. We know that \(\mathbb{E}(B)=\mu\) and \(\mathbb{E}(P)=1/n\). The generalised entropy index can be written as,

We know that inequality indices measure divergence from the uniform distribution and we can think of them as a system for ranking distributions from most to least fair. The most fair (and least uncertain) distribution, where everyone receives the mean benefit, has an index value of zero. In this case the benefit distribution has all of its weight at the mean \(\mu\). We can write the distribution of benefits in this case as \(\delta(b-\mu)\) where \(\delta\) is the delta function (see appendix A). To understand the role of the generalisation parameter in ranking, consider two closely related distributions with the same mean, illustrated in Figure 5.2. The first distribution \(f(b)\) is skewed, and the second is its reflection in the mean \(f(\mu-b)\).

Which distribution of benefits is preferred? The generalisation parameter \(\alpha\) determines the weight \(f_{\alpha}(b/\mu)\) applied to different parts of the distribution in calculating its ranking.

Observation 3.

From equation (5.2) we can show that, \[f_{\alpha}'(x) = \left\{ \begin{array}{cl} -1/x & \textrm{if}\quad \alpha=0 \\ 1+\ln x & \textrm{if}\quad \alpha=1 \\ x^{\alpha-1}/(\alpha-1) & \textrm{if}\quad \alpha\in\mathbb{R} \end{array}\right\} \qquad\textrm{and}\qquad f_{\alpha}''(x) = x^{\alpha-2}.\] Note that since \(x>0, \;f''_{\alpha}(x)>0\;\forall\;\alpha\), thus \(f_{\alpha}(x)\) is convex for all values of \(\alpha\). When \(\alpha=1\), the contribution to the inequality index is proportional to the individual benefit.

In Figure 5.3, we plot the function \(f_{\alpha}(x)\), for different choices of \(\alpha\).

We note that the contribution to the index, from individuals that receive the mean benefit, is always zero. As we increase \(\alpha\), the contribution to the index from the upper end of the benefit distribution grows, while the contribution from the lower end decays.

For \(\alpha<1\):

A fixed transfer in benefit (from rich to poor) at the low end of the distribution (where \(f_{\alpha}(x)\) is steeply declining), decreases the the value of the index more than at the top end (where \(f_{\alpha}(x)\) is flatter).

For \(\alpha>1\):

The reverse is true. A fixed transfer in benefit (from rich to poor) at the upper end of the distribution (where the \(f_{\alpha}(x)\) is steeply increasing), decreases the the value of the index more than at the lower end (where \(f_{\alpha}(x)\) is flatter).
One interpretation is that, for \(\alpha<1\), the index prioritises equality for the poor, while for \(\alpha>1\) equality is prioritised for the rich. Recall that Rawls’ maximin principle as the requirement that, social and economic inequalities must be of the greatest benefit to the least-advantaged members of society. As \(\alpha\rightarrow-\infty\), the associated rankings of distributions correspond to those implied by maximin principle[64].

Observation 4.

The contribution to the between-group component, from each subgroup, is weighted by the size of the subgroup. This serves to favour more prevalent groups in the data. It could be argued then, that the between-group component of the index (as an approach to measuring group fairness) is more aligned with utilitarian principles than those described in chapter 3 which do not account for group sizes. Arguably, this is intentional since aggregating as we do to calculate utility, can hide adverse impacts on underrepresented groups. Ignoring group sizes makes them less reliant on the assumption of representativeness of the data (with respect to those groups).

Observation 5.

The number of subgroups, greatly influences the size of the relative contributions from the between-group and within-group components of the inequality index. Notice that to calculate the between-group component, we first average the benefits over each group to get their means. We then calculate the value of the index on the means. The fewer subgroups, the fewer elements there are to sum in the between-group component. For large groups, \(\mu_g/\mu\) is close to unity and \(f_{\alpha}(\mu_g/\mu)\) is close to zero. Consider partitioning our population into subgroups of equal sizes. At one extreme, we have only a single group. In this case, the contribution from the between-group component is zero, and the index is equal to the within-group component. As the number of subgroups in the partition increases, the subgroups get smaller and the relative contribution to the index from the between-group component increases. Eventually, we have \(n\) groups, each composed of a single individual. In this case, the within-group component is zero, and the index is equal to the between-group component.

Observation 6.

For the values \(\alpha=0\) and \(\alpha=1\), the within-group component is a true weighted average of the index values for the subgroups, since the coefficients sum to one. For \(\alpha\in(0,1)\) the coefficients sum to less than unity, For \(\alpha>1\), the coefficients sum to more than unity. The sum of the coefficients is minimised for \(\alpha=1/2\).

Binary Benefits

All but the last benefit function in Table 5.1 result in binary arrays of benefits. Individuals either benefit from the system or they do not. For binary benefits, the distribution of benefits can be characterised with a single parameter, the mean benefit \(\mu\).

For binary benefits, the index is a monotonic increasing function of the mean benefit. The index is maximal where only one individual benefits. As we increase the proportion of people who benefit from from \(1/n\) to \(n\), the distribution of benefits approaches uniform. As the number of individuals grows, so does the maximal value of the index (as shown in section 5.2.5). In Figure 5.5, we plot the value of the index as a function of the mean benefit \(\mu\) for a variety of values of \(\alpha>0\).

For the benefit functions listed in Table 5.1 the only way to achieve overall fairness (index value zero) is if every individual under consideration receives a benefit of exactly one. We showed this to be the case for binary benefits above. For the fairness criterion in the last row of Table 5.1, comparing luck, benefits are no longer binary. We can also achieve a zero index value if all individuals receive a benefit of two; but this is only possible in the degenerate case where all predictions are false positives, that is, \(\hat{Y}=1\) and \(Y=0\). For each benefit function, the scenarios for which overall fairness is minimal (all individuals under consideration receive exactly one benefit) differs.

Comparing Outcomes

For benefit functions comparing outcomes, the higher the acceptance rate, the lower the value of the index. We achieve a perfectly fair model (zero index value) only by accepting everyone. Recall in the previous chapter where our model was stochastic, we saw that treating every individual the same, corresponded to mapping all individuals to the same the distribution over outcomes. Here, the only way to treat all individuals the same, is to accept them all. Notice that our metric is undefined in the case where we reject all individuals.

Comparing Errors

In the case where we translate group fairness criterion comparing errors, the accurate prediction is always defined as the beneficial one. In all cases we can achieve a zero index value with a 100% accurate model, i.e. \(\hat{Y}=Y\). For the benefit function corresponding to equal accuracy, our index is a monotonic function of accuracy and we can only achieve a zero index value, with a 100% accurate model. Neglecting to consider all the points means that in some cases, achieving 100% accuracy is no longer the only way to minimise the index. For example, consider the benefit function corresponding to equal false positive rates. For this we achieve a zero index value so long as the false positive rate is zero. Similar arguments apply to the benefit function corresponding to equal false negative rate, we need the false negative rate to be zero. We leave it to the reader to consider the remaining benefit functions in the Table 5.1.

Equal Luck

The final row of Table 5.1 shows the benefit function proposed by Speicher, Heidari et al. which distinguishes between false negative and false positive errors. We describe the criterion as requiring equal luck. It assumes a positive outcome to be the most advantageous to the individual. It assigns false negative predictions a benefit of zero (the least lucky), while a false positive prediction (the most lucky) is deemed twice as beneficial as a correct prediction. The benefits in this case are a measure of the discrepancy between the individuals assigned label (prediction) and the label deserved according to the ground truth. In fact the benefit is exactly one plus the error, i.e. \(b_i=\hat{y}_i-y_i+1\). It provides a measure of the relative prevalence of false positive to false negative errors; that is, if the model over or underestimates on average.

In the original paper the index value corresponding to equal luck is described as a measure of individual fairness in that "individuals deserving similar outcomes, receive similar outcomes". Recall that individual fairness (as described by Dwork et. al.[59] and discussed in the previous chapter), is the notion that in a fair system, similar people are treated similarly. For the benefit function associated with equal luck, the similarity of individuals is based solely on their associated error, \(\hat{y}_i-y_i\) (and not on their features, as described by Dwork et. al.[59]). The metric looks at the difference between the prediction and ground truth and thus clearly some measure of model performance. As demonstrated earlier, generalised entropy indices are subgroup decomposable loss functions and thus some measure of utility. That said, generalised entropy indices are a measure of individual fairness on some level, albeit one that makes a coarse comparison of individuals and places absolute faith in the data.

The Cost of an Error

Let us denote the cost of an error as, \[\Delta I^{\pm}_{\alpha}(\boldsymbol{b}) = I_{\alpha}(\boldsymbol{b^{\pm}}) - I_{\alpha}(\boldsymbol{b}).\] Here \(\boldsymbol{b^{\pm}}\) differs from \(\boldsymbol{b}\) by one prediction only, containing one less correct prediction, and one more erroneous one. For \(\boldsymbol{b^{+}}\), the additional error is a false positive. For \(\boldsymbol{b^{-}}\), the additional error is a false negative. An additional false negative error, reduces the total benefits by one; both the accuracy \(\lambda\) and the mean benefit \(\mu\) are reduced by \(1/n\). An additional false positive error, increases the total benefits by one; the accuracy \(\lambda\) is, once again, reduced by \(1/n\), and the mean benefit \(\mu\) increases by \(1/n\). Therefore, we can write,

The discrete grid of adjacent models we can reach through a small change in the model (given \(\mu\), \(\lambda\) and \(n\)), is shown in Figure 5.11.

Equation 5.12 provides an expression for \(I_{\alpha}(\mu,\lambda)\). Substituting for \(\lambda\) and \(\mu\) in the case \(\alpha=1\) gives, \[I_{\alpha}\left(\lambda-\frac{1}{n},\mu\pm\frac{1}{n}\right) = \left[1-\left(\frac{\lambda}{\mu}-\frac{1}{n\mu}\right) \left(1\pm\frac{1}{n\mu}\right)^{-1}\right]\ln2-\ln\mu - \ln\left(1\pm\frac{1}{n\mu}\right).\] For \(\alpha>0\), we get, \[I_{\alpha}\left(\lambda-\frac{1}{n},\mu\pm\frac{1}{n}\right) = \frac{1}{\alpha(\alpha-1)} \left[ \left(\frac{2}{\mu}\right)^{\alpha-1} \left(1\pm\frac{1}{n\mu}\right)^{1-\alpha} - \frac{(2^{\alpha-1}-1)}{\mu^{\alpha-1}} \left(\frac{\lambda}{\mu}-\frac{1}{n\mu}\right) \left(1\pm\frac{1}{n\mu}\right)^{-\alpha} - 1 \right].\] We showed earlier that we must have, \(\lambda\leq\mu\leq2-\lambda\), in addition, for most problems, any reasonable model should have \(0.5\leq\lambda\leq1\). We deduce that we must have \(0.5\leq\mu\leq1.5\) and so \(\mu=\mathrm{O}(1)\). Then for large \(n\), we can be sure that \(n\mu\) is large and its reciprocal is small. For large \(n\), we can write the cost of an error as \[\Delta I^{\pm}_{\alpha}(\mu,\lambda;n) = \xi_{\alpha}(\mu,\lambda) \left(\frac{1}{n\mu}\right) + \mathrm{O}\left(\frac{1}{n\mu}\right)^2\] where, \[\xi_{\alpha}(\mu,\lambda) = \left\{ \begin{array}{cl} \left(1\pm\dfrac{\lambda}{\mu}\right)\ln2\mp1 & \textrm{if}\quad\alpha = 1 \\ \rule{0em}{4.1ex} \dfrac{1}{\alpha(\alpha-1)\mu^{\alpha}}\bigg[ \Big((1\pm1\mp\alpha)2^{\alpha-1}-1\Big)\mu \pm \alpha(2^{\alpha-1}-1)\lambda\bigg] & \textrm{if}\quad \alpha>0. \end{array}\right.\]

From these expressions, we can get a clearer understanding of when the index deviates from simply being a measure of error. In particular, we want to know when an error, is preferable to an accurate prediction; that is, when the index change (resulting from an error) is negative. With a little effort we can show that,

where,

False Negative Errors

Let’s start by looking at \(h^-(\alpha)\), which we re-write as,

Equation (5.20) reveals that \(h^-(\alpha)\) is a strictly increasing function of \(\alpha\), for \(\alpha>0\) (since \(\alpha2^{\alpha-1}\) dominates \(\alpha\)). In addition, we can see that \(h^-(\alpha)\rightarrow1^-\) as \(\alpha\rightarrow\infty\). In Figure 5.12 we plot \(h^-(\alpha)\).

Earlier we showed that we must have \(\mu\geq\lambda\). Then from equation (5.18), for \(\Delta I^-_{\alpha}(\mu,\lambda;n)<0\) we need \(h^-(\alpha)>1\). Since \(h^-(\alpha)<1\) for all \(\alpha>0\), we know that making an additional false negative error, never decreases the value of the index. What about false positive errors?

False Positive Errors

We rewrite \(h^+(\alpha)\) as,

Equation (5.21) reveals that \(h^+(\alpha)\) is a decreasing function of \(\alpha\), since \(2^{1-\alpha}\) is a strictly decreasing function of \(\alpha\). In addition, we can see that \(h^+(\alpha)\rightarrow1^+\) as \(\alpha\rightarrow\infty.\) Earlier we showed that we must have \(\mu\leq2-\lambda\). Then from equation (5.18), for \(\Delta I^+_{\alpha}(\mu,\lambda;n)<0\) we need, \[h^+(\alpha)\lambda < 2-\lambda \quad\Leftrightarrow\quad \lambda < \bar{\lambda}(\alpha) = \frac{2}{1+h^+(\alpha)}.\] From what ew know about \(h^+(\alpha)\), we can deduce that \(\bar{\lambda}(\alpha)\) is an increasing function of \(\alpha\), and \(\bar{\lambda}(\alpha)\rightarrow1^-\) as \(\alpha\rightarrow\infty\). Since \(\bar{\lambda}(\alpha)<1\) for all \(\alpha>0\), we know there are indeed some circumstances, under which a false positive error, decreases the value of the index. In Figures 5.13, we plot \(h^+(\alpha)\) and \(\bar{\lambda}(\alpha)\).

The Deviation Region

We call the deviation region the part of the domain for which the index is not reduced by reducing the error rate, but instead reduced by increasing the error rate. For our benefit function equal luck, \[I_{\alpha}(\mu,\lambda): ([\lambda, 2-\lambda], [0.5,1]) \mapsto \mathbb{R}_{\geq0}.\] The only kind of error which is ever preferable to a correct prediction under this benefit function is a false positive error. This happens only when the mean benefit exceeds \(h^+(\alpha)\lambda\), that is when the ratio of lucky to unlucky people is sufficiently high. We note that for a model whose accuracy is greater than \(\bar{\lambda}(\alpha)\), it is not possible for the mean benefit (skew) to exceed the required level. That is, \[\begin{aligned} & \Delta I^-_{\alpha}(\mu,\lambda;n) > 0 \quad \forall\,\mu, \,\lambda, \, n \\ & \Delta I^+_{\alpha}(\mu,\lambda;n) < 0 \quad\Rightarrow\quad \mu > h^+(\alpha) \lambda \end{aligned}\] where, \[h^+(\alpha) = \left\{ \begin{array}{cl} \dfrac{\ln2}{1-\ln2} \approx 2.26 & \textrm{if}\quad\alpha = 1 \\ \rule{0em}{4.2ex} \dfrac{\alpha(2^{\alpha-1}-1)}{(\alpha-2)2^{\alpha-1}+1} & \textrm{if}\quad \alpha>0 \end{array}\right.\] This is only possible if the accuracy is sufficiently low, \[\lambda < \bar{\lambda}(\alpha)=\frac{2}{1+h^+(\alpha)}.\] The deviation region is then described as, \(\mu>h^+(\alpha)\lambda\), where \(h^+(\alpha)\) is given in equation (5.21). We mark the deviation region on the contour plot for \(I_{\alpha}(\mu, \lambda)\) in Figure 5.14.

For reference, in Table 5.2, we provide some values of \(\bar{\lambda}(\alpha)\) and \(h^+(\alpha)\).

Statlog (German Credit Data) Data Set

The Jupyter Notebook, mbml_german.ipynb, contains an example calculating some of the above fairness metrics on both a dataset and model output. It uses the Statlog (German Credit Data) Data Set, in which one thousand loan applicants are classified as representing ‘good’ or ‘bad’ credit risks based on features such as loan term, loan amount, age, gender, marital status and more.

Calculating Independence Metrics

In order to calculate our metrics on the data using AIF360, we must have it in the correct format; that is, in a Pandas DataFrame (data_df) containing only numeric data types. In code listing E.1, we calculate the rate at which male and female applicants are classified as being good credit risks (base_rate) along with the difference (mean_difference) and the ratio (disparate_impact) of these rates.

# Create a DataFrame to store results in
outcomes_df = pd.DataFrame(columns=[`female', `male',
                                    `difference', `ratio'],
                           index=[`data', `model',
                                  `train data', `train model',
                                  `test data', `test model'])

# Define privileged and unprivileged groups
privileged_groups = [{`sex_male':1}]
unprivileged_groups = [{`sex_male':0}]

# Create an instance of BinaryLabelDataset
data_ds = BinaryLabelDataset(df = data_df,
    label_names = [`goodcredit'],
    protected_attribute_names = [`sex'])

# Create an instance of BinaryLabelDatasetMetric
data_metric = BinaryLabelDatasetMetric(data_ds,
    privileged_groups = privileged_groups,
    unprivileged_groups = unprivileged_groups)

# Compute the metrics with data_metric and store them in outcomes_df
outcomes_df.at[`data', `female'] = data_metric.base_rate(privileged=0)
outcomes_df.at[`data', `male'] = data_metric.base_rate(privileged=1)
outcomes_df.at[`data', `difference'] = data_metric.mean_difference()
outcomes_df.at[`data', `ratio'] = data_metric.disparate_impact()

In the notebook we look at these metrics on both the data and the model output for three different sets of the data (the full dataset, the train set and the test set) with two different models (one trained on the full dataset and another trained only on a subset of the data - the training set). In code listing E.1, we create a DataFrame to display the results in (outcomes_df) and populate the first row of it. First we define our privileged and unprivileged groups.

[{`sex': 1, `age>=30': 1}, {`sex': 0}]

(data_df[`sex']==1 AND data_df[`age>=30']==1)  OR (data_df[`sex']==0)

Next we create a BinaryLabelDataset object (data_ds) which in turn is used to create a BinaryLabelDatasetMetric object (data_metric). We then calculate the fairness metrics from data_metric and store the results in outcomes_df.

Once we have trained a model and made predictions, similar code can be written to calculate independence metrics on the model predictions for the full dataset. Code listing E.2 shows how we do this using the predictions from the trained model clf.

# Create a DataFrame with the features and model predicted target
model_df = pd.concat([X, pd.Series(clf.predict(X), name=`goodcredit')],
	axis=1)

# Create an instance of BinaryLabelDataset
model_ds = BinaryLabelDataset(df = model_df,
    label_names = [`goodcredit'],
    protected_attribute_names = [`sex_male'])

# Create an instance of BinaryLabelDatasetMetric
model_metric = BinaryLabelDatasetMetric(model_ds,
    privileged_groups = privileged_groups,
    unprivileged_groups = unprivileged_groups)

# Compute the metrics with model_metric and store them in outcomes_df
outcomes_df.at[`model', `female'] = model_metric.base_rate(privileged=0)
outcomes_df.at[`model', `male'] = model_metric.base_rate(privileged=1)
outcomes_df.at[`model', `difference'] = model_metric.mean_difference()
outcomes_df.at[`model', `ratio'] = model_metric.disparate_impact()

Table E.1 shows the results of the calculations stored in outcomes_df from the notebook. From Table E.1 we note some variation in the rates at which men and women are predicted to present good credit risks for the model versus the data. In particular, the model acceptance rates are higher for both male and female applicants than those observed in the data. There are particularly big differences when we compare results for the test data versus the model on the test data (test model), which is not surprising since the mean difference and impact ratio for the train data and test data are markedly different. In addition we are aware that our model is overfitting. Without intervention, our model appears to be reducing the bias present in the data for the test set (as measured by our independence metrics).