Limits of the Quantitative Approach to Bias and Fairness

2024-03-14

The expectations for all blog posts apply!

What You Should Do

Quantitative methods for assessing discrimination and bias include techniques like:

Formal (mathematical) definitions of bias and fairness in terms.
Audits of machine learning algorithms, including things like confusion matrices and false positive rates.
Statistical tests of significance for effects related to race, gender, or other protected attributes.

In a recent speech, Narayanan (2022, 25) asserts that

“currently quantitative methods are primarily used to justify the status quo. I would argue that they do more harm than good.”

In a carefully-structured essay of approximately 1,500-2,000 words, engage this claim in conversation with the following scholarly sources:

Narayanan’s speech.
Fairness and Machine Learning: Barocas, Hardt, and Narayanan (2023).
Data Feminism: D’Ignazio and Klein (2023).
3 additional scholarly sources of your choosing.

Your essay should include:

A careful explanation of Narayanan’s position.
A careful explanation of the uses or benefits of quantitative methods, as described in one of your scholarly sources.
- Please include at least one example of a beneficial study of an algorithm or decision-process using quantitative techniques. Include a careful discussion of which quantitative notion(s) of fairness or lack of discrimination was/were used in the example, in both the technical language of Chapter 3 and the moral language of Chapter 4 of Barocas, Hardt, and Narayanan (2023).
A careful explanation of one of the limitations or drawbacks of quantitative methods, described in Narayanan’s speech or one of your scholarly sources.
- Please include at least one example of a limited, misleading, or otherwise disappinting study of an algorithm or decision-process using quantitative techniques. Include a careful discussion of which quantitative notion(s) of fairness or lack of discrimination was/were used in the example, in both the technical language of Chapter 3 and the moral language of Chapter 4 of Barocas, Hardt, and Narayanan (2023).
Appropriate supporting points from your other scholarly sources.
An argument in which you stake out a position on Narayanan’s claim of view. Do you agree? Disagree? Agree with qualifications? Which ones? Why?

References in Quarto

Appropriately formatted citations are a fundamental aspect of scholarly writing. A fundamental aspect of technical scholarly writing is learning to manage references using automated tools such as Quarto. Managing references in Quarto is very easy once you have followed the setup below! Blog posts that do not use Quarto’s citation system will receive at most Ms.

To manage references in Quarto, you need to create a .bib file (you can call it refs.bib). This file should live in the same directory as your blog post. Your .bib file is essentially a database of document information. Here’s an example of a a refs.bib file:

@book{hardtPatternsPredictionsActions2022,
  title = {Patterns, {{Predictions}}, and {{Actions}}},
  author = {Hardt, Moritz and Recht, Benjamin},
  year = {2022},
  publisher = {{Princeton University Press}},
  isbn = {978-0-691-23372-7},
  langid = {english}
}

@book{barocasFairnessMachineLearning2023,
  title = {Fairness and Machine Learning: Limitations and Opportunities},
  shorttitle = {Fairness and Machine Learning},
  author = {Barocas, Solon and Hardt, Moritz and Narayanan, Arvind},
  year = {2023},
  publisher = {{The MIT Press}},
  address = {{Cambridge, Massachusetts}}
}

@misc{narayanan2022limits,
  author       = {Narayanan, Arvind},
  howpublished = {Speech},
  title        = {The limits of the quantitative approach to discrimination},
  year         = {2022}
}

The simplest way to get entries for your references is to look them up on Google Scholar.

Search for the document you want.
Click the “Cite” link underneath and choose “Bibtex” from the options at the bottom.
Copy and paste the contents of the new page to your refs.bib file.

Once you’ve assembled your references, add the following line to your document metadata (the stuff in the top cell of your Jupyter notebook)

bibliography: refs.bib

Once you’ve followed these steps, you’re ready to cite! You can reference your documents using the @ symbol and their bibliographic key, which is the first entry for each document in the refs.bib file. For example, typing

@barocasFairnessMachineLearning2023

results in the reference

Barocas, Hardt, and Narayanan (2023)

as well as an entry in the “References” section at the end of your blog post, as illustrated below.

For more on how to handle citations in Quarto, check the Quarto documentation.

References

Barocas, Solon, Moritz Hardt, and Arvind Narayanan. 2023. Fairness and Machine Learning: Limitations and Opportunities. Cambridge, Massachusetts: The MIT Press.

D’Ignazio, Catherine, and Lauren F. Klein. 2023. Data Feminism. First MIT Press paperback edition. Cambridge, Massachusetts: The Mit Press.

Narayanan, Arvind. 2022. “The Limits of the Quantitative Approach to Discrimination.” Speech.