Street-Level Algorithms - A Theory at the Gaps Between Policy and Decisions

CHI, pp. 5302019.

Cited by: 0|Bibtex|Views29|Links
EI
Keywords:
artificial intelligence street-level algorithms street-level bureaucracies
Weibo:
By reasoning about street– level algorithms with the bene t of theoretical and historical background a orded by Lipsky’s discussion of street–level bureaucracies and the body of work that followed, we are con dent that we can make substantial progress toward designing and advanci...

Abstract:

Errors and biases are earning algorithms increasingly malignant reputations in society. A central challenge is that algorithms must bridge the gap between high-level policy and on-the-ground decisions, making inferences in novel situations where the policy or training data do not readily apply. In this paper, we draw on the theory of stre...More

Code:

Data:

0
Introduction
  • People have grown increasingly frustrated with the decisions that algorithmic systems make over their lives.
  • These decisions can have weighty consequences: they determine whether the authors are excluded from social environments [22, 23]; they decide whether the authors should be paid for the work [49]; they in uence whether the authors are sent to jail or released on bail [31].
  • It is exactly at these moments that the algorithm has to generalize: to “ ll in the gaps” between the policy implied by training data and a new case the likes of which it has never seen before
Highlights
  • People have grown increasingly frustrated with the decisions that algorithmic systems make over their lives
  • Across diverse applications of algorithmic systems, one aspect seems to come through: surprise and — quite often — frustration with these systems over the decisions they make. Researchers have approached these problems in algorithmic systems from a number of perspectives: some have interrogated the hegemonic in uence that these systems have over their users [63]; others have documented and called attention to the unfair and opaque decisions these systems can make [32]; others still have audited algorithms via the criteria of fairness, accountability and transparency [16]
  • Drawing on analogy to street–level bureaucracies, we identify opportunities for designers of algorithmic systems to incorporate mechanisms for recourse in the case of mistakes and for self-policing and audits
  • We argue in this paper that we would bene t from recognizing the agents in sociotechnical systems as analogous to Lipsky’s street–level bureaucrats: that we live in a world with street–level algorithms
  • It’s not our intention to imply that bureaucratic organizations are in any sense a panacea to any problems that we’ve discussed; instead, we hope that people can take this discussion and begin to apply a vocabulary that enriches future conversations about algorithmic systems and the decisions they make about us
  • By reasoning about street– level algorithms with the bene t of theoretical and historical background a orded by Lipsky’s discussion of street–level bureaucracies and the body of work that followed, we are con dent that we can make substantial progress toward designing and advancing systems that consider the needs of stakeholders and the potent in uence we have over their lives
Methods
  • DESIGN IMPLICATIONS

    The authors have discussed the demonetization on YouTube, the management of crowd work, and the bias of algorithmic justice, but the same undercurrent moves all of these cases.
  • Even the goodness or badness of that pattern must itself be taught.
  • The question the authors hope to answer is how to nd and identify circumstances for which algorithmic systems would not yield problematic outcomes, assuming that designers want to create a prosocial, fair system.
  • In some cases that will be easy — some classi cations’ con dence will be low, or the result will be ambiguous in some predictable way.
  • Often the system performs these erroneous classi cations with high con dence, because it does not recognize that the uniqueness of the input is di erent than other unique tokens or inputs
Conclusion
  • Street–level bureaucracies are not a perfect metaphor for the phenomena the authors have discussed.
  • The authors didn’t address in any capacity the fact that street–level bureaucrats sometimes diverge in unexpected ways from the prerogatives of their managers
  • This becomes the source of tension in Lipsky’s treatment of street–level bureaucracies, but in the discussion of the relationships between street–level algorithms and their stakeholders, the authors avoided the relationship between engineers and designers and the systems themselves.
  • By reasoning about street– level algorithms with the bene t of theoretical and historical background a orded by Lipsky’s discussion of street–level bureaucracies and the body of work that followed, the authors are con dent that the authors can make substantial progress toward designing and advancing systems that consider the needs of stakeholders and the potent in uence the authors have over their lives
Summary
  • Introduction:

    People have grown increasingly frustrated with the decisions that algorithmic systems make over their lives.
  • These decisions can have weighty consequences: they determine whether the authors are excluded from social environments [22, 23]; they decide whether the authors should be paid for the work [49]; they in uence whether the authors are sent to jail or released on bail [31].
  • It is exactly at these moments that the algorithm has to generalize: to “ ll in the gaps” between the policy implied by training data and a new case the likes of which it has never seen before
  • Methods:

    DESIGN IMPLICATIONS

    The authors have discussed the demonetization on YouTube, the management of crowd work, and the bias of algorithmic justice, but the same undercurrent moves all of these cases.
  • Even the goodness or badness of that pattern must itself be taught.
  • The question the authors hope to answer is how to nd and identify circumstances for which algorithmic systems would not yield problematic outcomes, assuming that designers want to create a prosocial, fair system.
  • In some cases that will be easy — some classi cations’ con dence will be low, or the result will be ambiguous in some predictable way.
  • Often the system performs these erroneous classi cations with high con dence, because it does not recognize that the uniqueness of the input is di erent than other unique tokens or inputs
  • Conclusion:

    Street–level bureaucracies are not a perfect metaphor for the phenomena the authors have discussed.
  • The authors didn’t address in any capacity the fact that street–level bureaucrats sometimes diverge in unexpected ways from the prerogatives of their managers
  • This becomes the source of tension in Lipsky’s treatment of street–level bureaucracies, but in the discussion of the relationships between street–level algorithms and their stakeholders, the authors avoided the relationship between engineers and designers and the systems themselves.
  • By reasoning about street– level algorithms with the bene t of theoretical and historical background a orded by Lipsky’s discussion of street–level bureaucracies and the body of work that followed, the authors are con dent that the authors can make substantial progress toward designing and advancing systems that consider the needs of stakeholders and the potent in uence the authors have over their lives
Funding
  • We would like to thank Os Keyes and Jingyi Li, among others, for volunteering their time and patience as they provided input to help us understand and discuss several of the sensitive topics with which we engaged. This work was supported by a National Science Foundation award IIS-1351131 and the Stanford Cyber Initiative
Reference
  • Paul S. Adler, Barbara Goldoftas, and David I. Levine. 1999. Flexibility Versus E ciency? A Case Study of Model Changeovers in the Toyota Production System. Organization Science 10, 1 (1999), 43–68. https://doi.org/10.1287/orsc.10.1.43
    Locate open access versionFindings
  • Ali Alkhatib, Michael S. Bernstein, and Margaret Levi. 2017. Examining Crowd Work and Gig Work Through The Historical Lens of Piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 4599–4616. https://doi.org/10.1145/3025453.3025974
    Locate open access versionFindings
  • Erin Anderson and David C. Schmittlein. 1984. Integration of the Sales Force: An Empirical Examination. The RAND Journal of Economics 15, 3 (1984), 385–395. http://www.jstor.org/stable/2555446
    Locate open access versionFindings
  • Julia Angwin, Je Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias. https://www.propublica.org/article/machine-bias-riskassessments-in-criminal-sentencing
    Findings
  • Josh Attenberg, Panagiotis G Ipeirotis, and Foster J Provost. 2011. Beat the Machine: Challenging Workers to Find the Unknown Unknowns. Human Computation 11, 11 (2011), 2–7.
    Google ScholarLocate open access versionFindings
  • Linda Babcock and Sara Laschever. 2009. Women don’t ask: Negotiation and the gender divide. Princeton University Press.
    Google ScholarFindings
  • Je rey P. Bigham, Michael S. Bernstein, and Eytan Adar. 2015. HumanComputer Interaction and Collective Intelligence. In Handbook of Collective Intelligence. MIT Press, 57–84. http://repository.cmu.edu/cgi/viewcontent.cgi?article=1264{&}context=hcii
    Locate open access versionFindings
  • Chris Bopp, Ellie Harmon, and Amy Voida. 2017. Disempowered by Data: Nonpro ts, Social Enterprises, and the Consequences of DataDriven Work. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 3608–3619. https://doi.org/10.1145/3025453.3025694
    Locate open access versionFindings
  • Nadia Boukhelifa, Marc-Emmanuel Perrin, Samuel Huron, and James Eagan. 2017. How Data Workers Cope with Uncertainty: A Task Characterisation Study. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 3645–3656. https://doi.org/10.1145/3025453.3025738
    Locate open access versionFindings
  • James Bridle. 2017. Something is wrong on the internet. https://medium.com/@jamesbridle/something-is-wrong-on-theinternet-c39c471271d2
    Findings
  • Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classi cation. In Conference on Fairness, Accountability and Transparency. 77–91.
    Google ScholarLocate open access versionFindings
  • Stevie Chancellor, Jessica Annette Pater, Trustin Clear, Eric Gilbert, and Munmun De Choudhury. 2016. # thyghgapp: Instagram content moderation and lexical variation in pro-eating disorder communities. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 1201–1213.
    Google ScholarLocate open access versionFindings
  • Lydia B. Chilton, John J. Horton, Robert C. Miller, and Shiri Azenkot. 2010. Task Search in a Human Computation Market. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP ’10). ACM, 1–9. https://doi.org/10.1145/1837885.1837889
    Locate open access versionFindings
  • Angèle Christin. 2017. Algorithms in practice: Comparing web journalism and criminal justice. Big Data & Society 4, 2 (2017), 2053951717718855. https://doi.org/10.1177/2053951717718855
    Findings
  • Jan Cohen-Cruz. 1998. Radical street performance: An international anthology. Cambridge Univ Press.
    Google ScholarFindings
  • Sam Corbett-Davies and Sharad Goel. 2018. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning. arXiv preprint arXiv:1808.00023 (2018).
    Findings
  • Motahhare Eslami, Aimee Rickman, Kristen Vaccaro, Amirhossein Aleyasen, Andy Vuong, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. "I Always Assumed That I Wasn’t Really That Close to [Her]": Reasoning About Invisible Algorithms in News Feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 153–162. https://doi.org/10.1145/2702123.2702556
    Locate open access versionFindings
  • Virginia Eubanks. 20Automating inequality: How high-tech tools pro le, police, and punish the poor. St. Martin’s Press.
    Google ScholarFindings
  • Megan Farokhmanesh. 2018. YouTube is still restricting and demonetizing LGBT videos — and adding anti-LGBT ads to some. https://www.theverge.com/2018/6/4/17424472/youtube-lgbtdemonetization-ads-algorithm
    Findings
  • R Stuart Geiger. 2018. The lives of bots. arXiv preprint arXiv:1810.09590 (2018).
    Findings
  • R. Stuart Geiger and Aaron Halfaker. 2013. When the Levee Breaks: Without Bots, What Happens to Wikipedia’s Quality Control Processes?. In Proceedings of the 9th International Symposium on Open Collaboration (WikiSym ’13). ACM, New York, NY, USA, Article 6, 6 pages. https://doi.org/10.1145/2491055.2491061
    Locate open access versionFindings
  • R Stuart Geiger and David Ribes. 2010. The work of sustaining order in wikipedia: the banning of a vandal. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 117–126.
    Google ScholarLocate open access versionFindings
  • Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.
    Google ScholarFindings
  • Mary L. Gray, Siddharth Suri, Syed Shoaib Ali, and Deepti Kulkarni. 2016. The Crowd is a Collaborative Network. In Proceedings of the 19th ACM Conference on Computer–Supported Cooperative Work & Social Computing (CSCW ’16). ACM, 134–147. https://doi.org/10.1145/2818048.2819942
    Locate open access versionFindings
  • Jan Grill. 2011. From street busking in Switzerland to meat factories in the UK: a comparative study of two Roma migration networks from Slovakia. Global Connections and emerging inequalities in Europe: perspectives on poverty and transnational migration (2011), 77–102.
    Google ScholarFindings
  • Aaron Halfaker, R Stuart Geiger, Jonathan T Morgan, and John Riedl. 2013. The rise and decline of an open collaboration system: How Wikipedias reaction to popularity is causing its decline. American Behavioral Scientist 57, 5 (2013), 664–688.
    Google ScholarLocate open access versionFindings
  • Kenneth Hanf. 1982. The implementation of regulatory policy: enforcement as bargaining. European Journal of Political Research 10, 2 (1982), 159–172.
    Google ScholarLocate open access versionFindings
  • Sally Harrison-Pepper. 1990. Drawing a circle in the square: street performing in New York’s Washington Square Park. Univ Pr of Mississippi.
    Google ScholarFindings
  • Alexis Hiniker, Hyewon Suh, Sabina Cao, and Julie A. Kientz. 2016. Screen Time Tantrums: How Families Manage Screen Media Experiences for Toddlers and Preschoolers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 648–660. https://doi.org/10.1145/2858036.2858278
    Locate open access versionFindings
  • John D Huber and Charles R Shipan. 2002. Deliberate discretion?: The institutional foundations of bureaucratic autonomy. Cambridge University Press.
    Google ScholarFindings
  • Aziz Z Huq. 2018. Racial Equity in Algorithmic Criminal Justice. (2018).
    Google ScholarFindings
  • Yasmin Ibrahim. 2010. The breastfeeding controversy and Facebook: Politicization of image, privacy and protest. International Journal of E-Politics (IJEP) 1, 2 (2010), 16–28.
    Google ScholarLocate open access versionFindings
  • Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, 611–620. https://doi.org/10.1145/2470654.2470742
    Locate open access versionFindings
  • B.M. Jewell. 1921. The problem of piece work. Number nos. 1-16 in The Problem of Piece Work. Bronson Canode Print. Co. https://books.google.com/books?id=NN5NAQAAIAAJ [35] Erik Kain.2017. YouTube Wants Content Creators To Appeal Demonetization, But It’s Not Always That Easy.https://www.forbes.com/
    Locate open access versionFindings
  • [36] Sanjay Kairam and Je rey Heer. 2016. Parting Crowds: Characterizing Divergent Interpretations in Crowdsourced Annotation Tasks. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW ’16). ACM, 1637–1648. https://doi.org/10.1145/2818048.2820016
    Locate open access versionFindings
  • [37] Eugene C Kim. 2007. YouTube: Testing the safe harbors of digital copyright law. S. Cal. Interdisc. LJ 17 (2007), 139.
    Google ScholarLocate open access versionFindings
  • [38] Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen, and Michael S Bernstein. 2017. Mechanical Novel: Crowdsourcing Complex Work through Revision. In Proceedings of the 20th ACM Conference on Computer Supported Cooperative Work \& Social Computing.
    Google ScholarLocate open access versionFindings
  • [39] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2017. Human decisions and machine predictions. The quarterly journal of economics 133, 1 (2017), 237–293.
    Google ScholarLocate open access versionFindings
  • [40] Yubo Kou, Xinning Gui, Shaozeng Zhang, and Bonnie Nardi. 2017. Managing Disruptive Behavior Through Non-Hierarchical Governance: Crowdsourcing in League of Legends and Weibo. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 62 (Dec. 2017), 17 pages. https://doi.org/10.1145/3134697
    Locate open access versionFindings
  • [41] Anja Lambrecht and Catherine E Tucker. 2018. Algorithmic bias? An empirical study into apparent gender-based discrimination in the display of STEM career ads. (2018).
    Google ScholarFindings
  • [42] John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. 2010. Ensuring quality in crowdsourced search relevance evaluation: The e ects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation. 21–26.
    Google ScholarFindings
  • [43] Roope Leppänen et al. 2017. The State of Video Hosting Websites from the Perspective of Content Creators. (2017).
    Google ScholarFindings
  • [44] Michael Lipsky. 1969. Toward a theory of street-level bureaucracy. Institute for Research on Poverty, University of Wisconsin.
    Google ScholarFindings
  • [45] Michael Lipsky. 1983. Street-Level Bureaucracy: The Dilemmas of the Individual in Public Service. Russell Sage Foundation.
    Google ScholarFindings
  • [46] David Martin, Benjamin V. Hanrahan, Jacki O’Neill, and Neha Gupta. 2014. Being a Turker. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’14). ACM, 224–235. https://doi.org/10.1145/2531602.2531663
    Locate open access versionFindings
  • [47] Bim Mason. 1992. Street theatre and other outdoor performance. Taylor & Francis.
    Google ScholarFindings
  • [48] Andreas Matthias. 2004. The responsibility gap: Ascribing responsibility for the actions of learning automata. Ethics and Information Technology 6, 3 (01 Sep 2004), 175–183. https://doi.org/10.1007/s10676004-3422-1
    Locate open access versionFindings
  • [49] Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing Around Rejection, Mistrust, Risk, and Workers’ Experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, 2271–2282. https://doi.org/10.1145/2858036.2858539
    Locate open access versionFindings
  • [50] Tanushree Mitra, C.J. Hutto, and Eric Gilbert. 2015. Comparing Personand Process–centric Strategies for Obtaining Quality Data on Amazon Mechanical Turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, 1345–1354. https://doi.org/10.1145/2702123.2702553
    Locate open access versionFindings
  • [51] Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016. The ethics of algorithms: Mapping the debate. Big Data & Society 3, 2 (2016), 2053951716679679. https://doi.org/10.1177/2053951716679679
    Findings
  • [52] Matthew G. O’Neill. 2014. Transgender Youth and YouTube Videos: Self-Representation and Five Identi able Trans Youth Narratives. Palgrave Macmillan UK, London, 34–45. https://doi.org/10.1057/9781137383556_3
    Findings
  • [53] Dražen Prelec. 2004. A Bayesian truth serum for subjective data. science 306, 5695 (2004), 462–466.
    Google ScholarLocate open access versionFindings
  • [54] Julia Quilter and Luke McNamara. 2015. Long May the Buskers Carry on Busking: Street Music and the Law in Melbourne and Sydney. Melb. UL Rev. 39 (2015), 539.
    Google ScholarLocate open access versionFindings
  • [55] Tobias Raun. 2016. Out online: Trans self-representation and community building on YouTube. Routledge.
    Google ScholarFindings
  • [56] Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST ’14). ACM, 75–85. https://doi.org/10.1145/2642918.2647409
    Locate open access versionFindings
  • [57] Richard Rothstein. 2017. The Color of Law: A Forgotten History of How Our Government Segregated America. Liveright Publishing.
    Google ScholarFindings
  • [58] Je rey M. Rzeszotarski and Aniket Kittur. 2011. Instrumenting the Crowd: Using Implicit Behavioral Measures to Predict Task Performance. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, 13–22. https://doi.org/10.1145/2047196.2047199
    Locate open access versionFindings
  • [59] Ari Schlesinger, W Keith Edwards, and Rebecca E Grinter. 2017. Intersectional HCI: Engaging identity through gender, race, and class. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 5412–5427.
    Google ScholarLocate open access versionFindings
  • [60] Aaron D. Shaw, John J. Horton, and Daniel L. Chen. 2011. Designing Incentives for Inexpert Human Raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work (CSCW ’11). ACM, 275–284. https://doi.org/10.1145/1958824.1958865
    Locate open access versionFindings
  • [61] Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’08). ACM, 614–622. https://doi.org/10.1145/1401890.1401965
    Locate open access versionFindings
  • [62] Jocelyn Summers. 2018. What Professors Dont Tell You About Social Media. (2018).
    Google ScholarFindings
  • [63] Zephyr Teachout and Lina Khan. 2014. Market Structure and Political Law: A Taxonomy of Power. Duke J. Const. L. & Pub. Pol’y 9 (2014), 37.
    Google ScholarLocate open access versionFindings
  • [64] Jacob Thebault-Spieker, Loren G Terveen, and Brent Hecht. 2015. Avoiding the south side and the suburbs: The geography of mobile crowdsourcing markets. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 265–275.
    Google ScholarLocate open access versionFindings
  • [65] Milena Tsvetkova, Ruth García-Gavilanes, Luciano Floridi, and Taha Yasseri. 2017. Even good bots ght: The case of Wikipedia. PLOS ONE 12, 2 (02 2017), 1–13. https://doi.org/10.1371/journal.pone.0171774
    Locate open access versionFindings
  • [66] J Richard Udry. 1994. The nature of gender. Demography 31, 4 (1994), 561–573.
    Google ScholarLocate open access versionFindings
  • [67] Francisco Valdes. 1994. Queers, Sissies, Dykes, and Tomboys: Deconstructing the Con ation of" Sex,"" Gender," and" Sexual Orientation" in Euro-American Law and Society. Ph.D. Dissertation. HeinOnline.
    Google ScholarFindings
  • [68] Melissa A. Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S. Bernstein. 2017. Flash Organizations: Crowdsourcing Complex Work by Structuring Crowds As Organizations. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY, USA, 3523–3537. https://doi.org/10.1145/3025453.3025811
    Locate open access versionFindings
  • [69] Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and Accountability Design Needs for Algorithmic Support in HighStakes Public Sector Decision-Making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA, Article 440, 14 pages. https://doi.org/10.1145/
    Locate open access versionFindings
  • [70] Rob Voigt, Nicholas P. Camp, Vinodkumar Prabhakaran, William L. Hamilton, Rebecca C. Hetey, Camilla M. Gri ths, David Jurgens, Dan Jurafsky, and Jennifer L. Eberhardt. 2017. Language from police body camera footage shows racial disparities in o cer respect. Proceedings of the National Academy of Sciences 114, 25 (2017), 6521–6526. https://doi.org/10.1073/pnas.1702413114 arXiv:http://www.pnas.org/content/114/25/6521.full.pdf
    Locate open access versionFindings
  • [71] Samuel Walker. 1993. Taming the system: The control of discretion in criminal justice, 1950-1990. Oxford University Press on Demand.
    Google ScholarFindings
  • [72] William Foote Whyte. 2012. Street corner society: The social structure of an Italian slum. University of Chicago Press.
    Google ScholarFindings
  • [73] William F. Whyte and Burleigh B. Gardner. 1945. The Position and Problems of the Foreman. Applied Anthropology 4, 2 (1945), 17–25. http://www.jstor.org/stable/44135127
    Locate open access versionFindings
  • [74] James Q Wilson. 1989. Bureaucracy: What government agencies do and why they do it. (1989).
    Google ScholarFindings
  • [75] Michael Winerip. 2012. Facing a robo-grader? Just keep obfuscating melli uously. New York Times 22 (2012).
    Google ScholarLocate open access versionFindings
  • [76] Johanna Wright. 2018. Restricted Mode: How it works and what we can do better. https://youtube-creators.googleblog.com/2017/03/restricted-mode-how-it-works-and-what.html
    Findings
  • [77] LLC YouTube. 2010. YouTube Community Guidelines.
    Google ScholarLocate open access versionFindings
  • [78] Haiyi Zhu and Loren Terveen. [n. d.]. Value Sensitive Algorithm
    Google ScholarFindings
Your rating :
0

 

Best Paper
Best Paper of CHI, 2019
Tags
Comments