Semi-Automated Coding for Qualitative Research: A User-Centered Inquiry and Initial Prototypes

Megh Marathe
Megh Marathe

CHI, pp. 1-12, 2018.

Cited by: 11|Bibtex|Views31|Links
EI
Keywords:
H.5.minter-rater reliabilityqualitative datum analysishuman computer interactionmachine learningMore(5+)
Weibo:
Disciplinary Variation in qualitative data analysis Our participants represent diverse disciplinary orientations: three work on social media and communication; four study human computer interaction; two focus on digital preservation and archives; another two are science and techn...

Abstract:

Qualitative researchers perform an important and painstaking data annotation process known as coding. However, much of the process can be tedious and repetitive, becoming prohibitive for large datasets. Could coding be partially automated, and should it be? To answer this question, we interviewed researchers and observed them code intervi...More

Code:

Data:

Introduction
  • Unstructured text forms most of the primary data in qualitative research, coming from sources such as transcribed interviews, field notes, and organizational reports.
  • There has been substantial research in the social sciences and the human computer interaction (HCI) community suggesting that NLP and machine learning (ML) techniques have the potential to assist certain types of analyses of qualitative data [1, 10, 11, 13, 17, 18, 27, 36, 39, 45]
  • Scholars from both fields note the paucity of research aimed at understanding the practices and needs of qualitative researchers, and as a result, existing tools for qualitative coding do not necessarily meet user needs
Highlights
  • Unstructured text forms most of the primary data in qualitative research, coming from sources such as transcribed interviews, field notes, and organizational reports
  • There has been substantial research in the social sciences and the human computer interaction (HCI) community suggesting that natural language processing and machine learning (ML) techniques have the potential to assist certain types of analyses of qualitative data [1, 10, 11, 13, 17, 18, 27, 36, 39, 45]. Scholars from both fields note the paucity of research aimed at understanding the practices and needs of qualitative researchers, and as a result, existing tools for qualitative coding do not necessarily meet user needs
  • Our novel contributions include the following: Through interviews with qualitative researchers, we found that across disciplines, researchers follow several practices well-suited to automation
  • (2) Implications for qualitative data analysis interface design highlights four user practices not supported by qualitative data analysis software
  • Disciplinary Variation in qualitative data analysis Our participants represent diverse disciplinary orientations: three work on social media and communication; four study human computer interaction; two focus on digital preservation and archives; another two are science and technology studies scholars; Emily is a sociologist whose research intersects with social work; Tatiana’s research centers on human rights; Caleb studies education; and Max is a linguistic anthropologist
  • We present implications for interface and algorithm design
Methods
  • Participant Recruitment and Data Collection The authors conducted two rounds of data collection with fifteen participants: in the first round, five qualitative researchers participated in three-hour long contextual inquiry sessions, and in the second, ten researchers participated in hour-long semistructured interviews.
  • Each session concluded with an indepth, semi-structured interview, during which the authors asked participants about their research experience; the parts they found the most interesting and the most tedious in coding; how they used the data from first-pass coding for further analysis; the features they most liked and disliked in QDA tools; their willingness to use software that partially automates coding; and under which conditions and when in the coding process they would appreciate such assistance.
Results
  • The authors detail three types of findings: (1) Disciplinary variation in QDA notes disciplinary differences in the purpose of coding, the importance of collaboration, and the use of QDA software. (2) Implications for QDA interface design highlights four user practices not supported by QDA software. (3) Implications for QDA algorithm design presents four recommendations for automated QDA assistants.

    Disciplinary Variation in QDA The authors' participants represent diverse disciplinary orientations: three work on social media and communication; four study human computer interaction; two focus on digital preservation and archives; another two are science and technology studies scholars; Emily is a sociologist whose research intersects with social work; Tatiana’s research centers on human rights; Caleb studies education; and Max is a linguistic anthropologist.
  • Except for in-vivo and to a small extent descriptive codes, the simple keyword technique performs poorly overall
  • This is expected: once the codes turn descriptive or meta, it fails to match most participant-created annotations.
  • While this drop in performance is present for demographic codes, Tom is a notable exception because his demographic codes bear in-vivo names.
  • The augmented keyword technique achieves IRR scores that are typical of human coders after the first pass of the inter-rater reliability process [7]
Conclusion
  • Avenues for Interface and Algorithm Design The authors confirm Fielding et al.’s report of researchers using QDA suites primarily as “electronic filing cabinets” [22, 23, 30]: despite using a QDA suite for multiple studies over several years and in one case being the departmental expert, participants used only basic coding functionality, choosing instead to export data out of it after first-pass coding.
  • This paper steps towards this goal, generating design implications based on research practices that are common across disciplines.Could parts of qualitative coding be automated, and should they be?
  • Researchers desire automation, but only after having developed a codebook and coded a subset of data, in extending their coding to unseen data.
  • They require any assistive tool to be transparent about its recommendations.
Summary
  • Introduction:

    Unstructured text forms most of the primary data in qualitative research, coming from sources such as transcribed interviews, field notes, and organizational reports.
  • There has been substantial research in the social sciences and the human computer interaction (HCI) community suggesting that NLP and machine learning (ML) techniques have the potential to assist certain types of analyses of qualitative data [1, 10, 11, 13, 17, 18, 27, 36, 39, 45]
  • Scholars from both fields note the paucity of research aimed at understanding the practices and needs of qualitative researchers, and as a result, existing tools for qualitative coding do not necessarily meet user needs
  • Methods:

    Participant Recruitment and Data Collection The authors conducted two rounds of data collection with fifteen participants: in the first round, five qualitative researchers participated in three-hour long contextual inquiry sessions, and in the second, ten researchers participated in hour-long semistructured interviews.
  • Each session concluded with an indepth, semi-structured interview, during which the authors asked participants about their research experience; the parts they found the most interesting and the most tedious in coding; how they used the data from first-pass coding for further analysis; the features they most liked and disliked in QDA tools; their willingness to use software that partially automates coding; and under which conditions and when in the coding process they would appreciate such assistance.
  • Results:

    The authors detail three types of findings: (1) Disciplinary variation in QDA notes disciplinary differences in the purpose of coding, the importance of collaboration, and the use of QDA software. (2) Implications for QDA interface design highlights four user practices not supported by QDA software. (3) Implications for QDA algorithm design presents four recommendations for automated QDA assistants.

    Disciplinary Variation in QDA The authors' participants represent diverse disciplinary orientations: three work on social media and communication; four study human computer interaction; two focus on digital preservation and archives; another two are science and technology studies scholars; Emily is a sociologist whose research intersects with social work; Tatiana’s research centers on human rights; Caleb studies education; and Max is a linguistic anthropologist.
  • Except for in-vivo and to a small extent descriptive codes, the simple keyword technique performs poorly overall
  • This is expected: once the codes turn descriptive or meta, it fails to match most participant-created annotations.
  • While this drop in performance is present for demographic codes, Tom is a notable exception because his demographic codes bear in-vivo names.
  • The augmented keyword technique achieves IRR scores that are typical of human coders after the first pass of the inter-rater reliability process [7]
  • Conclusion:

    Avenues for Interface and Algorithm Design The authors confirm Fielding et al.’s report of researchers using QDA suites primarily as “electronic filing cabinets” [22, 23, 30]: despite using a QDA suite for multiple studies over several years and in one case being the departmental expert, participants used only basic coding functionality, choosing instead to export data out of it after first-pass coding.
  • This paper steps towards this goal, generating design implications based on research practices that are common across disciplines.Could parts of qualitative coding be automated, and should they be?
  • Researchers desire automation, but only after having developed a codebook and coded a subset of data, in extending their coding to unseen data.
  • They require any assistive tool to be transparent about its recommendations.
Tables
  • Table1: A summary of participant characteristics, viz. their pseudonym, academic discipline, QDA method, primary QDA software, and the number of codes (Code) and annotations (Ann) they created during in situ observation. Code and Ann do not apply to second round participants because they did not code. We use ‘Media & com’ for media and communication studies, ‘Archives’ for digital preservation and archives, ‘STS’ for science and technology studies, ‘HCI’ for human computer interaction, and ‘Docs’ and ‘Sheets’ for Google Docs and Sheets
  • Table2: Performance of simple keyword, augmented keyword, and searchstyle query matching techniques averaged over participants
Download tables as Excel
Related work
  • Coding in Qualitative Data Analysis Qualitative research has a rich heritage originating in the humanities and social sciences, encompassing a range of approaches such as ethnography, grounded theory, and phe- Paper 348

    nomenology, and methods such as interviews, field observation, and text analysis [16]. While an in-depth review is beyond the scope of this paper, there are several excellent books and articles on research design [12, 33], methodology [9, 16, 19, 25], and methods [9, 34, 38, 43].

    In any case, much qualitative data analysis involves some form of coding, wherein researchers assign short labels or codes – typically single words or short phrases – to chunks of text to indicate something about their content [5,9,12,25,33,34,38,41]. Though researchers develop their own coding style over time, certain types of coding are common [34, 38]: descriptive coding annotates text with a code describing its high-level content; whereas in vivo coding uses respondents’ own words and phrases to create codes and highlight salient topics. Further, coding can be inductive and data-driven, where researchers read and reread data for emergent codes in the form of keywords, ideas and trends present in the data; or deductive and theory-driven, where codes are determined a priori based on theories or hypotheses before a close reading of the data; or a hybrid [34, 38]. Coding enables researchers to group data into categories and examine relationships between codes.
Funding
  • Interviewed researchers and observed them code interview transcripts
  • Found that across disciplines, researchers follow several practices well-suited to automation
  • Presents a first step to bridging this gap: conducts a researcher-centered design inquiry to identify implications for interface and algorithm design, and report initial results from building assistive tools for qualitative coding
Reference
  • Aneesha Bakharia, Peter Bruza, Jim Watters, Bhuva Narayan, and Laurianne Sitbon. 2016. Interactive Topic Modeling for aiding Qualitative Content Analysis. Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval - CHIIR ’16 (2016), 213–222. DOI: http://dx.doi.org/10.1145/2854946.2854960
    Locate open access versionFindings
  • Tehmina N. Basit. 2003. Manual or electronic? The role of coding in qualitative data analysis. Educational Research 45, 2 (2003), 143–154. DOI: http://dx.doi.org/10.1080/0013188032000133548
    Locate open access versionFindings
  • Peter M Bednar and Christine Welch. 2009. Contextual Inquiry and Requirements Shaping. Information Systems Development Challenges in Practice Theory and Education Vols 1and 2 1 (2009), 225–236. DOI: http://dx.doi.org/10.1007/978-0-387-68772-8_18
    Findings
  • Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media Inc. 479 pages.
    Google ScholarLocate open access versionFindings
  • V. Braun and V. Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3 (2006), 77–101. DOI: http://dx.doi.org/10.1191/1478088706qp063oa
    Locate open access versionFindings
  • Liora Bresler, Judy Davidson Wasser, Nancy B Hertzog, and Mary Lemons. 1996. Beyond the lone ranger researcher: Team work in qualitative research. Research Studies in Music Education 7, 1 (1996), 13–27.
    Google ScholarLocate open access versionFindings
  • John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013. Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement. Sociological Methods & Research 42, 3 (2013), 294–320. DOI: http://dx.doi.org/10.1177/0049124113500475
    Locate open access versionFindings
  • Matt Chaput. 2016. Whoosh: A fast, pure-Python full text indexing, search, and spell checking library. (2016). https://pypi.python.org/pypi/Whoosh/
    Findings
  • Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide Through Qualitative AnalysisâAR. Sage Publications, Thousand Oaks, CA.
    Google ScholarFindings
  • Nan-chen Chen, Michael Brooks, Rafal Kocielnik, Sungsoo Ray Hong, Jeff Smith, Sanny Lin, Zening Qu, and Cecilia Aragon. 2017. Lariat: A Visual Analytics Tool for Social Media Researchers to Explore Twitter Datasets. In Hawaii International Conference on System Sciences. Honolulu, HI. DOI: http://dx.doi.org/10125/41382
    Locate open access versionFindings
  • Nan-chen Chen, Rafal Kocielnik, Margaret Drouhard, Vanessa Peña-Araya, Jina Suh, Keting Cen, Xiangyi Zheng, and Cecilia Aragon. 2016. Challenges of Applying Machine Learning to Qualitative Coding. In ACM SIGCHI Workshop on Human-Centered Machine Learning. http://hcml2016.goldsmithsdigital.com/program/
    Locate open access versionFindings
  • J W Creswell. 2007. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. SAGE Publications, Thousand Oaks, CA.
    Google ScholarFindings
  • Gregory Dam and Stefan Kaufmann. 2008. Computer assessment of interview data using latent semantic analysis. Behavior Research Methods 40, 1 (2008), 8–20. DOI:http://dx.doi.org/10.3758/BRM.40.1.8
    Locate open access versionFindings
  • J. Davidson, T. Paulus, and K. Jackson. 2016. Speculating on the Future of Digital Tools for Qualitative Research. Qualitative Inquiry 22, 7 (2016), 606–610. DOI: http://dx.doi.org/10.1177/1077800415622505
    Locate open access versionFindings
  • S Deerwester, S Dumais, G Furnas, T Landauer, and R Harshman. 1990. Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science 41, 6 (1990), 391–407.
    Google ScholarLocate open access versionFindings
  • N.K. Denzin and Y.S. Lincoln. 1994. Handbook of Qualitative Research. Sage Publications, Thousand Oaks, CA.
    Google ScholarFindings
  • Karthik Dinakar, Jackie Chen, Henry Lieberman, Rosalind Picard, and Robert Filbin. 2015. Mixed-Initiative Real-Time Topic Modeling & Visualization for Crisis Counseling. Proceedings of the 20th International Conference on Intelligent User Interfaces - IUI ’15 (2015), 417–426. DOI: http://dx.doi.org/10.1145/2678025.2701395
    Locate open access versionFindings
  • Karthik Dinakar, Henry Lieberman, Allison J B Chaney, and David M Blei. 2014. Real-time Topic Models for Crisis Counseling. (2014), 1–4.
    Google ScholarFindings
  • Robert Emerson, Rachel Fretz, and Linda Shaw. 2011. Writing Ethnographic Fieldnotes. University of Chicago Press, Chicago, IL.
    Google ScholarFindings
  • Jerry Alan Fails and Dan R Olsen. 2003. Interactive machine learning. Proceedings of the 8th international conference on Intelligent user interfaces IUI 03 (2003), 39–45. DOI:http://dx.doi.org/10.1145/604045.604056
    Locate open access versionFindings
  • Ethan Fast, Binbin Chen, and Michael S. Bernstein. 2016. Empath: Understanding Topic Signals in Large-Scale Text. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16 (2016), 4647–4657. DOI: http://dx.doi.org/10.1145/2858036.2858535
    Locate open access versionFindings
  • Nigel Fielding and Raymond Lee. 2002. New Patterns in the Adoption and Use of Qualitative Software. Field Methods 14, 2 (2002), 197–216. DOI: http://dx.doi.org/10.1177/1525822X02014002005
    Locate open access versionFindings
  • N. G. Fielding. 2012. Triangulation and Mixed Methods Designs: Data Integration With New Research Technologies. Journal of Mixed Methods Research 6, 2 (2012), 124–136. DOI: http://dx.doi.org/10.1177/1558689812437101
    Locate open access versionFindings
  • Linda S. Gilbert, Kristi Jackson, and Silvana di Gregorio. 2014. Tools for Analyzing Qualitative Data: The History and Relevance of Qualitative Data Analysis Software. In Handbook of Research on Educational Communications and Technology. 221–236. DOI: http://dx.doi.org/10.1007/978-1-4614-3185-5
    Locate open access versionFindings
  • Barney G Glaser and Anselm L Strauss. 1967. The discovery of grounded theory. International Journal of Qualitative Methods 5 (1967), 1–10. DOI: http://dx.doi.org/10.2307/588533
    Locate open access versionFindings
  • Tony Hak and Ton Bernts. 1996. Coder training: Theoretical training or practical socialization? Qualitative Sociology 19, 2 (jun 1996), 235–257. DOI: http://dx.doi.org/10.1007/BF02393420
    Findings
  • Enamul Hoque and Giuseppe Carenini. 2015. ConVisIT: Interactive Topic Modeling for Exploring Asynchronous Online Conversations. Proceedings of the 20th International Conference on Intelligent User Interfaces IUI ’15 (2015), 169–180.
    Google ScholarLocate open access versionFindings
  • Paula Jarzabkowski, Rebecca Bednarek, and Laure Cabantous. 2014. Conducting global team-based ethnography: Methodological challenges and practical methods. Human Relations (2014), 0018726714535449.
    Google ScholarLocate open access versionFindings
  • Udo Kelle, Gerald Prein, and Katherine Bird. 1995. Computer-aided qualitative data analysis: theory, methods and practice. Sage Publications. 224 pages.
    Google ScholarFindings
  • Raymond M Lee and Nigel G Fielding. 1995. Users’ Experiences of Qualitative Data Analysis Software. In Computer-aided qualitative data analysis: theory, methods and practice, Udo Kelle, Gerald Prein, and Katherine Bird (Eds.). SAGE Publications, London; Thousand Oaks, Chapter 2, 29–.
    Google ScholarFindings
  • R. B. Lewis. 2004. NVivo 2.0 and ATLAS.ti 5.0: A Comparative Review of Two Popular Qualitative Data-Analysis Programs. Field Methods 16, 4 (2004), 439–464. DOI: http://dx.doi.org/10.1177/1525822X04269174
    Locate open access versionFindings
  • Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. 2008 1, c (2008), 496. DOI: http://dx.doi.org/10.1109/LPT.2009.2020494
    Findings
  • Joseph A Maxwell. 2010. Qualitative Research Design: An Interactive Approach. SAGE Publications, Thousand Oaks, CA.
    Google ScholarFindings
  • M B Miles, A M Huberman, and Johnny Saldaña. 2014. Qualitative Data Analysis: A Methods Sourcebook. Thousand Oaks, CA.
    Google ScholarFindings
  • Peter Axel Nielsen. 2012. Collaborative Coding of Qualitative Data. (2012).
    Google ScholarFindings
  • Pablo Paredes, Ana Rufino Ferreira, Cory Schillaci, Gene Yoo, Pierre Karashchuk, Dennis Xing, Coye Cheshire, and John Canny. 2017. Inquire: Large-scale Early Insight Discovery for Qualitative Research. In Proceedings of the 20th ACM Conference on Computer-Supported Cooperative Work & Social Computing. DOI: http://dx.doi.org/10.1145/2998181.2998363
    Locate open access versionFindings
  • Mary Elizabeth Raven and Alicia Flanders. 1996. Using contextual inquiry to learn about your audiences. ACM SIGDOC Asterisk Journal of Computer Documentation 20, 1 (1996), 1–13. DOI: http://dx.doi.org/10.1145/227614.227615
    Locate open access versionFindings
  • Johnny Saldaña. 2012. The Coding Manual for Qualitative Researchers. Sage Publications, Thousand Oaks, CA.
    Google ScholarFindings
  • Bruce Sherin. 2013. A Computational Study of Commonsense Science: An Exploration in the Automated Analysis of Clinical Interview Data. Journal of the Learning Sciences 22, 4 (2013), 600–638. DOI: http://dx.doi.org/10.1080/10508406.2013.836654
    Locate open access versionFindings
  • Amit Singhal. 2001. Modern Information Retrieval: A Brief Overview. Bulletin of the IEEEE Computer Society Technical Committee on Data Engineering 24, 4 (2001), 1–9.
    Google ScholarLocate open access versionFindings
  • Jonathan A Smith and Mike Osborn. 2015. Interpretative phenomenological analysis. In Qualitative Psychology: A Practical Guide to Research Methods. 54–80. DOI: http://dx.doi.org/10.1002/9780470776278.ch10
    Findings
  • Pontus Stenetorp, Sampo Pyysalo, Goran Topi, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012. BRAT: a Web-based Tool for NLP-Assisted Text Annotation. Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics (EACL ’12) Figure 1 (2012), 102–107. http://dl.acm.org/citation.cfm?id=2380921.2380942
    Locate open access versionFindings
  • David R Thomas. 2006. A general inductive approach for analyzing qualiative evaluation data. American Journal of Evaluation 27, 2 (2006), 237–246. DOI: http://dx.doi.org/10.1177/1098214005283748
    Locate open access versionFindings
  • Gregor Wiedemann. 2013. Opening up to big data: Computer-assisted analysis of textual data in social sciences. Historical Social Research 38, 4 (2013), 332–357.
    Google ScholarLocate open access versionFindings
  • Jasy Suet Liew Yan, Nancy McCracken, and Kevin Crowston. 2014. Semi-Automatic Content Analysis of Qualitative Data. In iConference Proceedings. DOI: http://dx.doi.org/10.9776/14399
    Locate open access versionFindings
Your rating :
0

 

Best Paper
Best Paper of CHI, 2018
Tags
Comments