Increasing the Transparency of Research Papers with Explorable Multiverse Analyses

Yvonne Jansen
Yvonne Jansen
Abhraneel Sarma
Abhraneel Sarma

CHI, pp. 652019.

Cited by: 15|Bibtex|Views50|Links
EI
Keywords:
explorable explanation interactive documents multiverse analysis statistics transparent reporting
Weibo:
We presented explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper

Abstract:

We present explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper itself. This approach draws from two recent ideas: i) multiverse analysis, a philosophy of statistical reporting where paper authors report the o...More

Code:

Data:

0
Introduction
  • The recent replication crisis in psychology and other disciplines has dealt a blow to the credibility of human-subject research and prompted a movement of methodological reform[70]
  • Much of this movement calls for more transparency in the way statistics are reported, so that findings become more trustworthy, more likely to be interpreted correctly, and easier to verify and replicate [29, 69, 72].
  • Undisclosed flexibility is damaging because it substantially increases the chances of reporting erroneous findings, while being invisible to the reader
Highlights
  • The recent replication crisis in psychology and other disciplines has dealt a blow to the credibility of human-subject research and prompted a movement of methodological reform[70]
  • Concern for transparency in statistical reporting has spread to the Human computer interaction community, which has published several articles [24, 31, 58] and hosted several workshops [56, 57, 96] on the topic
  • While there are many ways a statistical report can lack transparency, a common and damaging form of opacity is undisclosed flexibility, i.e., not reporting the different options that have been tried during the analysis [85, 98], or the options that would have been chosen had the data been different [40]
  • We presented explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper
  • Through examples and a design space analysis, we illustrated the many opportunities offered by emars, as well as the pending challenges
  • We hope our work will inspire more Human computer interaction research where the academic paper is treated as a user interface whose purpose is to convey scientific knowledge in an accurate and transparent manner
Conclusion
  • The authors first discuss limitations of the work, and the potential challenges involved in the adoption of emars.

    First of all, the design space is meant to capture elementary emar techniques, but many other more sophisticated techniques are possible to further enhance emars.
  • Other techniques and ideas could be borrowed from the domain of visual parameter space analysis, which shares many conceptual similarities with emars [82].The authors presented explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper.
  • Through examples and a design space analysis, the authors illustrated the many opportunities offered by emars, as well as the pending challenges.
  • The authors hope the work will inspire more HCI research where the academic paper is treated as a user interface whose purpose is to convey scientific knowledge in an accurate and transparent manner
Summary
  • Introduction:

    The recent replication crisis in psychology and other disciplines has dealt a blow to the credibility of human-subject research and prompted a movement of methodological reform[70]
  • Much of this movement calls for more transparency in the way statistics are reported, so that findings become more trustworthy, more likely to be interpreted correctly, and easier to verify and replicate [29, 69, 72].
  • Undisclosed flexibility is damaging because it substantially increases the chances of reporting erroneous findings, while being invisible to the reader
  • Conclusion:

    The authors first discuss limitations of the work, and the potential challenges involved in the adoption of emars.

    First of all, the design space is meant to capture elementary emar techniques, but many other more sophisticated techniques are possible to further enhance emars.
  • Other techniques and ideas could be borrowed from the domain of visual parameter space analysis, which shares many conceptual similarities with emars [82].The authors presented explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper.
  • Through examples and a design space analysis, the authors illustrated the many opportunities offered by emars, as well as the pending challenges.
  • The authors hope the work will inspire more HCI research where the academic paper is treated as a user interface whose purpose is to convey scientific knowledge in an accurate and transparent manner
Related work
  • We regard the research paper as an interactive medium and focus on how this medium can support the communication of multiverse analyses. We review prior literature on interactive documents and interactive statistical reports, and discuss the state of academic publication practices.

    Interactive Documents

    We use “document” to refer to any information artifact that is constructed around a textual narrative. Since the invention of hypertext [16], the HCI community has never stopped

    1available at https://explorablemultiverse.github.io/.

    to explore how interactivity can be used to enhance documents, e.g., for supporting annotation tasks [81, 101] or non-linear navigation within document content. Fluid documents [102, 103], for example, allow for supplemental content such as definitions and details to be revealed in-place and on demand. Document Cards [89] operate the opposite way by summarizing content into a set of curated figures in order to produce concise views that facilitate browsing of document collections. Elastic documents explore linking of text and tables to generated contextual visualizations [10]. Finally, explorable explanations are highly-interactive documents for which there exists a proof-of-concept toolkit, Tangle [94], and a comprehensive toolkit, Idyll [25], that was just recently released.
Funding
  • Presents explorable multiverse analysis reports, a new approach to statistical reporting where readers of research papers can explore alternative analysis options by interacting with the paper itself
  • Shows how combining those two ideas can complement existing reporting approaches and constitute a step towards more transparent research papers
  • A third approach, which proposes, consists of making the results section of the research paper interactive
  • Our goal with the present paper is to provide a better understanding of the design space of possible approaches
  • Reviews prior literature on interactive documents and interactive statistical reports, and discuss the state of academic publication practices
Reference
  • [n. d.]. Distill. https://distill.pub/. Accessed:2018-09-15.[2] [n.d.]. Jupyter.http://jupyter.org/. Accessed:2018-09-15.[3] [n.d.]. R Markdown Gallery.https://rmarkdown.rstudio.com/gallery. html. Accessed: 2018-09-15. [4] [n. d.]. Radix for R Markdown. https://rstudio.github.io/radix/. Accessed:2018-09-16.[5] [n.d.]. Shiny.https://shiny.rstudio.com/. Accessed:2018-09-15.
    Findings
  • [6] IJsbrand Jan Aalbersberg, Pilar Cos Alvarez, Julien Jomier, Charles Marion, and Elena Zudilova-Seinstra. 2014. Bringing 3D visualization into the online research article. Information Services & Use 34, 1-2 (2014), 27–37.
    Google ScholarLocate open access versionFindings
  • [7] Michael J Ackerman. 2015. The Educational Value of Truly Interactive Scientific Publishing. Journal of Electronic Publishing 18, 2 (2015).
    Google ScholarLocate open access versionFindings
  • [8] APA. 2010. The Publication manual of the APA (6th ed.). Washington, DC.
    Google ScholarFindings
  • [9] Christie Aschwanden. 201Science is not broken. https://fivethirtyeight.com/features/science-isnt-broken/. Accessed:201809-15.
    Findings
  • [10] Sriram Karthik Badam, Zhicheng Liu, and Niklas Elmqvist. 2019. Elastic Documents: Coupling Text and Tables through Contextual Visualizations for Enhanced Document Reading. IEEE Transactions on Visualization & Computer Graphics (2019). http://www.umiacs.umd.edu/~elm/projects/elastic-documents/elastic-documents.pdf
    Locate open access versionFindings
  • [11] David G Barnes and Christopher J Fluke. 2008. Incorporating interactive three-dimensional graphics in astronomy research papers. New Astronomy 13, 8 (2008), 599–605.
    Google ScholarLocate open access versionFindings
  • [12] Patrick Baudisch and Ruth Rosenholtz. 2003. Halo: a technique for visualizing off-screen objects. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 481–488.
    Google ScholarLocate open access versionFindings
  • [13] Patrick Baudisch, Desney Tan, Maxime Collomb, Dan Robbins, Ken Hinckley, Maneesh Agrawala, Shengdong Zhao, and Gonzalo Ramos. 2006. Phosphor: explaining transitions in the user interface using afterglow effects. In Proceedings of the 19th annual ACM symposium on User interface software and technology. ACM, 169–178.
    Google ScholarLocate open access versionFindings
  • [14] Michel Beaudouin-Lafon. 2000. Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 446–453.
    Google ScholarLocate open access versionFindings
  • [15] Daniel J Benjamin, James O Berger, Magnus Johannesson, Brian A Nosek, E-J Wagenmakers, Richard Berk, Kenneth A Bollen, Björn Brembs, Lawrence Brown, Colin Camerer, et al. 2018. Redefine statistical significance. Nature Human Behaviour 2, 1 (2018), 6.
    Google ScholarLocate open access versionFindings
  • [16] Timothy J Berners-Lee. 1989. Information management: A proposal. Technical Report.
    Google ScholarFindings
  • [17] Donald Berry. 2012. Multiplicities in cancer research: ubiquitous and necessary evils. Journal of the National Cancer Institute 104, 15 (2012), 1125–1133.
    Google ScholarLocate open access versionFindings
  • [18] Anastasia Bezerianos, Pierre Dragicevic, and Ravin Balakrishnan. 2006. Mnemonic rendering: an image-based approach for exposing hidden changes in dynamic displays. In Proceedings of the 19th annual ACM symposium on User interface software and technology. ACM, 159–168.
    Google ScholarLocate open access versionFindings
  • [19] Rolf Biehler. 1997. Software for learning and for doing statistics. International Statistical Review 65, 2 (1997), 167–189.
    Google ScholarLocate open access versionFindings
  • [20] Leen Breure, Hans Voorbij, and Maarten Hoogerwerf. [n. d.]. Rich Internet Publications:" Show What You Tell". Journal of Digital Information 12, 1 ([n. d.]).
    Google ScholarLocate open access versionFindings
  • [21] Jonathan B Buckheit and David L Donoho. 1995. Wavelab and reproducible research. In Wavelets and statistics. Springer, 55–81.
    Google ScholarFindings
  • [22] Dana R Carney, Amy JC Cuddy, and Andy J Yap. 2010. Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological science 21, 10 (2010), 1363–1368.
    Google ScholarLocate open access versionFindings
  • [23] Fanny Chevalier, Pierre Dragicevic, Anastasia Bezerianos, and Jean-Daniel Fekete. 2010. Using text animated transitions to support navigation in document histories. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 683–692.
    Google ScholarLocate open access versionFindings
  • [24] Andy Cockburn, Carl Gutwin, and Alan Dix. 2018. Hark no more: on the preregistration of CHI experiments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 141.
    Google ScholarLocate open access versionFindings
  • [25] Matt Conlen and Jeffrey Heer. 2018. Idyll: A Markup Language for Authoring and Publishing Interactive Articles on the Web. In ACM User Interface Software & Technology (UIST). http://idl.cs.washington.edu/papers/idyll
    Locate open access versionFindings
  • [26] Geoff Cumming. [n. d.]. Exploratory Software for Confidence Intervals. https://thenewstatistics.com/itns/esci/. Accessed:2018-09-15.
    Findings
  • [27] Geoff Cumming. 2002. Live figures: Interactive diagrams for statistical understanding. In Proceedings of the Sixth International Conference on Teaching of Statistics, Cape Town. Voorburg, The Netherlands: International Statistical Institute.
    Google ScholarLocate open access versionFindings
  • [28] Geoff Cumming. 2009. The dance of p-values (video). https://www.youtube.com/watch?v=ez4DgdurRPg
    Findings
  • [29] Geoff Cumming. 2014. The new statistics: Why and how. Psychological science 25, 1 (2014), 7–29.
    Google ScholarLocate open access versionFindings
  • [30] Geoff Cumming and Sue Finch. 2005. Inference by eye: confidence intervals and how to read pictures of data. American Psychologist 60, 2 (2005), 170.
    Google ScholarLocate open access versionFindings
  • [31] Pierre Dragicevic. 2016. Fair statistical communication in HCI. In Modern Statistical Methods for HCI. Springer, 291–330.
    Google ScholarFindings
  • [32] Pierre Dragicevic. 2017. Statistical Dances: Why no Statistical Analysis is Reliable and What to do About it (video). https://www.youtube.com/watch?v=UKX9iN0p5_A [33] Pierre Dragicevic.2018. Adding Inferential Information to Plots using Resampling and Animations. (2018).https://explorablemultiverse.github.io/examples/dance/
    Locate open access versionFindings
  • [34] Pierre Dragicevic. 2018. An Explorable Multiverse Analysis of Durante et al. (2013). (2018). https://explorablemultiverse.github.io/examples/dataverse/
    Locate open access versionFindings
  • [35] Pierre Dragicevic and Yvonne Jansen. 2018. Blinded with Science or Informed by Charts? A Replication Study. IEEE transactions on visualization and computer graphics 24, 1 (2018), 781–790.
    Google ScholarLocate open access versionFindings
  • [36] Pierre Dragicevic and Yvonne Jansen. 2018. Re-Evaluating the Efficiency of Physical Visualizations: A Simple Multiverse Analysis. (2018). https://explorablemultiverse.github.io/examples/frequentist/
    Findings
  • [37] Pierre Dragicevic, Gonzalo Ramos, Jacobo Bibliowitcz, Derek Nowrouzezahrai, Ravin Balakrishnan, and Karan Singh. 2008. Video browsing by direct manipulation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 237–246.
    Google ScholarLocate open access versionFindings
  • [38] Kristina M Durante, Ashley Rae, and Vladas Griskevicius. 2013. The fluctuating female vote: Politics, religion, and the ovulatory cycle. Psychological Science 24, 6 (2013), 1007–1016.
    Google ScholarLocate open access versionFindings
  • [39] Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics. Springer, 569–593.
    Google ScholarLocate open access versionFindings
  • [40] Andrew Gelman and Eric Loken. 2013. The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University (2013).
    Google ScholarLocate open access versionFindings
  • [41] Robert Gentleman and Duncan Temple Lang. 2007. Statistical analyses and reproducible research. Journal of Computational and Graphical Statistics 16, 1 (2007), 1–23.
    Google ScholarLocate open access versionFindings
  • [42] Roger Giner-Sorolla. 2012. Science or art? How aesthetic standards grease the way through the publication bottleneck but undermine science. Perspectives on Psychological Science 7, 6 (2012), 562–571.
    Google ScholarLocate open access versionFindings
  • [43] Roger Giner-Sorolla. 2012. Will we march to utopia, or be dragged there? Past failures and future hopes for publishing our science. Psychological Inquiry 23, 3 (2012), 263–266.
    Google ScholarLocate open access versionFindings
  • [44] Jonathan Gray, Lucy Chambers, and Liliana Bounegru. 2012. The data journalism handbook: How journalists can use data to improve the news. " O’Reilly Media, Inc.".
    Google ScholarFindings
  • [45] Quentin F Gronau, Henrik Singmann, and Eric-Jan Wagenmakers. 2017. Bridgesampling: an r package for estimating normalizing constants. arXiv preprint arXiv:1710.08162 (2017).
    Findings
  • [46] Quentin F Gronau, Sara Van Erp, Daniel W Heck, Joseph Cesario, Kai J Jonas, and Eric-Jan Wagenmakers. 2017. A Bayesian modelaveraged meta-analysis of the power pose effect with informed and default priors: The case of felt power. Comprehensive Results in Social Psychology 2, 1 (2017), 123–138.
    Google ScholarLocate open access versionFindings
  • [47] Tovi Grossman, Fanny Chevalier, and Rubaiat Habib Kazi. 2016. Bringing research articles to life with animated figures. interactions 23, 4 (2016), 52–57.
    Google ScholarLocate open access versionFindings
  • [48] Lane Harrison, Fumeng Yang, Steven Franconeri, and Remco Chang. 2014. Ranking Visualizations of Correlation Using Weber’s Law. IEEE Trans. Vis. Comput. Graph. 20, 12 (2014), 1943–1952.
    Google ScholarLocate open access versionFindings
  • [49] James Heathers. 2017. Life In The Tinderbox. Online. https://medium.com/@jamesheathers/life-in-the-tinderbox-6b2e9760f3aa.
    Findings
  • [50] Jessica Hullman, Paul Resnick, and Eytan Adar. 2015. Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering. PloS one 10, 11 (2015), e0142444.
    Google ScholarLocate open access versionFindings
  • [51] Yvonne Jansen, Pierre Dragicevic, and Jean-Daniel Fekete. 2013. Evaluating the efficiency of physical visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2593–2602.
    Google ScholarLocate open access versionFindings
  • [52] Yvonne Jansen and Kasper Hornbæk. 2018. How Relevant are Incidental Power Poses for HCI?. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 14.
    Google ScholarLocate open access versionFindings
  • [53] Alex Kale, Francis Nguyen, Matthew Kay, and Jessica Hullman. 2018. Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data. IEEE transactions on visualization and computer graphics (2018).
    Google ScholarLocate open access versionFindings
  • [54] Maurits Clemens Kaptein, Clifford Nass, and Panos Markopoulos. 2010. Powerful and Consistent Analysis of Likert-type Ratingscales. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10). ACM, New York, NY, USA, 2391–2394. https://doi.org/10.1145/1753326.1753686
    Locate open access versionFindings
  • [55] Matthew Kay and Pierre Dragicevic. 2018. A Multiverse Reanalysis of Likert-Type Responses. (2018). https://explorablemultiverse.github.io/examples/likert/
    Findings
  • [56] Matthew Kay, Steve Haroz, Shion Guha, and Pierre Dragicevic. 2016. Special interest group on transparent statistics in HCI. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1081–1084.
    Google ScholarLocate open access versionFindings
  • [57] Matthew Kay, Steve Haroz, Shion Guha, Pierre Dragicevic, and Chat Wacharamanotham. 2017. Moving transparent statistics forward at CHI. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 534–541.
    Google ScholarLocate open access versionFindings
  • [58] Matthew Kay, Gregory L Nelson, and Eric B Hekler. 2016. Researchercentered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4521–4532.
    Google ScholarLocate open access versionFindings
  • [59] Kris N Kirby and Daniel Gerlanc. 2013. BootES: An R package for bootstrap confidence intervals on effect sizes. Behavior research methods 45, 4 (2013), 905–927.
    Google ScholarLocate open access versionFindings
  • [60] Donald Ervin Knuth. 1984. Literate programming. Comput. J. 27, 2 (1984), 97–111.
    Google ScholarLocate open access versionFindings
  • [61] Hylke Koers, Ann Gabriel, and Rebecca Capone. 2013. Executable papers in computer science go live on ScienceDirect. Available at)(Accessed August 6, 2018) ElsevierConnect (2013).
    Google ScholarFindings
  • [62] David Koop, Emanuele Santos, Phillip Mates, Huy T Vo, Philippe Bonnet, Bela Bauer, Brigitte Surer, Matthias Troyer, Dean N Williams, Joel E Tohline, et al. 2011. A provenance-based infrastructure to support the life cycle of executable papers. Procedia Computer Science 4 (2011), 648–657.
    Google ScholarLocate open access versionFindings
  • [63] Daniel Kunin, Jingru Guo, Tyler Dae Devlin, and Daniel Xiang. [n. d.]. Seeing theory. https://students.brown.edu/seeing-theory/index.html. Accessed:2018-09-15.
    Findings
  • [64] Kristoffer Magnusson. [n. d.]. R Psychologist. http://rpsychologist.com/d3/CI/. Accessed:2018-09-15.
    Findings
  • [65] Emilie Marcus. 2009. Taming supplemental material. Immunity 31, 5 (2009), 691.
    Google ScholarLocate open access versionFindings
  • [66] Richard McElreath. 2018. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. CRC Press.
    Google ScholarFindings
  • [67] Amelia Ahlers McNamara. 2015. Bridging the gap between tools for learning and for doing statistics. Ph.D. Dissertation. UCLA.
    Google ScholarFindings
  • [68] Tara A Morgan, Adam E Flanders, William W Olmsted, Scott D Steenburg, and Eliot L Siegel. 2012. Challenges encountered and lessons learned in initial experience with the next generation of interactive radiology literature in RadioGraphics. Radiographics 32, 3 (2012), 929–934.
    Google ScholarLocate open access versionFindings
  • [69] Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christopher D Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J Ware, and John PA Ioannidis. 2017. A manifesto for reproducible science. Nature Human Behaviour 1, 1 (2017), 0021.
    Google ScholarLocate open access versionFindings
  • [70] Leif D Nelson, Joseph Simmons, and Uri Simonsohn. 2018. Psychology’s renaissance. Annual review of psychology 69 (2018).
    Google ScholarLocate open access versionFindings
  • [71] Deborah Nolan and Duncan Temple Lang. 2007. Dynamic, interactive documents for teaching statistical practice. International Statistical Review 75, 3 (2007), 295–321.
    Google ScholarLocate open access versionFindings
  • [72] Brian A Nosek, George Alter, George C Banks, Denny Borsboom, Sara D Bowman, Steven J Breckler, Stuart Buck, Christopher D Chambers, Gilbert Chin, Garret Christensen, et al. 2015. Promoting an open research culture. Science 348, 6242 (2015), 1422–1425.
    Google ScholarLocate open access versionFindings
  • [73] Daniel Nüst, Markus Konkol, Marc Schutzeichel, Edzer Pebesma, Christian Kray, Holger Przibytzin, and Jörg Lorenz. 2017. Opening the publication process with executable research compendia. D-Lib Magazine 23, 1/2 (2017).
    Google ScholarLocate open access versionFindings
  • [74] Steve Pettifer, Philip McDermott, James Marsh, David Thorne, Alice Villéger, and Terri K Attwood. 2011. Ceci n’est pas un hamburger: modelling and representing the scholarly article. Learned Publishing 24, 3 (2011), 207–220.
    Google ScholarFindings
  • [75] Victor Powell and Lewis Lehe. [n. d.]. Setosa. http://setosa.io/. Accessed:2018-09-15.
    Findings
  • [76] Jonathan C Roberts. 2007. State of the art: Coordinated & multiple views in exploratory visualization. In Coordinated and Multiple Views in Exploratory Visualization, 2007. CMV’07. Fifth International Conference on. IEEE, 61–71.
    Google ScholarLocate open access versionFindings
  • [77] Anthony Rossini and Friedrich Leisch. 2003. Literate statistical practice. (2003).
    Google ScholarFindings
  • [78] Abhraneel Sarma, Yvonne Jansen, and Matthew Kay. 2018. A Multiverse Analysis Considering Different Priors for Incidental Power Poses in HCI. (2018). https://explorablemultiverse.github.io/examples/prior/
    Findings
  • [79] Jeff Sauro and James R Lewis. 2010. Average task times in usability tests: what to report?. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2347–2350.
    Google ScholarLocate open access versionFindings
  • [80] Gnther Sawitzki. 2002. Keeping Statistics Alive in Documents. Computational Statistics 1, 17 (2002), 65–88.
    Google ScholarLocate open access versionFindings
  • [81] Bill N Schilit, Gene Golovchinsky, and Morgan N Price. 1998. Beyond paper: supporting active reading with free form digital ink annotations. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press/Addison-Wesley Publishing Co., 249–256.
    Google ScholarLocate open access versionFindings
  • [82] Michael Sedlmair, Christoph Heinzl, Stefan Bruckner, Harald Piringer, and Torsten Möller. 2014. Visual parameter space analysis: A conceptual framework. Visualization and Computer Graphics, IEEE Transactions on 99 (2014).
    Google ScholarLocate open access versionFindings
  • [83] David Shotton, Katie Portwin, Graham Klyne, and Alistair Miles. 2009. Adventures in semantic publishing: exemplar semantic enhancements of a research article. PLoS computational biology 5, 4 (2009), e1000361.
    Google ScholarLocate open access versionFindings
  • [84] Raphael Silberzahn, Eric Luis Uhlmann, Dan Martin, Pasquale Anselmi, Frederik Aust, Eli C Awtrey, Štěpán Bahník, Feng Bai, Colin Bannard, Evelina Bonnier, et al. 2017. Many analysts, one dataset: Making transparent how variations in analytical choices affect results. (2017).
    Google ScholarFindings
  • [85] Joseph P Simmons, Leif D Nelson, and Uri Simonsohn. 2011. Falsepositive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science 22, 11 (2011), 1359–1366.
    Google ScholarLocate open access versionFindings
  • [86] Uri Simonsohn, Joseph P Simmons, and Leif D Nelson. 2015. Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications. (2015).
    Google ScholarFindings
  • [87] Sara Steegen, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016. Increasing transparency through a multiverse analysis. Perspectives on Psychological Science 11, 5 (2016), 702–712.
    Google ScholarLocate open access versionFindings
  • [88] Rudolf J Strijkers, Reginald Cushing, Dmitry Vasyunin, Cees de Laat, Adam Belloum, Robert J Meijer, et al. 2011. Toward Executable Scientific Publications.. In ICCS. 707–715.
    Google ScholarLocate open access versionFindings
  • [89] Hendrik Strobelt, Daniela Oelke, Christian Rohrdantz, Andreas Stoffel, Daniel A Keim, and Oliver Deussen. 2009. Document cards: A top trumps visualization for documents. IEEE Transactions on Visualization and Computer Graphics 15, 6 (2009), 1145–1152.
    Google ScholarLocate open access versionFindings
  • [90] Kenji Takeda, Graeme Earl, Jeremy Frey, Simon Keay, and Alex Wade. 2013. Enhancing research publications using rich interactive narratives. Phil. Trans. R. Soc. A 371, 1983 (2013), 20120090.
    Google ScholarLocate open access versionFindings
  • [91] Stan Development Team. 2016. RStan: the R interface to Stan. R package version 2, 1 (2016).
    Google ScholarFindings
  • [92] George R Thoma, Glenn Ford, Sameer Antani, Dina Demner-Fushman, Michael Chung, and Matthew Simpson. 2010. Interactive publication: the document as a research tool. Web Semantics: Science, Services and Agents on the World Wide Web 8, 2-3 (2010), 145–150.
    Google ScholarLocate open access versionFindings
  • [93] Barbara Tversky, Julie Bauer Morrison, and Mireille Betrancourt. 2002. Animation: can it facilitate? International journal of human-computer studies 57, 4 (2002), 247–262.
    Google ScholarLocate open access versionFindings
  • [94] Bret Victor. 2011. Explorable Explanations. Online. http://worrydream.com/ExplorableExplanations/.
    Findings
  • [95] Bret Victor. 2011. Scientific Communication As Sequential Art. Online. http://worrydream.com/ScientificCommunicationAsSequentialArt/.
    Findings
  • [96] Chat Wacharamanotham, Matthew Kay, Steve Haroz, Shion Guha, and Pierre Dragicevic. 2018. Special Interest Group on Transparent Statistics Guidelines. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, SIG08.
    Google ScholarLocate open access versionFindings
  • [97] Chat Wacharamanotham, Krishna Subramanian, Sarah Theres Völkel, and Jan Borchers. 2015. Statsplorer: Guiding novices in statistical analysis. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2693–2702.
    Google ScholarLocate open access versionFindings
  • [98] Jelte M Wicherts, Coosje LS Veldkamp, Hilde EM Augusteijn, Marjan Bakker, Robbie Van Aert, and Marcel ALM Van Assen. 2016. Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology 7 (2016), 1832.
    Google ScholarLocate open access versionFindings
  • [99] Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA, 143–146. https://doi.org/10.1145/1978942.1978963
    Locate open access versionFindings
  • [100] Michael Wood. 2005. Bootstrapped confidence intervals as an approach to statistical inference. Organizational Research Methods 8, 4 (2005), 454–470.
    Google ScholarLocate open access versionFindings
  • [101] Dongwook Yoon, Nicholas Chen, and François Guimbretière. 2013. TextTearing: opening white space for digital ink annotation. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 107–112.
    Google ScholarLocate open access versionFindings
  • [102] Polle T Zellweger, Bay-Wei Chang, and Jock D Mackinlay. 1998. Fluid links for informed and incremental link transitions. In Proceedings of the ninth ACM conference on Hypertext and hypermedia: links, objects, time and space—structure in hypermedia systems: links, objects, time and space—structure in hypermedia systems. ACM, 50–57.
    Google ScholarLocate open access versionFindings
  • [103] Polle T Zellweger, Anne Mangen, and Paula Newman. 2002. Reading and writing fluid hypertext narratives. In Proceedings of the thirteenth ACM conference on Hypertext and hypermedia. ACM, 45–54.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Best Paper
Best Paper of CHI, 2019
Tags
Comments