Transparency in Practice in Qualitative Research

The discipline of political science has been engaged in discussion about when, why, and how to make scholarship more transparent for at least three decades. This piece argues that qualitative researchers can achieve transparency in diverse ways, using techniques and strategies that allow them to balance and optimize among competing considerations that affect the pursuit of transparency.. We begin by considering the “state of the debate,” briefly outlining the contours of the scholarship on transparency in political and other social sciences, which so far has focussed mostly on questions of “whether” and “what” to share. We investigate competing considerations that researchers have to consider when working towards transparent research. The heart of the piece considers various strategies, illustrated by exemplary applications, for making qualitative research more transparent.


Introduction
The discipline of political science has been engaged in vibrant debate about research transparency for more than three decades. In the abstract, augmenting transparency implies the same steps in all types of political science scholarship: making the empirical information that underpins our work meaningfully accessible; elucidating how that information was collected or generated; and describing how the information was interpreted and/or analyzed. 2 Nonetheless, the way in which transparency is pursued, and the type and severity of the challenges that pursuing it presents, vary across research traditions.
Scholars who collect, generate, and draw on qualitative evidence in their work are relative newcomers to the debate about transparency. Their more vigorous engagement over the last decade has brought important new voices and viewpoints to the conversation and raised new issues and questions. In particular, the recently completed Qualitative Transparency Deliberations (QTD, www.qualtd.net/), directed by Tim Büthe and Alan Jacobs, represent a crucial step forward. Called for during the business meeting of the Qualitative and Multi-Method Research (QMMR) section of the American Political Science Association (APSA) during the 2015 APSA conference, the QTD involved 13 working groups (and hundreds of political scientists beyond those groups), that ultimately produced 13 thoughtful final reports, which were published on SSRN in early 2019. Among their many lessons, the deliberations demonstrated that practices for making scholarship more transparent, and the challenges that doing so poses, vary among forms of qualitative inquiry. 3 In this short piece, we first briefly review the literature on transparency in qualitative inquiry, describing what we see as its evolution. We then highlight some of the considerations that scholars take into account when deciding how and how much to pursue transparency. The heart of the piece comprises a survey of exciting, creative techniques that scholars are developing and pioneering to enhance the transparency of qualitative research. We believe these strategies can help scholars to elucidate research practices, clarify the empirical underpinnings of their work, and facilitate its evaluation as well as optimize across some of the competing considerations that impinge on achieving transparency. The diversity of these emerging strategies demonstrates that transparency is not an all or nothing proposition, and can be pursued in many different ways. The conclusion summarizes and offers thoughts on the way forward.

State of the Debate
Over the last decade, political scientists who generate, collect, interpret, analyze, and publish scholarly work based on qualitative data have engaged in energetic dialogue about research transparency. 4 One way to characterize the arc of the debate is to suggest that it began with thoughtful consideration of "whether" scholars who employ qualitative data and methods can and should seek to make their work more transparent, progressed to the question of "what" information scholars should share about data production and analysis (and what data they should share) in pursuing transparency, and has just begun to consider "how" transparency can be achieved -what concrete techniques and strategies scholars can use to augment the transparency of their work.
We offer here a very general overview of the literature that addresses the first two of these questions; we take up the third in the fourth section of this paper, as it is to this conversation that we wish to contribute. The literature is rich and extensive, including interventions by scholars in various academic disciplines (e.g., political science, education, health, and sociology), and by practitioners in Information Schools, university libraries, and data repositories, and has been written by scholars from around the globe. Our review is by necessity incomplete given the length limitations on this piece. 5 The question of whether political scientists who engage in qualitative research can and should make their work more transparent mainly played out (in written form) in a series of journal symposia published since 2010. 6 In the opening pieces to these symposia (e.g. Büthe and Jacobs 2015;Golder and Golder 2016), and in contributions to those symposia, scholars have discussed the intellectual benefits -for producers and consumers of scholarship based on qualitative data and methods -of making such research more transparent, and the epistemological, ethical, legal, and practical challenges of doing so. In addition to the contributions to these symposia, several stand-alone pieces in various disciplines have taken on these questions, with some advocating for transparency (e.g., Corti 2006;Elman, Kapiszewski, and Lupia 2018;Gleditsch and Janz 2016;Miguel et al. 2014) and others registering concerns (e.g. Monroe 2018;Schwartz-Shea and Yanow 2016;Tripp 2018;Tsai et al. 2016); there are also pointed exchanges (e.g., Bishop 2005;Mauthner 2004, 2005).
The issue of what information about data generation and analysis, and which data, scholars should share has likewise been considered in a range of written work. The QTD advanced the debate in productive ways, offering novel insights on the meaning and "content" of transparency. In their overview piece, Jacobs et al. provide useful lists of information that can be shared to increase transparency (2019,(25)(26)(27), and various reports (e.g., Elkins, Spitzer, and Tallberg 2019; Schneider, Vis, and Koivu 2019, and others mentioned elsewhere in this piece) offer substantial guidance about what can be shared at low cost, low risk, and/or efficiently to achieve transparency. A few additional examples of work considering this question are Barnes and Weller's (2017) discussion of what information can elucidate analytic processes in process tracing work, and Tuval-Mashiach's (2017, suggestion that scholars answer three reflective questions in pursuit of transparency (what they did and how and why they did it).

Competing Considerations
Scholars -those who conduct qualitative and quantitative research -weigh and balance various factors and pressures as they consider how to make their work more transparent. Among these are two obligations whose fulfilment trumps other aims: pursuing transparency ethically (see, e.g., Carusi and Jirotka 2009; Chauvette, Schick-Makaroff, and Molzahn 2019) and legally. One example with regard to the first concerns the imperative to obtain informed consent from people a scholar involves in her research to share information they convey. Scholars and participants must reach 5 Over time we have gathered a large collection of the literature addressing transparency in qualitative research in a Zotero library that we have made public at https://www.zotero.org/groups/2379934/items; we invite readers to consult the collection. unambiguous agreement, ideally through a consultative process, on what information can be shared, when, where, how, and with whom, and scholars must adhere strictly to those agreements, without compromise. 7 An example related to pursuing research legally is scholars' inability to share work that is under copyright if permission to do so cannot be secured. We discuss these issues in more detail in the next section.
Scholars also consider various other factors when deciding how to pursue research transparency. These include (but are not limited to) personal intellectual considerations (how to pursue transparency in ways that will showcase the rigor and power of research, e.g., Fujii 2016, 25-26); public intellectual considerations (how to pursue transparency in ways that will showcase the rigor and power of research, e.g., Elman and Lupia 2016, 51); resource considerations and opportunity costs (how much time and money making research transparent will require, e.g., Saunders 2014, 694-97); and logistical considerations (how practically possible it is to make work transparent, e.g., Moravcsik 2012, 36). The need to adhere to ethical and legal imperatives, and to consider all of these additional factors, add to the challenges of pursuing transparency.

Transparency in Practice
In this section we outline a set of techniques that scholars can use to make qualitative research more transparent (beyond the information offered in their publications' text and footnotes), within ethical and legal constraints and in view of the competing considerations they face given the particularities of their project. Our discussion draws on the literature on transparency in qualitative research and our experiences working with scholars pursuing transparency in their work -but other exciting and promising techniques are surely being developed and used. Scholars should consider which strategies they may employ before beginning research, as their choices have implications for how they track the research process as they carry out their work.

Pre-registration
Pre-registration entails specifying a research project's rationale, hypotheses, design, and plan for data generation and analysis before initiating data collection. Interest in preregistration for qualitative work has been increasing (see, e.g., Haven and Grootel 2019; Jacobs forthcoming; Kern and Gleditsch 2017;Piñeiro and Rosenblatt 2016). 8 There are good reasons to be skeptical of the need and usefulness of preregistration in qualitative research given its often exploratory nature (see Haven and Grootel 2019, 6-8). Yet having a timestamped record of the original research and analysis plan and of changes made during the research process enhances transparency. A pioneering example of a preregistered qualitative case study is presented in Christensen, Hartman, and Samii (2019, 26-31), who use case study evidence to validate (and extend) quantitative findings.

Methodological Appendices
Methodological appendices discussing how an author collected, generated, and analyzed data can take multiple forms. 9 For instance, scholars who engage in ethnography or interpretive researchfor at least some of whom detailing "…data collection and analysis procedures, as well as my epistemological approach, has been a professional necessity…" (Cramer 2015, 18) -can offer information of this type in a methodological appendix to supplement what they convey in the spaceconstrained text of research articles. Reyes (2018) and Lester and Anders (2018) also discuss ethnographers' use of such appendices.
Scholars who conduct interviews can also increase the transparency of their work through including such appendices. A particularly impressive example is that provided by Shesterina, (2016) who describes in detail how she organized her fieldwork in Abkhazia, her interview settings and strategies, and how she recruited respondents and gained their trust. Bleich and Pekkanen (2013) propose a more formalized "Interview Method Appendix" comprising a descriptive list of all interviews conducted (including, e.g., the source of the interview contact, structure, and length).
The steps underlying archival research and process tracing likewise lend themselves to description in a methodological appendix. Scholars who conduct archival research typically keep a research log reflecting a full, ideally annotated, account of all sources consulted. Including in a methodological appendix a log describing how the primary historical sources cited in a publication were originally produced and how scholars selected them for inclusion can augment transparency, in particular of historically oriented research (see Gaikwad, Herrera, and Mickey 2019, 2). In addition, scholars who employ process tracing can generate online appendices to make analytic claims (and their role in an overall argument) more explicit (see, e.g., Bennett, Fairfield, and Soifer 2019, 9-10;Fairfield 2013).
Scholars whose work relies heavily on qualitative coding can enhance transparency by including with their publication a "coding appendix" detailing how they arrived at their initial coding, and resolved questions of inter-coder disagreements and refined that schema. In such methodological appendices or methodological companion pieces, for instance, Fuji Johnson et al. (2017)  Scholars in education research have pioneered two strategies for documenting the research process. Reflective journals allow scholars to record in detail the steps of the research process (Ortlipp 2008). Scholars with large research teams can periodically conduct "debriefing interviews" in which team members describe their decisions and actions throughout the research process (Collins et al. 2013).
Sharing such journals and interview transcripts as part of a methodological appendix could enhance transparency.
Finally and most generally, scholars might include as a methodological appendix information they assembled following "reporting guidelines" -checklists that set thresholds for information provision about data collection and analysis. While most often used in health-related areas, some guidelines should be familiar to political scientists, such as those on public opinion polls by the American Association of Public Opinion Research (AAPOR 2017). Guidelines also exist for in-depth interviews and focus groups (COREQ; Tong, Sainsbury, and Craig 2007); for synthesizing qualitative research (ENTREQ: Tong et al. 2012), and for qualitative research in general (SRQR; O'Brien et al. 2014). While they will need to be adapted in different ways by different political science research communities, these guidelines strike us as a potentially fruitful way to consider, organize, and systematize ideas about what should be shared in order to achieve transparency.

Annotation
Annotation can also help scholars to achieve transparency. Two forms of annotation that have been developed for political science inquiry are Active Citation (AC), which was pioneered by Moravcsik (e.g., Moravcsik 2014, forthcoming; see also Shesterinina, Pollack, and Arriola 2019, 23), and Annotation for Transparent Inquiry (ATI; www.qdr.org/ati) (see also Gaikwad, Herrera, and Mickey 2019, 15-17), developed by the Qualitative Data Repository (with which both authors are affiliated). ATI builds on AC, employing more sophisticated technology and placing additional emphasis on the value of sharing underlying data sources. ATI uses open annotation technology to allow researchers to link specific passages in a publication to digital annotations comprising "analytic notes" and extended excerpts from data sources, as well as to the data sources themselves when they can be shared ethically and legally (see Karcher and Weber 2019). Analytic notes can elucidate data generation or analysis, make the link between a source and a claim explicit, or discuss other aspects of the research process. Extended excerpts facilitate transparency even when sharing underlying data sources is not possible (Ellett 2016; and the discussion of this project in Shesterinina, Pollack, and Arriola 2019, 23). These annotations, plus an overview of the research process, comprise an ATI Data Supplement.
QDA Software Output Bringer, Johnston, and Brackenridge (2004) highlight how scholars can use qualitative data analysis (QDA) software, in particular the memo/note function, to provide an "electronic audit trail" of the research process and the development of a project. 10 Similarly, Sinkovics and Alfonsi (2012) argue, based on their work in international business and management, that QDA software can help capture the non-linear back-and-forth between data collection and analysis that is characteristic of much qualitative work, thereby improving its transparency and trustworthiness. QDA software can also provide a coherent image of the data as a whole. Corti and Gregory (2011) have long advocated for the sharing of QDA-generated qualitative data, and some researchers have shared excerpts from their QDA projects (see, e.g., Luigjes 2019;O'Neill 2017). While the proprietary nature of QDA file formats had stymied such efforts, an open exchange format for QDA data supported by the major software projects has recently emerged (www.qdasoftware.org), helping scholars to be transparent about the generation and analysis of data, and to share the QDA data themselves.

Data Sharing
Making accessible the data that underpin scholarship is a critical -and perhaps the most controversial -aspect of research transparency. We make three foundational points.
First, not all of the data that underlie a publication, and certainly not all of the data that were generated in the conduct of the research that led to the publication, need to be shared to achieve transparency. As noted earlier, ethical and legal constraints may limit what data can shared. An area of particular ethical concern is sharing information conveyed by individuals whom scholars involve in their research: if human participants do not consent to some or all of the information they offer in a research encounter being shared more broadly, it cannot be shared. A common legal challenge is determining whether documents or other work are under copyright or in the public domain. For example, Hitt (2019) shared papers that U.S. Supreme Court justices had dedicated to the public domain and can thus be freely shared. Scholars who wish to share copyrighted materials can petition the copyright owner for permission (see, e.g., newspaper articles shared in association with Holland (2019)). Should the owner refuse such requests or charge impractically high fees, those particular materials cannot be shared.
Second, data need not be shared in the exact form in which they were collected or generated. In order to protect human participants, for example, scholars may need to de-identify data, i.e., remove "direct identifiers" (pieces of information that are sufficient, on their own, to disclose an identity, such as proper names, addresses, and phone numbers) and "indirect identifiers" (contextual information that can be used, often in combination with other information, to identify a participant). 11 Contreras (2019, 11-16) explores three strategies for "partially disclosing" information about participants in dangerous research: semibiographical disclosure, partial spatial disclosure, and invitational disclosure (which involves inviting people to a field site to meet participants); see also Shesterinina, Pollack, and Arriola (2019, 15-16).
Third, sharing data does not mean making them "public". When data are shared in institutional venues, it is possible to limit the number or type of individuals to whom they are available by placing "access controls" on the data. For instance, Camp and Dunning (2015) shared transcripts of interviews with political brokers in Argentina, describing the general region but not the specific location where data were collected, and restricted access to the data to researchers with clearly specified research plans.

Conclusion: Continuing Forward
Debates about the challenges and benefits of research transparency, about the "content" of transparency, and about how precisely to achieve transparency in scholarly work, are proceeding across academic disciplines and geographies. Multiple innovative techniques have been developed to aid scholars to increase the transparency of their work within ethical and legal limits, and to help them balance and optimize among competing considerations that affect the pursuit of transparency. Their creation and use highlight that "transparency" is not an all-or-nothing prospect: most work is neither left completely opaque nor made completely transparent, but rather falls somewhere in between.
Indeed, it is important to keep in mind that transparency is a means to an end, not an end in itself. Transparency adds value by facilitating comprehension and assessment of our scholarship. The goal and necessity of assessment are in no way new: our research is assessed informally every day by individual scholars and more formally periodically through the peer review process. But increasing transparency can facilitate new ways to evaluate qualitative inquiry. What should new forms of evaluation look like? How can they be developed in ways that are appropriate for the epistemological commitments and methodological practices that make qualitative research the immensely powerful form of inquiry that it is?
It is critically important that scholars who employ qualitative data and methods continue to contribute to discussions of all of these topics (in oral and written form), to engage with each other within and across types of qualitative scholarship, and to listen to and learn from each other. Their ongoing involvement is crucial to the productivity of the conversation. Ultimately, however, we believe that the large and heterogeneous community of qualitative researchers will develop the best answers to the questions raised in this piece and the broader symposium by actively seeking to make their work more transparent, employing the techniques we discussed here and others that emerge. As they do so, research communities can draw on their examples to develop community-specific norms and practices for transparency, which funders, journal editors, and other institutions can then adopt. Continued conversation and engaged practice are both necessary for transparency to be deployed to its best purpose: to demonstrate the rigor of qualitative research and the value it delivers to political science.