Some Thoughts on Reviewing for Information Systems Research and Other Leading Information Systems Journals

Free AccessAboutSectionsView PDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked InEmail Go to SectionFree Access HomeInformation Systems ResearchVol. 34, No. 4 Editorial: Some Thoughts on Reviewing for Information Systems Research and Other Leading Information Systems JournalsSuprateek Sarker , Edgar A. Whitley , Khim-Yong Goh , Yili (Kevin) Hong , Magnus Mähring, Pallab Sanyal , Ning Su , Heng Xu , Jingjun David Xu , Jingjing Zhang , Huimin Zhao Suprateek Sarker , Edgar A. Whitley , Khim-Yong Goh , Yili (Kevin) Hong , Magnus Mähring, Pallab Sanyal , Ning Su , Heng Xu , Jingjun David Xu , Jingjing Zhang , Huimin Zhao Published Online:10 Nov 2023https://doi.org/10.1287/isre.2023.editorial.v34.n4IntroductionPeer review of research before publication is both an essential and an integral part of scientific knowledge production. For reputable journals, the peer review process distinguishes knowledge claims in journal articles from those in sources with unknown or varying veracity. The peer review process assures readers that the published work is credible (i.e., conducted in line with prescribed norms of research) and meets a certain threshold with respect to contributions and potential impact. Leading journals are perceived as such not only because the best research is submitted to them but also because of the efforts of the best reviewers and editors in evaluating and, when applicable, developing the initially submitted manuscripts1 to publishable form.The sustained quality of reviews is critical for journals such as Information Systems Research (ISR). With the number of submissions to ISR growing each year, as well as an explicit policy of encouraging and celebrating inclusive excellence (Sarker 2023), there is a need for more reviewers for the journal (and the discipline, more generally) who have the necessary expertise to evaluate submitted papers, who understand and are attuned to the norms of the different traditions and genres of work submitted, and who know how to craft reviews that ensure the review process supports effective knowledge production.In this editorial, we draw on the expertise of some of the experienced associate editors (AEs) at ISR2 who represent different research traditions to provide guidance on how ISR reviewers can contribute reviews that AEs and authors are likely to find valuable. The primary audience of this editorial is Ph.D. students and early career scholars who occasionally review for, or seek to review for, ISR and similar journals. Although experienced reviewers likely know most of what we will say in the next few pages, we are hopeful that the editorial can provide a useful recapitulation of characteristics of reviews that are appreciated by ISR editors, irrespective of the reviewers’ experience. Finally, revisiting what reviewers look for in manuscripts can prove helpful for authors submitting papers to journals such as ISR.Before proceeding, we would like to acknowledge the efforts of editors and editorial board members from various journals who have organized reviewer development workshops (e.g., Rai 2019, Whitley 2023), and reflections on the review process and effective reviews by notable scholars in our discipline (e.g., Lee 1995; Saunders 2005a, b; Straub 2009; Kohli and Straub 2011; Davison 2015; Rai 2016; Leidner et al. 2022); see Table 1. Our editorial does not seek to supplant this accumulated wisdom but seeks to add nuances to the various guidelines that have been offered in the past. We illustrate key points with examples from various research traditions.Table 1. Selected Articles Providing Reviewing Guidelines for Information SystemsTable 1. Selected Articles Providing Reviewing Guidelines for Information SystemsLee AS (1995) Reviewing a manuscript for publication. J. Oper. Management 13(1):87–92.Saunders C (2005a) Editor’s comments: From the trenches: Thoughts on developmental reviewing. MIS Quart. 29(2):iii–xii.Saunders C (2005b) Editor’s comments: Looking for diamond cutters. MIS Quart. 29(1):iii–viii.Straub D (2009) Editor’s comments: Diamond mining or coal mining? Which reviewing industry are we in? MIS Quart. 33(2):iii–viii.Kohli R, Straub D (2011) Editor’s comments: How reviews shape “MIS Quarterly”: A primer for reviewers and editors. MIS Quart. 35(3):iii–vii.Davison RM (2015) The art of constructive reviewing. Inform. Systems J. 25(5):429–432.Rai A (2016) Writing a virtuous review. MIS Quart. 40(3):iii–x.Leidner DE, Carte T, Chatterjee S, Chen D, Jones M, Preston D (2022) On civil critique: Reviewing for JAIS. J. Assoc. Inform. Systems 23(1):1–12.Why Does Academia Have Peer Review?Peer review is the process by which academia ensures the quality of work published in research journals such as ISR. The quality of scholarly work is not easy to pin down (see, e.g., Agarwal 2012), given the many research traditions and associated standards across and even within research traditions (see, e.g., Rai 2017, Sarker et al. 2018).When peer review is minimal or missing, the reader cannot be confident about the published findings, and the ideas are often poorly developed and presented. Moreover, a minimal review process does not contribute to advancing scholarship. In extreme cases of so-called predatory journals, the only activity that takes place before an article is published is payment of a publication fee (Safi 2014).One way to evaluate academic quality is offered by Gupta (2018), who builds on Ellison’s (2002) distinction of two dimensions of the quality of an academic paper. The q dimension relates to the quality of the idea or underlying message of the paper, whereas the r dimension reflects the quality of the execution of the paper. The peer review process can perform a number of key roles along both dimensions.The review process can improve the q dimension of the paper by helping to articulate the novelty of the contribution, although there are limits to how much the review process can do with respect to the fundamental ideas behind the study. Reviewers often need to make a judgment call regarding the adequacy of a manuscript on the q dimension.With respect to the r dimension, the review process ensures that the published research has no fundamental flaws that affect its contribution to knowledge. Thus, the review process might result in the authors undertaking additional analysis or gathering additional evidence to ensure that the findings accurately reflect the phenomena being examined. The r dimension may also include a cohesive presentation of the findings. This can range from clarifying the form of argument being followed, to contextualizing the work in the broader historical literature more effectively, to helping the authors articulate their contribution more clearly and persuasively (Sarker 2023, p. 2).When selecting reviewers for a manuscript, editors typically look for a mixture of expertise. Some reviewers may be specifically invited to focus on the q dimension: to assess the big picture and the profound implications the paper might have. Other reviewers might be chosen to assess and buttress the r dimension because of their familiarity with the specific techniques used in the paper or the phenomena under investigation. Doctoral students and early career faculty, having recently completed state-of-the-art methodological training, tend to have expertise that helps in assessing and strengthening the r dimension. Their reviews also tend to place a larger emphasis on the r dimension, sometimes without considering the q dimension.A variety of images have been used in relation to the objective of the review process. A traditional one is that of the review process playing a “gatekeeper” role, but those of “diamond cutter” and “champion” are also often mentioned (Sarker et al. 2015). For example, Saunders (2005b, p. iii) suggests that rather than acting as gatekeepers to publication in a journal, the review team should play a diamond cutter’s role, working with authors “in polishing manuscripts so that the gem can surface and shine.”The AEs who coauthored this editorial echoed many of these general points. For example, one AE saw their role as similar to the “coach” of a football team who manages and mentors a team of budding superstars to win games. Another saw their role as being a “coordinator, facilitator, helper, tutor, and advisor.” Yet another saw AEs as “developmental gatekeepers,” whereas a fourth saw editors and reviewers as “stewards” with responsibility for helping the research become the best it can be, thus creating value for a broad range of stakeholders: society, scholars, practitioners, and other consumers of research as well as authors, journals, and the academic community.Notwithstanding the positive imagery, Rai (2016, p. x) encourages reviewers to differentiate “between issues that are fatal, showstoppers” and those that are “not necessarily fatal.” Along similar lines, Saunders (2005b, p. iii) reminds us that “expert diamond cutters can camouflage some errors in the diamond” but also reassures us that most “fatal flaws are impossible to conceal.” The key point is that reviewing involves a thoughtful balancing act between being constructive and charitable on the one hand, and not being oblivious to fatal flaws and offering incisive critique based on relevant criteria on the other hand. This balancing act, when done effectively by the editorial team guided by the SE and the AE, ensures that type I errors (accepting papers that should have been rejected) and type II errors (rejecting papers that should have been accepted) in the review process are minimized (Kohli and Straub 2011).In short, the peer review process for leading journals is designed to provide assurance to the reader that articles published in the journal adequately address both q and r dimensions. This ensures the high standards of the journal and each article published in it. In disciplines such as information systems (IS), the peer review process also helps the authors construct and present their work in the most consumable and potentially impactful manner. The entire process needs to be undertaken in a professional manner, where objective, even critical, assessments are welcome and are generally desirable. However, the feedback has to be provided in a considerate and constructive manner so that, even if the manuscript is rejected, the authors can learn from the process and can improve the manuscript for submission elsewhere (Kohli and Straub 2011).This emphasizes the importance of being civil in the review process (Leidner et al. 2022). This is particularly important because ours is a relatively small discipline, and we see ourselves as members of a close-knit global community. Lee (1995, p. 87) suggests that academia is at its best when “reviewers rise to the occasion and give extensive help, even though the anonymous reviewing process promises them nothing in return for their efforts.” This is in marked contrast to the behavior where reviewers offer “negative remarks that they would not have the courage to voice in public” (Lee 1995, p. 87) because they can hide behind the anonymous review process.Why Review for ISR and Other Prominent Journals?Although the benefits of peer review are apparent to the authors of published papers and the readers of the journals, the incentives for reviewing are often not obvious. This can be problematic as reviewers need to expend significant effort on reviewing.For example, Bannister and Janssen (2019, p. 1) note rather bluntly that “reviewing can sometimes seem to be one of the least valued of tasks. Reviews are not published. They will never be cited. Usually, they will neither earn you promotion nor any recognition beyond that of a small circle of grateful editors and associate editors (AEs) and, occasionally, your co-reviewers.” According to Kohli and Straub (2011, p. iii), reviewers are the “Good Samaritans who remain anonymous.”Goes (2014, p. v) suggests that the act of reviewing manuscripts achieves a twofold objective, combining facets of “love and glory.” The element of “love” is manifested in the form of appreciation (often not visible) from authors who benefit from insightful and constructive feedback, as well as from AEs and SEs, who rely on these evaluations to make informed decisions on the manuscript’s suitability for publication. The “glory” aspect provides a venue to build and solidify academic standing within the scholarly community. This recognition can manifest itself in several ways: it can elevate one’s stature among senior colleagues in the discipline, making one a prime candidate for accolades such as Best Reviewer awards, and even pave the way for invitations to join prestigious editorial boards or highly visible conference program committees. Or, as Lee (1995) notes, the professional relationships that arise from working closely with journal editors can result in leading scholars in the field writing letters of support to the reviewers’ promotion and tenure committees (see also Rai 2016, p. iv).Lee (1995, p. 92) outlines a number of additional benefits of being a reviewer. First, he notes that “Doing a review … confers an insider’s view of the reviewing process. The reactions of the other reviewers and the editor all contain potential lessons for one’s own manuscripts to be submitted for publication. In reviewing manuscripts, one also gains access to invaluable bibliographies”. Second, Lee sees reviewing as a “socially significant gesture” that reciprocates “some of the help” received from colleagues who have been supporters, job contacts, or external letter writers. Finally, Lee (1995, p. 92) believes that participation in the review process allows for the opportunity to champion work, especially those related to marginalized research traditions, and inform those involved in the review/editorial process about the merits and appropriate standards—especially those who may be “hostile to and ignorant of the research traditions.”For researchers aspiring to benefit from participating in the review management process for papers aligned with their areas of expertise, the easiest way to do so is to create an account in the review management system that the journal uses (https://mc.manuscriptcentral.com/isr for ISR). These user accounts are commonly created when a researcher submits a paper to the journal but it is also feasible to create an account before submitting any papers, thus setting oneself up as a prospective reviewer for the journal. Beyond providing basic demographic and contact information, the researcher can specify their areas of expertise. These expertise keywords are often used by the AEs when they are looking for specialist reviewers for particular papers. Proactive engagement with relevant AEs—by informing them of one’s presence in the system and willingness to undertake review assignments—is another option. If a researcher is not registered in the review management system, they may still be invited to review a paper (and added to the system) by an AE who is already aware of their expertise.Although setting up an account is a good first step, and offering services to AEs with similar research interests can help, we should mention that, owing to the stature of the journal, the AEs are very selective in who they invite to serve as reviewers. Their invitations often hinge on the publication records of the potential reviewer in similar-quality journals, as well as past interactions at conferences and previous review processes at ISR and other journals.The Review Process—An OverviewEach manuscript undergoes initial screening at the editor-in-chief (EIC) and the managing editor (ME) level and, if it passes the screening, is then assigned to a senior editor (SE). The SE then checks the manuscript and identifies a suitable AE. The AE examines the manuscript and determines, in consultation with the SE, whether the manuscript should go out for review. If it is deemed suitable for review, the AE typically invites two or three reviewers for the manuscript.For ISR and many IS journals, the review process is double-blind. Reviewers typically do not know who the authors of the paper are, and the authors do not know who the reviewers are. This anonymity should be maintained throughout the review process to avoid perceptions of undue influence. The double-blind review process is intended to help ensure that the identities of the authors and reviewers do not influence it (positively or negatively).Reviews and DecisionsISR operates a tiered review management process with SEs making the final decisions on a submission based on their own reading of the manuscript and the recommendations made by AEs. The AEs typically base their recommendations on their independent reading of the manuscript along with the recommendations by the reviewers. Once the SE makes the decision, a decision letter is generated for the authors with the SE report, the AE report, and the reviews included. Reviewers will normally receive a blinded copy of the decision letter sent to the authors. They will also be thanked for their service as a reviewer, sometimes personally by the AE and SE, particularly if their review is truly outstanding.To learn from the review process, especially if you are relatively early in your career, we suggest you go over the entire package when a decision is made on the paper that you were a reviewer for. What did the other reviewers say? Did they have concerns and observations similar to those you had? Did they provide guidance or solutions to problems that you identified in your review? Did the AE use or refer to your comments in their report? Did the SE use or refer to your comments in their report? Did they agree or disagree with what you had to say?Please do not be disappointed if your recommendation (say “reject”) does not match the final decision of the SE. As a reviewer, you are one part of the process that decides the fate of the paper, and your review will typically have been very important for shaping the editors’ assessment of the paper, even if the editorial decision did not fully align with your recommendation.The SE and AE, who are typically more experienced and have the responsibility of publishing papers suitable for the journal, are informed by the reviews—they are not bound by reviewer recommendations. In particular, it is important to recognize that the editorial decision making is not a voting process. As an example, two “reject” recommendations and one “revise and resubmit” from the three reviewers may or may not result in the AE recommending to reject and/or the SE deciding to reject the manuscript.Additional Rounds of ReviewIf the paper is invited for resubmission after revisions, you (a reviewer in the earlier round) may be invited to review the revised version of the manuscript. At this point, you will need to read the reviews (including your own) from the previous round, the AE and SE reports, the response document that authors submit showing how they have addressed all the comments, and the revised manuscript. Then you will write reviews for this version of the paper.You will find that some of the issues that were pointed out in the earlier round have likely been resolved, some have been addressed but not to your satisfaction, some may have been countered by the authors, and, finally, new issues may have emerged. As a reviewer, although you do not have to necessarily agree with the AE’s and SE’s views regarding the (lack of) importance of some concerns that you had raised in the earlier round, it is important to consider their views carefully. If you feel very strongly about your concern, you may politely explain why you feel the issue is (still) critical. You may choose a confidential communication channel in the review management system with the AE/SE for this purpose, or reach out to the AE through email. This can also help to ensure there is no misunderstanding about these issues before you submit your report.Several AEs highlighted the importance of consistency across rounds of reviews. For example, one AE mentioned that reviews offering contradictory or changing suggestions in different rounds (they referred to this as “oscillations”) were a source of frustration for not only the authors but also the AEs and SEs.Collectively, the multiple rounds of review enable the manuscript to evolve and mature, and oftentimes, gradually reach closure. Thus, AEs urge reviewers to be conscious about not adding new layers of onerous concerns in each round, or asking for new studies to be conducted during the revision rounds unless they are critical for the work to stand. Here is an example where the reviewer acknowledges the progress of the manuscript, but seeks additional clarifications regarding some of the material added in the revised version (which is altogether appropriate):In this revision … the additional literature review, clarifying the constructs and hypotheses in the theoretical model, and the description of the empirical validation are well-received. … The earlier reviewer issues I had for this paper in areas of practical implications, empirical validation, and limitations of the empirical validation were all addressed in this round of revision. The hypotheses and the practical implications of this paper are clearer. That said, I have two more comments … (1) The most interesting finding is … The authors are suggested to elaborate on this finding and provide concrete cases or examples to illustrate … (2) on page…, the authors discuss the first theoretical contribution is … I cannot see where this theoretical contribution comes from. Please clarify the linkage between this conclusion and the supporting empirical findings. Also, what do you mean by … Do you mean…? Or…? These are missing in the discussion section.In addition, one of the AEs indicated that it is generally inappropriate to raise concerns of a fundamental nature, such as issues with the selection of cases or data samples, identification strategies, improper matching of treatment and control groups, or potential endogeneity issues, in later stages of the review process if these issues were already evident in earlier submissions. If evident, they really need to be brought up as soon as they are spotted. Obviously, in some cases, the significance of these factors becomes apparent only as the overall argument made by the authors becomes clearer. In such cases, these (emergent) concerns should be brought to the attention of the AE and SE through the review. However, wherever possible, flagging these issues in the initial rather than advanced rounds makes the whole process smoother, and fairer to the authors.Your Involvement in the Review ProcessHaving briefly described the overall review process, we now outline some good practices to follow as a reviewer.You Have Been Invited to Review, What Is the First Thing to Do?We recommend that you acknowledge and accept (or decline) the invitation as soon as possible. If the manuscript is completely outside your area of expertise,3 you perceive a conflict of interest (e.g., you know who the authors are, and feel that your participation in the process may weaken the integrity of the review process4), or your schedule simply does not permit you to take on the responsibility of submitting the review by the deadline, please write to the AE to explain the situation and, as applicable, ask to be excused from the review or request an extension (a week or so is usually fine). If you are unable to review for any reason, you may recommend a colleague who you believe has suitable qualifications and will do a good job. Please note that if you do not respond to the AE’s invitation promptly, you hold up the review process, which may result in the authors hearing back from the journal late, thereby losing precious time, which may have an impact on their career progression.As well as accepting invitations to review in a timely manner it is, of course, important to submit your review within the period indicated in the invitation to review that you accepted. Occasionally, external events will affect your ability to deliver the review on time, in which case you should inform the AE as soon as possible and renegotiate a new deadline. In some cases, the AE will need to move ahead without your review or will need to recruit an alternate reviewer.Practicalities Around Writing and Submitting Your ReviewWe recommend that you write your review in a word processor (or other program) and upload or paste the review text into the review management system. This can prevent the loss of the entire review that you may have typed if the website or browser experiences difficulty and shuts down. Alternatively, you can also submit your review as a Word or PDF document, especially if the review report contains mathematical equations or formulae that require specific formatting.Alongside the main review, you may also (optionally) provide brief comments to the editors which will not be shared with the authors. In these confidential comments, you may mention your overall impression about the manuscript, express serious reservations that you have (if any), and so on.However, your comments to authors should be consistent with the comments to the editors. Sometimes, AEs are left facing a perplexing situation where the review to be shared with the authors is positive, whereas the confidential comments to the editors are extremely negative (or vice versa), making it difficult to ascertain the reviewer’s overall assessment of the paper.Some Attributes of a Helpful Review from the AEs’ PerspectiveThe IS community has produced a number of excellent guidelines on the review process (see Table 1). We urge you to look at them. In this section, we complement these existing guidelines with some of the key issues—ethicality, approach, structure, and technique—highlighted by the AEs we invited to be part of this editorial. Among other inputs, they provided examples of suitably disguised reviewer comments from reviews of papers that they had managed in recent years. Excerpts from some of these sample reviews are included, with suitable modifications.Ethicality: Uphold Highest Standards of Ethics in the Review ProcessWe may regard our own behaviors, as reviewers of manuscripts in the “double-blind” reviewing process, to be a manifestation of the values that we hold as members of the community of scholars. (Lee 1995, p. 87)Given the role that peer review plays in ensuring that the published research is of the highest quality possible, it is important that we behave ethically in the review process. The kinds of unethical behavior that can affect peer review range from the conflation of personal bias with legitimate critique to deliberate attempts to subvert the integrity of the whole peer review process. Another concern arises when reviewers perceive the journal as favoring specific methodological or theoretical orientations and use this as a basis to reject papers. Such actions can often stem from reviewers not being clear about the journal’s editorial objectives.Although such biases cannot be eliminated from the review process, they must be actively managed to not unduly skew the process. As a reviewer, you must reflect on your own biases and strive to transcend them when writing a review.Sometimes, as a reviewer, you may disagree with the authors’ norms and values that underpin their research. This is to be expected, indeed, as one AE noted:The academic community would not be as interesting and valuable as it is today if we all agreed on everything.Such differences in norms and values will likely influence your assessment of the quality of the research (or open up additional areas for improvement of the work). However, it is unethical to disguise this intellectual disagreement by wrongly attributing a reject recommendation to other broad-brush reasons, such as a “lack of theoretical contribution” or a “sample size” issue. Rather, we advise that you disclose your own perspective and respectfully argue your point, while also respecting the authors’ perspective as they write their paper, even though it differs from your own.The kinds of unethical behaviors most threatening to peer review integrity are those that are hidden, including attempts to “game” the review process, for example, through quid pro quo arrangements where reviewers will be more supportive of particular authors’ papers in the expectation that they will reciprocate this support when reviewing the reviewers’ own papers. If you suspect attempts at gaming the review process, you should raise this with the AE immediately.We mentioned above the need to disclose conflicts of interest to the AE upon receiving a review request, as well as during any point of the review process when realizing that a conflict exists. This is an ethical duty of reviewers, and one AE gave a compelling illustration of a response to an invitation to review where the reviewer took a strong ethical stance:I declined the review because recently I’ve discovered who the authors were from a copy of the paper on SSRN. I know one of the authors. As a consequence, I don’t think it is appropriate for me to review this paper because I may have some positivity bias towards the work.Upholding high standards of ethics also includes being transparent about your limitations as a reviewer. For example, in the (common) situation where you do not have the expertise to authoritatively comment on every aspect of a manuscript that you have been invited to review, it is helpful to state what aspects of the manuscript you feel qualified to comment on. This may allow the AE and SE to decide whether they need to invite an additional reviewer to assess those aspects of the paper that you are unable to evaluate or how much weight your comments should carry regarding a particular aspect of