What makes a good PRIMUS review?

Post by Matt Boelkins, Editor-in-Chief

The purpose of this post is to offer some guidance for referees regarding what is most helpful to authors and editors in the review process, including how we interpret terms like “major revision”.

At PRIMUS, we are invested in the overall project of helping students learn collegiate mathematics effectively; we do so through the articles we publish for our readers, who are primarily teachers of collegiate mathematics themselves.  As noted in our Aims and Scope, our focus is on pedagogical initiatives that range in scale of application from individual students and courses to curricula and entire programs.  Articles we publish typically present a novel idea, demonstrate evidence that the idea supports student learning, and show how the idea may be implemented or transferred to other settings.  In addition, we sometimes publish survey articles, exploration or assessment of particular pedagogical approaches, and accounts of programmatic endeavors that affect multiple courses or courses across institutions.  All articles should be aimed at an audience of practitioners:  how can others put these ideas into practice?

Teaching is one of the best kinds of problems:  it doesn’t have a “solution”, and new ideas and approaches often lead to more questions.  When we teach ambitiously, there are hundreds of choices to consider, and all sorts of new ideas that are generated.  At PRIMUS, we seek an ongoing conversation about these new approaches, ideas, and insights.  And to that end, we are committed to helping authors bring good ideas and insights to publication.  Even if an initial submission isn’t suitable for publication and may need considerable revision, we work with the author through the review process to help them improve and strengthen their paper so that we can publish it.  We want our readers to benefit from the wealth of the collective by providing a venue in which as many good new ideas as possible are widely shared.

In a typical year, we receive 120-150 new submissions, and roughly the same number of resubmissions after revision (almost every paper we ultimately accept is revised and resubmitted at least once in order to respond to reviewer feedback).  Our volunteer referees make an enormous positive difference by providing feedback to authors and helping make every paper we receive even better.  

In our instructions for authors, we share the referee form, which details the main categories that constitute every review:  Brief Synopsis, Audience, Exposition, Mathematical Content, Professionalism, Fit, and Overall Merit.  When a referee completes the form, they are simultaneously providing feedback to both the author and our editorial staff.  

For the author, the most helpful feedback the referee can give is that which helps make the paper stronger.  Some common ways that papers are improved through the review process are by helping the author better identify their primary audience and speak directly to them; better organize the manuscript so that the most important ideas get articulated early in the paper; and improve the exposition so that the paper’s communication is as clear as possible.  When we assign referees, we often think of the reviewer as a primary candidate for being among the paper’s target audience.  Thus, it’s helpful when a referee thinks explicitly about how they personally might use the paper in their own teaching, and then ask natural questions for how the paper might be better.

Some brief examples of common feedback that are helpful:

• Audience.  “While you are writing about an intervention you used in a geometry course, I can see a similar approach being used in any undergraduate mathematics course for majors.  I encourage you to clarify your audience to be wider, to emphasize that you’re demonstrating this approach through a geometry class, and to think about how you can offer practical advice to practitioners beyond just those who teach geometry.”
  
• Organization.  “In your manuscript, you delay the discussion of results and conclusions until nearly the end of the paper.  You should share at least a summary of those results early on to draw your reader in and make them want to know more details.  Your work has compelling outcomes, so it’s important that the reader not have to get through 10+ pages before learning them.”
  
• Exposition. “While your paper is well-organized overall and has a great core idea, the exposition is not yet strong.  Many sentences are overly complicated.  You frequently use passive voice rather than active voice.  And in places, your phrasing is a bit vague and hard to follow.  I’ve attached a single marked-up paragraph to show some examples and make suggestions for how you might write this differently.” (Note: we don’t ever expect reviewers to line edit a full manuscript; but doing so for a small portion is often a big help to authors.)
  

For those of us on the editorial staff who read referee reports, while the feedback on items such as audience and organization/exposition are always helpful, perhaps the most important parts of the referee’s assessment are Fit and Overall Merit.  Regarding Fit, most PRIMUS articles have all or most of the following traits: (1) a novel idea for teaching and learning, (2) discussion of how that idea impacts student learning, (3) practical, transferable advice facilitating implementation by practitioners, and (4) exploration of at least one interesting problem in undergraduate mathematics pedagogy or curriculum.  It is extremely helpful when referees enumerate these traits of the paper and comment on them, and of course when they identify how one or more of these traits can be strengthened.  For Overall Merit, we are interested in the referee’s opinion of whether the paper makes a novel contribution to existing literature and is ultimately worthy of publication.  Overall Merit is closely tied to the recommendation the referee makes in one of the following classifications.

As referees assess papers, we ask them to keep the following broad classifications of papers in mind in their recommendations:

• Accept – this means that the paper is ready to publish as-is, and that no changes are needed other than a tiny number of typographical adjustments that could easily be made in the final review of the publisher’s typeset version of the manuscript.
• Minor Revision – when the editor classifies a paper as “minor revision” to an author, the journal is stating its expectation that successful implementation of these minor changes should lead to the revised version being published.  Such changes may or may not involve additional refereeing, and are typically more focused on issues like clarifying the audience, slight reorganization, and/or minor improvements to the exposition.  When a referee selects “minor revision”, this should indicate that that the paper demonstrates a clear fit with the journal and evident overall merit. 
• Major Revision – for a paper classified as “major revision”, we assert that there is a good core idea in the paper, but there is considerable work that needs to be done in revising the paper to bring that idea to readers in a suitable way.  This may involve a major reorganization of the manuscript, adding a new section to address a key omission, or a substantial overhaul to the prose.  Major revisions are always sent back to referees for an additional round of review after resubmission.  Subject to successful implementation of the requested changes, including affirmation from the referees, we expect to publish the revised manuscript.
• Revise & Resubmit – this classification is for papers that are not suitable for the journal in their current form, but that hint at a different, related paper that might be.  Here, rather than offering detailed feedback on what needs to be changed, it’s better to provide a short summary of why the current paper is not a good fit, and describe in broad terms a paper that might be, one that is presently hiding in the current paper.  This rating involves no presumption of future publication, and of course requires re-review by referees and editors.
• Reject – the paper is not suitable for PRIMUS and we don’t see a potential paper hidden in the current one.  A paper can be very good, but be rejected because PRIMUS is simply not the right venue for it.  For example, PRIMUS generally is not an appropriate journal for theoretical mathematics papers, student research papers, or data analysis from extended statistical studies.

Like a good paper itself, a good referee report is one that knows its audience (author and editor), that gets right to the point (is this paper a good fit? Is it novel? Is it interesting?), and that clearly articulates the suggestions the reviewer has to make the paper even better for its future readers.

Authors regularly offer effusive praise for the help they receive from referees:  like us, they want their paper to be the best it can be, and high quality referee feedback is crucial to making that happen.  We are deeply grateful to all of our volunteer referees for the time and investment they make in reviewing manuscripts in order that we may bring good new ideas to our readers.