Reviewing Another Subfield
January 2013
The peer review process can be a frustrating enigmatic process, but it can also help you produce a high quality paper and provide you with an outside perspective to your own work. I have reviewed enough papers lately and have received enough of my own rejections to at least say I have been initiated into the process. Based on advice from mentors, colleagues, and other authors; I am beginning to handle the rejection process better. On the other side of the process, I think I am also improving as a reviewer.
To be honest, there is not much in the way of training, instruction, or guidelines for being a reviewer. It seems that once you have been a reviewee, many venues view you as being qualified to be a reviewer. A recent blog post by Matt Might set out guidelines for new reviewers that seemed quite reasonable, and I try to follow them as best as I can. However, there is one question about reviewing that I have been struggling with lately—it is relevant both to the papers I have been reviewing and my own papers that are being reviewed. Should a paper be reviewed within the scope of its subfield or within the scope of the larger field?
Any field can be broken into a number of subfields and those subfields themselves can be subsequently divided. These boundaries are not immovable barriers; many researchers publish work that combines two smaller areas or work in multiple areas at the same time. Occasionally a great result is presented that contains implications for an entire field. More often, though, a paper provides an improvement or insight into techniques in a well-established subfield.
A particular journal may publish articles within a wide range of areas. Each subfield merrily publishes articles in the journal with little concern for those outside their area. One issue with subfields is that not everyone outside of that area will think it has merit. Come review time, this can have obvious implications if one of the reviewers does not think much of your particular subfield.
An Example in ASR
There is a subfield in automatic speech recognition (ASR) that focuses on recognition of speech in noisy environments—think of a conversation in a bar or on a noisy street. Even within that group, there is a subset of researchers who mainly publish results on the recognition of spoken digits in noisy environments. There are legitimate reasons for this narrow focus—a well-defined dataset exists for this problem and it allows researchers to focus on the noise by removing the added complications introduced by less restricted speech—but the narrow focus is also criticized by other researchers.
Two common complaints:
- The techniques do not scale to real speech recognition problems. The results may show that a method works for this spoken digit task, but it might not apply to actual speech recognition tasks that others care about.
- Improvements are not significant. Since this dataset has been studied for so many years, improvements are typically very small and may not pass any type of significance test used in other areas.
How to Review Another Subfield?
Given a paper in the previously described subfield, how do I review it? The viewpoint I take can entirely decide whether I reject or accept the paper. Either path has legitimate justifications.
Reject: The paper has characteristics that most people in the general field would view as serious flaws. The results are only interesting to people in a narrow subfield.
Accept: The paper improves upon previously published techniques. Several papers in this area have been previously published in this journal and this work clearly is of at least comparable quality.
In many ways it comes down to the expectations we have of the authors. Should we expect an author to justify the legitimacy of his entire field in every paper he writes? It may leave little room for presenting the novel contribution. To me, this is a very tricky question. You do not want to just assume the legitimacy of an area because one paper snuck through the review process, but it is unfair to require justification from authors that was clearly not required by their predecessors.
My current answer to this question is to include all of my objections in my review, however, I base my final decision on how the work compares to previously published work in the same area within similar venues. While I encourage the authors to address my concerns, I do not explicitly require it. I honestly do not know if this is the correct approach; I may just be reviewing in the way I wish other reviewers would review my work.