Conferences are no longer nice meetings among gentlemen who care about science. They are the center of a huge competition for thousands of scientists. An acceptance in a major conference can change the career of a scientist and the daily life of a laboratory. It is money and it is fame.
The goal of the peer-reviewing process has naturally shifted from validating the accuracy of a scientific work to selecting a subset of papers. The reviewers have mutated into hunters tracking any reason to crush a paper. All the gentle and polite manners from "guides of peer-reviewing" sound so XXth century. In computer sciences, the "hypercriticality" of scientists is well identified.
Unfortunately, all harsh reviewers are not gifted. Reasons for rejecting a paper frequently come from wrong beliefs, from major misunderstanding or from an obsession for a tiny flaw. These are negative badly-justified reviews, which sometimes make the authors speculate about a possible bias in the reviewing process: the reviewer was a competitor in the same domain, the reviewer was not an expert, the reviewer had personal grievances against the authors, the reviewer was a well-established scientist who does not want to see any newcomer out there...
What are the options for an angry author with a supposed unfairly-reviewed paper and its negative consequences (the money loss and the degraded career)? Let's call a lawyer.
The roles of every entity involved in the traditional academic process are well summarized in this nice paper. Sometimes, the editor of the venue describes the process in used. But the reality is that there is no well-defined rule, no engagement. In fact, it is written nowhere that every paper should be fairly reviewed. An author can only hope that the process is fair because the process is usually said to be fair. I have actually no idea what a lawyer could do versus a no-rule process.
I have always been astonished by Boeing (or Airbus) appealing a decision from an orderer when their bid is rejected. In my naive vision, how can anybody challenge the decision of a private company? When the orderer is the Air Force, there must be rules, for sure, but the idea is the same for a paper to, say, Infocom. But here is the point. There is no Infocom. More precisely, our beloved top-conferences might belong to anybody, including Infocom. I am not sure to know what a lawyer could do versus a no-entity.
Is is serious?
The rebuttal process is probably a good idea, which has the potential to reduce some unfair decisions. But it is still another prehistoric practice based on oral tradition and informal discussions between those who are supposed to be the owner of something that does not exist, except in author's mind. In medicine, some scientists have argued about the creation of a "supreme court" whose role would be to "judge the judges" (here and there). But the motivation is to prevent new scandals related to misbehaving editors. This court would deal with serious cases, not slightly unfair reviews. Maybe academics should look at what has been done in sports to legally organize the structures. Societies like IEEE or ACM could also consider to create a "mediator committee".
Otherwise a rejected paper will end up in court, then who knows who might be considered as responsible.
There is a lack of accountability in the system. A poor reviewer can simply continue to be one. I can't see the problem with open peer review processes where identities are known and reviews are published publicly. Others could then have the opportunity to 'sign' reviews with which they agree. With transparency comes accountability.
ReplyDelete