> How do you keep a ratings system from being a proxy for popularity, rather than a measure of novelty or utility? How do you keep it from being easily gamed?
To respond to this particular point, I'm not familiar with any characteristics of the academic journal system which prevent these problems either. Name recognition is often key to getting into competitive journals, which is just another way of calling it an easily gamed popularity contest.
Popularity plays a role. Here's a real life example: One of my quantum physics mentors submitted a paper to PRL (a high prestige physics journal) and it was rejected. Why? Because it came from a small, rather unknown university in Spain.
He was later admitted to Cambridge for a PhD and sent the exact same paper (same formatting, content, layout but different department/uni). It was immediately accepted. Does it necessarily mean that scientists in poorer countries are doing bad science? Or just that they didn't get their visa / couldn't leave their country / just want to stay there, etc.
The first filter most editors use is: "Do I know this research group? Are they part of the club? Have they published in top journals?". If not, it won't get read (unless it's a compelling and exciting title and abstract).
I haven't published there in a while but I don't think so.
> We are no longer able to accede to requests from authors that we withhold their identities from the referees. Such "double-blind" reviewing has been discontinued. [0]
Certainly the current system can be gamed, and is, and the complaints about problems with bibliometrics about 10 seconds after the first one was published.
But the argument is that "giving other researchers the ability to rate ... [would work] just as well as this journal reputation system".
My question is, simply, why should I believe this?
Bibliometrics often look at citations. Citations are, yes, a marker of popularity and social connections, but also have certain scholastic requirements. Editors and reviewers can and do point out missing references or highlight too many irrelevant self-citations. This is a type of moderation system. Imperfect certainly, but it does provide some grounding.
Also, it's not like the scientific publication system only contains original research papers. Review papers fill a curational role, with the advantage of a single voice (even if from co-authors) making the comparisons, so I at least have a relatively constant baseline.
I think a lot of ratings/reputation/moderation problems can be engineered around if you make all of the moderation metadata publicly available in a form that enables competition in the search functionality. Make articles, users, links, and comments up and down-votable in a way that is public and immutable forever, and de-couple the problem of mining the data.
Traditionally peer review is done anonymously from both sides. However, it is often possible to get a good idea of who the author is based on their related work, style and choice of references. This is particularly the case where the field is very specialised and there are only a few people qualified to judge papers.
This depends a lot on the field. Every neuro paper I have reviewed has the author's names and affiliations intact, along with unblinded citations like "Experiment was done as we described previously (SameFolks et al., 1999)."
On the other hand, I've always been suspicious that people can reliably guess the authors when doing a blinded review. I've seen one study that said 30% of reviewers (in a small sample) could do so, but that hardly seems like a reason to abandon the idea altogether.
To respond to this particular point, I'm not familiar with any characteristics of the academic journal system which prevent these problems either. Name recognition is often key to getting into competitive journals, which is just another way of calling it an easily gamed popularity contest.