arxiv postings should feature warning sign iconography. On one hand, it offers open access to research -- awesome. On the other hand, it offers us a chance to waste our time and energy studying material that may never survive peer review.
I get excited whenever I see what appears to be a really interesting, valuable piece of information gleamed through rigorous research. My immediate first emotion is excitement. However, that excitement casts a long shadow of doubt as my system 2 mode of reason kicks in and alerts me that the content of interest is the result of an arxiv posting that may fail to survive peer review. Of course, death-through-peer-review may not imply bunk science but I assume that it more often is the case (experts, please chime in).
This leads me to ask whether the HN community would be better without knowing of bleeding edge research on arxiv until it's graduated peer review --What do you think: Is it better to front page bleeding edge, unverified research?
I think the world would benefit by a service that indexes arxiv postings that have survived peer review. Is this feasible?
Surely the act of people reading it is peer review? Unless you assume that nobody on Hacker News is qualified to read scientific papers, which seems like a stretch.
No. Peer review is more than that. It's a vetting process. 90% if everything is crap, and so if you don't vet it, you'll get 90% crap.
Also, by conflating a forum full of lay people with experts in a specific field is a bit absurd. Simply put, not everyone's opinion should be given equal weight.
For any given arxiv article, I'd wager 98% of people on HN are unqualified to critically review the article. They're not aware of prior work that might contradict the article. They're not aware of the limitations of prior work that the article cites. They've never tried the experimental or theoretical methods the article uses, and so on. Yet these are the people upvoting and commenting on implications.
98% sounds like a made up stat, so if you grant me the same privilege then let's say HN has 1 million readers. So 2% of that would be 20,000 people qualified to judge the article, which is plenty of people.
1) HN has 300k daily uniques (https://news.ycombinator.com/item?id=9219581). Of those readers, nearly all do not comment (we don't have 300k comments, do we?). Of those commentors, only a fraction comment on a given article.
2) Of those commenters, most are not actual researchers, who are, IMO, much, much busier with real work (you know... research) than your average HN commenter.
Of course most don't comment, but commenting is not required for a paper to be peer reviewed, only reviewing it is. If someone were to see the paper here, and review it, and find an error, presumably they would contact the author.
300k * 0.02 = 6000 people who may be qualified. How many people are typically asked to peer review a paper? A handful? The odds are still good.
The original poster was talking about what would be better for the HN community. The HN community is a discussion forum, not a system of people privately emailing.
My rule of thumb is to always have a healthy dose of skepticism regardless of how a study was published, whether in a peer-reviewed journal, preprints, or even a blog post. About bleeding edge research, publishing machine learning methods immediately on arXiv seem to be hot among academics nowadays.
I get excited whenever I see what appears to be a really interesting, valuable piece of information gleamed through rigorous research. My immediate first emotion is excitement. However, that excitement casts a long shadow of doubt as my system 2 mode of reason kicks in and alerts me that the content of interest is the result of an arxiv posting that may fail to survive peer review. Of course, death-through-peer-review may not imply bunk science but I assume that it more often is the case (experts, please chime in).
This leads me to ask whether the HN community would be better without knowing of bleeding edge research on arxiv until it's graduated peer review --What do you think: Is it better to front page bleeding edge, unverified research?
I think the world would benefit by a service that indexes arxiv postings that have survived peer review. Is this feasible?