There is a thing in decision theory called the exploration/exploitation tradeoff. Exploitation is always choosing the option that is estimated to be best. Exploration is choosing other options to see if they might be even better. In this case showing new comments is exploration.
Hacker news has a nice solution to this problem. New comments appear at the top of the thread, and then slowly fall unless they get votes. I think this works very well, although I think it's still a bit too far on the exploitation tradeoff.
I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
There are other solutions, like treating it like a multi-armed bandit problem. And you can use something like https://en.wikipedia.org/wiki/Thompson_sampling . Create a probability distribution for the upvote/downvote ratio for every comment. Sample from that distribution, and rank the comments accordingly. This should be close to optimal, but it requires people to use the downvote button lots (because it measures comment quality by upvote/downvote ratio, which is what reddit does currently.) Without downvotes, you could use time again, and create a probability distribution for votes/time.
I would add that while it's plausible, it's by no means obvious that Reddit is leaving business value on the table. Imagine that they seek to minimize the bounce-rate on any thread such that they maximize overall time on site for each user. In this case, the important part could be that the top comment is 'not wrong' rather than 'right'. If the first thing users see is too factually incorrect, controversial, or emotionally upsetting, people could be driven people away from the site. Essentially this is the same mechanism that helps cause the Facebook news bubble effect -- people leave if you tell them what they don't want to hear, so you write an algorithm that underweights anything that might do so.
To put it into economic terms, if users have high loss aversion (from being upset, etc.) relative to their thirst for knowledge gained from great comments, you would expect exploitation to be overweighted vs exploration. The business value maximizing solution would then be that once an appropriate top comment is found, the thread should tend to stay in that equilibrium.
I really doubt there is any conscious decision by reddit to do this. For years there was a major bug in their ranking algorithm that hid posts with negative scores. People invented reasons why this was intentional and why reddit might want to do that. But eventually they fixed the bug and admitted it wasn't intentional.
The current comment sorting algorithm was just copied off the internet from some popular HN post. I don't think there was that much thought put into it.
Showing a few new comments closer to the top is not going to hurt anyone. I think if anything the massive improvement in comment quality would make users more loyal to the site. At the moment you can only see the good comments posted by the first 20 people or so, imagine if it was the best comments out of 1,000?
There are vastly more voters than commenters. If only one out of 5 people sees a new comment and votes on it, that would be more than enough to improve the comment quality 3 fold.
"I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
"
The traditional solution to this problem is wilson score intervals.
The real problem there is that most sites that use them only use one part of the bound, instead of using both.
Ideally, you use the lower bound to scale how good something is, and the upper bound to determine with what priority you show it order to get more votes on it to determine better bounds to be more confident of your lower bound.
This is what Google Moderator did, and pretty much never had these problems as a result.
(Is it perfect? No. You need a good place/way to show things for voting. But assuming you get that, it works very well in practice).
If you only use the lower bound, you get systems where you only ever become more confident in the scores of the things that show up first (like is happening here).
That said, it's still biased. Years ago, HN admins/mods marked my account as "low-quality content" so now my comments always quickly fall down to the bottom of the parent, no matter how many points they have. They admitted it in an email recently, and refused to take it off my account, despite the quality of my comments no longer being trollish. When you have human intervention like this, there's danger if it heavily biasing the entire forum for the worse.
Well moderators are necessary to remove trolls. If your comments really were so trollish they felt the need to do that, I wonder why they didn't just ban you outright.
Yep, banning would have been an alternative solution. But with the choice they made, it requires more human intervention, both on the shorter term (i.e. policing comments to find trolls and mark their accounts) and the longer term (i.e. policing marked accounts and unmarking them when their comments become consistently high quality), and that's a lot of work, and too subject to flawed bias, which is why it's a flawed choice.
This search for a better way to rank comments still, in my opinion, makes a large incorrect assumption; namely that it is desirable (and possible) to display comments in order from "best" to "worst". If you think of comments only from the prespective a viewer then yes ordering comments from best to worst makes sense, the viewer wants to get as much information as quickly as possible. But if you are looking at establishing a robust meaningful dicussion, part of that is that all comments should have a chance of getting responded to that is proportional to their value, not their rank. Realistically the top three comments in a large thread are about equally good, but the top comment is going the vast majority of views and replies. One system the facilities dicussion might effectivelly shard it's users into separate threads of managble size so they can have a real dicussion, though I'm fuzzy on the details.
Given all the worry we have about about how our online discussions work I think their is real societal (if not extrable monetary) value in discovering a better solution.
Yes-- would be nice if top level comments got collapsed once the thread reaches a certain size. Then you could scan the top level comments quickly and figure out which ones you want to drill into.
Hacker news has a nice solution to this problem. New comments appear at the top of the thread, and then slowly fall unless they get votes. I think this works very well, although I think it's still a bit too far on the exploitation tradeoff.
I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
There are other solutions, like treating it like a multi-armed bandit problem. And you can use something like https://en.wikipedia.org/wiki/Thompson_sampling . Create a probability distribution for the upvote/downvote ratio for every comment. Sample from that distribution, and rank the comments accordingly. This should be close to optimal, but it requires people to use the downvote button lots (because it measures comment quality by upvote/downvote ratio, which is what reddit does currently.) Without downvotes, you could use time again, and create a probability distribution for votes/time.