Norms and features for the Forum

Epistemic status: personal observations, personal preferences extrapolated. Uses one small random sample and one hard data source, but all else is subjective.

I think the Forum could use some epistemic healthcare. As Rohin Shah puts it

for any particular high-karma post, knowing nothing else about the post besides that it is high karma, there is a non-trivial probability that I would find significant reasoning issues in that post. You can't rely solely on karma as a strong signal of epistemics.

I randomly sampled 20 such posts and guess this probability is about 30% [90% CI: 10%, 50%].

But we say we value rigour to an unusual extent. So why aren't we rigorous?

Well, rigour is rare because it's hard. But I argue that our platform can make it easier.

Many posts aren't about reasoning

One innocuous point first: there are lots of reasons to write a post besides asserting a novel claim.

• Questions
• Admin like announcements
• New ideas to consider
• Summarising your understanding of other work
• Coordination, common knowledge
• Fiction! <3
• Practising writing

Karma means too much

Less innocuous is that our voting system doesn’t distinguish

• "I agree"
• "The argument is rigorous / the evidence is strong"
• "Makes a novel point"
• "This makes me feel good"
• "I encourage you to post more"

or, lately:

• "This post belongs to my faction on The Current Thing"
   

Is this a problem? That is: is anyone using karma as a measure of rigour or truth? 

Actually yes: via the halo effect (things good in one way get unthinkingly treated as good in other ways) I expect this to be somewhat true. And I expect high karma to cause a post to get read more, if only because of readers' fear of missing out.

I just want karma to mean "worth reading", which is a much easier objective than "is rigorously argued", but which we still often fail to meet. 

People have been arguing about having multiple types of upvote for a long time at  e.g. LessWrong ("I agree with this" vs "This persuaded me" vs "I want to encourage this person to keep posting", and the converses). I incline to think that this is too messy -but we can still try to decouple the truth economy from the vibe economy a bit, see below.

So here are some ideas. Some are unobtrusive enough that they're worth changing the UI for. Others are more like norms I'd like us to introduce, or reintroduce.

Feature ideas 

• UI. Add an extra editor text field for an "Epistemic status" at the start of the post. Time cost: 2 mins per post.
• UI: A Like button, separate from upvotes. To capture vibes and maybe leave karma to epistemics. Time cost: N/A.
• UI: Make a post summary required if the post is >1000 words. [EDIT: Can even integrate NLP here for a terrible 80/20 version.] Time cost: 10mins.
• UI: Add a “Limitations” text field to the editor. Time cost: 30 mins.
• Speculative: NLP for claim detection: the site asks you for your probabilities about the main claims. Time cost: 30 mins.
• Speculative: NLP to autosuggest prediction markets you could open. Time cost: 15 mins.
• Speculative: Two comment tabs, one for vibes and one for critique.

Norms

• You can implement all of the above yourself! The devs don't need to do it for us.
• e.g. If you liked the post but it doesn’t update your views, just comment on it saying "Nice!".
• Address criticism in advance. Simulate it if you are unfortunate enough to not have critics. 
  • If you mention someone / some specific view, send a draft to them before posting.
• To date, most throwaway accounts have been used to write magnificent hard truths we wouldn't otherwise hear. So maybe these are underused.
• I like Nuno's personal digest and want to see more of these as a workaround for the karma system not meaning what I want it to.

A big change, which I'll write up separately, is something like paid, open peer review for posts which seek to seriously update the community. Likely bringing in outsiders, likely post-publication. (This is close to red-teaming but using outsiders, come to think of it.) This will be expensive, but it passes a shallow cost-benefit test, to me.

These ideas increase the effort it takes to post something. I think it's well worth the time, but if you're on the margin of not posting, or if you're new and intimidated, it's likely better to skip them and just post. No excuse for the old-timers though.

Limitations

• The above is not mostly based on hard data; instead it's 7 years of reading the forum and a tiny random sample of posts.
• So I don't know that the above features will repay their time cost or even their dev cost.
• I am very opinionated.

PS: I used the GraphQL backend (thanks devs!) to get some basic stats:

• Average post length: 1235 words
• Average length w/ 125+ karma: 2600 words

If good evidence or arguments took longer to state, then this would be weakly reassuring. But probably that correlation is too weak to say anything.