# Questions

The current karma system of EA Forum, which is described __here__^{[1]}, is based on the assumption that karma is positively correlated with impact (of the post/comment). I think this is pretty reasonable, but how strong is the correlation?

I am also curious about the shape of the relationship between karma and impact. For example, if post Y has twice as much karma as post X, is Y roughly 2 times as valuable as X (__linear__ function), less than than (e.g. __logarithmic__), or more than that (e.g. __quadratic__)?

Moreover, what is the impact caused by a post with a given amount of karma? Ideally, one would want an answer in terms of the effect on the expected value of the future, but I understand this is hardly feasible. So, in practice, using heuristics may be better. Examples include donations to the __Long-Term Future Fund__ (or other), and the metric quality adjusted research papers (QARPs) used by Nuño Sempere __here__.

# Example answers

I tried to get a very preliminary sense of the answers to the questions above based on Nuño Sempere’s __analysis__ of the first 10 winners of the __EA Forum Prize__. I estimated the mean impact in QARPs of each of the posts from the mean between the lower and upper limit of the 80 % __confidence intervals__ provided by Nuño. The calculations are in tab “Posts” of __this__ Sheet.

The table below contains the slope and __correlation coefficient__ for the linear regression of the mean impact on various functions of the karma. It also has the __p-value__ for the null hypothesis that there is no correlation. The calculations are in tab “Statistics”.

Null intercept linear regression between mean impact (QARP) and... | Slope | Intercept | Correlation coefficient (R) | P-value |
---|---|---|---|---|

Logarithm of the karma | 0.244 | -1.04 | 0.445 | 0.269 |

Square root of the karma | 0.0492 | -0.407 | 0.440 | 0.276 |

Karma | 2.45 m | -0.164 | 0.433 | 0.284 |

Square of the karma | 11.8 µ | -0.0427 | 0.416 | 0.305 |

For the logarithm, square root and square of the karma, and karma alone, the correlation coefficient is 0.4, and the p-value 0.3. The differences are quite small given the sample size of 8^{[2]}. Interestingly, the above estimates for the correlation coefficient are similar to the 0.47 obtained by Nathan Young __here__ for the relationship between the inflation-adjusted karma and ranking of the posts of the __Decade Review__.

The table below shows the correspondence between the scale described __here__ by Nuño and the karma predicted by the linear regression of the mean impact on logarithm of karma, which has the best fit among the 4 presented above. The calculations are in tab “Predictions”.

Mean impact (QARP) | Description | Example | Predicted karma |
---|---|---|---|

0.1 m | “A thoughtful comment” | A thoughtful comment about the details of setting up a charity | 67.0 |

1 m | “A good blog post, a particularly good comment” | What considerations influence whether I have more influence over short or long timelines? | 67.3 |

10 m | “An excellent blog post” | Humans Who Are Not Concentrating Are Not General Intelligences | 71.0 |

100 m | “A fairly valuable paper” | Categorizing Variants of Goodhart's Law | 108 |

1 | “A particularly valuable paper” | The Vulnerable World Hypothesis | 475 |

10 to 100 | “A research agenda” | The Global Priorities Institute's Research Agenda | 4.15 k to 40.9 k |

100 to > 1000 | “A foundational popular book on a valuable topic” | Superintelligence, Thinking Fast and Slow | 40.9 k to > 408 k |

> 1000 | “A foundational research work” | Shannon’s "A Mathematical Theory of Communication" | > 408 k |

The predictions above are quite poor. For instance, they imply:

- An unreasonably small difference of:
- 0.367 karma between the impact of “a good blog post, a particularly good comment” and “a thoughtful comment”.
- 3.67 karma between the impact of “an excellent blog post” and “a good blog post, a particularly good comment”.

- “A particularly valuable paper” is worth 475 karma, whereas I think the right value is of the order of magnitude of 10 kkarma (with huge variation), i.e. 21.1 times as large.

In any case, one should certainly be mindful of __Goodhart's Law__, and do not start optimising posts just for karma!

^{^}^{^}Nuño evaluated 10 posts, but only scored 8 in terms of QARPs.