Upvotes or likes have become a standard way to filter information online. The quality of this filter is determined by the users handing out the upvotes.
For this reason, the archetypal pattern of online communities is one of gradual decay. People are more likely to join communities where users are more skilled than they are. As communities grow, the skill of the median user goes down. The capacity to filter for quality deteriorates. Simpler, more memetic content drives out more complex thinking. Malicious actors manipulate the rankings through fake votes and the like.
This is a problem that will get increasingly pressing as powerful AI models start coming online. To ensure our capacity to make intellectual progress under those conditions, we should take measures to future-proof our public communication channels.
One solution is redesigning the karma system in such a way that you can decide whose upvotes you see.
In this post, I’m going to detail a prototype of this type of karma system, which has been built by volunteers in Alignment Ecosystem Development. EigenKarma allows each user to define a personal trust graph based on their upvote history.
EigenKarma
At first glance, EigenKarma behaves like normal karma. If you like something, you upvote it.
The key difference is that in EigenKarma, every user has a personal trust graph. If you look at my profile, you will see the karma assigned to me by the people in your trust network. There is no global karma score.
If we imagine this trust graph powering a feed, and I have gamed the algorithm and gotten a million upvotes, that doesn’t matter; my blog post won’t filter through to you anyway, since you do not put any weight on the judgment of the anonymous masses.
If you upvote someone you don’t know, they are attached to your trust graph. This can be interpreted as a tiny signal that you trust them:
That trust will also spread to the users they trust in turn. If they trust user X, for example, you too trust X—a little:
This is how we intuitively reason about trust when thinking about our friends and the friends of our friends. Only EigenKarma being a database, it can remember and compile more data than you, so it can keep track of more than a Dunbar’s number of relationships. It scales trust. Karma propagates outward through the network from trusted node to trusted node.
Once you’ve given out a few upvotes, you can look up people you have never interacted with, like K., and see if people you “trust” think highly of them. If several people you “trust” have upvoted K., the karma they have given to K. is compiled together. The more you “trust” someone, the more karma they will be able to confer:
I have written about trust networks and scaling them before, and there’s been plenty of research suggesting that this type of “transitivity of trust” is a highly desired property of a trust metric. But until now, we haven’t seen a serious attempt to build such a system. It is interesting to see it put to use in the wild.
Currently, you access EigenKarma through a Discord bot or the website. But the underlying trust graph is platform-independent. You can connect the API (which you can find here) to any platform and bring your trust graph with you.
Now, what does a design like this allow us to do?
EigenKarma is a primitive
EigenKarma is a primitive. It can be inserted into other tools. Once you start to curate a personal trust graph, it can be used to improve the quality of filtering in many contexts.
- It can, as mentioned, be used to evaluate content.
- This lets you curate better personal feeds.
- It can also be used as a forum moderation tool.
- What should be shown? Work that is trusted by the core team, perhaps, or work trusted by the user accessing the forum?
- Or an H-index, which lets you evaluate researchers by only counting citations by authors you trust.
- This can filter out citation rings and other ways of gaming the system, allowing online research communities to avoid some of the problems that plague universities.
- Extending this capacity to evaluate content, you can also use EigenKarma to index trustworthy web pages. This can form the basis of a search engine that is more resistant to SEO.
- In this context, you can have hyperlinks count as upvotes.
- Another way you can use EigenKarma is as a way to automate who gets privileges in a forum or Discord server - whoever is trusted by the core team. Or trusted by someone they trust.
- If you are running a grant program, EigenKarma might increase the number of applicants you can correctly evaluate.
- Researchers channel their trust to the people they think are doing good work. Then the grantmakers can ask questions such as: "conditioning on interpretability-focused researchers as the seed group, which candidates score highly?" Or, they'll notice that someone has been working for two years but no one trusted thinks what they're doing is useful, which is suspicious.
- This does not replace due diligence, but it could reduce the amount of time needed to assess the technical details of a proposal or how the person is perceived.
- You can also use it to coordinate work in distributed research groups. If you enter a community that runs EigenKarma, you can see who is highly trusted, and what type of work they value. By doing the work that gives you upvotes from valuable users, you increase your reputation.
- With normal upvote systems, the incentives tend to push people to collect “random” upvotes. Since likes and upvotes, unweighted by their importance, are what is tracked on pages like Reddit and Twitter, it is emotionally rewarding to make those numbers go up, even if it is not in your best interest. With EigenKarma this is not an effective strategy, and so you get more alignment around clear visions emanating from high-agency individuals.
- Naturally, if EigenKarma was used by everyone, which we are not aiming for, a lot of people would coalesce around charismatic leaders too. But to the extent that happens, these dysfunctional bubbles are isolated from the more well-functioning parts of the trust graph, since users who are good at evaluating whom to trust will by doing this sever the connections.
- EigenKarma also makes it easier to navigate new communities, since you can see who is trusted by people you trust, even if you have not interacted with them yet. This might improve onboarding.
- You could, in theory, connect it to Twitter and have your likes and retweets counted as updates to your personal karma graph. Or you could import your upvote history from LessWrong or the AI alignment forum. And these forums can, if they want to, use the algorithm, or the API, to power their internal karma systems.
- By keeping your trust graph separate from particular services, it could allow you to more broadly filter your own trusted subsection of the internet.
If you are interested
We’re currently test-running it on Superlinear Prizes, Apart Research, and in a few other communities. If you want to use EigenKarma in a Discord server or a forum, I encourage you to talk with plex on the Alignment Ecosystem Development Discord server. (Or just comment here and I’ll route you.)
There is work to be done if you want to join as a developer, especially optimizing the core algorithm’s linear algebra to be able to handle scale. If you are a grantor and want to fund the work, the lead developer would love to be able to rejoin the project full-time for a year for $75k (and open to part-time for a proportional fraction, or scaling the team with more).
We’ll have an open call on Tuesday the 14th of February if you want to ask questions (link to Discord event).
As we progress toward increasingly capable AI systems, our information channels will be subject to ever larger numbers of bots and malicious actors flooding our information commons. To ensure that we can make intellectual progress under these conditions, we need algorithms that can effectively allocate attention and coordinate work on pressing issues.
As a mathematician I think this is cool and interesting and I'd be glad to know what comes out of these experiments.
As a citizen I'm concerned about the potential to increase gatekeeping, groupthink and polarisation, and most of all of the major privacy risk. Like, if I open a new account and upvote a single other user, can I now figure out exactly who they have upvoted? Even if I can't in this manner, what can I glean from looking, as your example suggests, at the most trusted individuals in a community etc.?
As it is currently set up, you could start a blank account and give someone a single upvote and then you would see something pretty similar to their trust graph. You would see whom they trust.
It could, I guess, be used to figure out attack vectors for a person - someone trusted that can be compromised. This does not seem like something that would be problematic in the contexts where this system would realistically be implemented over a short to medium term. But it is something to keep in mind as we iterate on the system with more users onboard.
It does seem like an important point that your trust graph is effectively public even if you don't expose it in the API.
I think the privacy risk is more prominent when you don't know that information is being made public. A system like this would obviously tell users that their upvotes are counted towards the trust system and therefore people would upvote based on that. I guess we could make it all hidden, similar to Youtube's algorythm where the trust graph is not publically available and only works in the backgrounds to change the upvotes you see for the people you trust (so it changes the upvotes but you don't see how it changes and other people can't see yours), but I believe that would be more concerning and likely to be abused then if the information is always available for everyone.
As for the gatekeeping and groupthink, I think that already happens regardlessly of a trust system, and is a consequence of tribalism in the real world rather than the cause of it. Honestly, I see it beneficial that people who despise each other get more separated, and usually it's the forceful interaction between opposing groups that results in violent outcomes.
The literature on differential privacy might be helpful here. I think I may know a few people in the field, although none of them are close.
I have put some thought into the privacy aspect, and there are ways to make it non-trivial or even fairly difficult to extract someone's trust graph, but nothing which actually hides it perfectly. That's why the network would have to be opt-in, and likely would not cover negative votes.
I'd be interested to hear the unpacked version of your worries about "gatekeeping, groupthink and polarisation".
I don't have time to write it in detail, but basically I'm referring to two ideas here regarding what a user sees:
Both lead to gatekeeping because new users aren't trusted by anyone so their content can't get through.