Comment Permalink

Do you post on the EA subreddit? Everyone's vote power is equal there:

Yes, I do post there. It's...fine. I don't exactly love it, but it at least doesn't give me an active feeling of disgust every time I use it (which the forum does).

Retributive downvoting appears to be a bannable offense, according to the forum guide:

This is unenforceable. In fact that whole section is unenforceable:

Additionally, please avoid:
Asking your friends or coworkers to vote on a post, especially if you might be biased (e.g. because the post is criticizing your work, or because your friend wrote the post)
We think sharing a post in a public channel and saying “Hey, I quite like this post that summarizes my organization’s work is cool, check it out” is fine. If you see a message like this, evaluate the post on its own merits; don’t just go upvote because someone you know wrote it.
But posting — or even worse, saying this on a call — “Hey, everyone, please go upvote this post that our organization just shared, we need everyone on the Forum to see it” is bad. Even worse is asking people to downvote criticism of something you work on.
Deferring entirely to someone else (your vote should be your own)
Additionally, please try to judge each post or comment on its own merits; don’t just vote based on whether or not you like the poster’s other activity.
Other than that, you can vote using your preferred criteria. Here are our suggestions:
Action If… Not if…
Strong-upvote
Reading this will help people do good.
You learned something important.
You think many more people might benefit from seeing it.
You want to signal that this sort of behavior adds a lot of value.
“I agree and want others to see this opinion first.”
Upvote
You think it adds something to the conversation, or you found it useful.
People should imitate some aspect of the behavior in the future.
You want others to see it.
You just generally like it.
“Oh, I like the poster, they’re cool.”
Downvote
There’s a relevant error.
The comment or post didn’t add to the conversation, and maybe actually distracted.
“There are grammatical errors in this comment.”
Strong-downvote
It contains many factual errors and bad reasoning
It’s manipulative or breaks our norms in significant ways (consider reporting it)
It’s literally spam (consider reporting it)
“I disagree with this opinion.”

I agree that if everyone followed these norms there wouldn't be a problem, however they aren't, and there isn't a way to make them. Worse, the incentives work against a lot of these norms:

If someone is spreading opinions you disagree with, then the karma system makes strong-downvoting them an excellent way to hinder their ability to do so.
If your friend makes mediocre posts but he also upvotes all your mediocre posts, then upvoting them is a great way to ensure your posts get more exposure.
If an HBDer sees me sharing studies that undermine HBD, then a great way to lessen my ability to do so is downvote other posts and comments I've written (and downvote the posts and comments I write in the future too). Worst case scenario, my reach decreases. Best case scenario, I start to self-censor or leave the forum (aka what happened).

Voting is anonymous, so unless you "mass" vote it will remain undetected. It's good that they have written down these norms, but it'll barely do anything even if it was better known. The karma system simply works against it. It's not enough to say "pretty-please don't act on the bad incentives we've created", you have to actually give people good incentives.

See in context

Poll: Should people with more forum karma have more powerful votes?

by Henry Howard🔸

Apr 281 min read 43

48

Building effective altruismCommunityDIY Debate WeekEffective Altruism Forum

Frontpage

My upvotes/downvotes are worth 2 points each and my supervotes are worth 6. A person with between 10 and 100 karma on the forum has an upvote worth 1 and a supervote worth 2 (the scaling system is described in this code here I think)

My concern is that this system lends itself to groupthink, whereby the dominant views or topics are liable to get more karma, giving holders of those views more voting power, giving users that makes posts they agree with or that they see as relevant more karma, etc.

Dissenting opinions or posts not of interest to the in-group are liable to be downvoted (although karma is meant to reflect quality or relevance of a post or comment, this is of course misused), which both hides those comments but also puts off dissenting voices from commenting/posting in the future.

The justification for the current system is that people with more karma are more likely to be have better understanding and judgement, less likely to be sockpuppets or trolls and so are better positioned to vote. This is a system ported over from LessWrong (described here).
Concerns about the scaling system have been discussed on the forum previously, for example here.

Is this system more beneficial than harmful?

Vote power should scale with karma

disagree

agree

48 Reactions

Comments43

Sorted by

New & upvoted

Click to highlight new comments since: Today at 7:44 AM

NickLaingApr 2818

22% agree

Vote power should scale with karma

I think it should scale but less than it does now. The power goes to my head....

In all seriousness I thin + 2 for a new user strong upvote and +8 for a high Karma user is too much of a gap. Maybe +4 or +5 would be enough? I know this is a specific case, but I feel sometimes deep in an animal welfare threads you can make a reasonable comment and be downvoted to oblivion pretty quickly by a couple of heavy downvotes. But I would say that....

As anecdata here's an example of a fairly reasonable comment from a few hours ago that would be negative karma without my strong upvote....

Toby Tremlett🔹Apr 2818

78% agree

Vote power should scale with karma

A few quick thoughts:

Scaled voting power is part of why moderation on the Forum is sustainable. When I see posts downvoted past zero I agree the majority of the time.
Karma is an imperfect but still very helpful measure of how useful a writer has been to the EA Forum community. Therefore, high karma correlates pretty well with people having good Forum content taste.

I disagree with the "groupthink" idea because I don't see strong downvotes being used to suppress good, on-topic content the majority of the time (when I do, I strong upvote it). I can imagine a world where this system did go wrong (if high karma users represent an old orthodoxy that new users wanted to move past for example). But I don't see this issue in reality.

Definitely open to hearing more about this and seeing examples of the system going wrong! And thanks for raising the question Henry :)

titotalApr 2812

Scaled voting power is part of why moderation on the Forum is sustainable

On a typical day the forum has like 10 posts and like 30 comments. A subreddit like r/excel is roughly 3 times as active as this forum, and is moderated entirely by volunteers. I do not think it would be very difficult to moderate the forum without a karma system if people chose to do so.

Therefore, high karma correlates pretty well with people having good Forum content taste.

I would say the people with the most karma are the people who comment a lot with a content that is in line with the tone of the forum. This seems like it serves to reinforce the status quo of forum norms. Whether this is a good or bad thing will depend on your opinion of said norms: for example, I would prefer this place be a bit more tolerant of humour and levity.

JasonApr 283

I directionally agree, but the system does need to also perform well in spikes like FTX, the Bostrom controversy, critical major news stories, and so on. I doubt those are an issue on r/excel.

NickLaingApr 282

Am hoping the new "made me laugh" reaction might help on the humour and levity front, but haven't seen it used much yet outside of April Fools.

SofiaBaldersonApr 2910

As someone who moderates a large community I agree with this! Moderation is hard and can be a thankless task, how I wish Slack had a similar system:)

JasonApr 288

I'll call the role of voters in voting posts/comments below zero / off the frontpage / to be collapsed in comments "cloture voting" for short. As the name implies, I see that role as cutting off or at least curtailing discussion -- which is sometimes a necessary function, of course.

Scaled voting power is part of why moderation on the Forum is sustainable. When I see posts downvoted past zero I agree the majority of the time.

While I agree that cloture voting serves a pseudo-moderation function, is there evidence that the results are better with heavily scaled voting power than they would be with less-scaled power?

As applied to cloture voting, I have mixed feelings on the degree of scaling in the abstract. In practice, I think many of the downsides come (1) the ability of users to arbitrarily decide when and how often to cast strongvotes and (2) net karma being the mere result of adding up votes.

On point 1, I note that someone with 100 karma could exercise more influence over vote totals than I do with a +9 strongvote, simply by strongvoting significantly more than I do. This would be even easier with 1000 karma, because the voter would have the same standard vote as I. In the end, people can self-nominate themselves for greater power merely by increasing their willingness to click-and-hold (or click twice on mobile). I find that more concerning than the scaling issue.

On point 2, the following sample equations seem generally undesirable to me:

(A) three strongvotes at -9, -8, and -7, combined with nine +2 standard votes = -6 net karma
(B) five strongvotes at -6, combined with four strongvotes at +6 = -6 net karma

There's a reason cloture requires a supermajority vote in most parliamentary manuals. And those reasons may be even more pronounced here, where the early votes are only a fraction of the total potential votes -- and I sense are not always representative either!

In (2A), there appears to be a minority viewpoint whose adherents are using strongvotes to hide content the significant majority of voters believe to be a positive contribution. Yes, those voters could respond with strongvotes of their own. But they don't know they are in the majority and that their viewpoint is being overridden by 1/3 or less of a strongvoter's opinion.

In (2B), the community is closely divided and there is no consensus for cloture. But the use of strongvotes makes the karma total come out negative enough to hide a comment (IIRC).

One could envision encoding special rules to mitigate these concerns, such as:

A post or comment's display is governed by its cloture-adjusted karma, in which at most one-third of the votes on either side count as strong. So where the only downvotes are -9, -8, -7, they would count as -9, -2, -2.
In addition to negative karma, cloture requires a greater number of downvotes than upvotes, the exact fraction varying a bit based on the total votes cast. For example, I don't think 4-3 should be enough for cloture, but 40-30 would be.

Bob JacobsApr 2916

90% disagree

Not only that, it incentivizes detractors to go back and downvote your other stuff as well. When I was coming out against HBD, older things I had written also got downvoted (and I lost voting power).
This doesn't make sense on other forums but here it's perfectly reasonable since with karma you're not just deciding "how good is this post/comment?" but also "who gets voting power?". So if you want the forum to remain dominated by your ingroup, better upvote your ingroup's posts/comments while downvoting everything by the outgroup. Not necessarily because you want to, but just because that's how the system is set up.

The only reason why I don't go full disagree is because I could see a system akin to "liquid democracy" where you can give proxies or where once in a while we vote on which people will have more voting power for the next term.

In any case, we should expect some heavy survivorship bias here in favor of the status-quo since EAs or potential EAs who get turned off by the karma system will either fully or largely leave the forum (e.g. me).

Ebenezer DukakisApr 3019

Retributive downvoting appears to be a bannable offense, according to the forum guide:

https://forum.effectivealtruism.org/posts/yND9aGJgobm5dEXqF/guide-to-norms-on-the-forum#Voting_norms

I suggest you take your case up with the admins.

More generally, perhaps it would be valuable to publicize the voting guide better? E.g. every time my mouse hovers over a voting widget, a random voting guideline could pop up, so over time I would learn all of the guidelines. @Sarah Cheng

I think the risk of groupthink death spirals is real, and I suspect I've been on the receiving end of it. "With great power comes great responsibility."

In any case, we should expect some heavy survivorship bias here in favor of the status-quo since EAs or potential EAs who get turned off by the karma system will either fully or largely leave the forum (e.g. me).

Do you post on the EA subreddit? Everyone's vote power is equal there:

https://reddit.com/r/EffectiveAltruism/

IMO, the discussion quality on the subreddit is not great. I'm unsure if that's because it lacks scaled vote power, or simply because it has fewer serious EAs and more random redditors. I wonder what would happen if serious EAs made a dedicated effort to post on the subreddit, and bring the random redditors up to speed more etc.

Sarah ChengApr 305

I appreciate the suggestion! :) I've added it to our backlog. My current guess is that, given the limited resources we have to spend, this probably won't meet our bar for being cost-effective enough to implement.

Ebenezer DukakisMay 12

Up to you. But I think voting does a tremendous amount to influence the forum's culture. Nudging people towards voting wisely, and talking about how to vote, seems pretty high-leverage to me. Right now, my sense is we're in a bit of a bad place, where people take karma scores too seriously given the low amount of thought that goes into them.

Bob JacobsApr 304

Do you post on the EA subreddit? Everyone's vote power is equal there:

Yes, I do post there. It's...fine. I don't exactly love it, but it at least doesn't give me an active feeling of disgust every time I use it (which the forum does).

Retributive downvoting appears to be a bannable offense, according to the forum guide:

This is unenforceable. In fact that whole section is unenforceable:

Additionally, please avoid:
Asking your friends or coworkers to vote on a post, especially if you might be biased (e.g. because the post is criticizing your work, or because your friend wrote the post)
We think sharing a post in a public channel and saying “Hey, I quite like this post that summarizes my organization’s work is cool, check it out” is fine. If you see a message like this, evaluate the post on its own merits; don’t just go upvote because someone you know wrote it.
But posting — or even worse, saying this on a call — “Hey, everyone, please go upvote this post that our organization just shared, we need everyone on the Forum to see it” is bad. Even worse is asking people to downvote criticism of something you work on.
Deferring entirely to someone else (your vote should be your own)
Additionally, please try to judge each post or comment on its own merits; don’t just vote based on whether or not you like the poster’s other activity.
Other than that, you can vote using your preferred criteria. Here are our suggestions:
Action If… Not if…
Strong-upvote
Reading this will help people do good.
You learned something important.
You think many more people might benefit from seeing it.
You want to signal that this sort of behavior adds a lot of value.
“I agree and want others to see this opinion first.”
Upvote
You think it adds something to the conversation, or you found it useful.
People should imitate some aspect of the behavior in the future.
You want others to see it.
You just generally like it.
“Oh, I like the poster, they’re cool.”
Downvote
There’s a relevant error.
The comment or post didn’t add to the conversation, and maybe actually distracted.
“There are grammatical errors in this comment.”
Strong-downvote
It contains many factual errors and bad reasoning
It’s manipulative or breaks our norms in significant ways (consider reporting it)
It’s literally spam (consider reporting it)
“I disagree with this opinion.”

I agree that if everyone followed these norms there wouldn't be a problem, however they aren't, and there isn't a way to make them. Worse, the incentives work against a lot of these norms:

Ebenezer DukakisMay 12

Voting is anonymous

I don't believe that is true for admins:

We will try to maximally protect privacy and pseudonymity, as long as it does not seriously interfere with our ability to enforce important norms on the Forum.

https://forum.effectivealtruism.org/posts/yND9aGJgobm5dEXqF/guide-to-norms-on-the-forum

This forum is fairly small. It seems relatively feasible for the admins to enforce norms manually.

But in any case, I encourage you to prove me wrong. I encourage you to reach out to the admins, and then report back here when nothing useful happens, as you seem to be predicting.

Bob JacobsMay 12

I don't believe that is true for admins

They literally say so:

Voting activity is generally private (even admins don't know who voted on what), but if we have reason to believe that someone is violating norms around voting (e.g. by mass-downvoting many of a different user's comments and posts), we reserve the right to check what account is doing this.

That's why I said:

Voting is anonymous, so unless you "mass" vote it will remain undetected.

The examples I gave --downvoting based on opinion not content, downvoting based on ideology, upvoting your ingroup, upvoting because they're you friends-- are all things that can be done while staying anonymous.

But in any case, I encourage you to prove me wrong. I encourage you to reach out to the admins, and then report back here when nothing useful happens, as you seem to be predicting.

You think I haven't done that? I even send a comment to Ben West publicly and people downvoted me for it:

Ebenezer DukakisMay 12

The examples I gave --downvoting based on opinion not content, downvoting based on ideology, upvoting your ingroup, upvoting because they're you friends-- are all things that can be done while staying anonymous.

Your initial complaint was mass-downvoting, which is explicitly called out in the FAQ (based on your own quote!) as something the admins are willing to de-anonymize for, no?

You think I haven't done that?

If you had done it, I would expect your initial comment to contain something along the lines of: "I reached out privately to the admins, through standard channels, to complain about mass-downvoting. Despite the forum guidelines, they didn't do anything. Their stated reason was X."

Bob JacobsMay 14

My complaint was the incentive structure:

Not necessarily because you want to, but just because that's how the system is set up.

I used a personal example, but the complaint was about people being incentivized to downvote (past and future) stuff by the outgroup while upvoting the ingroup, whether or not it's "mass" voting:

it incentivizes detractors to go back and downvote your other stuff as well. [...]
So if you want the forum to remain dominated by your ingroup, better upvote your ingroup's posts/comments

which I then expanded on with examples like:

If someone is spreading opinions you disagree with, then the karma system makes strong-downvoting them an excellent way to hinder their ability to do so.
If your friend makes mediocre posts but he also upvotes all your mediocre posts, then upvoting them is a great way to ensure your posts get more exposure.

If you had done it

I had done it, see: my screenshotted comment.

abrahamroweApr 2814

67% agree

Vote power should scale with karma

I think that the upside of the system is high, and that EA Forum posts have been pretty effective in changing community direction in the past, so the downside risk seems low. My impression (as someone who has posted things that aren't particularly popular at times) is that well reasoned-but-disagreed-with posts still get lots of upvotes.

Robi RahmanApr 284

EA Forum posts have been pretty effective in changing community direction in the past, so the downside risk seems low

But giving more voting power to people with lots of karma entrenches the position/influence of people who are already high in the community based on its current direction, so it would be an obstacle to the possibility of influencing the community through forum posts.

If you think it's important for forum posts to be able to change community direction, you should be against vote power scaling with karma.

Neel NandaApr 2920

This presupposes that the way something gets to change community direction is by having high karma, while I think it's actually about being well reasoned and persuasive AND being viewed. Being high karma helps it be viewed, but this is neutral to actively negative if the post is low quality/flawed - that just entrenches people in their positions more/makes them think less of the forum. So in order for this change to help, there must be valuable posts that are low karma that would be high karma if voting was more democratic - I personally think that the current system is better at selecting for quality and this outweighs any penalty to dissenting opinions, which I would guess is fairly minor in practice

abrahamroweApr 289

I think my view is that while I agree in principle it could be an issue, the voting has worked this way for long enough that I'd expect more evidence of entrenching to exist. Instead, I still see controversial ideas change people's minds on the forum pretty regularly and not be downvoted to oblivion, and see low quality or bad faith posts/comments get negative karma, and I think that's the upside of the system working well.

Denis Apr 3011

80% agree

People who have devoted more time and energy to EA and have a deeper grasp of it should have a bigger role in defining what is or isn't worth other people reading. It's not just judgment (is it right or wrong), it's also originality - is this a new opinion for EA's to think about? is this a topic which EA's haven't really engaged? It's hard for a new person to make these calls.

Karma is a reasonably good indicator of meaningful engagement with the EA forum - as good as any other that can be quickly and fairly calculated.

I would add one caveat: to use a more powerful supervote, a person should be required to add a comment. From personal experience, I am very happy to have dissenting opinions and arguments against my posts, but it's frustrating to get downvotes without any explanation.

Knight LeeApr 28*9

22% agree

My silly idea is that your voting power should not scale with your karma directly, but should scale with the number of unique upvotes minus the number of unique downvotes you received. This prevents circular feedback.

Reasons

Hypothetically, if you had two factions which consistently upvote themselves, A with 67 people, and B with 33 people. People in A will have twice as many unique upvotes as people in B, and their comments can have up to 4 times more karma (in the simplistic case where voting power scales linearly with karma).

However, if voting power depends not on unique upvotes but on karma, then at first people in A will still have twice as many unique upvotes as people in B, and their comments will still have more than 4 times more karma. But then, (in the simplistic case where voting power scales linearly with karma), their comments will have 8 times more karma. Which further causes their comments to have 16 times more karma.

This doesn't happen in practice because voting power doesn't scale linearly with karma (thank goodness), but circular feedback is still partially a problem.

Stan PinsentApr 287

44% agree

Vote power should scale with karma

It's likely that karma correlates with good ability to judge Forum content. There is a risk that this gives too much power to incumbents, but this dynamic probably served the Forum well when usage exploded in 2021/2022.

CB🔸Apr 296

56% agree

Vote power should scale with karma

I find the content on the Forum exceptionally good when compared to all other online spaces I've been to. I don't know what share the karma votes played in that, but I tend to assume it contributed.

Guy RavehApr 286

44% disagree

Vote power should scale with karma

It's Ok to give users with really small karma less power, but otherwise EA has the wrong idea that if someone has read much/thought a lot about something it means they understand it better.

MichaelDickensApr 289

That makes sense to me—I would be more okay with vote power scaling if it was capped. Maybe it scales linearly up to 1000 karma and then stops scaling.

Jobst Heitzig (vodle.it)Apr 305

100% disagree

We need to push back on echo bubbles.

Bob JacobsApr 302

Are all the voting theorists on the 'disagree' side or is there a voting theorist on the 'agree' side I don't know about?

Rían O'M May 14

~~90%~~ ➔ 80% agree

Vote power should scale with karma

On group think: I think this worry can mostly be ignored if the elite-karma accounts have sufficiently diverse views. That being the case would mean that (a) diverse views aren't obviously being punished and (b) to voters with the most individual leverage are less likely to all vote in the same direction. If they top karma accounts were all aligned in how they vote or were even colluding to suppress comments / posts, then the downsides of group-think would be more pronounced.

I guess It feels like this should be a testable claim: do the most upvoted posts / comments conform to the views of the highest karma users? Given how diverse the viewpoints are of the +5000 karma plus users (even just the top twenty), I'm not even sure their is a single coherent view among the karma-elite.

Among the karma-elite are a few OpenPhil / CEA accounts, whilst the second highest upvoted account is Habryka (arguably OpenPhil's antichrist^[1] and one of CEA's largest critique).

I haven't really thought about all the angles though and can imagine something like "the EA forum voting overly favours the bucket of people who have ~1000 karma and use the form everyday."

Probably the risk of group-think is more dependant on who the forum users are, rather than the specific mechanics of the forum.

^{^}
I'm using this phrase as a tongue-and-cheek reference to Habryka hailing a kind of end time for OpenPhil; "I think OpenPhil is kind, is like dead in the sense of the historical role it has played within the AI safety and EA ecosystem."

AustinApr 294

~~80%~~ ➔ 90% agree

Vote power should scale with karma

This gives EA Forum and LessWrong a very useful property of markets: more influence accrues to individuals who have a good track record of posting.

SofiaBaldersonApr 294

60% agree

Vote power should scale with karma

I think community members with more experience of engagement can help upvote posts which are in line with the topics that the community might value more. This helps all community members identify valuable posts, as there is a lot to read. However I think that at some point your vote weight should stop growing and be capped at a certain number, but I’m unsure how this works currently.

Ebenezer DukakisMay 12

20% agree

Vote power should scale with karma

I think vote power has helped maintain a culture of expertise here on the forum. It's hard to find quality discussion on the internet.

However, I think concerns about groupthink are valid. We could try having a section of the forum with no voting as a "control group", for instance. Or tweak the algorithm to penalize feedback loops.

Joseph_ChuApr 302

10% disagree

Vote power should scale with karma

I'm ambivalent about this. On the one hand, I'm partial to the ideal of "one person, one vote" that modern liberal democracies are built on. On the other hand, I do find scaling with karma in some way to be an interesting use of karma that makes it more important than just for bragging rights, which I like from a system design perspective.

Elliot Billingsley 🔸Apr 302

30% agree

Vote power should scale with karma somewhat.

Elliot Billingsley 🔸Apr 301

this slider poll DIY debate thing is SO COOL

MichaelDickensApr 282

56% disagree

Vote power should scale with karma

I dislike strong votes and I personally never use them. I agree with OP's criticisms.

(The exception is that my own posts are strong-upvoted by default, and I don't undo this because that would be unfairly penalizing my own posts relative to other people's.)

There is an argument that perhaps power users are "better" at voting. But from the voting patterns I've observed, my impression is that power users are still too liable to upvote shallow posts, drama-y posts, etc. I have some uncertainty about this because it could be that the median user's voting patterns are even worse.

Henry Howard🔸Apr 282

44% disagree

Vote power should scale with karma

Risks groupthink (this is hard to prove). Also bad optics to outsiders (looks liable to groupthink, cultish).
The benefit that it makes sock-puppeteering harder means I'm a little ambivalent

Marcus Abramovitch 🔸Apr 282

78% agree

Vote power should scale with karma

To a point, maybe a bit less than it does currently but in general it seems to work well.

jackvaApr 282

56% agree

Vote power should scale with karma

KaleemApr 282

56% agree

Vote power should scale with karma

Robi RahmanApr 282

11% agree

Vote power should scale with karma

EdoAradApr 282

56% agree

Vote power should scale with karma

Assuming at least half of the upvote value of high-karma posts-comments is from people with base upvote value.itd be interesting to check how that is actually distributed

Eli Svoboda🔸Apr 291

33% agree

Vote power should scale with karma

[comment deleted]Apr 282

33% agree

Deleted by JWS 🔸, 04/28/2025

[comment deleted]Apr 291

50% agree

Deleted by cb, 04/29/2025

Action	If…	Not if…
Strong-upvote	Reading this will help people do good. You learned something important. You think many more people might benefit from seeing it. You want to signal that this sort of behavior adds a lot of value.	“I agree and want others to see this opinion first.”
Upvote	You think it adds something to the conversation, or you found it useful. People should imitate some aspect of the behavior in the future. You want others to see it. You just generally like it.	“Oh, I like the poster, they’re cool.”
Downvote	There’s a relevant error. The comment or post didn’t add to the conversation, and maybe actually distracted.	“There are grammatical errors in this comment.”
Strong-downvote	It contains many factual errors and bad reasoning It’s manipulative or breaks our norms in significant ways (consider reporting it) It’s literally spam (consider reporting it)	“I disagree with this opinion.”