Thanks, great points. I agree that we should only be interested in good faith arguments — we should be clear about that in the judging criteria, and clear about what counts as a bad faith criticism. I think the Forum guidelines are really good on this.

Of course, it is possible to strongly disagree with a claim without resorting to bad faith arguments, and I'm hopeful that the best entrants can lead by example.

I had a very similar reaction. Here's a comment I'd previously offered in response to this idea:

I think "sceptical critiques" tend to be less good and useful, on average than "impassive, unbiased questioning". Commissioning critiques is especially awkward, because then authors have to stick with the X is bad conclusion, even if that's not where their investigation leads them. A better framing IMO is to say that you want people to question major premises, arguments, and conclusions of EA, and that you will accept pure sceptical critiques as a part of that, but that if, from investigating an issue, an author end up with a defense of an argument that fortifies EA, then that's great too.

Worth noting that the post mentions that minimal-trust investigations are in scope. From that link:

The basic idea of a minimal-trust investigation is suspending one's trust in others' judgments and trying to understand the case for and against some claim oneself, ideally to the point where one can (within the narrow slice one has investigated) keep up with experts.

I think a minimal trust investigation can end up being positive or negative. I suppose one could start off with a minimal trust investigation which could then turn into a red team if one disagrees with the generally accepted viewpoint.

I personally feel uncomfortable with a criterion being "critical" for the reasons you and others have mentioned.

Thank you, this is a really good point. By 'critical' I definitely intended to convey something more like "beginning with a critical mindset" (per JackM's comment) and less like "definitely ending with a negative conclusion in cases where you're critically assessing a claim you're initially unsure about". 

This might not always be relevant. For instance, you might set out to find the strongest case against some claim, whether or not you end up endorsing it. As long as that's explicit, it seems fine.

But in cases where someone is embarking on something like a minimal-trust investigation — approaching an uncertain claim from first principles — we should be incentivising the process, not the conclusion!

We'll try to make sure to be clear about that in the proper announcement.

I think a list like this might be useful for other purposes too:

• raising the profile of critiques that matter amongst EAs, thus hopefully improving people's thinking
• signalling that criticism genuinely is welcome and seen as useful

Yes, totally. I think a bunch of the ideas in the comments on that post would be a great fit for this contest.

“Not being easy to criticise even if the criticism is valid” seems like an excellent critique of effective altruism.

Your description of practical critiques being difficult to steelman with only anecdata available feels like the classic challenge of balancing type I and type II error when reality is underpowered.

 In the context of a contest encouraging effective altruism critiques, I think we maybe want to have a much higher tolerance than usual for type I error in order to get less type II error (I am thinking of the null hypothesis as “critique is false”, so the type I error would be accepting a false critique and type II error would be rejecting a true critique). 

Obviously, there needs to be some chance that the critique holds. However, it seems very valuable to encourage critiques that would be a big deal if true even if we’re very uncertain about the assumptions, especially if the assumptions are clear and possible to test with some amount of further investment (eg. by adding a question to next year’s EA survey or getting some local groups to ask their new attendees to fill out an anonymous survey on their impressions of the group).

This makes me think that maybe a good format for EA critiques is a list of assumptions (maybe even with the authors’ credences that they all hold and their reasoning), and then the outlined critique if those assumptions are true. If criticisms clearly lay out their assumptions, even if,  say, we guess that there is a, say, 70% chance that the assumptions don’t hold, in the 30% of possible worlds where they do hold up (assuming our guess was well-calibrated :P), having the hypothetical implications written up still seems very valuable (to help us work out if it's worth investigating these assumptions further/to get us to pay more attention to evidence for and against the hypothesis that we live in that 30% world/to get us to think about whether there are low-cost actions we can take just in case we live in that 30% world). 

Hi EKillian! Could you provide some more context on what you're interested in? Anyone will be welcome to write a submission. If you're more interested in helping others with their work, you could say a bit more about that here in the comments, and then perhaps someone will reach out.

In terms of serving as a judge in the competition, we haven't finalised the process for selecting judges – but it would be helpful if you could DM with some more information.