Paying candidates to complete a test task likely increases inequality, credentialism and decreases candidate quality. If you pay candidates for their time, you're likely to accept less candidates and lower variance candidates into the test task stage. Orgs can continue to pay top candidates to complete the test task, if they believe it measurably decreases the attrition rate, but give all candidates that pass an anonymised screening bar the chance to complete a test task.
Orgs can continue to pay top candidates to complete the test task, if they believe it measurably decreases the attrition rate, but give all candidates that pass an anonymised screening bar the chance to complete a test task.
My guess is that, for many orgs, the time cost of assessing the test task is larger than the financial cost of paying candidates to complete the test task, and that significant reasons for wanting to compensate applicants are (i) a sense of justice, (ii) wanting to avoid the appearance of unreasonably demanding lots of unpaid labour from applicants, not just wanting to encourage applicants to complete the tasks[1].
So I agree that there are good reasons for wanting more people to be able to complete test tasks. But I think that doing so would potentially significantly increase costs to orgs, and that not compensating applicants would reduce costs to orgs by less than one might imagine.
I also think the justice-implications of compensating applicants are unclear (offering pay for longer tasks may make them more accessible to poorer applicants)
It takes a significant amount of time to mark a test task. But this can be fixed by just adjusting the height of the screening bar, as opposed to using credentialist and biased methods (like looking at someone's LinkedIn profile or CV).
My guess is that, for many orgs, the time cost of assessing the test task is larger than the financial cost of paying candidates to complete the test task
This is an empirical question, and I suspect is not true. For example, it took me 10 minutes to mark each candidates 1 hour test task. So my salary would need to be 6* higher (per unit time) than the test task payment for this to be true.
I also think the justice-implications of compensating applicants are unclear (offering pay for longer tasks may make them more accessible to poorer applicants)
So my salary would need to be 6* higher (per unit time) than the test task payment for this to be true.
Strictly speaking your salary is the wrong number here. At a minimum, you want to use the cost to the org of your work, which is your salary + other costs of employing you (and I've seen estimates of the other costs at 50-100% of salary). In reality, the org of course values your work more highly than the amount they pay to acquire it (otherwise... why would they acquire it at that rate) so your value per hour is higher still. Keeping in mind that the pay for work tasks generally isn't that high, it seems pretty plausible to me that the assessment cost is primarily staff time and not money.
I see a dynamic playing out here, where a user has made a falsifiable claim, I have attempted to falsify it, and you've attempted to deny that the claim is falsifiable at all.
I recognise it's easy to stumble into these dynamics, but we must acknowledge that this is epistemically destructive.
Strictly speaking your salary is the wrong number here.
I don't think we should dismiss empirical data so quickly when it's brought to the table - that sets a bad precedent.
other costs of employing you (and I've seen estimates of the other costs at 50-100% of salary
I can also provide empirical data on this if that is the crux here?
Notice that we are discussing a concrete empirical data point, that represents a 600% difference, while you've given a theoretical upper bound of 100%. That leaves a 500% delta.
Keeping in mind that the pay for work tasks generally isn't that high
Would you be able to provide any concrete figures here?
In reality, the org of course values your work more highly than the amount they pay to acquire it
I view pointing to opportunity cost in the abstract as essentially an appeal to ignorance.
Not to say that opportunity costs do not exist, but you've failed to concretise them in a way, and that makes it hard to find the truth here.
I could make similar appeals to ignorance in support of my argument, like the idea the benefit of getting a better candidate is very high, as candidate performance is fat-tailed ect. - but I believe this is similarly epistemically destructive. If I were to make a similar claim, I would likely attempt to concretise it.
I see a dynamic playing out here, where a user has made a falsifiable claim, I have attempted to falsify it, and you've attempted to deny that the claim is falsifiable at all.
My claim is that the org values your time at a rate that is significantly higher than the rate they pay you for it, because the cost of employment is higher than just salary and because the employer needs to value your work above its cost for them to want to hire you. I don't see how this is unfalsifiable. Mostly you could falsify them by asking orgs how they think about the cost of staff time, though I guess some wouldn't model it as explicitly as this.
They do mean that we're forced to estimate the relevant threshold instead of having a precise number, but a precise wrong number isn't better than an imprecise (closer to) correct number.
Notice that we are discussing a concrete empirical data point, that represents a 600% difference, while you've given a theoretical upper bound of 100%. That leaves a 500% delta.
No, if you're comparing the cost of doing 10 minutes of work at salary X and 60 minutes of work compensated by Y, but I argue that salary X underestimates the cost of your work by a factor of 2, your salary now only needs to be more than 3 times larger than the work trial compensation, not 5 times.
When it comes to concretising "how much does employee value exceed employee costs", it probably varies a lot from organisation to organisation. I think there are several employers in EA who believe that after a point, paying more doesn't really get you better people. This allows their estimates of value of staff time to exceed employee costs by enormous margins, because there's no mechanism to couple the two together. I think when these differences are very extreme we should be suspicious if they're really true, but as someone who has multiple times had to compare earning to give with direct work, I've frequently asked an org "how much in donations would you need to prefer the money over hiring me?" and for difficult-to-hire roles they frequently say numbers dramatically larger than the salary they are offering.
This means that your argument is not going to be uniform across organisations, but I don't know why you'd expect it to be: surely you weren't saying that no organisation should ever pay for a test task, but only that organisations shouldn't pay for test tasks when doing so increases their costs of assessment to the point where they choose to assess fewer people.
My expectation is that if you asked orgs about this, they would say that they already don't choose to assess fewer people based on cost of paying them. This seems testable, and if true, it seems to me that it makes pretty much all of the other discussion irrelevant.
It takes a significant amount of time to mark a test task. But this can be fixed by just adjusting the height of the screening bar, as opposed to using credentialist and biased methods (like looking at someone's LinkedIn profile or CV).
Whether or not to use "credentialist and biased methods (like looking at someone's LinkedIn profile or CV)" seems orthogonal to the discussion at hand?
The key issue seems to be that if you raise the screening bar, then you would be admitting fewer applicants to the task (the opposite of the original intention).
This is an empirical question, and I suspect is not true. For example, it took me 10 minutes to mark each candidates 1 hour test task. So my salary would need to be 6* higher (per unit time) than the test task payment for this to be true.
This will definitely vary by org and by task. But many EA orgs report valuing their staff's time extremely highly. And my impression is that both grading longer tasks and then processing the additional applicants (many orgs will also feel compelled to offer at least some feedback if a candidate has completed a multi-hour task) will often take much longer than 10 minutes total.
Feature request: discussion threads on episodes of the 80k podcasts.
I consistently generate a lot of thoughts listening to the 80k pocast. I could imagine leaving these as comments, reading others thoughts, and identifying cruxes. To further steelman the need of podcast threads: the 80k podcast is the most widely consumed, indepth media produced in the EA sphere at the moment.
Episode highlights tend to be shared under this account, which might be a good place to leave thoughts.
Just FYI: feature suggestions for the Forum could also go on the Forum feature suggestion thread. (I'm not sure if this is meant to be a feature request for the Forum or for 80,000 Hours, though! If it's the latter, you might want to get in touch more directly.)
Anyone can create a linkpost for an 80k episode. Though it might be extra convenient to have a way to automatically create a linkpost with a pre-filled summary of the linked page and a top-level comment with your thoughts.
Couldn't the comment section under the episode announcement posts (like this one) serve the same purpose? Or are you imagining a different kind of discussion thread here?
That is preciously what I am looking for. I feel a bit silly now, because I haven't noticed these previously.
FWIW, @80000_Hours I think the formatting "[#Ep] [podcast title]" is a much better title/formatting than how they were linked previous to the last episode.
Biological life forms experience unequal (asymmetrical) amounts of pleasure and pain. This asymmetry is important. It's why you cannot make up for starving someone for a week by giving them food for a week.
This is true for biological life, because a selection pressure was applied (evolution by natural selection). This selection pressure is necessitated by entropy, because it's easier to die than it is to live. Many circumstances result in death, only a narrow band of circumstances results in life. Incidentally, this is why you spend most of your life in a temperature controlled environment.
The crux: there's no reason to think that a similar selection effect is being applied to AI models. LLMs, if they were sentient, would be equally as likely to enjoy predicting the next token as to dislike predicting the next token.
Paying candidates to complete a test task likely increases inequality, credentialism and decreases candidate quality. If you pay candidates for their time, you're likely to accept less candidates and lower variance candidates into the test task stage. Orgs can continue to pay top candidates to complete the test task, if they believe it measurably decreases the attrition rate, but give all candidates that pass an anonymised screening bar the chance to complete a test task.
My guess is that, for many orgs, the time cost of assessing the test task is larger than the financial cost of paying candidates to complete the test task, and that significant reasons for wanting to compensate applicants are (i) a sense of justice, (ii) wanting to avoid the appearance of unreasonably demanding lots of unpaid labour from applicants, not just wanting to encourage applicants to complete the tasks[1].
So I agree that there are good reasons for wanting more people to be able to complete test tasks. But I think that doing so would potentially significantly increase costs to orgs, and that not compensating applicants would reduce costs to orgs by less than one might imagine.
I also think the justice-implications of compensating applicants are unclear (offering pay for longer tasks may make them more accessible to poorer applicants)
I think that many applicants are highly motivated to complete tasks, in order to have a chance of getting the job.
It takes a significant amount of time to mark a test task. But this can be fixed by just adjusting the height of the screening bar, as opposed to using credentialist and biased methods (like looking at someone's LinkedIn profile or CV).
This is an empirical question, and I suspect is not true. For example, it took me 10 minutes to mark each candidates 1 hour test task. So my salary would need to be 6* higher (per unit time) than the test task payment for this to be true.
This is a good point.
Strictly speaking your salary is the wrong number here. At a minimum, you want to use the cost to the org of your work, which is your salary + other costs of employing you (and I've seen estimates of the other costs at 50-100% of salary). In reality, the org of course values your work more highly than the amount they pay to acquire it (otherwise... why would they acquire it at that rate) so your value per hour is higher still. Keeping in mind that the pay for work tasks generally isn't that high, it seems pretty plausible to me that the assessment cost is primarily staff time and not money.
I see a dynamic playing out here, where a user has made a falsifiable claim, I have attempted to falsify it, and you've attempted to deny that the claim is falsifiable at all.
I recognise it's easy to stumble into these dynamics, but we must acknowledge that this is epistemically destructive.
I don't think we should dismiss empirical data so quickly when it's brought to the table - that sets a bad precedent.
I can also provide empirical data on this if that is the crux here?
Notice that we are discussing a concrete empirical data point, that represents a 600% difference, while you've given a theoretical upper bound of 100%. That leaves a 500% delta.
Would you be able to provide any concrete figures here?
I view pointing to opportunity cost in the abstract as essentially an appeal to ignorance.
Not to say that opportunity costs do not exist, but you've failed to concretise them in a way, and that makes it hard to find the truth here.
I could make similar appeals to ignorance in support of my argument, like the idea the benefit of getting a better candidate is very high, as candidate performance is fat-tailed ect. - but I believe this is similarly epistemically destructive. If I were to make a similar claim, I would likely attempt to concretise it.
My claim is that the org values your time at a rate that is significantly higher than the rate they pay you for it, because the cost of employment is higher than just salary and because the employer needs to value your work above its cost for them to want to hire you. I don't see how this is unfalsifiable. Mostly you could falsify them by asking orgs how they think about the cost of staff time, though I guess some wouldn't model it as explicitly as this.
They do mean that we're forced to estimate the relevant threshold instead of having a precise number, but a precise wrong number isn't better than an imprecise (closer to) correct number.
No, if you're comparing the cost of doing 10 minutes of work at salary X and 60 minutes of work compensated by Y, but I argue that salary X underestimates the cost of your work by a factor of 2, your salary now only needs to be more than 3 times larger than the work trial compensation, not 5 times.
When it comes to concretising "how much does employee value exceed employee costs", it probably varies a lot from organisation to organisation. I think there are several employers in EA who believe that after a point, paying more doesn't really get you better people. This allows their estimates of value of staff time to exceed employee costs by enormous margins, because there's no mechanism to couple the two together. I think when these differences are very extreme we should be suspicious if they're really true, but as someone who has multiple times had to compare earning to give with direct work, I've frequently asked an org "how much in donations would you need to prefer the money over hiring me?" and for difficult-to-hire roles they frequently say numbers dramatically larger than the salary they are offering.
This means that your argument is not going to be uniform across organisations, but I don't know why you'd expect it to be: surely you weren't saying that no organisation should ever pay for a test task, but only that organisations shouldn't pay for test tasks when doing so increases their costs of assessment to the point where they choose to assess fewer people.
My expectation is that if you asked orgs about this, they would say that they already don't choose to assess fewer people based on cost of paying them. This seems testable, and if true, it seems to me that it makes pretty much all of the other discussion irrelevant.
Whether or not to use "credentialist and biased methods (like looking at someone's LinkedIn profile or CV)" seems orthogonal to the discussion at hand?
The key issue seems to be that if you raise the screening bar, then you would be admitting fewer applicants to the task (the opposite of the original intention).
This will definitely vary by org and by task. But many EA orgs report valuing their staff's time extremely highly. And my impression is that both grading longer tasks and then processing the additional applicants (many orgs will also feel compelled to offer at least some feedback if a candidate has completed a multi-hour task) will often take much longer than 10 minutes total.
80K podcast discussion threads.
Feature request: discussion threads on episodes of the 80k podcasts.
I consistently generate a lot of thoughts listening to the 80k pocast. I could imagine leaving these as comments, reading others thoughts, and identifying cruxes. To further steelman the need of podcast threads: the 80k podcast is the most widely consumed, indepth media produced in the EA sphere at the moment.
Episode highlights tend to be shared under this account, which might be a good place to leave thoughts.
Just FYI: feature suggestions for the Forum could also go on the Forum feature suggestion thread. (I'm not sure if this is meant to be a feature request for the Forum or for 80,000 Hours, though! If it's the latter, you might want to get in touch more directly.)
That's totally it, thanks for flagging, I have not seen these previously (not sure why).
Anyone can create a linkpost for an 80k episode. Though it might be extra convenient to have a way to automatically create a linkpost with a pre-filled summary of the linked page and a top-level comment with your thoughts.
Couldn't the comment section under the episode announcement posts (like this one) serve the same purpose? Or are you imagining a different kind of discussion thread here?
That is preciously what I am looking for. I feel a bit silly now, because I haven't noticed these previously.
FWIW, @80000_Hours I think the formatting "[#Ep] [podcast title]" is a much better title/formatting than how they were linked previous to the last episode.
Sentient AI ≠ AI Suffering.
Biological life forms experience unequal (asymmetrical) amounts of pleasure and pain. This asymmetry is important. It's why you cannot make up for starving someone for a week by giving them food for a week.
This is true for biological life, because a selection pressure was applied (evolution by natural selection). This selection pressure is necessitated by entropy, because it's easier to die than it is to live. Many circumstances result in death, only a narrow band of circumstances results in life. Incidentally, this is why you spend most of your life in a temperature controlled environment.
The crux: there's no reason to think that a similar selection effect is being applied to AI models. LLMs, if they were sentient, would be equally as likely to enjoy predicting the next token as to dislike predicting the next token.