I used an LLM to help draft this post, but I’ve edited/rewritten it extensively and endorse it.
AI in Context is a channel about transformative AI and its risks, published by 80,000 Hours.
Writing up our current approach to thumbnails, which is nowhere near perfect, for easy shareability and cross-pollination of lessons. Would love to hear what other people are trying!
Making thumbnails
We're lucky enough to have folks at 80k with great design instincts. We work with them as well as with some external folks, but finding great people is harder than we expected. Let us know if this is something you or someone you know would be great at!
We iterate way more than people expect
Every video gets ~dozens of thumbnail variations, most of which are made after launch. You can see the full set of data on our IABIED thumbnails here.
I believe 2/3 of our winning thumbnails (maybe all 3) were made after launch. It's pretty hard to predict ahead of time which thumbnails will do well.
We launch with a few thumbnails we're excited about, a/b/c testing them
We tried pre-launch testing via paid ads once, didn't correlate well, but we haven't tried super intensely
We iterate from there. If one is doing well, or the video is doing well, we do new ones similar to what we've tried. If the views are lower or nothing is breaking out, we try more variance. This could be from thumbnails we had ready, or new ones we make with the new information. We also swap out titles, usually not at the same time so we get the full information.
We have someone checking ~continuously through the first few days.
Tests run for about six hours. For our view rate, tests stabilize around six hours. If you check at three hours and again at six, the numbers often shift meaningfully. After six, they tend to settle.
YouTube's AB testing data updates roughly every 30 minutes, not in real time. You'll see the same numbers for a while and then a sudden jump.
We have a Slack thread for every video where we dump every thumbnail iteration and every test result. It's kind of unhinged and there's probably a better way to do it.
If one thumbnail is clearly tanking, we'll cut the test early rather than wait for statistical significance
When views are low and the test is running but not informative, sometimes the right move is to stop testing entirely for a while and just let the winning thumbnail run since the algorithm is sometimes looking for your audience.
What we've learned
Big text is good
Graphs are surprisingly strong
Host face helps, but not always
e.g. this is our winning MechaHitler thumbnail
Having a "glowing" comment in the thumbnail (that is, grabbing a really positive comment on the video and putting it in the thumbnail) sometimes works but not always. Veritasium and 3B1B do this. We have a theory that it needs to be really specific.
BLUF:
* To determine whether AI is ‘improving exponentially’, ‘hitting the wall’, or any other claim which involves a quantity or magnitude (e.g. ‘This model was a big leap/small increment’). We need a good y-axis: an interval scale of AI capability which means +1 unit always represents the same degree of ‘how much better’, in the same way +1 degree Celsius is always the same amount of ‘how much hotter’.
* Yet there is no good y-axis for AI capability. All our...
Summary
* The animal welfare movement has already seen an influx in funding and should prepare for the possibility of more.
* The EA Animal Welfare Fund is encouraging those working in animal advocacy to actively set aside time and resources now to concretely plan for scaling sustainably, and we’ll support you in doing that.
* We’re requesting advocates set concrete ambitious goals and submit plans t...
Public service announcement
1. Applications are now open for our first ever round of the Charity Entrepreneurship Incubation Program dedicated exclusively to animal welfare. Learn more about what’s different this round here and apply...
