Dr. in math. AI Alignment and Safety researcher. Bayesian.
Science YouTuber, podcaster, writer.
Author of the books "The Equation of Knowledge", "Le Fabuleux Chantier" and "Turing à la plage
NB: I've edited the sentence to clarify what I meant.
The argument here is more that recommender systems are maximization algorithms, and that, if you buy the "orthogonality thesis", there is no reason to think that there cannot go AGI. In particular, you should not judge the capability of an algorithm by the simplicity of the task it is given.
Of course, you may reject the orthogonality thesis. If so, please ignore the first argument.
I would just like to point out three "classical EA" arguments for taking recommender systems very seriously.
1) The dangerousness of AGI has been argued to be orthogonal from the purpose of AGI, as illustrated by the paperclip maximizers. If you accept this "orthogonality thesis" and if you are concerned about AGI, then you should be concerned about the most sophisticated maximization algorithms. Recommender systems seem to be today's most sophisticated maximization algorithms (a lot more money and computing power has been invested in optimizing recommender systems than in GPT-3). Given the enormous economic incentives, we should probably not discard the probability that they will remain the most sophisticated maximization algorithms in the future.
As a result, arguments of the form "I don't see how recommender systems can pose an existential threat" seem akin to arguments of the form "I don't see how AGI can pose an existential threat".
(of course, if you reject the latter, I can see why you could reject the former 🙂)
2) Yudkowsky argues that “By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.” Today's recommender systems are typical examples of something "that people conclude too early that they understand it". Such algorithms learn from enormous amounts of data which will definitely bias them in ways that no one can understand, since no one can view even a iota of what the YouTube algorithm sees. After all, YouTube receives 500 hours of new video per minute (!!), which it processes at least for copyrights, hate speech filtering and automated captioning.
As a result, arguments of the form "I don't think the YouTube recommender system is intelligent/sophisticated" might be signs that, perhaps, you may be underestimating today's algorithms. If so, then you might be prey to Yudkowsky's "greatest danger". At the very least, discarding the dangerousness of large-scale algorithms without an adequate understanding of them should probably be regarded as a bad habit.
3) Toby Ord's latest book stresses the problem of risk factors. Typically, if everybody cared about political scandals while a deadly pandemic (much worse than COVID-19) is going on, then, surely, the probability of mitigating pandemic risks will be greatly diminish. Arguably, recommender systems are major risk factors, because they point billions of individuals' attentions away from the most pressing problems. Including the attention of the brightest of us.
Bill Gates seems to have given a lot of importance to the risk factor of exposure to poor information, or to the lack of quality information, as his foundation has been investing a lot in "solutions journalism". Perhaps more interestingly still, he has decided to be a YouTuber himself. His channel has 2.3M views (!!) and 450 videos (!!). He publishes several videos per week, especially during this COVID-19 pandemic, probably because he considers that the battle of information is a major cause area! At the very least, he seems to believe that this huge investment is worth this (very valuable) time.
I guessed the post strongly insisted on the scale and neglectedness of short-term AI alignment. But I can dwell more on this. There are now more views on YouTube than searches on Google, 70% of which are results of recommendation. And a few studies show (cited here) suggest that the influence of repeated exposure to some kind of information has a strong effect on beliefs, preferences and habits. Since this has major impacts on all other EA causes, I'd say the scale of the problem is at least that of any other EA cause.
I believe that alignment is extremely neglected in academia and industry, and short-term alignment is still greatly neglected within EA.
The harder point to estimate is tractability. It is noteworthy that Google, Facebook and Twitter have undertaken a lot of measures recently towards more ethical algorithms, which suggest that it may be possible to get them to increase the amount of ethics in their algorithms. The other hard part is technical. While it might be possible to upgrade some videos "by hand", it seems desirable to have more robust sustainable solutions to robustly beneficial recommendation. I think that having a near-perfect recommender is technically way out of reach (it's essentially solving AGI safety!). But there are likely numerous small tweaks that can greatly improve how robustly beneficial the recommender system is.
Of course, all of this is a lot more complex to discuss. I've only presented a glance of what we discuss in our book or in our podcast. And I'm very much aware of the extent of my ignorance, which is unfortunately huge...
The importance of ethics in YouTube recommendation seems to have grown significantly over the last two years (see this for instance). This suggests that there are pressures both from outside and inside that may be effective in making YouTube care about recommending quality information.
Now, YouTube's effort seems to have been mostly about removing (or less recommending) undesirable contents so far (though as an outsider it's hard for me to say). Perhaps they can be convinced to also recommend more desirable contents too.
It's unfortunately very hard to quantify the impact of recommender systems. But here's one experiment that may update your prior on the effectiveness of targeted video recommendations.
In 2013, Facebook did a large-scale experiment where they tweaked their newsfeeds. For some of their users, they removed 10% of posts with negative contents. For others, they removed 10% of posts with positive contents. And there was also a control group. After only one week, they observed a change in users' behaviors: the first group posted more positive contents, and the second posted more negative contents. Here's our discussion of the paper.
I cannot stress enough the fact that Facebook's intervention was minimal. A 2020 aligned recommender system could completely upset the entire newsfeed. Also, it would do so for longer than just a week — and other studies suggest that it takes months for users to learn even basic adaptation to tweaks of the algorithm.
In terms of healthcare in developing countries, it still seems to me that most philanthropists or small donators neglect the effectiveness of healthcare interventions in developing countries. Yet videos (not ads!) that do discuss this are not necessarily sufficiently promoted by the YouTube algorithm. As an example, this excellent video by Looking Glass Universe only has a fraction of the views of her other videos. Similarly, this video by Bill Gates has a lot less views than his videos on, say, climate change.
Note that this other post tried to quantify aligning recommender systems in terms of the common scale Scale/Tractability/Neglectedness. But the authors acknowledge themselves that they have low confidence in their estimates. But I'd argue that this goes on showing just how neglectedness this cause area is. Scientists both within and outside these companies have a hard time estimating the impact and tractability of (making progress in) aligning recommender systems (most don't even know what "alignment" is).
This is a fair point. We do not discuss much the global improvement of the world. I guess that we try to avoid upsetting those who have a negative vision of AI so far.
However, Chapter 5 does greatly insist on the opportunities of (aligned) AIs, in a very large number of fields. In fact, we argue that there is a compelling argument to say that fighting AI progress is morally wrong (though, of course, there is the equally compelling flip-side of the argument if one is concerned about powerful AIs...).
We should probably add something about the personification of AI. This indeed has negative side effects. But if pondered adequately, especially for reinforcement learning AIs, it is a very useful way to think about AIs and to anticipate their actions.
This is a good point. The book focuses a lot on research questions indeed.
We do see value in many corporations discussing AI ethics. In particular, there seems to be a rise of ethical discussions within the big tech companies, which we hope to encourage. In fact, in Chapter 7, we urge AI companies like Google and Facebook to, not only take part of the AI ethics discussion and research, but to actively motivate, organize and coordinate it, typically by sharing their AI ethics dilemmas and perhaps parts of their AI codes. In a sense, they already started to do so.
Another point is that, given our perceived urgency of AI Safety, it seems that it may be useful to reach out to academic talents in many different ways. Targeted discussions do improve the quality of the discussions. But we fear that they may not "scale" sufficiently. We feel that some academics might be quite receptive to reflecting on the public discussion. But we may be underestimating the difficulty to make this discussion productive...
(I have given a large number of public talks, and found it quite easy to raise the concerns of the book for all sorts of audiences, including start-ups / tech companies, but I do greatly fear what could happen with medias...)
I should add that the book really goes on and on to encourage calm thinking and fruitful discussions on the topic. We even added a section in Chapter 1, where we apologize for the title and clarify the purpose of the book. We also ask readers to be themselves pedagogical and benevolent when criticizing or defending the theses of the book. But clearly, such contents of the book will only have an impact on those who actually read the book.
Anyways, thanks for your comment. We're definitely pondering it!
The book will be published by EDP Sciences. They focus a lot on textbooks. But they also work on outreach books. I published my first book with them on Bayesianism.
We hope to reach out to all sorts of people who are intrigued by AI but do not have any background on the topic. We also hope that more technical readers will be interested in the book to have an overview on AI Safety.
I should point out that I run a YouTube channel, whose audience will likely be the base audience of the book too.
Thanks! This is reassuring. I met someone last week who does his PhD in post-quantum cryptography and he did tell me about an ongoing competition to set the standards of such a cryptography. The transition seems on its way!
NB: I've edited the sentence to clarify what I meant.
The argument here is more that recommender systems are maximization algorithms, and that, if you buy the "orthogonality thesis", there is no reason to think that there cannot go AGI. In particular, you should not judge the capability of an algorithm by the simplicity of the task it is given.
Of course, you may reject the orthogonality thesis. If so, please ignore the first argument.
I would just like to point out three "classical EA" arguments for taking recommender systems very seriously.
1) The dangerousness of AGI has been argued to be orthogonal from the purpose of AGI, as illustrated by the paperclip maximizers. If you accept this "orthogonality thesis" and if you are concerned about AGI, then you should be concerned about the most sophisticated maximization algorithms. Recommender systems seem to be today's most sophisticated maximization algorithms (a lot more money and computing power has been invested in optimizing recommender systems than in GPT-3). Given the enormous economic incentives, we should probably not discard the probability that they will remain the most sophisticated maximization algorithms in the future.
As a result, arguments of the form "I don't see how recommender systems can pose an existential threat" seem akin to arguments of the form "I don't see how AGI can pose an existential threat".
(of course, if you reject the latter, I can see why you could reject the former 🙂)
2) Yudkowsky argues that “By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.” Today's recommender systems are typical examples of something "that people conclude too early that they understand it". Such algorithms learn from enormous amounts of data which will definitely bias them in ways that no one can understand, since no one can view even a iota of what the YouTube algorithm sees. After all, YouTube receives 500 hours of new video per minute (!!), which it processes at least for copyrights, hate speech filtering and automated captioning.
As a result, arguments of the form "I don't think the YouTube recommender system is intelligent/sophisticated" might be signs that, perhaps, you may be underestimating today's algorithms. If so, then you might be prey to Yudkowsky's "greatest danger". At the very least, discarding the dangerousness of large-scale algorithms without an adequate understanding of them should probably be regarded as a bad habit.
3) Toby Ord's latest book stresses the problem of risk factors. Typically, if everybody cared about political scandals while a deadly pandemic (much worse than COVID-19) is going on, then, surely, the probability of mitigating pandemic risks will be greatly diminish. Arguably, recommender systems are major risk factors, because they point billions of individuals' attentions away from the most pressing problems. Including the attention of the brightest of us.
Bill Gates seems to have given a lot of importance to the risk factor of exposure to poor information, or to the lack of quality information, as his foundation has been investing a lot in "solutions journalism". Perhaps more interestingly still, he has decided to be a YouTuber himself. His channel has 2.3M views (!!) and 450 videos (!!). He publishes several videos per week, especially during this COVID-19 pandemic, probably because he considers that the battle of information is a major cause area! At the very least, he seems to believe that this huge investment is worth this (very valuable) time.
I guessed the post strongly insisted on the scale and neglectedness of short-term AI alignment. But I can dwell more on this. There are now more views on YouTube than searches on Google, 70% of which are results of recommendation. And a few studies show (cited here) suggest that the influence of repeated exposure to some kind of information has a strong effect on beliefs, preferences and habits. Since this has major impacts on all other EA causes, I'd say the scale of the problem is at least that of any other EA cause.
I believe that alignment is extremely neglected in academia and industry, and short-term alignment is still greatly neglected within EA.
The harder point to estimate is tractability. It is noteworthy that Google, Facebook and Twitter have undertaken a lot of measures recently towards more ethical algorithms, which suggest that it may be possible to get them to increase the amount of ethics in their algorithms. The other hard part is technical. While it might be possible to upgrade some videos "by hand", it seems desirable to have more robust sustainable solutions to robustly beneficial recommendation. I think that having a near-perfect recommender is technically way out of reach (it's essentially solving AGI safety!). But there are likely numerous small tweaks that can greatly improve how robustly beneficial the recommender system is.
Of course, all of this is a lot more complex to discuss. I've only presented a glance of what we discuss in our book or in our podcast. And I'm very much aware of the extent of my ignorance, which is unfortunately huge...
The importance of ethics in YouTube recommendation seems to have grown significantly over the last two years (see this for instance). This suggests that there are pressures both from outside and inside that may be effective in making YouTube care about recommending quality information.
Now, YouTube's effort seems to have been mostly about removing (or less recommending) undesirable contents so far (though as an outsider it's hard for me to say). Perhaps they can be convinced to also recommend more desirable contents too.
It's unfortunately very hard to quantify the impact of recommender systems. But here's one experiment that may update your prior on the effectiveness of targeted video recommendations.
In 2013, Facebook did a large-scale experiment where they tweaked their newsfeeds. For some of their users, they removed 10% of posts with negative contents. For others, they removed 10% of posts with positive contents. And there was also a control group. After only one week, they observed a change in users' behaviors: the first group posted more positive contents, and the second posted more negative contents. Here's our discussion of the paper.
I cannot stress enough the fact that Facebook's intervention was minimal. A 2020 aligned recommender system could completely upset the entire newsfeed. Also, it would do so for longer than just a week — and other studies suggest that it takes months for users to learn even basic adaptation to tweaks of the algorithm.
In terms of healthcare in developing countries, it still seems to me that most philanthropists or small donators neglect the effectiveness of healthcare interventions in developing countries. Yet videos (not ads!) that do discuss this are not necessarily sufficiently promoted by the YouTube algorithm. As an example, this excellent video by Looking Glass Universe only has a fraction of the views of her other videos. Similarly, this video by Bill Gates has a lot less views than his videos on, say, climate change.
Note that this other post tried to quantify aligning recommender systems in terms of the common scale Scale/Tractability/Neglectedness. But the authors acknowledge themselves that they have low confidence in their estimates. But I'd argue that this goes on showing just how neglectedness this cause area is. Scientists both within and outside these companies have a hard time estimating the impact and tractability of (making progress in) aligning recommender systems (most don't even know what "alignment" is).
Well, you were more than right to do so! You (and others) have convinced us. We changed the title of the book :)
This is a fair point. We do not discuss much the global improvement of the world. I guess that we try to avoid upsetting those who have a negative vision of AI so far.
However, Chapter 5 does greatly insist on the opportunities of (aligned) AIs, in a very large number of fields. In fact, we argue that there is a compelling argument to say that fighting AI progress is morally wrong (though, of course, there is the equally compelling flip-side of the argument if one is concerned about powerful AIs...).
We should probably add something about the personification of AI. This indeed has negative side effects. But if pondered adequately, especially for reinforcement learning AIs, it is a very useful way to think about AIs and to anticipate their actions.
Thanks for the comment, Paul!
This is a good point. The book focuses a lot on research questions indeed.
We do see value in many corporations discussing AI ethics. In particular, there seems to be a rise of ethical discussions within the big tech companies, which we hope to encourage. In fact, in Chapter 7, we urge AI companies like Google and Facebook to, not only take part of the AI ethics discussion and research, but to actively motivate, organize and coordinate it, typically by sharing their AI ethics dilemmas and perhaps parts of their AI codes. In a sense, they already started to do so.
Another point is that, given our perceived urgency of AI Safety, it seems that it may be useful to reach out to academic talents in many different ways. Targeted discussions do improve the quality of the discussions. But we fear that they may not "scale" sufficiently. We feel that some academics might be quite receptive to reflecting on the public discussion. But we may be underestimating the difficulty to make this discussion productive...
(I have given a large number of public talks, and found it quite easy to raise the concerns of the book for all sorts of audiences, including start-ups / tech companies, but I do greatly fear what could happen with medias...)
I should add that the book really goes on and on to encourage calm thinking and fruitful discussions on the topic. We even added a section in Chapter 1, where we apologize for the title and clarify the purpose of the book. We also ask readers to be themselves pedagogical and benevolent when criticizing or defending the theses of the book. But clearly, such contents of the book will only have an impact on those who actually read the book.
Anyways, thanks for your comment. We're definitely pondering it!
The book will be published by EDP Sciences. They focus a lot on textbooks. But they also work on outreach books. I published my first book with them on Bayesianism.
We hope to reach out to all sorts of people who are intrigued by AI but do not have any background on the topic. We also hope that more technical readers will be interested in the book to have an overview on AI Safety.
I should point out that I run a YouTube channel, whose audience will likely be the base audience of the book too.
Thanks! This is reassuring. I met someone last week who does his PhD in post-quantum cryptography and he did tell me about an ongoing competition to set the standards of such a cryptography. The transition seems on its way!