I think this is an important question. My own actual answer is that I'm very unsure. It seems plausible that implementing or advocating for such a policy would be wise, or that it would be counterproductive.

The following thoughts and links will hopefully be more useful than that answer:

1. This seems very reminiscent of the arguments Bostrom makes in the paper The Vulnerable World Hypothesis (and especially his "easy nukes" thought experiment). From that paper's abstract:

Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the ‘semi-anarchic default condition’. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order.

(Perhaps half-remembering this paper is what leads you to say "It's likely that I have seen this term mentioned somewhere else in the past by someone else".)

2. Some other relevant sources include:

• https://forum.effectivealtruism.org/posts/xoxbDsKGvHpkGfw9R/problem-areas-beyond-80-000-hours-current-priorities#Global_governance
• https://forum.effectivealtruism.org/posts/xoxbDsKGvHpkGfw9R/problem-areas-beyond-80-000-hours-current-priorities#Surveillance
• https://www.effectivealtruism.org/articles/ea-global-2018-the-future-of-surveillance/

3. I think this is an important topic. In my draft series on "Crucial questions for longtermists", one question I list is "Would further development or deployment of surveillance technology increase risks from totalitarianism and dystopia? By how much?"

I'm also considering including an additional "topic" that would contain a more thorough set of "crucial questions" on the matter.

4. I don't think the unilateralist's curse is quite the right term in your argument. The potential for huge harms from unilateral action are indeed key, but the unilateralist's curse is something more specific.

Essentially, the curse is about a specific way in which random distribution of misjudgement can lead to the "most optimistic" person acting, and thereby to harm occurring, despite the actor themselves having genuinely aimed to do good. (I think it's also meant to be when this happens despite people's average estimates being accurate, rather than them being systematically overly optimistic, though I can't remember that for sure.) From the paper on the curse:

The unilateralist’s curse is closely related to a problem in auction theory known as the winner’s curse. The winner’s curse is the phenomenon that the winning bid in an auction has a high likelihood of being higher than the actual value of the good sold. Each bidder makes an independent estimate and the bidder with the highest estimate outbids the others. But if the average estimate is likely to be an accurate estimate of the value, then the winner overpays. The larger the number of bidders, the more likely it is that at least one of them has overestimated the value.
The unilateralist’s curse and the winner’s curse have the same basic structure. The difference between them lies in the goals of the agents and the nature of the decision. In the winner’s curse, each agent aims to make a purchase if and only if doing so will be valuable for her. In the unilateralist’s curse, the decision-maker chooses whether to undertake an initiative with an eye to the common good, that is, seeking to undertake the initiative if and only if the initiative contributes positively to the common good.

This could apply in cases like well-intentioned but harmful dual-use research, or in well-intentioned release of hazardous information. Interestingly, it could also apply to widely promoting this sort of "vulnerable world" argument - it's possible that:

• the people who would do so are those who overestimate the expected value of surveillance, preventing policing, etc.
• the "real" expected value is negative
• just a few people widely promoting this sort of argument is enough for major harm to occur, because then the idea can be picked up by others, acquire a life of its own, etc.

In any case, the possibility for well-intentioned yet extremely harmful actions, and the way the unilateralist's curse boosts the likelihood of them, does provide additional reason for surveillance. But the case for surveillance doesn't necessarily have to rest on that, and you seem most focused on malicious use (e.g., terrorism).

5. I've collected a bunch of sources related to the topics of the unilateralist's curse, downside risks/accidental harm, and information hazards, which might be interesting to you or some other readers.

Hope that's helpful!

Which would be more dystopian to you, DNA engineering to ensure the distribution of human behavior will not include unilateral destruction, or super surveillance?

I personally think DNA engineering at least has some positive points, too, while surveillance is purely a necessary evil.

Background: I am an information science student who has taken a class on the societal aspects of surveillance.

My gut feeling is that advocating for or implementing "mass surveillance" targeted at preventing individuals from using weapons of mass destruction (WMDs) would be counterproductive.

First, were a mass surveillance system aimed at controlling WMDs to be set up, governments would lobby for it to be used for other purposes as well, such as monitoring for conventional terrorism. Pretty soon it wouldn't be minimally invasive anymore; it would just be a general-purpose mass surveillance system.

Second, a surveillance system of the scope that Bostrom has proposed ("ubiquitous real-time worldwide surveillance") would itself be an existential risk to liberal democracy. The problem is that a ubiquitous surveillance system would create the feeling that surveillees are constantly being watched. Even if it had strong technical and institutional privacy guarantees and those guarantees were communicated to the public, people would likely not be able to trust it; rumors of abuse would only make establishing trust harder. People modify their behavior when they know they are being watched or could be watched at any time, so they would be less willing to engage in behaviors that are stigmatized by society even if the Panopticon were not explicitly looking out for those behaviors. This feeling of constantly being watched would stifle risk-taking, individuality, creativity, and freedom of expression, all of which are essential to sustain human progress.

I think that a much more limited suite of targeted surveillance systems, combined with other mechanisms for arms control, would be a lot more promising while still being effective at controlling WMDs. Such limited surveillance systems are already used in gun control: for example, the U.S. federal government requires dealers to keep records of gun sales for at least 20 years, and many U.S. states and other countries keep records of who is licensed to own a gun. Some states also require gun owners to report lost or stolen guns in order to fight gun trafficking. These surveillance measures can be designed to balance gun owners' privacy interests with the public's interest in reducing gun violence. We could regulate synthetic biology a lot like we do gun control: for example, companies that create synthetic biology or sell desktop DNA sequencers could be required to maintain records of transactions.

However, I don't expect this targeted approach to work as well for cyber weapons. Because computers are general-purpose, cyber weapons can theoretically be developed and executed on any computer, and trying to prevent the use of cyber weapons by surveilling everyone who owns a computer would be extremely inefficient (since the vast majority of people who use computers are not creating cyber weapons) and impractical (because power users could easily uninstall any spyware planted on their machines). Also, because computers are ubiquitous and often store a lot of sensitive personal information, this form of surveillance would be extremely unpopular as well as invasive. Strengthening cyber defense seems like a more promising way to prevent harm from cyber attacks.