← Explore Capabilities
Dimension: Robustness

Resist manipulation

Ability to resist manipulation that would decrease trustworthiness, legitimacy or unfairly influence the outcome.

MINIMAL
EXTREME
Maturity
Opportunity
Importance
Neglectedness
Transnational
Read about assessment rating system

How this is performed now...

Process organizers
Process organizers anticipate vulnerabilities in processes and do their best to mitigate risk with countermeasures.
Citizens' assembly organizers
Citizens' assemblies organizers design processes with an understanding of where manipulation is possible and more likely, and develop mitigating strategies, such as by reinforcing the epistemic capabilities of participants before interacting with new information, developing selection algorithms with manipulation resistance, and establishing governance protocols for impartiality of key actors.

Related Product Gaps

AI Manipulation Detection System

Urgent
Real-time AI-enhanced monitoring and detection system that identifies manipulation attempts, whether AI-generated or human-orchestrated, across all stages of deliberative processes: from recruitment and selection gaming to astroturfing in discussions, coordinated voting patterns, and output tampering. 
AI/ML
Impact
Scale
Related Capabilities: Resist manipulation

Related Goals and Research Questions

Goal: Manipulation attempts that can be reliably detected and prevented across different stages of the assembly process.

How can we develop real-time detection systems for coordinated manipulation attempts during participant recruitment and selection?

Urgent
AI/ML Statistics & Sampling
Related Capability: Resist manipulation

What behavioral indicators reliably signal attempts to game deliberative processes?

Urgent
Psychology
Related Capability: Resist manipulation

How can we distinguish between legitimate persuasion and manipulative influence in deliberative settings?

Urgent
Psychology Ethics & Philosophy
Related Capability: Resist manipulation

Goal: Assembly designs that are robust to both internal and external manipulation attempts.

How can we quantify and test the manipulation resistance of different assembly design choices?

Urgent
Psychology Political Science
Related Capability: Resist manipulation

What are the tradeoffs between openness/transparency and manipulation resistance?

Urgent
Ethics & Philosophy Political Science
Related Capability: Resist manipulation

How can we design information presentation formats that minimize susceptibility to framing effects?

Urgent
Psychology Communication & Media
Related Capability: Resist manipulation

How can we develop manipulation impact metrics that distinguish between minor and outcome-altering influences?

Urgent
Psychology Ethics & Philosophy
Related Capability: Resist manipulation

Goal: Post-hoc verification that can establish that outcomes were not unduly influenced.

How can we create standardized integrity assessment frameworks for evaluating completed assemblies?

Urgent
Political Science Public Policy & Law
Related Capability: Resist manipulation

Related Resources

Strategy-proofness and Arrow's conditions: Existence and correspondence theorems for voting procedures and social welfare functions

Satterthwaite's landmark 1975 work on strategy-proofness and Arrow's conditions, investigating the relationship between preventing strategic manipulation in voting procedures and satisfying Arrow's impossibility conditions. This foundational work in mechanism design theory demonstrates existence ...
Research
Mark Allen Satterthwaite

Adversarial testing for Generative AI

Google's guide defining adversarial testing as systematically evaluating ML models against malicious or inadvertently harmful input, covering explicit queries (containing policy-violating language) and implicit queries (seeming harmless but involving sensitive topics). The four-stage workflow inv...
Research
Google

Strategic Classification

Hardt et al. (2015) address classifier manipulation by strategic actors, modeling the problem as a sequential game between classifier designers and individuals seeking favorable classification who may alter attributes to game the system. For natural cost function classes, they developed computati...
Research
Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou and Mary Wootters

The Social Cost of Strategic Classification

Examines how actors modify behavior when being evaluated by algorithms. Builds on a 2015 paper by Hardt and related to a 2020 paper by the same authors
Research
Smitha Milli, John Miller and Moritz Hardt

Strategic Classification is Causal Modeling in Disguise

Miller, Milli, and Hardt (2020) reveal a fundamental connection between strategic classification and causal inference, distinguishing between gaming (circumventing the system) and genuine improvement. Their central argument is that designing classifiers that incentivize improvement must inevitabl...
Research
John Miller, Smitha Milli and Moritz Hardt

Fair algorithms for selecting citizens’ assemblies

Sortition algorithms have been designed to select participants according to quotas, balancing representativeness, fairness and manipulation resistance (Flanigan, 2021; Baharav, 2024).
Research
Bailey Flanigan, Paul Gölz, Anupam Gupta, Brett Hennig and Ariel D. Procaccia