AI Manipulation Detection System
What
Real-time AI-enhanced monitoring and detection system that identifies manipulation attempts, whether AI-generated or human-orchestrated, across all stages of deliberative processes: from recruitment and selection gaming to astroturfing in discussions, coordinated voting patterns, and output tampering.
Why
Protect process integrity against threats ↑, build resilience into democratic infrastructure ↑, and maintain public trust ↑ that deliberative outputs reflect genuine citizen input rather than coordinated influence campaigns.
Problem Definition
Processes have built in defenses to manipulation but strong, coordinated efforts may overcome these. Organizers have limited ability to quickly detect and understand the scale/origins of manipulation threats across the lifecycle of deliberative process.
Definition of Success
Effectively understand the level/scale of threats targeting elements of deliberative processes, and the degree of threat (e.g. a group of citizens trying to influence the output of the deliberative process, coordinated manipulation on a geopolitically/socially sensitive topic). Ultimately could also make large-scale online deliberation far more defensible. Prevents invalidation of $100K-$1M+ assemblies compromised by manipulation.
Requirements
- ML detection models for each online attack vector (bot detection, coordinated behavior analysis, text authenticity assessment, voting pattern anomalies).
- Low latency for flagging suspicious activity.
- Alert system for review from process organizers.
- Detection system must resist manipulation by sophisticated attackers.
- Monitor behavior patterns without invasive surveillance of legitimate participants.
- Provide evidence for why flagged activity is suspicious
- Challenges to address: distinguishing coordinated manipulation from legitimate grassroots consensus or passionate individuals; attackers will probably use AI to evade detection; and effective monitoring requires analyzing participant behavior without creating surveillance infrastructure (legal/ethical boundaries).
Existing Limitations
Process organizers anticipate vulnerabilities in processes and do their best to mitigate risk with countermeasures. Citizens’ assembly organizers design processes with an understanding of where manipulation is possible and more likely, and develop mitigating strategies, such as reinforcing the epistemic capabilities of participants before interacting with new information, developing selection algorithms with manipulation resistance, and establishing governance protocols for impartiality of key actors.
Milestones
- In collaboration with process organizers, security researchers document attack vectors across process stages, define detection requirements, establish critical boundaries
- Build ML models for bot detection, coordinated behavior, voting anomalies
- Design cross-platform monitoring system, real-time alert infrastructure, and process organizers dashboard
- Hire security researchers hackers to attempt manipulation and improve detection robustness
Starting Points
Convene deliberative practitioners and security researchers to brainstorm attack scenarios across deliberation lifecycle.