Elite polarization,
measured from parliamentary speech

This project aggregates parliamentary data worldwide to measure elite polarization by analyzing what politicians say about one another inside parliaments. It employs an agent-AI-driven framework for parliamentary data collection, homologation, and analysis, orchestrated through the OpenClaw framework.

The system collects transcripts from academic archives across many countries. When corpora exist only on parliamentary websites, it uses web-scraping techniques to acquire them. Because many parliaments publish proceedings as PDF scans, the pipeline separates born-digital PDFs, where text is extracted directly, from scanned-image PDFs, where optical character recognition is applied. For video-only records, it extracts official captions or subtitles; otherwise, it tests automated speech recognition on a sample before transcription.

OpenClaw coordinates the workflow across collection, normalization, affiliation recovery, extraction, validation, and escalation. Collected transcripts are normalized into a unified format in which each row is one speaker’s turn, with country, date, speaker name, party, and speech text. Since speaker–party affiliation is often missing, a recovery algorithm uses registry and metadata joins, deterministic date-window rules for known affiliations over time, and fuzzy matching against biographical and parliamentary records. Unresolved cases then fall back to language-model lookup.

The extraction stage uses the Google Gemma 4 26B LLM to identify passages where a speaker evaluates another named politician, distinguishing attacks or praise from neutral and procedural references. Each mention is coded as in-group or out-group using recovered coalition affiliation. The output is a cross-national dataset of elite-hostility relationships, aggregated by party-pair and time period, covering over 42 million speeches across 82 countries.