Insights into the accuracy of social scientists' forecasts of societal change
NATURE HUMAN BEHAVIOUR
How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists' forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.
View details for DOI 10.1038/s41562-022-01517-1
View details for Web of Science ID 000931761000002
View details for PubMedID 36759585
Three families of automated text analysis.
Social science research
2022; 108: 102798
Since the beginning of this millennium, data in the form of human-generated text in a machine-readable format has become increasingly available to social scientists, presenting a unique window into social life. However, harnessing vast quantities of this highly unstructured data in a systematic way presents a unique combination of analytical and methodological challenges. Luckily, our understanding of how to overcome these challenges has also developed greatly over this same period. In this article, I present a novel typology of the methods social scientists have used to analyze text data at scale in the interest of testing and developing social theory. I describe three "families" of methods: analyses of (1) term frequency, (2) document structure, and (3) semantic similarity. For each family of methods, I discuss their logical and statistical foundations, analytical strengths and weaknesses, as well as prominent variants and applications.
View details for DOI 10.1016/j.ssresearch.2022.102798
View details for PubMedID 36334926
Word Embeddings Reveal How Fundamental Sentiments Structure Natural Language
AMERICAN BEHAVIORAL SCIENTIST
View details for DOI 10.1177/00027642211066046
View details for Web of Science ID 000762967000001
Measuring the predictability of life outcomes with a scientific mass collaboration.
Proceedings of the National Academy of Sciences of the United States of America
How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.
View details for DOI 10.1073/pnas.1915006117
View details for PubMedID 32229555
Virtual reality perspective-taking increases cognitive empathy for specific others
2018; 13 (8)
View details for DOI 10.1371/journal.pone.0202442
View details for Web of Science ID 000443388900030