Impact of a Severe Drought on Girls' Attendance and Learning (American Educational Research Journal, 2025) - with Christopher Cotton
Abstract How will climate change impact education since near-universal primary education has been achieved in many developing countries? We begin to answer this by studying a recent severe drought in Southern Africa. Using data from a large cluster-randomized control trial involving girls and satellite data measuring drought intensity, we find drought exposure increased attendance, especially for the poorest girls. Girls experiencing food insecurity see declines in learning assessment scores while others see no gains in assessment scores, on average, compared to the counterfactual. The analysis highlights the importance of considering both attendance and performance in analyses. It also suggests the potential importance of ensuring schools have the resources to deal with higher attendance and providing food assistance to students during droughts.
Can Discussions about Girls’ Education Improve Academic Outcomes? Evidence from a Randomized Development Project (World Bank Economic Review, 2024) — with Christopher S. Cotton, Jordan Nanowski, and Eric Richert
Abstract This article evaluates the impact that facilitated discussions about girls’ education have on education outcomes for students in rural Zimbabwe. The staggered implementation of components of a randomized education project allowed for the causal analysis of a dialogue-based engagement campaign. This campaign involved regular discussions between trained facilitators and parents, teachers, and youth about girls’ rights, the importance of attending school, and the barriers girls face in pursuing education. The campaigns increased mathematics performance and enrollment in the year after implementation. There was no similar improvement in literacy performance during this period. Longer-term data on the broader project suggest that adding additional education-focused interventions did not further increase mathematics performance and enrollment beyond what can be attributable to the dialogue campaigns alone.
Correcting for bias in hot hand analysis: An application to youth golf (Journal of Economic Psychology, 2018) - with Christopher Cotton, Frank McIntyre, Ardyn Nordstrom, and Joseph Price
Abstract This paper illustrates the problems that arise with traditional tests for the hot hand and proposes instead using a consistent dynamic panel data estimator, which corrects for these problems and is easy to implement. The issue is demonstrated by performing regression analysis on a sample of simulated data for junior golfers that does not include any dependence in a golfer’s performance across holes. The traditional regression analysis finds evidence of both hot and cold hand effects, even though the data is known to have no such effects. We resolve this problem by applying the consistent dynamic panel data estimator to a large dataset of amateur, youth golfers, to find no evidence of either hot or cold hand effects overall. When we restrict attention to the most-amateur of the golfers in our data, we do see weak evidence of a small hot hand. Thus, casual athletes may experience small hot hands, but the effect does not persist among more serious athletes. This may give insight into why the belief in the hot hand in professional sports exists, even when the evidence suggests otherwise.
The Marginal Benefit of Bicycles in Education Programs: An Evaluation Using Optimal Full Matching (link)— with Christopher S. Cotton, Zachary Robb [Under Review]
Abstract We evaluate the impact that bicycle provision has on education and empowerment outcomes for girls in remote communities. Bicycles were provided as a sub-component of a large multidimensional girls’ education program implemented across rural Zimbabwe. The bicycles and the larger program were designed to address barriers experienced by the most marginalized children in these communities, including girls who have to travel long distances to get to school. To isolate the marginal impact of the bicycle program, we identify a counterfactual group from the larger program’s treatment group using a novel matching approach referred to as ``optimal full matching''. Optimal full matching leads to a more balanced match across pre-treatment characteristics than more common matching exercises that use one-to-one or one-to-many matching. This improved match quality has implications for the resulting treatment effects. Using this approach, we show that the bicycle program improved attendance, but there was no evidence of improved learning or girls’ sense of empowerment overall.
Understanding Education Outcomes in Zimbabwe: A Machine Learning Approach to Informing Program and Evaluation Design - with Aditya Maheshwari [Under Review]
Abstract Many development projects are designed to improve outcomes for the most marginalized members of society. Despite development agencies collecting unprecedented levels of evaluation data, many development programs fail to effectively reach individuals with the greatest need for support. To evaluate how widely-implemented life skills and financial support programs impact students from different marginalized groups, we use innovative machine learning methods to evaluate the impact of a Girls' Education Challenge program implemented in Zimbabwe and Tanzania. By estimating the conditional average treatment effects for different subgroups, we find that the program was—on average—effective at improving math test scores for the ``most marginalized'' students, based on a set of indicators for common risk factors. This included girls, who experienced larger improvements for both learning and advancement. However, overall, the CAMFED interventions was less effective for the most marginalized students, including students from relatively poorer households. If the interventions were targeted to individuals with the highest levels of marginalization then the program's overall impact on advancement would have been 3.7 percentage points \textit{lower} than it was in practice. This has important policy implications for how development interventions are designed and targeted to reach different objectives.
Evaluating Participant Experiences in Public Health Surveys: A Mixed-Methods Analysis of Qualitative Data using Text Mining - with Aditya Maheshwari, Rheann Quenneville, and Grace Shen-Tu [Under Review]
Background Engagement in surveys is vital to health and social sciences research. This study uses a novel mixed-methods approach to identify themes related to participant engagement in a large public health survey.
Methods The study employed a combination of text mining methods (sentiment analysis, topic modelling) to process qualitative data from 15,000 surveys collected as part of the Alberta’s Tomorrow Project 2023 survey. This was done alongside a manual thematic analysis of a purposive sample of 852 responses to validate the computational methods and identify themes not captured by the topic model.
Results Many participants found the survey easy to complete and valued having the opportunity to participate. Challenges experienced included difficulty with recall-based questions, irrelevant response options, and program functionality issues. The findings highlight the importance of customizable communication methods to adapt to individual preferences.
Interpretation Computational and manual thematic analysis of qualitative data in surveys typically generated similar conclusions about participant experiences. Addressing survey issues related to program functionality, communication strategies, and may improve engagement and data quality in longitudinal surveys.
Canadian Public Health Experiences with Evidence during COVID-19 and a New Framework for Assessing Evidence in Emergencies - with Christopher S. Cotton and Monica LaBarge [Under Review]
It Takes a Village: The Impact of Community Mobilization Campaigns on Attitudes and Education (former job market paper—link)
Abstract This paper uses a quasi-randomized field experiment in Zimbabwe to assess the impact of large-scale community mobilization campaigns to build support for girls and marginalized groups in rural communities. I analyze the impact that the program has had on attitudes, the behaviour of teachers and caregivers, and the learning and progression outcomes of at-risk youth. The quantitative survey and learning assessment data I use for this is complemented by transcripts from focus groups and interviews, which I analyze using innovative text mining methods to measure changes in community sentiment towards marginalized groups. I find that the program improved community attitudes toward girls' education by 0.560 SD over the three and a half year project. This contributed to a 20.9 percentage point increase in the likelihood that students in the treatment group reported receiving enough support from their community to continue learning during COVID-19 school closures, along with other changes in the behaviours of community members and families. The program facilitated better learning and progression outcomes, with marginalized students performing 0.28 SD better on learning assessments after the project. These findings lead to two important conclusions about the efficacy of interventions designed to mobilize communities to reshape community attitudes and support marginalized students. The first is that community attitudes can be influenced in a relatively short time to become more supportive towards marginalized groups. The second is that these interventions can support education outcomes. This paper also demonstrates the usefulness of qualitative methods and text mining techniques for future experimental work.
Using Images as Covariates: Measuring Curb Appeal with Deep Learning (link) - with Morgan Nordstrom and Matthew D. Webb
Abstract This paper details an innovative methodology to integrate image data into traditional econometric models. Motivated by forecasting sales prices for residential real estate, we harness the power of deep learning to add "information" contained in images as covariates. Specifically, images of homes were categorized and encoded using an ensemble of image classifiers (ResNet-50, VGG16, MobileNet, and Inception V3). Unique features presented within each image were further encoded through panoptic segmentation. Forecasts from a neural network trained on the encoded data results in improved out-of-sample predictive power. We also combine these image-based forecasts with standard hedonic real estate property and location characteristics, resulting in a unified dataset. We show that image-based forecasts increase the accuracy of hedonic forecasts when encoded features are regarded as additional covariates. We also attempt to "explain" which covariates the image-based forecasts are most highly correlated with. The study exemplifies the benefits of interdisciplinary methodologies, merging machine learning and econometrics to harness untapped data sources for more accurate forecasting.
Mass Reproducibility and Replicability: A New Hope - with Abel Brodeur, Derek Mikola, and Nikolai Cook (and others) (link) [Submitted]
Abstract This study pushes our understanding of research reliability by reproducing and replicating claims from 110 papers in leading economic and political science journals. The analysis involves computational reproducibility checks and robustness assessments. It reveals several patterns. First, we uncover a high rate of fully computationally reproducible results (over 85%). Second, excluding minor issues like missing packages or broken pathways, we uncover coding errors for about 25% of studies, with some studies containing multiple errors. Third, we test the robustness of the results to 5,511 re-analyses. We find a robustness reproducibility of about 70%. Robustness reproducibility rates are relatively higher for re-analyses that introduce new data and lower for re-analyses that change the sample or the definition of the dependent variable. Fourth, 52% of re-analysis effect size estimates are smaller than the original published estimates and the average statistical significance of a re-analysis is 77% of the original. Lastly, we rely on six teams of researchers working independently to answer eight additional research questions on the determinants of robustness reproducibility. Most teams find a negative relationship between replicators’ experience and reproducibility, while finding no relationship between reproducibility and the provision of intermediate or even raw data combined with the necessary cleaning codes.
Comparing Human-Only, AI-Assisted, and AI-Led Teams on Assessing Research Reproducibility in Quantitative Social Science - with Abel Brodeur, David Valenta, and Alexandru Marcoci (and others) (link) [Submitted]
Abstract This study evaluates the effectiveness of varying levels of human and artificial intelligence (AI) integration in reproducibility assessments of quantitative social science research. We computationally reproduced quantitative results from published articles in the social sciences with 288 researchers, randomly assigned to 103 teams across three groups — human-only teams, AI-assisted teams and teams whose task was to minimally guide an AI to conduct reproducibility checks (the “AI-led” approach). Findings reveal that when working independently, human teams matched the reproducibility success rates of teams using AI assistance, while both groups substantially outperformed AI-led approaches (with human teams achieving 57 percentage points higher success rates than AI-led teams, 𝒑 < 0.001). Human teams were particularly effective at identifying serious problems in the analysis: they found significantly more major errors compared to both AI-assisted teams (0.7 more errors per team, 𝒑 = 0.017) and AI-led teams (1.1 more errors per team, 𝒑 < 0.001). AI-assisted teams demonstrated an advantage over more automated approaches, detecting 0.4 more major errors per team than AI-led teams ( 𝒑 = 0.029), though still significantly fewer than human-only teams. Finally, both human and AI-assisted teams significantly outperformed AI-led approaches in both proposing (25 percentage points difference, 𝒑 = 0.017) and implementing (33 percentage points difference, 𝒑 = 0.005) comprehensive robustness checks. These results underscore both the strengths and limitations of AI assistance in research reproduction and suggest that despite impressive advancements in AI capability, key aspects of the research publication process still require human substantial human involvement.
Trade, poverty, and food security: A survey with implications for East Africa (link) — with Huw Lloyd-Ellis
Abstract We survey the latest research on the linkages between international trade, regional integration, poverty and food security in developing economies and draw out its implications for East Africa and future research. While there is now an extensive literature on the impacts of trade reform on poverty outcomes, research on the actual and potential effects of trade and regional integration on food security is much more limited. This reflects inconsistencies in the definition and measurement of food security, substantial data limitations, and the complexity of food systems themselves. Nevertheless, we argue that there is an urgent need and considerable scope for further research on these linkages.
"Online Information is not a Substitute for Sex Education" - with Laura Derksen and Catherine Michaud-Leclerc
"Inductive Analysis and Text Mining: Complements or Substitutes?" - with Rheann Quenneville, Aditya Maheshwari, and Grace Shen-Tu
"Evaluating Effectiveness under the Policy on Results" - with Jaimi Plater
"The Economic Value of Better Biking Infrastructure" — with Morgan Nordstrom
"Vocational Training for Out-of-School students: A Qualitative Evaluation Using Text Mining in Zimbabwe" – with Christopher Cotton, Zachary Robb, Shannon Veenstra, and Lindsay Wallace
“Estimating the causal impact of trade liberalization on food security” — with Huw Lloyd-Ellis (based on published report “An Empirical Analysis of the Relationships Between Trade and Food Security” — see above)
"Focusing on foundational skills: Understanding literacy and numeracy skill acquisition in Zimbabwe" — with Christopher Cotton
Endline Evaluation of IGATE-T (2021) - for World Vision UK, with Christopher S. Cotton, Shannon Veenstra, Jay MacKinnon, Lindsay Wallace, and Zachary Robb
Cost-Benefit Analysis: Improving the Quality of Primary School Education in Malawi (2021) — for Copenhagen Consensus Center and the African Institute for Development Policy, with colleagues
Cost-Benefit Analysis: Reducing Secondary School Dropout Rates in Malawi (2021) — for Copenhagen Consensus Center and the African Institute for Development Policy, with colleagues
An Empirical Analysis of the Relationships Between Trade and Food Security (2020) — for USAID Kenya/East Africa, with Huw-Lloyd Ellis, Edward Carr, and Deanna Gordon
Understanding the Relationships Between Trade and Food Security: A Landscape Map (2020) — for USAID Kenya/East Africa, with Huw-Lloyd Ellis, Edward Carr, Anthony Cambas, and Deanna Gordon
Midline Evaluation of IGATE-T (2020) — for World Vision UK, with Christopher S. Cotton and Shannon Davis
Cost Benefit Analysis of Youth Ready Programs in Central America (2019) — for World Vision Canada, with Bahman Kashi
Baseline Evaluation of IGATE-T (2018) — for World Vision UK, with Christopher S. Cotton, Bahman Kashi, and Jay MacKinnon
Response bias in voluntary surveys: An empirical analysis of the Canadian Census (2016) — with Kerry Nield
Identifying unvaccinated individuals in Canada: A predictive model (2016) — with Kevin Dick, arXiv