Research

Can Discussions about Girls’ Education Improve Academic Outcomes? Evidence from a Randomized Development Project (World Bank Economic Review, 2024) — with Christopher S. Cotton, Jordan Nanowski, and Eric Richert 

• Abstract This article evaluates the impact that facilitated discussions about girls’ education have on education outcomes for students in rural Zimbabwe. The staggered implementation of components of a randomized education project allowed for the causal analysis of a dialogue-based engagement campaign. This campaign involved regular discussions between trained facilitators and parents, teachers, and youth about girls’ rights, the importance of attending school, and the barriers girls face in pursuing education. The campaigns increased mathematics performance and enrollment in the year after implementation. There was no similar improvement in literacy performance during this period. Longer-term data on the broader project suggest that adding additional education-focused interventions did not further increase mathematics performance and enrollment beyond what can be attributable to the dialogue campaigns alone.

Correcting for bias in hot hand analysis: An application to youth golf (Journal of Economic Psychology, 2018) - with Christopher Cotton, Frank McIntyre, Ardyn Nordstrom, and Joseph Price

• Abstract This paper illustrates the problems that arise with traditional tests for the hot hand and proposes instead using a consistent dynamic panel data estimator, which corrects for these problems and is easy to implement. The issue is demonstrated by performing regression analysis on a sample of simulated data for junior golfers that does not include any dependence in a golfer’s performance across holes. The traditional regression analysis finds evidence of both hot and cold hand effects, even though the data is known to have no such effects. We resolve this problem by applying the consistent dynamic panel data estimator to a large dataset of amateur, youth golfers, to find no evidence of either hot or cold hand effects overall. When we restrict attention to the most-amateur of the golfers in our data, we do see weak evidence of a small hot hand. Thus, casual athletes may experience small hot hands, but the effect does not persist among more serious athletes. This may give insight into why the belief in the hot hand in professional sports exists, even when the evidence suggests otherwise.

Impact of a Severe Drought on Education: More Schooling Does Not Imply More Learning (link) - with Christopher Cotton [Requested revision submitted]

• Abstract In 2015 and 2016, Southern Africa experienced one of the severest droughts on record. The drought's intensity varied significantly across locations in unanticipated ways, providing a natural experiment to estimate the effect of large, negative agricultural shocks. We consider the impact of these shocks on children’s educational outcomes using data from a major education evaluation that was ongoing in rural Zimbabwe at the time of the drought, supplemented by remote-sensing satellite data to measure the drought intensity within each community. Droughts cause decreases in income and food access, which can affect household resource allocation and school attendance as well as learning. We find that drought exposure increased attendance, especially among the poorest students, which is consistent with droughts reducing work opportunities. At the same time, however, we do not see improvements in learning. The drought led to a significant decline in performance on literacy assessments for students who were experiencing food insecurity, and no significant impact on learning overall. This suggests that studies can arrive at very different conclusions depending on their measure of education performance. While most studies focus on either quantity of schooling or quality of learning, we highlight the importance of including both measures in an analysis.​

Pedal-Powered Progress: Evaluating Multidimensional Girls' Education Programs with Optimal Full Matching (link)— with Christopher S. Cotton, Zachary Robb [Submitted]

• Abstract We evaluate the impact that bicycle provision has on education and empowerment outcomes for girls in remote communities. Bicycles were provided as a sub-component of a large multidimensional girls’ education program implemented across rural Zimbabwe. The bicycles and the larger program were designed to address barriers experienced by the most marginalized children in these communities, including girls who have to travel long distances to get to school. To isolate the marginal impact of the bicycle program, we identify a counterfactual group from the larger program’s treatment group using a novel matching approach referred to as ``optimal full matching''. Optimal full matching leads to a more balanced match across pre-treatment characteristics than more common matching exercises that use one-to-one or one-to-many matching. This improved match quality has implications for the resulting treatment effects. Using this approach, we show that the bicycle program improved attendance, but there was no evidence of improved learning or girls’ sense of empowerment overall. 

Using Images as Covariates: Measuring Curb Appeal with Deep Learning (link) - with Morgan Nordstrom and Matthew D. Webb [Submitted]

• Abstract This paper details an innovative methodology to integrate image data into traditional econometric models. Motivated by forecasting sales prices for residential real estate, we harness the power of deep learning to add "information" contained in images as covariates. Specifically, images of homes were categorized and encoded using an ensemble of image classifiers (ResNet-50, VGG16, MobileNet, and Inception V3). Unique features presented within each image were further encoded through panoptic segmentation. Forecasts from a neural network trained on the encoded data results in improved out-of-sample predictive power. We also combine these image-based forecasts with standard hedonic real estate property and location characteristics, resulting in a unified dataset. We show that image-based forecasts increase the accuracy of hedonic forecasts when encoded features are regarded as additional covariates. We also attempt to "explain" which covariates the image-based forecasts are most highly correlated with. The study exemplifies the benefits of interdisciplinary methodologies, merging machine learning and econometrics to harness untapped data sources for more accurate forecasting.

It Takes a Village: The Impact of Community Mobilization Campaigns on Attitudes and Education (former job market paper—link) 

• Abstract This paper uses a quasi-randomized field experiment in Zimbabwe to assess the impact of large-scale community mobilization campaigns to build support for girls and marginalized groups in rural communities. I analyze the impact that the program has had on attitudes, the behaviour of teachers and caregivers, and the learning and progression outcomes of at-risk youth. The quantitative survey and learning assessment data I use for this is complemented by transcripts from focus groups and interviews, which I analyze using innovative text mining methods to measure changes in community sentiment towards marginalized groups. I find that the program improved community attitudes toward girls' education by 0.560 SD over the three and a half year project. This contributed to a 20.9 percentage point increase in the likelihood that students in the treatment group reported receiving enough support from their community to continue learning during COVID-19 school closures, along with other changes in the behaviours of community members and families. The program facilitated better learning and progression outcomes, with marginalized students performing 0.28 SD better on learning assessments after the project. These findings lead to two important conclusions about the efficacy of interventions designed to mobilize communities to reshape community attitudes and support marginalized students. The first is that community attitudes can be influenced in a relatively short time to become more supportive towards marginalized groups. The second is that these interventions can support education outcomes. This paper also demonstrates the usefulness of qualitative methods and text mining techniques for future experimental work.​

Understanding Education Outcomes in Zimbabwe: A Machine Learning Approach to Informing Program and Evaluation Design - with Aditya Maheshwari

• Abstract This paper uses multiple machine learning methods to describe the barriers and characteristics associated with different education outcomes. The data collected for this project is uniquely high-dimensional for a project of this scope, providing an opportunity to apply model shrinkage and tree-based methods that can identify the ``most important'' barriers and characteristics that explain education outcomes. This serves two main objectives. The first is to define the characteristics associated with negative education outcomes. As shown in Nordstrom (2022), effectively reaching the most marginalized students can be particularly important for education programs and policies to be effective in developing context. Therefore, identifying the characteristics and barriers associated with poor academic performance can help design projects that are better suited to addressing the barriers and characteristics of marginalized students. The second objective builds on this by specifically exploring the characteristics associated with poor progression outcomes. This is an area where the program this data came from did not see significant impact, and understanding the predictors of failed or successful progression outcomes may help inform future program design. I find that lack of support from households and schools, as well as pregnancy and motherhood, present the largest barriers to progression outcomes. While these are also important barriers to learning, other factors such as a student’s self-confidence were relatively more important. Learning outcomes are also important for explaining progression outcomes, however, the opposite does not appear to be true.​

Mass Reproducibility and Replicability: A New Hope - with Brodeur, Abel, Derek Mikola, and Nikolai Cook (and others) (link) [Submitted]

• Abstract This study pushes our understanding of research reliability by reproducing and replicating claims from 110 papers in leading economic and political science journals. The analysis involves computational reproducibility checks and robustness assessments. It reveals several patterns. First, we uncover a high rate of fully computationally reproducible results (over 85%). Second, excluding minor issues like missing packages or broken pathways, we uncover coding errors for about 25% of studies, with some studies containing multiple errors. Third, we test the robustness of the results to 5,511 re-analyses. We find a robustness reproducibility of about 70%. Robustness reproducibility rates are relatively higher for re-analyses that introduce new data and lower for re-analyses that change the sample or the definition of the dependent variable. Fourth, 52% of re-analysis effect size estimates are smaller than the original published estimates and the average statistical significance of a re-analysis is 77% of the original. Lastly, we rely on six teams of researchers working independently to answer eight additional research questions on the determinants of robustness reproducibility. Most teams find a negative relationship between replicators’ experience and reproducibility, while finding no relationship between reproducibility and the provision of intermediate or even raw data combined with the necessary cleaning codes.

Trade, poverty, and food security: A survey with implications for East Africa (link) — with Huw Lloyd-Ellis

• Abstract We survey the latest research on the linkages between international trade, regional integration, poverty and food security in developing economies and draw out its implications for East Africa and future research. While there is now an extensive literature on the impacts of trade reform on poverty outcomes, research on the actual and potential effects of trade and regional integration on food security is much more limited. This reflects inconsistencies in the definition and measurement of food security, substantial data limitations, and the complexity of food systems themselves. Nevertheless, we argue that there is an urgent need and considerable scope for further research on these linkages.

• "What Affects the Education Outcomes of Marginalized Students? A Machine Learning Approach to Inform Program Design"

• "Text Mining Approaches to Efficiently Select a Purposeful Qualitative Sample" — with Christopher Cotton

• "Understanding the impact of Crises in Public Health Decision Making" — with Christopher Cotton and Monica LaBarge [data collection in progress]

• "The Economic Value of Better Biking Infrastructure" — with Morgan Nordstrom

• "Vocational Training for Out-of-School students: A Qualitative Evaluation Using Text Mining in Zimbabwe" – with Christopher Cotton, Zachary Robb, Shannon Veenstra, and Lindsay Wallace

• “Estimating the causal impact of trade liberalization on food security” — with Huw Lloyd-Ellis (based on published report “An Empirical Analysis of the Relationships Between Trade and Food Security” — see above)

• “Quantifying qualitative data: An alternative to content analysis in survey data” — with Christopher Cotton

• "Focusing on foundational skills: Understanding literacy and numeracy skill acquisition in Zimbabwe" — with Christopher Cotton

• "Monetizing education benefits in cost-benefit evaluations" — with Christopher Cotton

• Endline Evaluation of IGATE-T (2021) - for World Vision UK, with Christopher S. Cotton, Shannon Veenstra, Jay MacKinnon, Lindsay Wallace, and Zachary Robb

• Cost-Benefit Analysis: Improving the Quality of Primary School Education in Malawi (2021) — for Copenhagen Consensus Center and the African Institute for Development Policy, with colleagues

• Cost-Benefit Analysis: Reducing Secondary School Dropout Rates in Malawi (2021) — for Copenhagen Consensus Center and the African Institute for Development Policy, with colleagues

• An Empirical Analysis of the Relationships Between Trade and Food Security (2020) — for USAID Kenya/East Africa, with Huw-Lloyd Ellis, Edward Carr, and Deanna Gordon

• Understanding the Relationships Between Trade and Food Security: A Landscape Map (2020) — for USAID Kenya/East Africa, with Huw-Lloyd Ellis, Edward Carr, Anthony Cambas, and Deanna Gordon

• Midline Evaluation of IGATE-T (2020) — for World Vision UK, with Christopher S. Cotton and Shannon Davis

• Cost Benefit Analysis of Youth Ready Programs in Central America (2019) — for World Vision Canada, with Bahman Kashi

• Baseline Evaluation of IGATE-T (2018) — for World Vision UK, with Christopher S. Cotton, Bahman Kashi, and Jay MacKinnon

• Response bias in voluntary surveys: An empirical analysis of the Canadian Census (2016) — with Kerry Nield​

• Identifying unvaccinated individuals in Canada: A predictive model (2016) — with Kevin Dick, arXiv