Data inanities

I feel like half the time I read something from one of these data news explainer sites, I want to blog about how silly it is. So while I’ve been wrestling with what to write here regarding a series of terrible NYT op-eds (no, I won’t link to them, but you know which ones I’m talking about), I will take a minute to call out 538 for publishing this article complaining that giving students free lunch is going to make data analysis difficult.

It’s absolutely true that students receiving free lunches is a proxy for student poverty. In fact, in my own teaching, we talk about proxy variables by examining a data set of school characteristics and students achievement scores. We actually run regressions where I encourage students to think about socioeconomic status and poverty through school lunch programs (along with other measures). But it’s also a rather coarse measure. In the way that school lunch programs have traditionally been applied, if you fail to meet some income threshold, you get free lunch, and in some cases, free breakfast. In Colorado, for a family of four, it’s $44,123. While it’s useful for looking at broad categories, it doesn’t tell you anything about the heterogeneity within those categories. The number of kids qualifying for free lunch could be the same at two different schools, but if one school is in a relatively homogenous district with most families hovering around the cutoff point and the other pulls from one very rich area and one very poor area, looking at those schools as the same actually “muddies” the waters” more than diluting the program.

So, it’s not actually a great measure, anyway, which we’ve kind of already covered by calling it a “proxy.” So why not look for better measures? The article mentions education levels of parents; that’s a good one. Or economic variables of the surrounding districts could work. Property values, for instance, are widely available and could be linked to school district. This is a little more work perhaps, because often these variables aren’t automatically linked to school quality data.

It’s true; we don’t like change. And changing a commonly used measure of poverty means looking for new answers, and that trends over time will be a bit difficult to determine for awhile, but with a little hard work and ingenuity, the new answers should be better. Decrying the end of a poor measure of socioeconomic status when its expansion will actually help a lot of kids at the margin is just not very useful. Why not spend a little more time thinking about how we can make data better, answer questions more fully, and ultimately improve school experiences for kids?

Advertisements

The second life of RCTs and implications for child development

In the last few weeks, I’ve come upon two research programs (each with a few related papers) that utilize a combination of an RCT or phased-in intervention and follow-up data 7-10 years on to examine new research questions. They both happen to be focused on the lasting effects of childhood health and wellbeing initiatives, but I doubt that this trend will be confined to child health and literacy. Barham, Macours and Maluccio have a few papers (gated) that use the phasing in of a conditional cash transfer program in Nicaragua to test later childhood cognitive and non-cognitive outcomes, distinguishing effects by timing of the intervention. A working paper out last week shows that deworming programs in Uganda not only increased short-term anthropomorphic outcomes, but also contributed to children’s numeracy and literacy several years later.

In short, we’re seeing more evidence that these early health and wellbeing interventions can have profound impacts not just on the immediate outcomes–Under-5 mortaility, school attendance, etc–but also on future outcomes. I think it’s a neat use of experimental design to examine questions we might not have thought about when the programs were first put in place.

India-bound!

It’s (almost) official! I think I actually have a ticket and am leaving for India and the Philippines for the rest of the summer on Friday. I’ll post updates here as the mood strikes me, but feel free to follow @ekfletch and @EPoDHarvard on twitter for more frequent (and perhaps less related) content (pictures of all the momos I’m going to eat? Anyone?).

For now, I’ll leave you with the World Bank’s new project to determine the economic cost of child marriage, a well-funded, but really huge undertaking:

What is the economic cost of child marriage? We don’t really know. Studies – including those by the World Bank – suggest a range of negative impacts of child marriage on human development outcomes. For example, Bank staff have estimated that in sub-Saharan Africa child marriage may account in some countries for up to one-fifth of drop-outs among girls at the secondary level, and each additional year of delay in the age of (child) marriage could potentially increase the likelihood of literacy and secondary school completion by several percentage points for the affected girls. Another study published a few years ago in the Journal of Political Economy suggests similar impacts in the case of Bangladesh.

Breastfeeding and mommy wars

Last week, a paper came out in a relatively obscure journal and got a lot of attention (or at least got its own #slatepitches headline). A sociologist at Ohio State published a study saying that the positive effects of breastfeeding essentially disappear if you look at within-sibling differences. That is to say if you compare two siblings from the same parents, one of whom was breastfed and one of whom wasn’t, there isn’t much in the way of statistically significant differences in their educational achievement, health status, or intelligence as measured by standardized tests.

In many ways, this isn’t surprising. We already know that the vast majority of our later life outcomes are determined by our parents’ incomes and education levels, where we grow up, how many words are said to us before we can even talk, and the myriad investments our parents make in our parents’ health. That breastfeeding doesn’t make a very big difference among siblings shouldn’t surprise us.

Perhaps even more important though, and this is really the kicker when trying to identify the effects of early childhood interventions, is that we don’t know what else the parents did differently for these two children. Given what we know about mothers who breastfeed—they tend to be wealthier and more educated, they get more assistance with breastfeeding education, they tend to have more flexible jobs that allow them to breastfeed for longer, or are staying at home with their kids, they’ve been told for years, etc.—we also expect them to be conscientious if they for some reason are not breastfeeding one of their children. Colen’s analysis can only control for time-invariant, mother-specific characteristics. An important omission is that she can’t control for mothers who supplement bottle-feeding with additional doctor’s visits, vitamins, extra care, more time spent together, or any other activity or characteristic that would act as a complement to breastfeeding. It ignores anything that might cancel out the fact the child isn’t getting the extra nutrients and other stuff that we ascribe to breastfeeding.

This omission is only important if it’s correlated with breastfeeding, and in all likelihood, it is. If you thought your child was missing out by not being breastfed, mothers with the ability to might try to compensate by increasing other investments.

It’s also worth noting that there are other studies using the same methodology that do find within-family effects for breastfeeding. This one, for instance, by Rees and Sabia.

Ultimately Colen wants to use the results to push for more family-friendly social policy, like increased maternity leave and more. But this is not the last word on breastfeeding. One momy blogger called it a “suspect methodology.” It’s not, it’s a perfectly valid methodology, but we need to be careful about what it’s actually showing.

Cited:

Rees, Danial and Joseph Sabia. 2009. “The Effect of Breastfeeding on Educational Attainment: Evidence from Sibling Data.” University of Colorado at Denver Working Paper 09-03.

Colen, Cynthia and DM Ramey. 2014. “Is Breast Truly Best?:Estimating the effects of breastfeeding on long-term child health in the United States using sibling comparisons.”Social Science and Medicine.

Hart, B. & Risley, T.R. “The Early Catastrophe” (2004). Education Review, 77 (1), 100-118.

Cognitive effects of poverty

A new paper by Anandi Mani, Sendhil Mullainathan, Eldar Shafir, and Jiaying Zhao shows some pretty profound effects of poverty on cognition and decision making. The paper says that poverty is equivalent to pulling all-nighters in terms of its effect on your ability to perform routine tasks and make good decisions. It reminded me of a conversation I had with Mark Hecker, director of Reach, Inc., a nonprofit working on literacy in DC, about children who’ve been abused. He asked me to think about that feeling of indigence and anger that shoots up when someone bumps into you. It’s startling, difficult to process, and affects everything we do next. Children who’ve suffered abuse feel that way all the time, which puts additional stress on them to make good decisions, to concentrate in school, and more.

It’s a good reminder that putting ourselves in someone else’ shoes is often impossible; someone who has grown up middle class never worrying about money is not going to approach large expenditures the same way that someone who grew up poor will. Analyzing decision making of poor and disadvantaged individuals is subject to so many more constraints that we realize.

Reading to girls

An Atlantic piece today outlines some current research that is very much in line with my own.

The researchers found a gender difference in what they call “teaching activities” that build cognitive skills in children as young as nine months old. Girls, not boys, in all three countries received more time from parents on three activities: reading, storytelling, and teaching letters and numbers. Baker and Milligan scrutinized data for first-born children, to control for differences arising when parents slack off after baby number two or three arrives. They also examined parents’ time spent with boy-girl twins and again found boys receiving less time than girls on the three teaching activities.

I’ve found a small, but statistically significant difference in the amount of time parents spend reading to girls at ages one, three, and five as part of a paper focused on relationship quality and investments in children.

They’ve got a very economist-y explanation for the behavior: “It is just more costly to provide a unit of reading to a boy than to a girl because the boy doesn’t sit still, you know, doesn’t pay attention,” Michael Baker told NPR (on his research with Kevin Milligan).

Costs are not just about money, people.

More on adolescent girls, because, yeah

I just realized that I never shared this work with you all. This post was written almost four months ago, but I think it’s still relevant. And even more so now as the papers are all live on the Girl Effect website. I hope you enjoy it!

My coauthor and I spent the last week finishing up our issues paper on adolescent girls for DFID and the Nike Foundation. It’s been this super crazy, whirlwind kind of project where I’ve learned so much and met so many amazing people. It’s exciting, but it sure was exhausting. I’m really excited to be able to share our findings, here they are!

So, what do we find? For the most part, programs that seek to use social norms to reduce societal discrimination against adolescent girls aren’t very well-studied. With the exception of a very small number of programs, both quantitative and qualitative analysis are lacking; overall there has been little effort to sufficiently randomize participants and perform rigorous pre- and post-intervention analysis. Thus the ability to causally identify statistically significant effects of these programs is incredibly limited.

There are a few rays of light, however. We found three programs–Tostan, Meena Communication Initiative, and Promises–that promote gender-equitable behaviors and discourage violence and discrimination against adolescent girls using social norms language or methods. All three of these programs employ multifaceted interventions. That is to say that while each has a goal of reducing discrimination or ending FGM/C, the actual process includes community conversations, social norms marketing through popular culture medium such as comic books and television shows, community declarations, school programming and more.

It seems that this is the way programs in the developing world are going. Recently, Markus Goldstein posted about his new paper on a child club program to promote the status and welfare of adolescent girls in Uganda. Though it doesn’t seem to have a strong social norms component, ELA is multifaceted, and thus multi-outcome.

In terms of sexual behavior, the girls who participate in the clubs show significantly better HIV and pregnancy knowledge than the control group.   They are also 12.6 percentage points more likely to report always using a condom when they have sex (which matches up with a reduction in those reporting often or occasional use of a condom).   They also experience a striking reduction in fertility – at follow up, treatment girls are 2.7 percentage points less likely to have a kid (26 percent of the baseline mean).   Now since they also report no increase in use of other forms of contraception, these things taken together strongly suggest that they are markedly reducing their risk of exposure to HIV.

My favorite part of reading this paper was this interactive effect. It’s very cool and I think will provide an strong template going forward for programs that wish to engage communities and have profound, lasting effects. Both Markus’ research and ours suggest that the narrowly focused, difficult-to-replicate, difficult-to-scale-up RCTs such as those heralded in Poor Economics and More Than Good Intentions have some growing to do.