9/14/15

Scientific reproducibility and impact investing

Reproducibility in science is a topic that's getting quite a bit of attention.

Three articles I'd encourage reading for different perspectives:

Given the impact investment sector's often reliance on the social sciences to guide their decision making—in pre-investment valuation & due diligence, post-investment outcomes monitoring, etc—I thought it would be worth briefly digging into why the reproducibility discussion should matter to impact investors.

The reproducibility discussion in science essentially boils down to the reality that even slight variations in one or more parts of the scientific method—be it experimental design/conditions, study participant characteristics, data collection/cleaning methods, and data analysis methods—can lead to material differences in a study's ultimate findings (as illustrated by the example below).

Are soccer players with darker skin more likely to receive red cards from referees?

A practical takeaway from this is to remind ourselves that when reading statements like: "intervention X, reduces outcome Y, by Z%", we need to be open to the idea that the cited "Z%" may not be a singular definitive value, as much as it may be a range of values.

A crude interpretation of this could be that variations in the scientific method are "a proxy" for the underlying sensitivity / risk associated with a certain finding. This type of mindset should push impact investors to think more carefully about how to interpret the statistics—especially in different contexts and use cases.

For example, in applications where absolute outcomes are imperative to an investment—like in pay for success contracts / social impact bonds—it may be appropriate to "stress test" the scientific method to uncover variations.

This may mean going beyond traditional literature research and looking into things like different analysis methods on the same raw data (third-party model validation), and/or running additional studies (targeted field-experiments). This is a step towards turning uncertainty into risk, and helps avoid expectation misalignments amongst pre- and post- investment stakeholders.

Accounting for variations in the scientific method—or at the very least, being aware of the possibility that they exist—must be part of the toolkit for impact investors relying on social science data to drive their decision making. If anything, the reproducibility discussion in science should serve to reinforce the need for impact investors to be cautious when extrapolating scientific conclusions from the lab into practice.