EA - Predicting the cost-effectiveness of running a randomized controlled trial by Falk Lieder

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Predicting the cost-effectiveness of running a randomized controlled trial, published by Falk Lieder on April 17, 2023 on The Effective Altruism Forum.TLDR: Research is underrated. Running an RCT to evaluate a digital intervention for promoting altruism could be more than 10x as cost-effective as the best charities working on global health and wellbeing.In the previous post, we found that – in expectation – Baumsteiger’s (2019) intervention for promoting altruism is about 4x as cost-effective as GiveDirectly but lower than the cost-effectiveness of the Against Malaria Foundation or StrongMinds. However, the uncertainty about the actual cost-effectiveness of this intervention is still extremely high.The uncertainty is, in fact, so high that the 95% credible interval on the cost-effectiveness of the new intervention ranges from -0.5 WELLBYs/$1000 to 88 WELLBYs/$1000. The upper bound of this credible interval is close to the cost-effectiveness of the presumably most cost-effective mental health charity StrongMinds (90 WELLBYs/$1000; Plant, 2022), and more than twice the cost-effectiveness of the Against Malaria Foundation (39 WELLBYs/$1000; Plant, 2022). Based on these estimates, there is a 5% chance that the intervention might be harmful and a more than 5% chance that it might be at least as cost-effective as the charities recommended by GiveWell and the Happier Lives Institute.Because of this high uncertainty, any decisions based on the current state of knowledge could be highly suboptimal compared to what we would do if we had additional information. However, information can be costly, especially when running a randomized controlled trial (RCT). And the more money we spend on information, the less we can spend on saving lives. This dilemma raises the question, “When is it worthwhile to run an RCT to gather more data, and when should we exploit what we already know?” To answer this question, we introduce a new method for predicting the cost-effectiveness of gaining new information through an RCT and comparing it to the cost-effectiveness of cash transfers and directly promoting global health and well-being. We illustrate this method using the intervention by Baumsteiger (2019) as an example.However, the approach we are illustrating is more general and can also be applied to RCTs on established, emerging, and yet unknown EA interventions, including deworming, motivating parents to vaccinate their children, water purification, and interventions for improving mental health.We develop our method in two steps. First, we apply the established Value of Information framework (Howard, 1966) to obtain an upper bound on the cost-effectiveness of running an RCT. Then, we replace this method’s unrealistic assumption of perfect information with more realistic assumptions about the imperfect information generated by an RCT. This yields a new method that can provide more accurate estimates of the cost-effectiveness of evaluation research. As a proof of concept, we apply this method to predict how cost-effective it would be to evaluate the altruism intervention based on Baumsteiger (2019) in RCTs with different numbers of participants. Our method predicts that running such an RCT with 1200 participants would be highly cost-effective. This post is a brief summary of the longer report presented in this interactive notebook.How valuable would it be to know the true exact value of the cost-effectiveness of the intervention by Baumsteiger (2019)?To obtain an upper bound on how valuable it might be to evaluate the intervention by Baumsteiger (2019), I first calculate the value of obtaining perfect information about its cost-effectiveness. The value of perfect information is an established mathematical concept introduced by Howard (1966). It has recently been applied to charity evalu...