When is the evidence strong enough? Using the A-value to decide whether to collect more data

Suppose a government receives the results of an experiment evaluating an anti-poverty program. The estimated welfare effect is positive, but its p-value is 0.08. The researchers describe the result as “not statistically significant”. What should the government do? Implement the program? Retain the status quo? Ask the researchers to collect more evidence?

This is an economic policy question which the p-value is not formally designed to answer. A statistically uncertain estimate may call for additional evidence if a follow-up study is relatively inexpensive and the policy will affect many people. At the same time, however, it may represent the best evidence we can get if an additional experiment is prohibitively expensive.

In our recent paper, “When is the statistical evidence strong enough? Using hypothesis tests to value data collection”, we develop a framework for making this decision. Our proposal is a statistic that researchers can report alongside conventional results: the abstention-value, or A-value. The A-value is the break-even welfare cost of conducting an additional experiment versus making an immediate policy decision using existing evidence.

Consider an initial (experimental) study that provides an unbiased estimate of welfare effects. A government or research funding agency faces three possible responses: (1) recommend the policy, (2) retain the status quo, or (3) defer a recommendation and collect more evidence. The third decision is an investment with up-front costs and potential future benefits: further evidence generates expected welfare gains by reducing the chances of a wrong decision. We evaluate decision rules by minimizing the welfare loss of making the wrong decision, worst-case over possible welfare effects.

The optimal rule corresponds to a standard t-test where the threshold depends on the cost of collecting additional evidence. Our solution illustrates why conventional significance thresholds are ill-suited to reconcile the economic trade-offs involved. Even a statistically significant result can warrant further investigation (“deemed insignificant”) when another study is inexpensive. Conversely, policymakers may take an immediate policy action even with an insignificant but positive result when collecting additional evidence is too costly to justify waiting for more data collection.

What should researchers report to facilitate these high-stakes decisions? Our core idea is to invert this rule and give the A-value as the break-even cost at which the optimal action switches from an immediate decision to deferral. Researchers should recommend collecting more evidence if and only if the experimentation costs are below the A-value, and make an immediate policy decision otherwise.

In summary, the A-value captures two features of the evidence. Holding the standard error and relative precision of the follow-up fixed, it increases with the p-value. Less decisive evidence creates more scope for useful learning. But, holding the p-value fixed, the A-value also increases with the absolute magnitude of the effect, measured in welfare units. An inconclusive estimate about a potentially large effect can justify more research than an equally statistically inconclusive estimate about a much smaller effect.

The framework can also help funding agencies choose among competing research proposals when resources are limited. We propose funding follow-ups with the largest A-values per unit of research expenditure, and show formal theoretical guarantees.

We illustrate the approach using 42 evaluations of unconditional cash-transfer programs, drawing on harmonised data from Crosta et al. (2024), and using monthly household consumption gains as the welfare criterion for illustrative purposes.

Two programs illustrate why the A-value can improve research funding decisions. An intervention in Bangladesh delivers a p-value of 0.277, while analogous results from Kenya have a p-value of 0.377. Neither result passes conventional significance tests. A naive rule prioritising the largest p-values would direct resources to the Kenyan program for follow-up study. Yet under our calibration, the monetary A-values are approximately $198,000 for the Bangladeshi program and $30,000 for the Kenyan program due to larger standard errors and welfare at stake of the Bangladeshi program. In short, there is an economic wedge between the A-value and p-value rankings: while the Kenyan estimate is statistically less decisive, the unresolved uncertainty surrounding the Bangladeshi program concerns substantially larger welfare consequences.

Overall, how should researchers use the A-value in practice? We suggest four steps.

1. Define the decision. Specify the intervention, the status quo, and the outcome that matters for the decision. Incorporate the implementation costs which are pertinent to the policy choice (e.g., monetary implementation cost, welfare opportunity cost).
2. Specify the follow-up study. The value of additional evidence depends on how informative it will be. When no particular study has been designed, an independent follow-up with the same standard error provides a useful reporting benchmark. Researchers can then examine alternative precisions and their associated research costs.
3. Calculate and calibrate the A-value. Convert the welfare threshold into interpretable economic units using transparent assumptions. When feasible, we encourage researchers to use a monetary calibration, something we discuss in our applications.
4. Report the A-value alongside the usual results. Include the point estimate, standard error, confidence interval, p-value, and the (assumed) follow-up precision. We also recommend reporting a standardised A-value, obtained by dividing by an explicitly stated reference outcome standard deviation. This expresses the break-even cost in outcome-standard-deviation units and makes the statistic invariant to changes in measurement units.