{"id":5248,"date":"2026-08-19T15:17:00","date_gmt":"2026-08-19T13:17:00","guid":{"rendered":"https:\/\/blog.cerge-ei.cz\/?p=5248"},"modified":"2026-08-19T15:22:31","modified_gmt":"2026-08-19T13:22:31","slug":"to-understand-heterogeneity-design-for-it","status":"publish","type":"post","link":"https:\/\/blog.cerge-ei.cz\/?p=5248","title":{"rendered":"Designing Experiments for Better Policy"},"content":{"rendered":"

Policy effects can differ substantially across people, especially when individuals decide for themselves whether to participate. Recent CERGE-EI PhD graduate Maksim Smirnov<\/a>, winner of this year\u2019s Best Conference Paper Award<\/a> at the Royal Economic Society Annual Conference 2026<\/a> for his paper Treatment effects identification and testing via reduced-form projections<\/strong>, develops new statistical methods for testing how the effects of policies differ across people, with practical applications for policy evaluation. His new statistical test shows how experiments with multiple levels of incentives can reveal whether treatment effects differ across otherwise unobserved groups, offering researchers a simple way to assess how far their findings may generalize. In the following text, Maksim Smirnov<\/a> presents the key ideas behind his award-winning paper and explains their implications for policy analysis and design.<\/em><\/p>\n

Why varying incentives can reveal who benefits from a policy\u2014and help design better ones<\/strong><\/h5>\n

The broad motivation for the paper Treatment effects identification and testing via reduced-form projections<\/em> comes from trying to understand the degree of heterogeneity in individual-level responses to some policy, intervention, or treatment. This heterogeneity of individual effects is especially important when people select into a policy or treatment, such as participation in job training programs or college attendance.<\/p>\n

Selection is a common phenomenon in economics, and unfortunately it rules out some policy analysis techniques that are regarded as the gold standard; e.g. randomized controlled trials, which are great tools, but if there is selection, then their scope is limited. Even worse than selection is selection on unobservable characteristics. The classical example is the quest to estimate the return to college; people might select into college if their ability is high, so direct comparisons of average wages between college attendees and non-attendees are meaningless.<\/p>\n

An instinctive urge to somehow get back to randomization is useful here: indeed, you cannot randomize education, but you can randomize incentives. For example, you can randomize vouchers that reduce tuition costs. Then suddenly you can compute a meaningful causal parameter \u2014 the average effect for those people who comply with the given voucher \u2014 i.e. those who would not attend college without this voucher but do if they randomly get it. The big question then, and this is finally where I would like to place my paper, is whether these effects for compliers are relevant for other subpopulations.<\/p>\n

In general, it is not possible to identify compliers from the data, so you cannot check if their characteristics are somewhat representative of the whole population. However, if you want to scale the program or change some of its characteristics, there is no reason to expect the observable effect to scale.<\/p>\n

Suppose you randomized not one type of voucher but, say, two types, so that there are three groups of people: those who receive no voucher, a small voucher, or a large voucher. Then there are suddenly two groups of compliers \u2014 those who comply with the small voucher and those who do not comply with the small voucher but comply with the larger voucher. In principle, you can compute average effects for these two subpopulations of compliers and compare them.<\/p>\n

Obviously, large-voucher compliers need a stronger incentive to select into college, while small-voucher compliers need a much weaker incentive, so they are different in their willingness to participate. If you find that the effects are very different in these two groups of compliers, then it would indicate a strong association between the willingness to participate and individual effects, and hence a lot of heterogeneity between subpopulations of compliers.<\/p>\n

Alternatively, if the effects are similar in these two groups of compliers even though they are different in their willingness to participate, then there is no evidence of strong heterogeneity between subpopulations of compliers.<\/p>\n

\"\"<\/a>
Maksim Smirnov, winner of the Best Conference Paper Award at the Royal Economic Society Annual Conference 2026. Source: Royal Economic Society.<\/figcaption><\/figure>\n

The main contribution of my paper is to provide a formal statistical procedure that can compare average effects across different unobserved subpopulations and therefore discover if there is meaningful heterogeneity. The main advantage is that my test is simple to run and fast to execute using common software.<\/p>\n

Apart from the test itself, or more in addition to it, the main lesson of the paper is that if you want to understand heterogeneity, you should design heterogeneity. Back to the voucher example \u2014 if you design an experiment in which you randomize only one kind of incentive, say a 5% tuition cost voucher, there is no way you can learn something about heterogeneity between unobserved subpopulations or whether the effect you compute using this particular voucher is in any way generalizable.<\/p>\n

But if you randomize just two types of vouchers, you can learn something about unobserved heterogeneity. And the longer the menu of different incentives you randomize is, the more you can learn about heterogeneity. And the more you learn about heterogeneity, the more flexible and predictable policies you can design.<\/p>\n

For instance, if you know that the net effect of some policy is negative for 90% of the population, you will try to design a policy that focuses on the other 10%. Another advantage of having multiple different incentives is that, back to my test, you identify effects for a larger number of subpopulations, and if you still find no evidence of heterogeneity, the conclusion of the test is more powerful and credible.<\/p>\n

Maksim Smirnov on his future plans<\/h5>\n

As for the future, I am now a postdoctoral researcher at the Aarhus Center for Econometrics (ACE)<\/a>, and one of the directions in which I would like to take this paper further is to explore how the optimal policy, from the point of view of maximizing some welfare criterion, depends on the amount of heterogeneity. In particular, I want to measure how a simple test, which I proposed in this paper, can make the lives of policy-makers easier compared to general-case welfare maximization, which requires heavy machinery with slow computation, and reliance on noisy estimates of nuisance parameters. I feel that this might be another meaningful application of my work, with important implications for policy analysis in general and policy design in particular.<\/p>\n

Would you like to learn about Maksim Smirnov\u2019s research stay at MIT? Read more about it here.<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

Policy effects can differ substantially across people, especially when individuals decide for themselves whether to participate. Recent CERGE-EI PhD graduate Maksim Smirnov, winner of this year\u2019s Best Conference Paper Award at the Royal Economic Society Annual Conference 2026 for his paper Treatment effects identification and testing via reduced-form projections, develops new statistical methods for testing … Continue reading Designing Experiments for Better Policy<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":10,"featured_media":5249,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[149,1,11,13,256],"tags":[16,20,22],"class_list":["post-5248","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-alumni","category-uncategorized","category-phd","category-research","category-research-2","tag-cerge-ei","tag-economics","tag-ph-d-in-economics"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t