Score manipulation, density continuity and intent‐to‐treat effect for regression discontinuity

• Published date: 01 October 2023
• Author: Jin‐young Choi,Myoung‐jae Lee
• Date: 01 October 2023
• DOI: http://doi.org/10.1111/1468-0106.12380

ORIGINAL ARTICLE

Score manipulation, density continuity and

intent-to-treat effect for regression

discontinuity

Jin-young Choi

1

| Myoung-jae Lee

2

1

WISE and the School of Economics,

Xiamen University, Xiamen, China

2

Department of Economics, Korea

University, Seoul, South Korea

Correspondence

Myoung-jae Lee, Department of

Economics, Korea University, Seoul

02841, South Korea.

Email: myoungjae@korea.ac.kr

Funding information

Fundamental Research Funds for the

Central Universities, Grant/Award

Number: 20720201046; National Research

Foundation of Korea, Grant/Award

Number: 2020R1A2C1A0100778611

Abstract

Regressiondiscontinuity(RD)iswidelyusedinmanydis-

ciplines of science to find treatment effect when the treat-

ment is determined by an underlying running variable

(‘score') Scrossing a cutoff cor not. The main attraction

of RD is local randomization around c,whichis,how-

ever, often ruined by manipulation on S. To detect

manipulation, the continuity of score density function

fSsðÞat cis routinely tested in practice. In this paper, we

examine how informative fSis for RD, and show the fol-

lowing. First, for incumbency effect in election to which

RD has been heavily applied, fSmay have no informa-

tion content. Second, for RD in general, the fScontinuity

is neither necessary nor sufficient for RD validity. Third,

if the treatment cannot be implemented without manipu-

lation of S, then the manipulation had better be consid-

ered as part of the treatment effect, much as in ‘intent-to-

treat effect’for clinical trials. These findings call for

relying less on fScontinuity tests and, instead, thinking

moreabouthowsubjectsreacttothetreatmenttomodify

their S, how to design the treatment to lessen manipu-

lation, and what to take as the desired treatment effect.

1|INTRODUCTION

Regression discontinuity (RD) is one of the most popular approaches for treatment effect

analysis with observational data: see Imbens and Lemieux (2008), Lee and

Received: 8 December 2020 Revised: 2 June 2021 Accepted: 17 October 2021

DOI: 10.1111/1468-0106.12380

552 ©2021 John Wiley & Sons Australia, Ltd Pac Econ Rev. 2023;28:552–569.wileyonlinelibrary.com/journal/paer

Lemieux (2010), Lee (2016), Choi and Lee (2017,2021), Cattaneo et al. (2019), and refer-

encestherein.InadditiontotheseRDreviewsoneconomics,statisticsandpoliticalsci-

ence, there are also many RD studies in behavioural and educational sciences, as can be

seen in Bloom (2012), Wong et al. (2013), Tahamont (2019), Sales and Hansen (2020)

and Bloom et al. (2020), among many others.

In typical RD, a subject is assigned to a treatment or control group, depending on a running

variable or ‘score’Scrossing a cutoff cor not. A localization around c(i.e., using only some

local subjects with S’c) then gives local treatment and control groups, which are thought to be

randomized in the sense that all covariates are balanced, observed (X) or unobserved (ε). The

two local groups have the same mean (or distribution) of Xand ε, which is covariate balance or

local randomization.

One problem in this local randomization scenario is that often Sis manipulated to result in

covariate imbalances, and thus no local randomization. ‘Manipulation’exists if ‘G≠S’, where

Gis the ‘Genuine score’that would prevail if there were no treatment and thus no incentive for

manipulation. In RD, covariate balance is implied by the continuity of EXjS¼sðÞand

EεjS¼sðÞat c, because the continuity around camounts to no change in EXjS¼sðÞand

EεjS¼sðÞaround c. We use the expression ‘balance’instead of ‘continuity’, however, because

what is needed eventually is balance.

To detect manipulation causing covariate imbalances, the continuity of score density fSsðÞ

at cis routinely tested in RD practice. Since the balance of Xcan be easily checked out by com-

paring the means (or distributions) of Xacross the two groups, the main use of fScontinuity

test has been for the balance of the unobserved ε, under the belief that the f Scontinuity at c is

equivalent to the balance of ε. Because Scan be redefined as Sc, set c¼0 unless otherwise

mentioned. Since breaks in fSsðÞand EjS¼sðÞmatter only at s¼0, we often omit the qualifier

‘at s¼0’henceforth.

Facing the problem of score manipulation, it might be possible to find Gusing extra infor-

mation. In revisiting the issue of class size effect on test scores (Angrist & Lavy, 1999), Angrist

et al. (2019) were able to find Gbased on birthdays, whereas class size Sis based on a reported

number from schools. Using both Gand Sin data over 2002–2011, Angrist et al. (2019) con-

cluded no class size effect on test scores, differently from Angrist and Lavy (1999). However,

this way of overcoming a score manipulation problem should be an exception, not a rule,

because most researchers would take the data at hand as given to work only with what is

available.

There appeared several tests for fSbreak: McCrary (2008) using a histogram and local linear

regression with two bandwidths and data-prebinning, and Cattaneo et al. (2020) based on a

local polynomial density estimator with one bandwidth and no data-prebinning. Also, Otsu

et al. (2013) proposed an empirical likelihood-based test, and Frandsen (2017) proposed a test

for manipulation with integer S. Among these tests, almost all existing RD studies used the

McCrary (2008) test, and RD studies with the fScontinuity rejected by McCrary test are

extremely rare (e.g., Urquiola & Verhoogen, 2009). This reflects how strong the belief is that the

fScontinuity is equivalent to the balance of ε.

Contrary to the common belief, however, the fScontinuity is neither necessary nor suffi-

cient for the balance of ε, that is, for RD local randomization success given the balance of X.

Indeed, McCrary (2008, p. 701) himself provided two verbal counter-examples to state ‘a run-

ning variable with a continuous density is neither necessary nor sufficient for identification

except under auxiliary assumptions’, which was also mentioned in Bloom (2012), Sekhon and

Titiunik (2016) and Frölich and Sperlich (2019). The first example in McCrary (2008) is for a

CHOI AND LEE 553