MODEL COMPARISONS IN UNSTABLE ENVIRONMENTS

• DOI: http://doi.org/10.1111/iere.12161
• Published date: 01 May 2016
• Author: Barbara Rossi,Raffaella Giacomini
• Date: 01 May 2016

INTERNATIONAL ECONOMIC REVIEW

Vol. 57, No. 2, May 2016

MODEL COMPARISONS IN UNSTABLE ENVIRONMENTS∗

BYRAFFAELLA GIACOMINI AND BARBARA ROSSI1

UCL and CeMMAP, U.K.; ICREA - Univ. Pompeu Fabra, Barcelona GSE and CREI, Spain

The goal of this article is to develop formal tests to evaluate the relative in-sample performance of two competing,

misspecified, nonnested models in the presence of possible data instability. Compared to previous approaches to model

selection, which are based on measures of global performance, we focus on the local relative performance of the models.

We propose tests that are based on different measures of local performance and that correspond to different null and

alternative hypotheses. The empirical application provides insights into the time variation in the performance of a

representative Euro-area Dynamic Stochastic General Equilibrium model relative to that of VARs.

1. INTRODUCTION

The problem of detecting time variation in the parameters of econometric models has been

widely investigated for several decades, and empirical applications have documented that struc-

tural instability is widespread.

In this article, we depart from the literature by focusing on investigating instability in the

performance of models, instead of focusing solely on instability in their parameters. The idea is

simple: In the presence of structural change, it is plausible that the performance of a model may

itself be changing over time even if the model’s parameters remain constant. In particular, when

the problem is that of comparing the performance of competing models, it would be useful to

understand which model performed better at which point in time.

The goal of this article is therefore to develop formal techniques for conducting inference

about the relative performance of two models over time and to propose tests that can detect

time variation in relative performance even when the parameters are constant. Existing model

selection tests such as Rivers and Vuong (2002) are inadequate for answering this question, since

they work under the assumption that there exists a globally best model. The central idea of our

method is instead to propose a measure of the models’ local relative performance: the “local

relative Kullback–Leibler Information Criterion” (local relative KLIC), which represents the

relative distance of the two (misspecified) likelihoods from the true likelihood at a particular

point in time. We then investigate ways to conduct inference about the local relative KLIC and

construct tests of the joint null hypothesis that the relative performance and the parameters of

the models are constant over time.

We propose two tests, which correspond to different assumptions about the parameters and

the relative performance under the null and alternative hypotheses: (1) a “one-time reversal”

∗Manuscript received April 2011; revised January 2015.

1We are grateful to D. Kristensen for helpful comments and M. del Negro, F. Smets, R. Wouters, W.B. Wu, and

Z. Zhao for sharing their codes. We also thank the editor, three anonymous referees, seminar participants at the

Empirical Macro Study Group at Duke University, Atlanta Fed, UC Berkeley, UC Davis, University of Michigan,

NYU Stern, Boston University, University of Montreal, UNC Chapel Hill, University of Wisconsin, UCI, LSE, UCL,

Stanford’s 2006 SITE workshop, the 2006 Cleveland Fed workshop, the 2006 Triangle Econometrics workshop, the

Fifth ECB Workshop, the 2006 Cass Business School Conference on Structural Breaks and Persistence, the 2007 NBER

Summer Institute, the 2009 NBER-NSF Time Series Conference, and the 2012 AEA Meetings for useful comments and

suggestions. Support by NSF grants 0647627 and 0647770 is gratefully acknowledged. Please address correspondence

to: Barbara Rossi, Centre de Recerca en Economia Internacional (CREI), Universitat Pompeu Fabra (UPF), carrer

Ramon Trias Fargas, 25-27, Merc`

e Rodoreda bldg., Barcelona, 08005, Spain (ES). Phone: +34 93 542 1655. Fax: ++34

93 542 2826. E-mail: barbara.rossi@upf.edu.

369

C

(2016) by the Economics Department of the University of Pennsylvania and the Osaka University Institute of Social

and Economic Research Association

370 GIACOMINI AND ROSSI

test against a one-time change in models’ performance and parameters and (2) a “fluctuation

test” against smooth changes in both performance and parameters. The first test is based on

estimating the parameters and the relative performance before and after potential change

dates, whereas the second is based on nonparametric estimates of local performance and local

parameters. The fluctuation test is based on a fixed-bandwidth approximation; unreported

results show that it delivers a better finite-sample performance than a test based on a standard

shrinking bandwidth approximation (e.g., Wu and Zhao, 2007).

For both tests, we show that the dependence of the local performance on unobserved param-

eters does not affect the asymptotic distribution of the test statistic as long as the parameters

are also estimated locally.

Our research is related to several papers in the literature, in particular Rossi (2005) and,

more distantly, to Muller and Petalas (2010), Elliott and Muller (2006), Andrews and Ploberger

(1994), and Andrews (1993). Rossi (2005) proposes a test that is similar to our one-time reversal

test but focuses on the case of nested and correctly specified models. Here, we consider the

more general case of nonnested and misspecified models and propose one additional test. In

a companion paper, Giacomini and Rossi (2010) investigate the problem of testing the time

variation in the relative performance of models in an out-of-sample forecasting context. Even

though some of the techniques are similar, the additional complication in the in-sample context

considered in this article is that the measure of relative performance depends on estimated

parameters, which needs to be taken into account when performing inference. The dependence

on parameter estimates can instead be ignored in an out-of-sample context, provided one adopts

the asymptotic approximation with finite estimation window considered by Giacomini and Rossi

(2010).

Our approach in this article is also related to the literature on parameter instability testing

(e.g., Brown et al., 1975; Ploberger and Kramer, 1992; Andrews, 1993; Andrews and Ploberger,

1994; Elliott and Muller, 2006; Muller and Petalas, 2010) in that we adapt the tools developed in

that literature to our different context where the null hypothesis of interest is a joint hypothesis

that the relative performance of the models is equal at each point in time and that the parameters

are constant.

The fact that parameters are constant under our null hypothesis means that we are not

considering the potentially relevant case in which the performance of two models is equal in

spite of their parameters changing over time. The reason for excluding this case is a pragmatic

one. In principle, one could have developed versions of our tests that allow for some time

variation in parameters under the null hypothesis. Doing so would, however, be costly in terms

of the general applicability of our techniques, as it would require us to impose additional

restrictions on the type of time variation under the null hypothesis, the properties of the data,

and the models that are compatible with the assumptions on which the tests are based. We

illustrate this point more concretely when discussing the assumptions of each test in the body

of the article.

One important limitation of our approach is that our methods are not applicable when the

competing models are nested, which is common in the literature on model selection testing

based on Kullback–Leibler type of measures. See Rivers and Vuong (2002) for an in-depth

discussion of this issue.

The article is structured as follows: The next section discusses a motivating example that

illustrates the procedures proposed in this article. Section 3 defines the null hypotheses, and

Section 4 describes the tests. Section 5 evaluates the small sample properties of our proposed

procedures in a Monte Carlo experiment, and Section 6 presents the empirical results. Section

7 concludes. The proofs are collected in the Appendix.

2. MOTIVATING EXAMPLE

Let yt=β0

txt+γ0

tzt+ut,with ut∼i.i.d.N(0,1),xt,ztindependent N(0,σ

2

x,t) and N(0,σ

2

z,t),

respectively, independent of each other and of utfor t=1,...,T, so that the true conditional