Shades of Grey: Guidelines for Working with the Grey Literature in Systematic Reviews for Management and Organizational Studies

• Date: 01 October 2017
• Author: Anne Sigismund Huff,Palie Smart,Richard J. Adams
• DOI: http://doi.org/10.1111/ijmr.12102
• Published date: 01 October 2017

International Journal of Management Reviews, Vol. 19, 432–454 (2017)

DOI: 10.1111/ijmr.12102

Shades of Grey: Guidelines for Working

with the Grey Literature in Systematic

Reviews for Management and

Organizational Studies

Richard J. Adams, Palie Smart1and Anne Sigismund Huff2

Surrey University Business School, University of Surrey, Guildford, GU2 7XH, UK, 1Cranfield School of

Management, Cranfield University, Bedfordshire, MK43 0AL, UK, and 2School of Business, Maynooth University,

Maynooth, Co. Kildare, Ireland

Corresponding author email: correspond.adams@btinternet.com

This paper suggests how the ‘grey literature’, the diverse and heterogeneous body of

material that is made public outside, and not subject to, traditional academic peer-

review processes,can be used to increase the relevance and impact of management and

organization studies (MOS). The authors clarify the possibilities by reviewing 140 sys-

tematic reviews published in academic and practitioner outlets to answer the following

three questions: (i) Why is greyliterature excluded from/included in systematic reviews

in MOS? (ii) What types of grey material have been included in systematic reviews

since guidelines for practice were first established in this discipline? (iii) How is the

grey literature treatedcurrently to advance management and organization scholarship

and knowledge? This investigation updates previous guidelines for more inclusive sys-

tematic reviews that respond to criticisms of currentreview practices and the needs of

evidence-based management.

Introduction

Summaries of past research are widely used both to

inform new inquiries in many research disciplines and

to influence professional practice (Briner et al. 2009;

Shepperd et al. 2013). Procedures for systematic re-

This work was conducted with support from the Engineer-

ing and Physical Sciences Research Council (EPSRC) Cen-

tre of Excellence for Industrial Sustainability grant number

EP/I033351/1 (a research collaboration of Cambridge, Cran-

field,Imperial and Loughborough Universities). No new data

were created in the course of this work. The authors are

grateful to Greg Boulton, Technology Enhanced Learning

Designer at Cranfield University, for his help in creating

Figure 1. The authors also acknowledge the pioneering work

of Emeritus Professor David Tranfield who laid the founda-

tions for systematic review in Management and Organiza-

tional Studies which continues to inspire.

view and evidence-based decision-making based on

review were developed for medicine by the Cochrane

Collaboration (http://www.cochrane.org/) and re-

ceived a good deal of attention from researchers from

other fields in the early 1990s. A decade later, these

ideas were adapted in the field of management and

organization studies (MOS) (Denyer and Tranfield

2009; Rousseau 2006: 2012; Tranfield et al. 2003).

A critical feature of systematic review is com-

prehensive, rule-based search operations for collect-

ing and synthesizing relevant evidence. Following

the definition of a review question, identification

of relevant knowledge is typically initiated by key-

word/search string searches of electronic databases of

scholarly publications. If these sources uncover a co-

herent body of high-quality, relevant, peer-reviewed

articles, so-called ‘white literature’ (Lawrence et al.

2014), it is possible to proceed with scholarly

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution

and reproduction in any medium, provided the original workis properly cited.

C2016 The Authors

International Journal of Management Reviews published by British Academy of Management and John Wiley & Sons Ltd.

Published by John Wiley & Sons Ltd, 9600 Garsington Road, Oxford OX4 2DQ, UK and 350 Main Street, Malden, MA

02148, USA

Shades of Grey 433

inquiry on a firm foundation. However, scholars are

increasingly recognizing instances whereit seems ap-

propriate to broaden the evidence search beyond the

limits of academic journals to incorporate ‘grey liter-

ature’ (Adams et al. 2015; Sharma et al. 2015).

Incorporating grey literature – the diverse and het-

erogeneous body of material available outside, and

not subject to, traditional academic peer-review pro-

cesses – can make a variety of positive contributions

to subsequent inquiry and practice. The review of

reviews summarized in this paper shows that a sig-

nificant number of MOS scholars already assert that

grey material has relevance to their research ques-

tions and objectives. Our analysis reveals that these

MOS scholars have used the grey literature to extend

the scope of findings in their reviews by incorporat-

ing relevant contemporary material in dynamic and

applied topic areas where scholarship lags; they have

explored novelfields of enquir y, and have validated or

corroborated findings from the academic literature.

Scholars active in other disciplines, with longer

traditions of deploying systematic review, report ad-

ditional benefits such as addressing publication bias

(Hopewell et al. 2007), of which there is some evi-

dence in MOS (Kepes et al. 2012), but limited in our

sample. A number of reviewers from our sample who

exclude the grey literature because of its challenges

and the time required also believe that their conclu-

sions may be poorer for its absence (e.g. Levy and

Williams 2004). On the basis of these observations,

we argue that there is strong justification for greater

consideration of including the grey literature in future

MOS systematic reviews.

While the potential contributions of grey litera-

ture are becoming apparent (Benzies et al. 2006;

Rothstein and Hopewell 2009), little methodologi-

cal guidance exists. For example, in the index of the

460 page Oxford Handbook of Evidence-Based Man-

agement (Rousseau 2012) ‘grey literature’ is referred

to once. We believe that more specific guidelines for

scholars on including grey literature in MOS reviews

are important as the practice of systematic review in

our field continues to mature. This paper contributes

to discussion of the purpose and methods of litera-

ture review (e.g. Hart 1998; Jones and Gatrell 2014;

Tranfield et al. 2003; Webster and Watson 2002) by

suggesting how grey literature can be handled more

systematically,even though its diversity also requires

considerable flexibility.

Our purpose is to consider how current rules

for MOS systematic review might be systematically

broadened to incorporate grey literature, where ‘grey

literature’ includes a wide variety of potentially rel-

evant material, from specialist journals to blogs and

other informal communications. Drawing on an anal-

ysis of 140 MOS systematic reviews published since

Tranfield et al. (2003), we develop additional guide-

lines by considering three questions aimed at better

understanding current use of this material:

1. Why is grey literature excluded from/included in

systematic reviews in MOS?

2. What types of grey material have been included

in systematic reviews since guidelines for practice

were first established in this discipline?

3. How is the grey literature currently treated to ad-

vance management and organization scholarship

and knowledge?

The analysis reported here more firmly connects

systematic review to its original pragmatic and

practice-oriented purpose (Tranfield et al. 2003) in

the evidence-based management (EBMgt) paradigm

(Briner et al. 2009; Huff et al. 2006; Rousseau 2012).

A primary aim of the paper is to suggest further tools

for pragmatic, ‘evidence-informed’ (Tranfield et al.

2003) research in MOS; ‘pragmatic’ in the sense that

they inform future management practice by present-

ing a range of evaluated alternatives developed in the

service of action (Pascal et al. 2013).

We begin by defining grey literature in relation to

scientific/academic literature. Next, we describe our

review methodology and present our synthesis of the

evidence relating to our why, what and how ques-

tions about grey literature. Motivated by the need

for improved management of the grey literature in

systematic MOS reviews, our discussion draws our

findings together in a set of practice guidelines, sim-

ilar in intent to more general guidelines developed

by Rousseau et al. (2008). A concluding discussion

considers the use of grey literature in future reviews

that take into account current critiques of systematic

review practices.

Characterizations and challenges

of grey literature

As a general definition, grey literature is composed

of knowledge artefacts that are not the product of

peer-review processes characterizing publication in

scientific journals (Lawrence et al. 2014). Grey lit-

erature has been more specifically conceptualized in

C2016 The Authors

International Journal of Management Reviews published by British Academy of Management and John Wiley & Sons Ltd.