Evidence use in State policymaking: A bibliometric analysis of two consequential policy areas

Evidence based or evidence informed decision making is the paradigm by which public decision makers leverage knowledge from a variety of sources in order to craft public policies and programs that are fair, efficient, and effective. Evidence can come in a multitude of packages and be leveraged with different calculuses about what matters most.  In this work we build on Head’s three lenses of evidence based policy where he articulates that evidence can come from science (what we traditionally think of at “evidence”), from practice (the tacit and earned knowledge of front line public servants), and from the political sphere (knowledge about political dynamics, authority, and feasibility) by examining policy documents from two consequential public policy domains in the United States: autonomous vehicles and state level opioid strategic plans.

 

Policy reports and plans can be used as artifacts of policy processes that capture the decisions and dynamics of complex policy activities, rather than as an end of the policy process (Freeman and Maybin 2011).  As such, a detailing of the people and sources used to create the policy document can reveal important aspects of how the policy evolves in a location. We use policy documents from the two domains mentioned above to identify important aspects of how the policy deliberations occurred and the impact it had on the outcome (report construction). We created unique datasets for this investigation.

 

Data and methods. All reports from the 50 states and Washington DC relative to the two substantive areas were scraped from publicly available sources on the internet. All reports from 2015 until 2020, when the COVID-19 pandemic started) were considered for this investigation.  Reports were then reviewed for their focus on program implementation and governance, rather than a technical focus.  A technical focus for the autonomous vehicle area would be things such as road construction and other engineering aspects of auto manufacturing. A technical focus on the opioid response issue would be the specific evidence for one intervention or another (e.g. methadone clinics). There were 68 reports on the AV side and 72 in the opioid case.

 

Reports were then analyzed on a number of dimensions to understand how they used evidence in their work. Our methods were centered in bibliometric techniques which catalogs and enumerates the contents of scientific documents.  We assessed images and figures, references, word/page count, topics covered using natural language processing, authors, and committee/task force membership. We also ran clustering models to find distinct patterns of reports.

 

Findings. We found three distinct clusters of reports within each of the two substantive areas.  While the specifics of the clusters across the two areas are slightly different, they are remarkable in that they have the hallmarks of Head’s three lenses of evidence for policy making.  In the AV side we see three clusters that are Expert- engineering (practice evidence), expert academic (scientific evidence), and convening, legislative focus (political evidence). For the opioid reports we see that reports that have alot of private organizations (rather than state agencies) participating show characteristics of emphasizing practice based knowledge, while those reports that are oriented by task forces display characteristics of political knowledge, and finally, those reports authored by groups with alot of domain expertise (representation from health and public health departments) display the scientific knowledge discussed by Head.  

 

Conclusion. While all three lenses of evidence in policymaking are valid, the kinds of evidence used within policymaking bodies show clear patterns related to who participates and what kind of information is synthesized. This impacts what issues are emphasized and examined in these processes and the policies and programs resulting from these processes. Therefore when constructing working bodies to analyze and develop policies and programs of public import, the construction of those working bodies is greatly important since it will determine the nature of the results yielded from that process.