Evidence and its Importance for Policy Making

In the process of gathering information, data or evidence to facilitate policy development processes, the parameters placed on the gathering of such, has the potential to influence the outcomes. In today’s world, policy built upon science-based evidence is the goal of all governance systems, be it to develop policy for the environment, industry, health or food. To develop policy rooted in scientific evidence, the policy process adopts norms to ensure that the assessment process is as broadly representative as possible, thereby mitigating the potential for bias to enter the process and adversely impact the outcome. In other words, the goal of good evidence-based policy is looking to cast a net out wide enough to collect all views on a topic, in order to make a balanced and informed policy decision. Sounds reasonable and responsible.

Launching the project

It is estimated that approximately 1.6 million journal articles are published annually, with a total volume of 50 million published articles. Clearly, no policy development team could review this many publications in their search for relevant evidence, so methods are used to refine the search and filter the relevant literature. Along with colleagues Peter Phillips and David Castle, we were commissioned by the academic publishing firm Edward Elgar to produce a research collection on the 50 most significant articles relating to agriculture, biotechnology and development. To ensure that our personal bias didn’t influence the selection process, we engaged three different methods to determine the 50 articles. We utilized an expert survey, citation analysis and a social network analysis, as methods to assess the data that was gathered.

The three of us are well connected across the fields of agriculture, biotechnology and development, with over 60 years of academic research on these topics. For this reason, Edward Elgar reached out to us to carry out this research, and we used our networks to begin the search by reaching out to roughly 100 experts we were aware of. We then asked these experts to pass the invitation along to other colleagues and to suggest significant articles based on their own experience and expertise, creating a snowball effect and casting a wider net to collect our data. This resulted in 421 articles being recommended, which we shared with the 283 experts, requesting them to vote on the ones they felt were the most significant. This expert survey method resulted in a total of 51 articles that had 9 votes or more.

Evaluating the data

Amongst the 421 articles, the citation analysis found 55 to have been cited over a 100 times. These articles were typically published for nearly 14 years and had an average citation rate of 554. The citation analysis on the 51 articles identified through the expert survey and voting process reveals an average publication time of 8 years and 94 citations. This method identified 16 articles as significant.

Social network analysis (SNA) is a method that can identify leading expert authors or leading clusters of expert collaborators. Not only did we want to know what articles were the most significant, but who was behind them. To do this, we utilized SNA to look at the networks, co-authorships, regions and institutes leading the research. This process identified 23 significant articles, based on factors like centrality and eigenvectors. The largest cluster of authors came from the USA, with leading institutions being UC Berkeley, Washington State University and the International Food Policy Research Institute based in Washington, DC.

As editors of the volume, for interest’s sake, the three of us selected the most significant articles from our own research perspectives resulting in 94 articles from the pool of 421. Only 19 of these articles were selected by two or more of the three of us. The total of all four methods resulted in 156 articles of significance. However, only four of these articles were common across all four methods.

With these four articles as the basis, we then selected another 39 articles from the expert opinion, citation analysis and SNA, for a total of 43 articles. The results of our different assessment methods found that they produced very different, yet significant results. Each method provided a body of literature that would provide policymakers with different measures of evidence. We were able to confirm that evidence is based on robust science-based methodologies, rigorous and statistical data analysis and results that are supported by data. Given the importance of science-based evidence for policy-makers, our research highlights the importance of engaging multiple methodologies in the assessment of data in order to reduce the unintended bias inherent in the methodologies.