A meta-analysis of field experiment data

Pollinators play a crucial role in the production of food crops. While many crops are open pollinated, meaning they don’t require pollinators to produce the crop, numerous crops benefit from them. Fruit and vegetable crops require pollinators (e.g., apples, beans, cantaloupe, cucumbers, grapes, strawberries) and field crops like alfalfa, almonds, clover, flax and mustard. A 2009 estimate placed a value of up to US$29 billion annually on the contribution of pollinators for American farm incomes. Given the important role and value of pollinators, ensuring their health is of great importance, which is why studies reporting harmful effects on pollinator populations are important.

Over the past decade, mainstream and social media stories have had many stories and posts about threats to bee populations. Environmental non-governmental organizations (ENGOs) use these stories to put political pressure on politicians and policy makers to enact new legislation to better protect bees. Much of this pressure was targeted at getting agricultural chemical restrictions and bans implemented. The ENGO pressure proved effective as between 2010 and 2015, numerous countries either banned or dramatically restricted the use of some chemicals that play significant roles in agriculture, especially neonicotinoids. Neonicotinoids provide plants with protection from insects that damage crops. Many of these bans and restrictions occurred in Europe, resulting in farmers losing access to these chemicals. Oilseed rape production in the UK dropped from 1.8 million acres in 2012, to 1 million in 2020. Neonicotinoid bans in France resulted in substantial sugar beet losses of up to 50% by 2020 and the bans on neonicotinoid use were rescinded.

The questions arising from policies that restrict or ban chemicals, is whether these should be based solely on laboratory experiments, with little or limited field study evidence? That’s because lab studies aren’t the best representation of field conditions bees encounter. In the field bees:

1. choose which plants to pollinate and how much/often they do so;
2. are exposed to bacteria, viruses, parasites and predators that negatively impact their health; and
3. have complete freedom to move about fields (some go feral).

These are just a few of the factors that laboratory experiments don’t contemplate when bees are exposed to neonics in typically controlled settings. Field experiments are expensive to undertake, and comprehensive protocols to do them have only recently begun to be discussed. But they need to be part of policy designs, especially when these could negatively impact food security.

The current state of bee populations

While bees are important for agriculture and honey production, little attention has been paid to their populations. It wasn’t until 2008 that the COLOSS monitoring group began annually surveying beekeepers (mostly in European countries) about bee overwintering losses. One study that examined bee losses in 2018-19, reported average losses ranged from 6-32%. American research reports a 2007-08 average overwintering honeybee colony loss of 14% in the Pacific Northwest, while in the New England region, overwintering losses of 15% to 30% are normal. Colony losses ranging from 30% to 90% were reported through late 2006 and early 2007, which was termed Colony Collapse Disorder (CCD), and its cause or causes remain unknown.

The US House Agriculture Committee recognized that CCD posed a potential threat to food security and held hearings to investigate CCD in honeybee colonies across the US and the United Nations Environment Program published a report highlighting the importance of bees to modern agriculture, highlighting the threats they face. At the time CCD was first being reported, experts debated whether it was caused by any of the myriad of pesticides and metabolites, bacteria, fungi, parasites, viruses or any combinations thereof that were present in symptomatic colonies. Research quantified and explored 61 variables that could potentially be the cause, but failed to identify a single factor with enough consistency, to suggest it was responsible for the losses. In the absence of sufficient rigorous data on the health of domesticated honeybees, and even fewer on wild pollinator populations, other experts questioned whether reported losses were a new phenomenon at all. The absence of historical data meant it wasn’t possible to adequately measure the bee losses experienced in 2006-07 when CCD. The nature and severity of the losses was ultimately built on the perceptions of experienced beekeepers.

Research into the potential effects of neonicotinoids on honeybees increased in 2006, following the initial wave of CCD reports. Much of research focused on the impacts of imidacloprid on honeybees, even though there are 7 neonicotinoid insecticides available and over 18,000 bee species. Suspicion that neonicotinoids were responsible for the perceived decline of pollinators, resulted in governments in Ontario, France, the United Kingdom and Germany enacting restrictions and complete bans on these chemicals.

Contrary to media perceptions, from 1961 to 2019 managed honeybee colonies increased by 83% worldwide. While managed colonies are not the most suitable metric with which to assess honeybee populations, the lack of data prevents the use of more adequate metrics.

Worldwide number of honeybee colonies

Source: based on FAOSTAT (2021).

There is a growing body of literature reporting on the effects of neonicotinoids on bees, including Chensheng et al., 2014, Faucon et al., 2005, Johnson et al., 2010, Shi et al., 2017 and Wu-Smart & Spivak, 2016. These effects need to be taken into account when regulatory agencies are approving pesticides for agricultural use, especially as many of the publications are solely based on laboratory experiments and fail to represent chemical exposures in field settings. Woodcock et al. (2017) show that while neonicotinoids may have negative effects on bees, these effects are not uniform across countries. Moreover, screening primary field studies for this meta-analysis revealed the wide-ranging methodologies researchers use to assess the impact of neonicotinoids on bees. The developmental stage of bees, plot topography and size, duration of experiment, nearby crops and plants, bee genetics, age of queens as well as the country of their origin, all differ among studies, making it difficult to combine their results.

Searching for field evidence of neonicotinoids impacts on bees

The criteria to justify including an experiment in the meta-analysis dataset, required that it:

1. be peer reviewed;
2. report results of an experiment in which there was a treatment and control group;
3. measure survival or mortality;
4. explicitly report sample size and some measure of statistical dispersion that could be used to calculate the magnitude of the effect being studied;
5. satisfied the rule nt + nc > 2, where, nt is the sample size of the treatment group and nc is the sample size of the control group; and
6. be written in English.

To incorporate as many articles as possible, no distinction was made between bee genera.

The figure depicts the screening process followed to reach the number of studies that fully complied with the inclusion criteria detailed above. The literature search yielded 625 initial results. After removing duplicates, irrelevant abstracts and studies not assessing bee survival or mortality, a total of 128 papers were fully considered for inclusion. Upon scrutinizing these papers, a further 119 papers did not fully comply with the inclusion criteria laid out above and were not incorporated into the meta-analysis.

Analysis flow diagram

Hedge’s d, magnitude of the effect, was computed for every study included in this meta-analysis. Effect sizes were calculated using the Practical Meta-Analysis Effect Size Calculator. Data to calculate these were obtained from sample sizes and any measures of dispersion found in journal article sections that included: methodology, material sections, tables, or graphs. If an experiment reported the impact of multiple agents under study, on an equal number of bee species, Hedge’s d was calculated for each instance and was considered a distinct data point. To avoid issues of non-independence, if an experiment spanned months or years or underwent many replications, only the final result reported was incorporated into this meta-analysis. In total, 15 data points were extracted from 9 field experiments.

Field evidence of neonicotinoids impacts on bees

The overall weighted-mean effect size (diamond shape) of field studies assessing the impact of neonicotinoids on bee mortality is -0.27, and the 95% confidence interval is -0.73 – 0.19. The weighted-mean overlaps the vertical ‘line of no effect’, indicating no difference in mortality between the groups exposed to neonicotinoids and those that were not (control groups). By using a random-effects model in this meta-analysis, an assumption that different studies estimate different, but related, interventions was made. This can also be intuitively understood, as studies did not employ the same methodology with which to assess the lethality of neonicotinoids on bees, which itself has been the subject of extensive debate among experts.

Meta-analysis forest plot

The harmful effects of neonicotinoid concentrations used in laboratory-based studies, which were instrumental in the EU’s neonicotinoid ban, aren’t observed when these same concentrations are used in field-level studies. The results don’t deviate from those of other comparable analyses. Blacquiere et al. (2012) undertook a review on the effects of neonicotinoids on bees and found that many of the lethal and sub-lethal effects neonicotinoids have on bees in laboratory studies are absent in experiments with field-realistic dosages. Cresswell (2011) undertook a meta-analysis of 14 laboratory and semi-field experiments measuring the effects of imidacloprid on honeybees and found that trace dietary presence of imidacloprid has no lethal effects but does reduce honeybee performance.

This is not a defense of neonicotinoids; it’s possible field experiment protocols aren’t designed in ways that capture the true effects of neonicotinoids. Publication bias, or the publication of favorable outcomes may also be affecting field studies. However, only a small number of field experiments with comparable protocols were found, partly because undertaking these experiments is logistically challenging and financially expensive.

Summary

Neonicotinoids were suspected of being, if not the single causal agent, definitely among the principal causing agents of CCD. Laboratory experiment results were instrumental in the decision of numerous, predominantly European governments to ban neonicotinoids from agricultural application. However, these studies don’t provide a thorough perspective of the situation, and nonetheless resulted in economic losses in the hundreds of millions of dollars for farmers. Knee-jerk policies like neonicotinoid bans, result in significant economic costs for farmers through reduced yields and consumers through higher food prices, as there are few, if any, chemical pest control alternatives. More field experiments with comparable protocols would better inform policies about the impacts of neonics on bees in general, and honeybees in particular.