On statistics, and how to use them with care

Table of Contents

“We have only 60 harvests left
due to soil erosion.”

We’ve probably all heard a statistic something like this at some point, though the numbers might vary, from 30 to 100. Maybe you’ve even quoted it. Stats like these are so commonly invoked that their veracity is rarely questioned. The trouble is, try as they might, several researchers have tried to find the original source for this stat and come up short.¹ As far as we know, it doesn’t exist. If it were ever derived from a study, it was presumably based on an average figure for a specific location before being (misleadingly) universalised.

What happens when statistics like these are unthinkingly removed from their original context, inflated, generalised, and repeated to such an extent that they reach near-mythological status? What might be glossed over in their apparent simplicity, and what if things are actually a bit more complex than they suggest? What is at stake when powerful institutions like the United Nations Food and Agriculture Organisation (FAO) unquestioningly parrot, and even create them? What other widely used statistics have similarly murky origins or more complex realities?

The reality of soil degradation is much more complicated. It is a serious problem, largely driven by extractive industrial agriculture; yet the actual rate of degradation varies hugely globally.² And whilst we sympathise with the implication this stat urges us to consider—that something must be done!—it does not suggest what exactly that something must be. Consequently, this stat can be and is indeed used to justify wildly different food futures: from soil-healing agroecology to techno-solutions that aim to entirely decouple food production from land use. And when such stats are mobilised to lend seemingly neutral credibility, even inevitability, to ideological food futures for which there are always alternatives, their usefulness and accuracy can easily become obscured. What do we stand to gain if we slow down, question them, recontextualise them, and start to unpack how they were even arrived at in the first place?

Statistics is a powerful tool for helping make sense of patterns larger than what we can observe directly. But, we would propose, in order to be meaningful, they must remain connected to their context and their methodology. In many ways this argument is not controversial; many statisticians would probably agree. We don’t intend to engage with the minutiae of the branch of mathematics called ‘statistics’ here. Instead, we use ‘statistics’ in its broader, more popular sense: any instance in which quantitative data is mobilised to support an argument. In this essay we outline some of the key ways we’ve come across statistics being (we think) misused, including in the reductionism of carbon tunnel vision; how average figures removed from their context oversimplify important variation; the over-extrapolation from single studies; the obscuring effects of blackboxing; and the impossibility of making like-for-like comparisons between what is and what could be.

We illustrate this essay with our vision for the future of the food system: one in which agroecology, food innovation and certain forms of technology co-exist to build a more flavourful, equitable, and ecological future. Whilst we’ve chosen to illustrate these statistical misuses in the context of the future of food, many of these points apply more broadly. 

This essay is not about pointing fingers, but about discussing some patterns we’ve noticed, which we are all—even us—prone to. We’re concerned about that. Our aim is not to whip up paranoia and unreflexive criticality; it is rather to help cultivate a more discerning, sensitive engagement with statistics and their power. To do that, we reflect on some reasons that might explain (but not justify) why each misuse occurs, before offering some simple ways to resist them. Many are basic scientific conventions that in principle should be followed, but are often, surprisingly, not. We finish with an antidote: a suggested research agenda to invite more pluralistic, nuanced discussions on food system transformation, in which careful use of statistics are part of the solution.

i. Escaping carbon tunnel vision

Sustainability means different things to different people, and can only be fully understood as operating within and across multiple paradigms. The parameters for ‘sustainability’ used in studies of food system transformation are typically reductionist, limiting the conclusions that can be drawn from them and often leading to oversimplified or misleading claims.

For example, as important as climate change is, it is insufficient to say that one food is ‘more sustainable’ than another simply because it has lower GHG emissions. To fully grasp the impact of our foods, we need to escape ‘carbon tunnel vision’, in which all ecological problems are reduced to carbon.³ Not every study needs or can have dozens of collectively holistic parameters, but we need to be very clear about what conclusions we can and cannot draw based on the parameters that are chosen.

Figure 1: Carbon Tunnel Vision. Matthew Miller for Future Earth. Adapted from the original Jan Konietzko diagram.

An influential and widely cited study by Poore and Nemecek (2018) used five impact indicators to characterise the global food system: GHG emissions, land use, freshwater withdrawals weighted by local water scarcity, eutrophying emissions, and acidifying emissions.⁴ These are all useful impact indicators that reveal more than carbon alone. Still, they are limited in their breadth and were chosen because they are typical of life cycle assessment (LCA) studies and were common across studies included in the meta-analysis, rather than what they reveal about sustainability as a whole. 

To fully grasp the impact of our foods, we
need to escape ‘carbon tunnel vision’, in which
all ecological problems are reduced to carbon.

A wider set of indicators like biodiversity, soil health, and agrobiodiversity could be used in studies to better characterise different agricultural production systems. Though natural scientific measures are quite well-established for some of these indicators—like the Shannon, Simpson and Chao indices for biodiversity—there currently is a lack of consensus on how best to measure these indicators in ways that are less cost- and labour-prohibitive for farmers.⁵ Much promising work is being conducted in this space, however. Developments in sensor technology using eDNA and acoustics may allow for biodiversity data to be collected much more quickly and cheaply compared to conventional methods.⁶ Meanwhile, efforts like the Soil Health Monitoring Framework being developed by the Soil Health Benchmarks consortium may help to standardise data-rich ways of characterising soil health.⁷ These types of indicators are increasingly being explored as potential reporting metrics for multinational food companies working with farmers to transition to more regenerative agriculture.

Yield is another metric commonly used to justify extractive, industrial forms of agriculture over others. Such claims are weakened if the total output of food produced per area is described using metrics that reflect the diversity and quality of food being produced e.g. total macro- and micronutrients, protein digestibility-corrected amino acid score (PDCAAS), and/or even the number of humans and non-human species nourished. Other land-based outputs—like fibre and other materials—should also be included, where relevant. Farmer Mark Shepard demonstrated how one acre of diverse silvopasture, producing fruits, nuts, animal products and mushrooms, could yield almost double the calories and vastly more varied nutrition than an acre of industrial corn—even without factoring in the additional potential that upcycling by-products or technology might unlock.⁸

Other methods like True Cost Accounting could also used to account for all of the positive and negative externalities associated with food production, factoring them into an adjusted ‘true’ price of food.⁹ If this approach were more widely used, it would quickly become apparent that ‘cheap food is not cheap’.¹⁰

The social sciences and humanities should be employed to try to account for the more human, qualitative sides of agriculture, which are too often ignored because they are difficult to quantify and aggregate.¹¹ How can a food system be determined to be, for example, regenerative (or not), if we don’t try to find ways to better ways to account for indicators like equality, justice, food sovereignty, food culture-appropriateness, land tenure, well-being, health, community resilience, dignity, spirituality and self-actualisation? One of the great challenges here is finding ways not only to account for them methodologically but to show their equal importance to more natural-scientific and/or quantitative indicators.

Not all of these more holistic indicators are easy to measure or have well-agreed-upon conventions for doing so. These indicators are often ignored precisely because they are difficult to quantify. As farmer and social scientist Chris Smaje puts it: ‘it’s not always easy or even desirable to quantify every cost or every benefit… these approaches involve counting the uncountable and comparing the incomparable’.¹² But this difficulty is not a reason they should be excluded, which is hard to see how it would help; on the contrary, we think it invites us to develop better methods that facilitate their inclusion in the discussion. 

Failing to account for such more holistic indicators can lead to perverse conclusions. For example, intensively produced animal products may be more ‘efficient’ in terms of GHG emissions and land use, and cheaper for consumers.¹³ Yet to conclude that such products are ‘more sustainable’, without considering a more holistic set of natural- and social-scientific metrics, is grossly misleading.

We’re not arguing that all studies must include an exhaustive range of natural- and social-scientific metrics to be valid, or even that more parameters are always better than fewer. Rather, studies should clearly define which parameters are being used and which are not, and acknowledge how the parameters they select limit the conclusions they can draw.

Our suspicion is that the benefits of an agroecological food system would be much more obvious against a more holistic set of parameters than is typically used in research today, since using such a narrow set of parameters under-describes both the harms caused by more extractive forms of agriculture and the benefits of more agroecological forms of production. Instead of claiming this on ideological but as-yet scientifically unsubstantiated grounds, which might do more harm than good, we instead, in the final part of the essay, present a possible research agenda to help test it empirically.

ii. Handling averages

When discussing the types of foods we could produce and how we could produce them in the future, it’s common practice to use average numbers derived from global or regional data. We’re sympathetic as to why: researchers usually need to rely on available data, using single figures for concision and because they are usually easier to work with in models than ranges. And research is hard enough as it is.

When introducing our work on plant cheese, we describe how ‘industrial cheese has a high ecological footprint (8.4kg CO2eq/kg), producing more greenhouse gas (GHG) emissions on average than chicken (4.3kg CO2eq/kg) and pork (6.5kg CO2eq/kg)’, based Poore and Nemecek (2018).¹⁴ We cautiously present that data to contextualise the impact that industrial cheese has, on average, compared to other typically high-impact animal products. The same study also suggests that many ingredients that can be used to make plant cheese, like beans (0.65kg CO2eq/kg), peas (0.36kg CO2eq/kg), and nuts (-0.8kg CO2eq/kg), or that are otherwise analogous to it, like tofu (1.6kg CO2eq/kg), have a much lower impact on average than dairy cheese.¹⁵

It might be tempting to conclude based on these figures—and many people have and do—that plant cheese is categorically ‘more sustainable’ than dairy cheese, and perhaps that all animal products should be replaced with lower-impact plant ingredients. We strongly push back against such overly simplistic conclusions based on average data for reductionist metrics. Indeed it is not the conclusion that Poore and Nemecek (2018) reach. We think both plant and animal products have an important, pluralistic role to play in a future food system—a future not of ‘either/or’ but of ‘both–and’. 

Averages can be handy, but they can also
conceal the complexity of the real world.

Such averages can be handy, but they also conceal the complexity of the real world. There is considerable variation in the GHG emissions impact of foods caused by different production methods, with large ranges around the mean figures presented above and considerable overlap in the ranges of impacts between many ingredients.¹⁶ For example, the most emissions-intensive forms of cheese generate almost six times more emissions than the least emissions-intensive.¹⁷ Perhaps surprisingly, the least emissions-intensive forms of chicken and pork produce fewer emissions than the most emissions-intensive forms of tofu (Figure 2, shaded in pink and green respectively), and the least emissions-intensive forms of chicken produce the same emissions as the most emissions-intensive beans and pulses (Figure 2, shaded in blue).¹⁸

Figure 2: Some surprising overlaps between average GHG emissions ranges for common proteins. Adapted from Poore & Nemecek (2018).

Without additional context on how the foods were produced, it's difficult to judge how to read Figure 2. Might the area shaded in blue be comparing industrially-produced chicken (that seems ultra-efficient if only measured against reductionist metrics) with agroecologically-produced beans and pulses, or vice versa? This is a good example of where having more qualitative data on the production context would enrich the findings. Either way, these examples are extremes—overlapping tails rather than typical examples of each food.¹⁹ But, while it would be remiss to draw too many conclusions from such outliers, they do illustrate how using averages obscures crucial and often wide variations in production methods. 

Our problem with the Poore and Nemecek (2018) study is less with its methodology (even if the parameters selected are quite reductionist) or the author's conclusions, but rather how data from the study has been taken out of context by others. The supplementary materials go to great lengths to detail the statistical significance of the data, further illustrating the complexity at hand. Yet the widespread use of average figures from this study without their original statistical nuance does a disservice not only to the careful work of Poore and Nemecek but to the complexity of the world. Clarity is nice, but not at the expense of truth.

Clarity is nice,
but not at the expense of truth.

Many basic scientific conventions like providing ranges, error bars, standard deviation and statistical significance help provide this context, though are surprisingly often not given. We suggest it would help us all engage with statistics better to include this full context, both when conducting our own research and when citing the work of others.

iii. Avoiding overextrapolation

Since there are relatively few existing studies that adequately characterise different food futures, it can be tempting to over-extrapolate from those few that do exist to try to fill in these knowledge gaps. This might stem from a desire to make futures more tractable for practical applications, technology or policy, or an impulse to make things manageable or governable as soon as possible. We’d caution against this impulse, using a well-known example of research on regenerative agriculture to illustrate why.

One of the better examples of peer-reviewed research on regenerative farming is a study on White Oak Pastures (WOP) in Georgia, USA by Rowntree et al. (2020).²⁰ WOP’s approach to animal agriculture is a well-known example of regenerative grazing, where cattle, poultry, pigs and other animals are grazed as ‘stacked enterprises’ in a rotational multi-species pasture system within the same landscape. Unfortunately, this study has been misinterpreted both by those who wish to overstate its claims and by those who wish to invalidate its results, with the former’s motivation only serving to strengthen the latter’s critiques.

The study demonstrates the complexities of allocating soil carbon sequestration to different food products produced within the same landscape, and the importance of reaching a consensus on the best practices for doing so (discussed further in the next section). The overall emissions intensity of meat produced in this system (not just beef, but an aggregate of poultry, pork and beef) is 20.8kg CO2eq/kg, but factoring in soil carbon sequestration reduces that figure by 80% to 4.1kg CO2eq/kg. This aggregate number might be frustrating at first: since carbon sequestration was not allocated to each product individually (a difficult thing to do accurately), we cannot make a fair comparison of poultry, pork or beef with the figures for those same products shown in Poore and Nemecek’s study. Yet in some ways, the act of allocating relative proportions of carbon sequestration to specific forms of flesh is less important than understanding the agroecosystem and the relations within it as a whole—something that is lost in reducing impact to single numbers allocated to single species.

It would be wrong to presume that
all regenerative animal production systems,
regardless of size, complexity, and location,
would have the same outcomes.

Rowntree et al. mention in passing that if all the soil carbon sequestration were allocated to cattle, then beef would be carbon negative, with emissions of -4.4kg CO2eq/kg. This is misleading since cattle are not the only animals in the system and hence can’t credibly be claimed as the sole cause of increased soil carbon sequestration. Relatedly, an earlier LCA on WOP also made such a claim, which gained a lot of media traction, and was used by WOP to advertise their beef as carbon-negative, though they no longer do so.²¹ Beef produced at WOP is not carbon-neutral or carbon-negative.²² And that’s okay. Livestock can still be a valuable part of the food system, even without a negative emissions footprint. However, determining what that healthy carrying capacity might be is far from straightforward. Perhaps in some contexts, this is possible, but it should not be extrapolated from this study that expanding cattle production is a climate mitigation solution.

Overall, whilst there is no ‘typical farm’, WOP might be considered fairly atypical. It’s a relatively large farm (1214ha) that grazes ten types of animals in rotation, in the humid subtropical climate of Georgia, USA—so we should be cautious of drawing too generalised conclusions and applying them to farms that are smaller, with less livestock diversity, and/or that are located in other bioregions e.g. Northern and Western Europe.²³ It would be overly simplistic, and indeed harmful to the regenerative agriculture movement, to presume based on this one study that all regenerative animal production systems, regardless of size, complexity, and location, would have the same outcomes.

The temptation to over-extrapolate can only be resolved with more high-quality studies, that use a range of holistic parameters from a variety of contexts to characterise food production systems. Until then, it’s important that we don’t make inflated claims about particular food futures based on a handful of studies, to always contextualise where the data that do exist come from, and what their constraints might be.

iv. Opening up the black-box

The quotation of sourceless received wisdom at the start of this article, that ‘we only have 60 harvests left due to soil erosion’, is an example of blackboxing.

A term originally coined by science studies scholars Michel Callon and Bruno Latour, blackboxes ‘contain that which no longer needs to be reconsidered, those things whose contents have become a matter of indifference.’²⁴ It is through this same phenomenon that statistics are removed from their original context, lose their complexity, qualification, and uncertainty, and, when repeated often enough and left unquestioned, come to be treated as received wisdom, even reaching near-mythological status. Blackboxing might result from an understandable desire to reduce complexity to make stats more ‘usable’ or ’practical’, or to free up ‘cognitive load’ for other problems. It is important to note that critical engagement with complexity and nuance is often present in the discussion where the statistic originates, but that nuance is often lost when the statistic is divorced from its context and circulated as a fact. Much is at stake with how we negotiate blackboxing, not least how we should design the food system to ensure that everyone can be fed well.

Claims that ‘lab-grown meat can produce 96% fewer GHG emissions and use 99% less land than conventional meat’ have been widely repeated.²⁵ In addition to falling prey to the same pitfalls outlined in sections 1 and 2, statistics like these, if taken at face value, can be even more problematic when used to justify food futures that are different to what many people might prefer.²⁶ Such removal of context and qualification lends certainty and scientific credibility to particular ideologies, and gives certain futures the illusion of necessity or even inevitability—in this case, the replacement of animal agriculture with techno-solutions. Of course, if the ‘96% fewer GHG emissions and 99% less land’ stat were true and all that mattered, it might make sense to replace all animal agriculture with lab-grown meat. All that changes when the wider context of these statistics is revealed. Crucially, these statistics are based on purely theoretical data and production systems that don’t yet exist, and involve many optimistic assumptions about as-yet-unresolved barriers facing the technology. Vital factors like energy consumption are glossed over, as are many more inadequacies that have been skillfully unpacked elsewhere.²⁷ Critical food-cultural factors like food acceptance are wholly ignored. Far from actually ‘containing that which no longer needs to be reconsidered, those things whose contents have become a matter of indifference’, statistics like these are highly contestable, which is not apparent when blackboxed. 

Blackboxes ‘contain that which no longer
needs to be reconsidered, those things whose
contents have become a matter of
indifference.’

- Michel Callon & Bruno Latour

This blackboxed statistic is not an isolated example. The ‘need to double food production by 2050’ is another problematically blackboxed imperative that is used to justify everything from the ‘sustainable intensification’ of food production to the necessity that everyone eat insects to the inevitability of lab-grown meat, and any other techno-solution you might think of. This statistic was originally derived from a modelling exercise as one of several possible future scenarios that aimed to ‘describe the future as it is likely to be, not as it ought to be’.²⁸ As Tomlinson, a scholar who has unearthed this history, states, the original report ‘does not present an agenda for what we need to do to “feed the world” and it is clearly not intended to do so’.²⁹ Yet somehow this statistic has warped from a descriptive datapoint on possible futures to a ‘normative goal for policy’, as articulated by institutions like the UK Government, the FAO, and the UN—despite the fact that we already produce more than enough food to feed everyone on the planet.³⁰ Globally speaking the problem is not gross food production but food production that is appropriate to different biocultural circumstances, and effective, equitable food distribution. Simply increasing food production doesn’t solve, and in fact exacerbates, many of the entangled problems in the food system like climate change, land use change, access to nutrition, inequality and more.³¹ More is not always more. Tomlinson goes on to suggest that if the question ‘how can we double food production by 2050?’ is the wrong one to ask, then a better question might be ‘how do we feed people and communities with healthy food whilst ensuring food sovereignty, securing livelihoods and sustaining the biosphere?’ Reframing the question permits us to imagine a future food system that is much better equipped to tackle these latter issues—a future the former question’s framing can easily preclude.³²

For these reasons, we see it as vital that we all cultivate greater criticality around the blackboxing of statistics and how they are used—sometimes knowingly, but often blithely—to justify certain ideologies or futures and to dismiss others. As proponents of a transition to a flavourful, equitable, and ecological food system, we must also take care not to fall into the same trap ourselves, and always offer context, criticality and nuance in any statistic or data point that we share. Rather than blindly quoting stats, even if they come from a seemingly reputable source, we should strive to always clearly define their parameters and assumptions, check their cited references, their degree of uncertainty, and discuss possible alternative options or projections proposed in the original study.

v. Comparing industrial apples and regenerative pears

When seeking visions of desirable futures, it’s natural to look to hopeful examples that exist today, however small, early-stage, or marginalised in the current food system—like pioneers of agroecology and upcycling. They can act as ‘signals of the future’ to help imagine a better world.³³ Yet data yielded from these contemporary examples, perhaps about their impacts (even if they escape carbon tunnel vision) or costs, are of limited use when trying to characterise a future food system, because the context in which they currently exist is so different to the context they strive for.

In the current food system, conventional, industrial agriculture is heavily subsidised, backed by a favourable policy environment and powerful lobbies. The vast majority of supporting infrastructure is oriented around it, including fertiliser, pesticide and seed production, machinery, advisory services, farmer education and a host of other institutions and infrastructure. In contrast, agroecological production exists in a system in which the economic, political and infrastructural context is stacked against it. The infrastructure supporting agroecology tends to be fragmented, underfunded, localised, and reliant on support from passionate but relatively unpowerful NGOs, pressure groups, and grassroots networks. Neither positive nor negative externalities are priced into the current food system based on extractive industrial agriculture, which incentivises poor land stewardship, degradation of resources, and low value use or outright waste of food by-products, making industrially-produced food artificially cheap and agroecologically-produced food more expensive.

This raises the question: ‘What would agroecology look like if it was as financially, politically, and infrastructurally well supported as today's extractive agriculture?’

Such a food system could take many forms, but would likely be radically different from the current food system. All food would be produced agroecologically, including food-culture appropriate animal and plant ingredients produced symbiotically in the same landscape. Regenerative land stewardship would be incentivised through policies and financial mechanisms that, amongst other things, would reward the provision of public goods generated by agroecology and disincentivise negative externalities. Access to land and the ownership models of farms, supporting business and infrastructure might look very different, particularly if justice and equity are guiding principles. The types of seeds, tools, machinery, other inputs that are produced and financial support would all be optimised for agroecological production. Heterogeneity would abound. Perhaps the untapped potential of food by-products would be unleashed through upcycling, enabled by technological innovation and new infrastructure like local biorefineries producing enzymes and co-located production facilities based on the principles of industrial symbiosis. We intend to explore what this possible future could look like in more detail in future work.

What would agroecology
look like if it was as financially,
politically, and infrastructurally well-
supported as today's extractive industrial
agriculture?

In such a system, industrial extractive agriculture might be totally unviable. No longer subsidised, without supportive infrastructure totally oriented around it, and heavily disincentivised by a policy environment in which its numerous negative impacts must be paid for, many of the conditions essential for extractive agriculture to be viable simply don’t exist.

This comparison nicely illustrates how the 'effectiveness' and 'efficiency' of a certain approach to agriculture depend so much on the systems that support it, and it's hard to imagine extractive agriculture being viable in anything other than a system designed to prop it up. Therefore any attempt to compare the possibilities of a future agroecological food system with today's dominant form of extractive agriculture is inherently skewed, because the only data points available to describe agroecology are from marginal examples based within a hegemony hostile to it. In other words, using data on ‘what is’ to describe what ‘could be’ is inherently limiting, and ‘what is’ must always be critically examined within the current context and both enables and constrains it. The ‘playing field’ between more extractive forms of agriculture and more agroecological forms of agriculture today is wholly uneven, and comparisons of food systems should be made between like for like. That is, we should compare industrial apples with agroecological apples, not industrial apples with agroecological pears (no offence to pears).

Of course, it’s not possible to collect data on a future food system until it exists. Since the current food system wouldn’t exist in that circumstance, it would be metaphysically impossible to ever directly compare the two from the same position. Research methods and discourse would also have changed radically between then and now. Therefore, at the very least, we must make do with what data we can glean from the current system, ensure that the peer-reviewed studies that describe and compare production systems do so cautiously and with nuance, and always take care to acknowledge the difficulties of comparing present and future holistic contexts.

The use of data from the present day when talking about what the future might look like is logical, to an extent: it is all we have available to us, and we can’t measure things that don’t exist yet. But it’s not the only way we can try to describe what the future might look like in an evidence-based way. ‘Futures thinking’ and ‘speculative design’ are two related disciplines that can provide us with the tools to reveal much about possible futures that quantitative data alone cannot: for example through forecasting; constructing different future scenarios; exploring more inclusive, preferred futures; and telling ‘creative yet rational, evidence-based stories’ about those futures.³⁴ We find that scientists (especially natural scientists) are generally suspicious of more speculative or theoretical work since it goes against much of their empiricist, positivistic training: it is easily dismissed as ‘unscientific’ or at least as the domain of others. Yet in this case, if done discerningly, we would argue that it is actually the more appropriate mode for creating knowledge about agroecological food systems in conditions of comparable infrastructural support, however provisional or conditional that knowledge be. After all: ‘the most wondrous thing about the future, is that there are no facts about the future’.³⁵ We intend to explore what this approach could look like in more detail in future essays. 

vi. Best practices, and an agenda for future research

Beyond more meticulously applying well-established statistical best practices, we think there are a few things that we as researchers could do to better describe the current food system, and describe what it could look like in the future, in less ideological and more rigorous ways.

To frame these suggestions, it makes sense to briefly note some limitations of the current literature. Few peer-reviewed academic studies characterise agricultural production in genuinely holistic or transdisciplinary ways. Those studies that do exist only characterise very specific forms of food production in very specific contexts (and often still in a quite reductionist way) within the wider context of the contemporary food system. Meta-analyses are by their nature limited to the existing literature, yet the full range of possible food production systems is vast, highly context-specific and place-based. Since studies like Poore and Nemecek’s are based on only a narrow range of possible production systems that exist today, they are of limited use in characterising the full breadth of possible future food systems. Their widespread use contributes to an inherently limiting way of thinking, and restricting ourselves to this approach may partly explain the crisis of imagination we face when imagining possible futures.³⁶

To address some of these limitations, In Table 1 we suggest a non-exhaustive agenda for future research to better characterise the types of food we do and could produce, and how we do and could produce them. Little of this is new, but we haven’t seen it all laid out satisfactorily in one place. We thought doing so could be a contribution to the conversation on food futures. A future metastudy on food systems, similar to Poore & Nemecek’s (2018) but based on this research agenda, whose results people used and discussed with care, could be transformative for how we talk about the future of food.

1. Use a more holistic selection of parameters to characterise food systems	Researchers should use a much more holistic range of parameters to better characterise the impacts of different foods and food production systems. These parameters should be as transdisciplinary as possible: ideally a mixture of natural- and social-scientific, and quantitative and qualitative parameters. The research agenda should prioritise developing consensus or at least convergence for the ‘best’ way to characterise these parameters, including the selection of appropriate units or metrics, where this has yet to be reached.Importantly, though studies should try to use more holistic parameters, more isn’t always better. Studies that use fewer parameters can still be valuable, as long as their results and limitations are appropriately qualified and contextualised.
2. Use statistical best practices to open up the ‘black box’	When presenting statistics, researchers should take care to provide ranges, error bars, standard deviation and statistical significance. These are basic scientific conventions which should be followed but surprisingly often are not, especially when citing the work of others.Additionally, researchers should cultivate greater criticality around the use of statistics in general. We should ensure that stats retain their context and criticality by defining the parameters and assumptions used, checking their cited references, and discussing possible alternative options or projections proposed in the original study. Taking these steps can go a long way in pushing back against the use of statistics to justify ideologies that can make particular futures seem inevitable.
3. Reflect the diversity of practices and contexts	Researchers should systematically characterise a much wider range of possible production systems for different foods in different contexts, to account for their diversity. Rather than making broad generalisations about diverse categories like ‘regenerative agriculture’, the characteristics of food productions under study should be thoroughly described so that like-for-like comparisons can be made, and assumptions can be more clearly laid out.Using the example of dairy, different production practices that might equally claim to be ‘agroecological’ or ‘regenerative’ should be studied in the literature e.g. rotational grazing, silvopasture, cow-and-calf management, pasture cropping, and different combinations of each. Each should be described in detail, noting important factors like e.g. climatic factors, enterprise stacking (if any), supplementary feed (if any), stocking density, frequency and duration of grazing rotation, tree species within silvopasture, plant species richness of pasture, breed of cattle and nutritional composition of milk produced, among others.
4. Develop methods that better reflect the realities of complex systems	Researchers should develop methods that don’t try to atomise or simplify agricultural systems in a reductionist manner, but rather try to account for their inherent complexity. For example, there are challenges in applying Life Cycle Assessment and similar quantitative methods to more complex food production systems, like agroecosystems where many types of food are produced in the same landscape, or production systems designed using industrial symbiosis principles to produce food from multiple upcycled food by-products. Researchers should develop a consensus around best practices for the allocation of impacts and benefits that reflects the realities of producing food in complex systems like these. And/or, great care should be taken about communicating the overall system, rather than over-allocating or under-allocating to a particular product or another. Guidelines should be established on when it would be advisable to take each approach.
5. Embrace the discerning use of speculative methods	Since it's impossible to truly quantify the future of food (‘what could be’) using data from the present (‘what is’), researchers should be open to using additional speculative methods that aim to complement quantitative methods, rather than wholly replace them. ‘Futures thinking’ and ‘speculative design’ are two related disciplines that can reveal much about possible futures that quantitative data alone cannot. Many of the same caveats to using statistics from the present apply here too: results should be developed rigorously and presented cautiously, remain in their context with their assumptions and parameters laid out, and be clearly labelled as speculative.

Table 1: A research agenda for better characterising agricultural production systems.

The research agenda proposed above would be most fruitful if performed in partnership with farmers and other stewards of the land, as funded partners on research projects, to ‘open up the ivory tower of research’.³⁷ However, if those farmers and stewards are only included as partners in science-as-usual, that too risks, as social scientist Jack Kloppenburg Jr. writes, ‘obscuring important connections between organisms and phenomena, and actively inhibit[ing] achievement of holistic understanding of ecological systems’.³⁸ Most of the statistical and scientific practices discussed in this essay, whether sloppy or careful, are all ultimately based on a deeper shared epistemology, in that they aim to find truth by abstracting away from specific single contexts. Whilst this approach is incredibly powerful and valuable, the belief that (natural) scientific knowledge is the only form of knowledge that matters, otherwise known as scientism, is harmful and limiting.³⁹ Another approach to knowledge that goes in the other direction—toward and deeply into single specific contexts—is equally valid and necessary. We need to find not just better ways to abstract, but also ways to engage different yet complementary modes of making sense of the world.

We need to find not just better ways to
abstract, but also ways to engage different yet
complementary modes of making sense of the
world.

We posit that the benefits of more agroecological forms of agricultural production are also evident in the rich knowledge that farmers have of their land gleaned through observation, lived experience, and other sensory and tacit means. Yet this knowledge generally isn’t as highly regarded as science is, particularly the natural sciences, since it is often more qualitative, idiosyncratic, or ‘fuzzy’, and is therefore too often ignored.⁴⁰ Whilst not every study needs to include this type of parameter to be valuable, when they are not included, it must be acknowledged what is being left out. To achieve a truly holistic understanding of food and agriculture, these oft-marginalised knowledges must be acknowledged as valuable alongside the more academic disciplines of the natural sciences, social sciences, and humanities. Only together can they create the kind of knowledge we need to build the food systems we want.

We aim to explore these other forms of knowledge, and how they might be woven into and transform science to facilitate richer food system transformation and discussion, in a future essay.

Contributions & acknowledgements

Eliot and Josh conceived this essay together. Eliot wrote the first draft, Josh provided editorial input, and they developed it further together after valuable feedback from Caroline, Mathieu, and Tiff Mak.

Header image credit: Harold Fisk (1944). ‘Meander Maps of the Lower Mississippi’.

Carbon tunnel vision image credit: Matthew Miller for Future Earth. Adapted from the original Jan Konietzko diagram. 

Eliot drew the graph.

Related posts

Endnotes

[1] James Wong (2019), ‘The idea that there are only 100 harvests left is just a fantasy’, New Scientist.

[2] Hannah Ritchie (2021), ‘Do we only have 60 harvests left?’, Our World in Data.

[3] Jan Konietzko (2022), ‘Moving beyond carbon tunnel vision with a sustainability data strategy’. Cognizant.

[4] Joseph Poore and Thomas Nemecek (2018), ‘Reducing food’s environmental impacts through producers and consumers’, Science. All figures are given as CO2eq/kg using 100-year time horizon global warming potential (GWP100), a method of converting all GHG emissions to equivalent units of CO2. This is a widely cited study, with 1895 citations at the time of writing.

[5] Even seemingly well-established metrics like the Shannon, Simpson and Chao indices for biodiversity are the subject of much debate. See: Michael Roswell, Jonathan Dushoff and Rachael Winfree (2021), ‘A conceptual guide to measuring species diversity’, Nordic Society Oikos.

[6] Mathis Rillig, Karine Bonneval and Johannes Lehmann (2019), ‘Sounds of Soil: A New World of Interactions under Our Feet?’, Soil Systems; and Kathrin Pascher, Vid Svara and Michael Jungmeier (2022), ‘Environmental DNA-Based Methods in Biodiversity Monitoring of Protected Areas: Application Range, Limitations, and Needs’, Diversity.

[7] Soil Health Benchmarks, EU Horizon.

[8] Mark Shepard (2013), Restoration agriculture: real-world permaculture for farmers, Acres, USA.

[9] Sheryl Hendriks et al. (2023), ‘The True Cost of Food: A Preliminary Assessment’, in: Joachim von Braun, Koasar Afsana, Louise Fresco, Mohamed Hag Ali Hassan (eds), ‘Science and Innovations for Food Systems Transformation’, Springer, Cham, Switzerland.

[10] True Cost Accounting, Sustainable Food Trust.

[11] In one notable corporate framework that attempted to quantify regeneration, by OP2B, a single metric for social indicators of regeneration was publicly left as ‘tbc’ for several years in their framework, whilst a wider array of natural scientific indicators were much better fleshed out.

[12] Chris Smaje (2023), ‘Saying No to a Farm Free Future’, Chelsea Green.  

[13] Tara Garnett (2010), ‘Intensive vs extensive livestock systems and greenhouse gas emissions’, Table debates, Food & Climate Research Network briefing paper.

[14] Joseph Poore and Thomas Nemecek (2018), ‘Reducing food’s environmental impacts through producers and consumers’, Science.

[15] Ibid. Note that this is a slightly unfair comparison as it compares two processed foods (dairy cheese and tofu) with three broad categories of raw ingredients (peas, beans, nuts), so the latter three are not including the impacts of processing that would occur when turning peas, beans and nuts into cheese and we would expect such cheese to have a higher impact.

[16] Hannah Ritchie (2020), ‘Less meat is nearly always better than sustainable meat, to reduce your carbon footprint’, Our World in Data. Note: we don’t necessarily agree with the conclusion made in the title of this reference.

[17] Joseph Poore and Thomas Nemecek (2017), ‘Reducing food’s environmental impacts through producers and consumers - Supplementary Materials S2’, Science, Cheese - 5th percentile (10.2kg CO2eq/kg) vs 95th percentile (58.8kg CO2eq/kg).

[18] Ibid. Pork - 5th percentile (6.9kg CO2eq/kg) vs tofu - 95th percentile (7.3kg CO2eq/kg); chicken - 5th percentile (4.0kg CO2eq/kg) vs beans and pulses - 95th percentile (4.0kg CO2eq/kg).

[19] Though we suspect that since the full diversity of production systems that could exist in an agroecological regenerative food system are not included, and that many of these systems might have a much lower impact than the average, in future those tails might be rendered as less of statistical outliers than they might appear today.

[20] Jason Rowntree et al (2020), ‘Ecosystem Impacts and Productive Capacity of a Multi-Species Pastured Livestock System’, Frontiers in Sustainable Food Systems. Though better in terms of the type of farm being characterised, the study still falls prey to carbon tunnel vision. This same farm was also the subject of a different controversial non-peer-reviewed LCA study: Carbon footprint evaluation of regenerative grazing at White Oak Pastures. Quantis.

[21]  Study: White Oak Pastures beef reduces atmospheric carbon, White Oak Pastures blog.

[22] Chris Smaje discusses some of the factors that should be considered in ‘Saying No To A Farm-Free Future’ p.83-104, including Simon Farlie’s work on ‘default livestock’. The Rowntree et al. (2020) study discussed here shows how regenerative production can have a higher land footprint (in this case 2.5x higher) for the same amount of food produced, suggesting it may not be desirable or even possible to attempt to replicate the volume of animal products produced by extractive forms of agriculture today while remaining within the carrying capacity of healthy agroecosystems. For this reason, research and development that is sensitive to diverse land stewardship practices and food cultures, on topics like umamification and plant cheese, is critical to support a ‘both–and’ approach to ‘less but better’ animal products supplemented with ‘alt’ analogues.

[23] Based on the Köppen climate classification system.

[24] Michel Callon and Bruno Latour (1981), ‘Unscrewing the Big Leviathan: How Actors Macro- Structure Reality and How Sociologists Help Them To Do So’, in: Karin Knorr Cetina and A.V. Cicourel (eds), ‘Advances in Social Theory and Methodology: Toward an Integration of Micro- and Macro-Sociologies’, Routledge, London, UK.

[25] Hanna Tuomisto and M. Joost Teixeira de Mattos (2011), ‘Environmental Impacts of Cultured Meat Production’, Environmental Science and Technology.

[26] Here is a speculative example future in which techno-solutions are imposed on people against their will.

[27] Chris Smaje wrote ‘Saying No to a Farm Free Future’ in response to George Monbiots ‘Regenesis’, in which he debunked many of the latter’s blackboxed Ecomodernist claims. This essay, ‘George and the Food System Dragon’ by Jim Thomas, summarises the debate well.

[28] FAO (2006), ‘World Agriculture: Towards 2030/2050, Interim Report’, FAO, Rome.

[29] Isobel Tomlinson (2011), ‘Doubling food production to feed the 9 billion: A critical perspective on a key discourse of food security in the UK’, Journal of Rural Studies.

[30] Eric Holt-Giménez, Annie Shattuck, Miguel Altieri, Hans Herren and Steve Gliessman (2012), ‘We Already Grow Enough Food for 10 Billion People … and Still Can’t End Hunger’, Journal of Sustainable Agriculture.

[31] Many famines, for example, are not the result of insufficient production but rather of social and economic collapse that result from inappropriate policy. See Amartya Sen (1983), 'Poverty and Famines’, Clarendon Press, Oxford, UK.

[32] Isobel Tomlinson (2011), ‘Doubling food production to feed the 9 billion: A critical perspective on a key discourse of food security in the UK’, Journal of Rural Studies; Hans Herren (2010), Supporting a True Agricultural Revolution, The Embassy.

[33] Sabrina Howard (2021), Drivers and signals: how are they different?, Institute for the Future.

[34] IFTF (2023), Institute for the Future ‘Scenario Building’ course

[35] Marina Gorbis (2023), Institute for the Future ‘Scenario Building’ course

[36] Hanna Thomas Uose (2023), The most creative look to the future, UN Pulse.

[37] EARA (2023), Together for Regenerative Agrifood Ecosystems, European Alliance for Regenerative Agriculture.   

[38]  Jack Kloppenburg Jr. (1991), Social Theory and the De/Reconstruction of Agricultural Science: Local Knowledge for an Alternative Agriculture, Rural Sociology.

[39] Massimmo Pigliucci (2018), The problem with Scientism, Blog of the American Philosophical Association.

[40] Arielle Johnson (2020), ‘Marxist analysis, in my food technology? Fuzzy legibility, flavor connections, and the recent dialectical emergence of post-modernity in cuisine’, in: ‘Food and Power: Proceedings of the Oxford Symposium on Food and Cookery’, Prospect Books, London, UK.