A foundational report on the methodology we use at Animal Ask to measure the impact of lobbying campaigns.

EXECUTIVE SUMMARY

Legislative lobbying has led to numerous positive outcomes in many policy areas and social movements. For example, lobbying in the United States is responsible for lower taxes on solar power, increased taxes on tobacco, and the establishment of a program to detect asteroids (Lerner 2020). In the animal advocacy movement specifically, one recent victory is the 'End the Cage Age' initiative. This lobbying campaign led to a commitment by the European Commission to phase out the use of cages for hens, sows, calves, and numerous other farmed animals (European Commission 2021).

When evaluating campaign strategy, it is important to have a good way to project the success of competing legislative opportunities. Ideally, this would involve predicting the degree to which lobbying effort makes a legislative opportunity more likely to succeed, compared to the scenario where that additional effort does not take place. We refer to this as the counterfactual impact of lobbying. Understanding this would provide insight into how campaigning can be most effective.

Currently, we do not know the best way to measure the counterfactual impact of lobbying. One promising option is to use the academic literature on lobbying. The published studies on lobbying could be used to derive estimates of policy success and, ideally, the counterfactual impact of lobbying.

The purpose of this report is to find out whether this academic literature contains quantitative estimates that could be useful in gauging the counterfactual impact of lobbying. We answer this question by reviewing the literature on lobbying and examining studies that have published estimates of the rates of policy success.

Overall policy success is made up of two components: a baseline rate of policy success, and the counterfactual impact of lobbying. Therefore, to measure the counterfactual impact of lobbying, we need to be able to do two things: 1) measure overall policy success, and 2) break down this overall policy success into the baseline rate of success and the counterfactual impact of lobbying (see Figure 1 below).

We find that measuring overall policy success using the academic literature is unlikely to work. The lobbying literature contains many well-known weaknesses which cast doubt on researchers' ability to measure policy success. As well as this, our systematic review found a strange result, which suggests that the published estimates of policy success should not be taken at face value.

We find that there is insufficient evidence to break down overall policy success into the baseline rate of success and the counterfactual impact of lobbying. There is only a single study that identifies the counterfactual impact of lobbying. That study was limited to a single policy issue and a single context, and so its findings need to be replicated across other issues and contexts before we can draw any sound conclusions.

In summary, the academic literature does not contain sufficient information to gauge the counterfactual impact of lobbying. For these reasons, we recommend against using estimates from the published literature when modelling the effects of lobbying. Instead, we recommend that researchers choose a different option for incorporating information on the prospects of success. We conclude by highlighting two promising options that deserve further exploration: expert judgement and panels of superforecasters.

Although we conducted this research to guide our own research process at Animal Ask, this report may also be useful to other researchers. In particular, we expect this to prove useful to researchers who forecast legislative outcomes, as well as those interested more generally in the academic literature on lobbying.

Figure 1: The overall observed success rates of policies are made up of two parts: the baseline success rates of policies (grey puzzle piece) and the counterfactual impact of lobbying (blue puzzle piece). Assessing the counterfactual impact of lobbying would be very useful for evaluating campaign strategies. However, academic research has mostly focused on studying the overall success rates of policies, and that research has encountered serious difficulties. There is currently insufficient evidence to gauge the counterfactual impact of lobbying based on the academic literature.

INTRODUCTION

When conducting research to guide policy and social movement strategy, a common goal is to recommend which campaign opportunities have the greatest prospect of success¹. This often involves systematically comparing different campaign opportunities. Many campaigns involve lobbying government officials, and so research into this often involves trying to predict the outcome of lobbying government officials for particular policy goals.

Overall policy success is made up of two components. First, there is a baseline rate of policy success. Bills introduced to a legislature naturally have some baseline probability of passing without any additional effort. Second, there is the counterfactual impact of lobbying. This describes the degree to which lobbying effort increases the probability of policy success. For example, lobbying legislators to support a bill might make it more likely to pass by a certain percentage.

When evaluating campaign opportunities, it would be very helpful to understand the counterfactual impact of lobbying. If there are multiple competing campaign opportunities, and one opportunity appears particularly receptive to lobbying, then it would make sense to focus lobbying resources on that particular opportunity. This would help limited resources be used most effectively.

How might we go about assessing the counterfactual impact of lobbying? The principal option that we explore in this report is to use information published in the academic literature on lobbying.

To measure the counterfactual impact of lobbying from the academic literature, we need to be able to achieve two goals: 1) measure overall policy success, and 2) split this overall policy success into the baseline rate of success and the counterfactual impact of lobbying.

Currently, it is unclear whether the academic literature contains sufficient information to achieve these two goals. To determine this, we review the academic literature on lobbying, beginning with a broad overview of the lobbying literature and the serious difficulties that have been encountered in trying to measure overall policy success. We then elaborate on the need to understand the counterfactual impact of lobbying, before turning to our systematic review of published studies.

MEASURING POLICY SUCCESS: A SUMMARY OF THE LITERATURE

Broadly speaking, the academic research on lobbying has identified a few well-supported findings. Perhaps most importantly, this includes the finding that lobbying is often successful in general (de Figueiredo and Richter 2014; Lerner 2020), as well as for animal advocacy in particular (Animal Charity Evaluators 2022). However, measuring the effectiveness of lobbying is 'extraordinarily challenging' (de Figueiredo and Richter 2014).

There are numerous methods available to researchers who seek to measure the effectiveness of lobbying (Oeri, Rinscheid, and Kachi 2021). Bernhagen, Dür, and Marshall (2014) provide a useful way to classify these methods, based on categorisation as subjective or objective data. Subjective data may involve surveys that ask lobbyists to estimate the success of themselves and others, while objective data involves assessing the actual outcomes of lobbying efforts. It has been documented that when lobbyists estimate their own success, the resulting estimates are higher than would be calculated using objective data (Lyons, McKay, and Reifler 2020).

Separately, the scale of measurement can be binary or continuous. A binary scale generally considers lobbying efforts to be either 'successful' or 'not successful' (or a limited number of intermediate options), while a continuous scale considers the level of success along a continuum². The validity of particular methods remains under debate (Adriana Bunea and Ibenskas 2015).

Problems with studying influence and estimating success

In the academic literature on lobbying and interest groups, many studies have attempted to measure the influence or success of lobbying. However, these studies have very mixed results (Anzia 2019; Lerner 2020). It has been shown multiple times that many of these studies - sometimes, even a majority - have found results that indicate either no effect or an effect that is different to what is predicted by theory³ (Burstein 2019; Burstein and Linton 2002; Lowery 2013; Uba 2009; Burstein 2011). Substantial results occur only a small proportion of the time (Burstein 2011).

Nevertheless, the belief that lobbying is a worthwhile endeavour remains widespread among academics, the public, and lobbyists themselves. Lobbying is a multibillion dollar industry because it involves large opportunities and risks (Baumgartner et al. 2009). While nobody seriously believes that lobbying is ineffective (Leech 2010), academic research has found it difficult to prove its effectiveness - a scientific challenge that dates back to at least the 1960s.

There are many barriers faced by researchers on this topic. As two primary examples, this topic is inherently difficult to research (substantive challenges), and there are weaknesses in researchers' approach (methodological challenges). We summarise these challenges in Table 1 and we discuss them in further detail below.

Table 1: The main challenges in the academic literature on lobbying.

Studying influence: Substantive challenges

When trying to measure the success of lobbying efforts, researchers face a number of substantive challenges. Some of these challenges are so profound that they might prevent researchers from ever being able to measure lobbying success, even in principle. However, some authors, like Dür (2008b), are more optimistic. Here, we will list the substantive challenges that researchers face, in no particular order.

Success in lobbying is highly contingent and context-dependent (Leech 2010). Lobbying involves complex interactions in dynamic systems and depends on exogenous forces like timing, tactics, targets, salience, party power, political mood, lobbying resources, dominant issue frames, the other issues on the agenda, and the behaviour of other interest groups (Leech 2010; Lowery 2013; Sentience Institute 2020; de Figueiredo and Richter 2014). Many of these forces are inherently unobservable (de Figueiredo and Richter 2014).

In particular, there is a powerful influence of the status quo (Baumgartner et al. 2009; Mahoney 2008). The status quo is strong, and lobbying to change it is difficult (Lowery 2013). There are several reasons for this: policymaker attention is scarce; lobbyists in favour of the status quo can effectively raise general doubts and fear of costs associated with change; the difficulties in creating bipartisan coalitions; the interests that gatekeepers have in maintaining existing policy; and the idea that the status quo already reflects the relative abilities of competing lobbyists (Baumgartner et al. 2009). Lerner (2020) concludes that lobbying for the status quo is more likely to succeed than lobbying against the status quo. Lerner also proposes a way to take advantage of this phenomenon by lobbying to retain beneficial policies that already exist. Relatedly, one study found that lobbying against a proposal is more effective than lobbying for a proposal (McKay 2012a). However, this may be an artefact of studies failing to consider the counterfactual. If the status quo is likely to remain, then lobbying in favour of the status quo might make no difference. In this case, a small amount of lobbying might appear to have a very high probability of success, making lobbying in favour of the status quo appear more effective than it is. Separately, businesses are more likely than advocacy groups to lobby in favour of the status quo (Garlick 2016; Mahoney 2008). If businesses are better at lobbying than advocacy groups, this might confound the relationship between status quo and success, though we emphasise that we have seen no convincing evidence that businesses are better at lobbying than advocacy groups.

Defining influence and power is inherently very difficult. Theorists hold that there are several 'faces' of power (Lukes 2004). The first face of power is to do with who wins and loses on particular policy issues. The second face is to do with who can control the agenda on which those issues appear or do not appear. The third face is to do with manipulating the preferences of others, such as via ideology (Dür 2008a; Finger 2019; Dür 2008b). Through the second and third faces, many actors can exercise influence in a way that is totally invisible to researchers (Lowery 2013). For example, if an interest group wants to maintain the status quo, they may lobby to keep a potential reform off of the agenda. If this group succeeds, by definition there would be very little observable behaviour (Lerner 2020). Despite the importance of the second and third faces in understanding lobbying success, few studies have attempted to unravel these (Leech 2010), though some studies do exist (Pedersen and Binderkrantz 2014; Fellowes, Gray, and Lowery 2006; Garlick 2016; Binderkrantz and Rasmussen 2015; Yackee 2012).

Dür (2008a, [b] 2008) also identifies numerous pathways to influence. Studies often only focus on one or a handful of these pathways, limiting researchers' ability to detect the full scope of influence. These pathways to influence include access (directly expressing information or demands to decision makers), selection (helping aligned people to win power by influencing the selection of politicians, bureaucrats, etc.), voice (the use of outside lobbying methods, including media statements, rallies, and petitions), and structural coercion (the ability of businesses to influence politicians with minimal or no effort because politicians are incentivised to create a favourable environment towards business). A business can also have substantial structural power due to the popularity of its products with consumers. When a product is perceived as important by consumers, the line between a business's commercial activity and political activity becomes blurred. In this case, deliberate activity like lobbying may not be a useful way to think about political influence (Woll 2019).

Similarly, lobbyists have a number of techniques available by which to exercise their influence. These include authority and respect, friendship and benevolence, rational persuasion, selling and suggestion, fraud and deception, and coercion (Lowery 2013; Banfield 1961).

There are many reasons why it might be difficult or impossible to detect true influence. Lowery (2013) identifies many such reasons: some lobbying efforts are inherently more difficult to succeed in than others; groups have other goals than winning (such as organisational survival); professional lobbyists may be incentivised to misrepresent their clients' interests; and, of course, interest groups might actually be weak, with the weak evidence simply reflecting this. Another important reason is that lobbying happens over long periods of time, with earlier efforts affecting later efforts (Baumgartner et al. 2009; Sentience Institute 2020; Lowery 2013; Mahoney 2008). There may even be substantial time lags between cause and effect (Selling 2021), further complicating the process of assessing influence.

Interest groups are more likely to lobby when they think they are likely to succeed. This is a form of selection bias, and it means the decision to lobby is non-random (de Figueiredo and Richter 2014; Marie Hojnacki and Kimball 1998). This is particularly problematic for the inferences that we seek to make. Studies cannot analyse lobbying efforts that were never attempted, though some true experimental work has been conducted (Bergan 2009; Cluverius 2017; Sekar 2020). This means that these studies can only make valid inferences about lobbying efforts that were attempted. However, research in effective altruism and policy often focuses on hypothetical lobbying efforts, some of which have limited or no precedent in public policy. Even if robust estimates of the base-rate of success of lobbying were available, it would not be clear that these estimates would apply to the hypothetical lobbying efforts that we generally consider.

Lobbying is two-sided, and in the vast majority of cases, there are lobbyists on two or more opposing sides of an issue (Baumgartner et al. 2009). A failure for one group often means a success for an opposing group (Lowery 2013; Leech 2010).

Given these factors, it is very difficult to determine the counterfactual - in other words, what would have happened if a certain group did not lobby (Lowery 2013). There are studies that attempt to circumvent this problem, either by looking at all decisions across a given issue or by identifying prior positions (e.g. Yackee and Yackee 2006; Gamboa, Segovia, and Avendaño 2016). However, due to the complex nature of lobbying, with many actors interacting in complex ways, even these methods may struggle to determine the counterfactual. A true experimental study may overcome this limitation, but there have been few such studies (e.g. Bergan 2009; Cluverius 2017; Sekar 2020).

When an actor takes an initial position on some issue, that position is shaped by the anticipated reactions of other actors. This shaping of the initial position of one actor represents true influence on the part of those other actors, yet lobbying research cannot feasibly capture this dynamic (Lowery 2013). Lastly, influence is not a one-way street, and politicians can often manipulate interest groups (Dür 2008a; Lowery 2013).

Studying influence: Methodological challenges

In essence, research on lobbying has focused up to now on two broad questions. Firstly, there is the question of influence, looking at how interest group behaviour affects policy outcomes. Secondly, there is the question of success. This involves looking at the extent to which policy outcomes satisfy the preferences of groups (Bernhagen, Dür, and Marshall 2014).

Even if answering these two questions is possible in principle, there are many problems that have been identified with the methodology of lobbying research in practice. These methodological challenges cast doubt on whether such measures can be extracted from the existing literature given the current state of research.

The existing literature suffers from fragmented methodology, frameworks, and vocabulary (Beyers, Eising, and Maloney 2008; Burstein 2019). There is also a large number of exploratory, descriptive studies, making generalisations problematic (Bunea and Baumgartner 2014). There are numerous scales of measurement, and the choice of scale affects a study's results (Bernhagen, Dür, and Marshall 2014; McKay 2012b). Together, these factors make synthesising the literature and drawing conclusions very difficult.

Additionally, there is a strong bias towards focusing on wealthy democracies, such as the United States and the European Union, making it difficult to assess the value of lobbying in other places⁴ (Bunea and Baumgartner 2014; M. Hojnacki and Kimball 2012; Burstein 2019).

Meanwhile, there are several research biases that may lead to lobbying influence being underestimated. Within research on the United States, there is a strong focus on the federal government (Anzia 2019). Separately, there is a strong focus on controversial, highly salient issues with lots of interest groups present (Uba 2009; Leech 2010; Anzia 2019; Lowery 2013). In both of those contexts, there are theoretical reasons to expect that lobbying would be less effective. Added to this, there is a consistent focus on studying the interest groups' behaviour, rather than measuring influence from other perspectives (Lowery 2013; Bernhagen, Dür, and Marshall 2014). Finally, there is also a focus on the size of resources and the way they are deployed; this is at odds with the deployment of resources itself, which theory predicts to be important (Burstein and Linton 2002).

There are also research biases that may lead to lobbying influence being overestimated. There is a bias against publishing null findings (Burstein and Linton 2002; Leech 2010), as well as a common failure to include potentially important variables like public opinion (Burstein and Linton 2002).

Studying influence: Overall outlook for measuring policy success

Based on the considerable substantive and methodological challenges, some researchers argue that estimating rates of policy success is fundamentally the wrong approach (Leech 2010; Sentience Institute 2020). These researchers maintain that the world of lobbying is governed by uncertainty between lobbying behaviour and outcomes (Lowery 2013; Heinz et al. 1993), with context proving critical (Baumgartner et al. 2009). One such researcher, Frank Baumgartner, compared lobbying to the weather. Under this analogy, lobbying research is more like climate science than weather forecasting: while understanding the complex dynamics at play is both possible and useful, giving point predictions is impossible (Sentience Institute 2020). A different analogy comes from Robert Salisury, who argued that we might as well ask, 'Who is the most influential member of the U.S. Senate?' or 'What is the influence rank among the Supreme Court Justices?'. In other words, since the success of lobbying depends so strongly on context, we are posing a question that is utterly meaningless (Salisbury 1994 cited in Leech 2010)

Similarly, Lerner (2020) concludes, in the context of effective altruism, that 'it's not advisable to attempt to extract any quantitative estimate of the effectiveness of lobbying from the published work'. This conclusion seems to be based on the literature's methodological challenges. Under this view, achieving a detailed understanding of policy success may be possible in principle, but would not be feasible until the research field advances.

However, Dür (2008b) is more optimistic, concluding that 'the difficulty of measuring influence should not be exaggerated'. They argue that measuring policy success is not substantially different from other questions commonly tackled in academic research.

THE COUNTERFACTUAL IS CRITICAL

The challenges outlined above provide numerous reasons why the academic literature might not contain sound estimates of the rates of policy success. But even if the literature does contain such estimates, we need to go one step further and understand the counterfactual impact of lobbying.

It is helpful to think of policy success as the sum of two components (Figure 2). Firstly, there is the baseline success rate of policies (grey puzzle piece). This might represent the general probability of success for a bill to pass without any additional effort from interest groups. Secondly, there is the counterfactual impact of lobbying (blue puzzle piece). This might correspond to the additional probability of success that can be attributed to extra lobbying effort. These two components both make up the overall, observed rates of policy success⁵.

For the purposes of evaluating campaign strategy and comparing different opportunities, the most useful piece of information is the counterfactual impact of lobbying. It is this counterfactual impact that best determines where limited lobbying resources should be focused. On this basis, even if the academic literature contains sound estimates of policy success, we need to see if any studies go one step further and uncover the counterfactual impact of lobbying.

Figure 2: The overall, observed success rates of policies are made up of two parts: the baseline success rates of policies (grey puzzle piece) and the counterfactual impact of lobbying (blue puzzle piece).

OUR SYSTEMATIC REVIEW OF THE LOBBYING LITERATURE

The purpose of this report is to find out whether the academic literature on lobbying contains quantitative estimates of policy success - and, ideally, the counterfactual impact of lobbying.

To gather all available published studies on success rates of policy, we performed a systematic review of the literature (for a detailed explanation of methods and results, see Appendix 2). From our systematic review, we obtained 16 papers that estimated a probability of success of lobbying. These studies mostly focused on the United States and Europe, although many issues and types of lobbying (i.e. lobbying for both social causes and economic causes) were represented. Of the 16 papers we obtained, 14 were based on observational studies, and two were based on experimental studies. Details about the 16 papers, including their methodologies, are summarised in the Appendix 2 (Table 2). We do not list each paper's estimate of policy success rates as we do not want to encourage the reader to place a high weight on the evidence we extracted from these studies.

Is the academic literature sufficient to assess rates of policy success?

In the 14 observational studies, the estimated rates of policy success were narrowly distributed around 50%. This pattern was unexpected and strange. There is a hypothesis that could explain this pattern, though we note that we are speculating here, and hypothesising after having seen our results. For any particular issue, there are usually both winners and losers (Baumgartner et al. 2009). A single policy outcome might be recorded as a success for one group and a failure for an opposing group. Indeed, many of the observational studies highlighted that most issues were contested by multiple groups. So, across many issues and a large sample size, a researcher might naturally observe that around 50% of lobbying efforts succeed (Lowery 2013; Leech 2010; Baumgartner et al. 2009). If this reason is indeed why we have observed that the probability estimates tend to sit around 50%, then our evidence might contain useful information on the big picture of the world of lobbying, but it would not contain useful information on the success rate of any particular policy or group of policies.

Is the academic literature sufficient to estimate the counterfactual impact of lobbying?

Our systematic review identified two studies that are experimental rather than observational. In general, experimental studies are ideal when seeking to uncover counterfactual effects (Robinson, McNulty, and Krasno 2009). Accordingly, the two experimental studies that we identified provide the best prospects for obtaining evidence on the counterfactual impact of lobbying.

In one experimental study, Bergan (2009) conducted an experiment in which emails from an anti-smoking coalition were delivered to state legislators in New Hampshire, United States. There was a treatment group consisting of 72 legislators, who did receive emails, and a control group of 71 legislators, who did not receive emails. Bergan analysed the actual votes of legislators during two pivotal votes on a smoke-free workplace bill. In the vote on whether to table the bill, 48% of legislators in the control group voted in favour of the bill, compared to 62% in the treatment group⁶. In the vote on passing the bill, 42% of legislators in the control group voted in favour, compared to 50% in the treatment group. It is unclear whether it is more appropriate to view this experiment as a single lobbying effort targeting one bill, or a series of many lobbying efforts, each targeting one legislator. Therefore, while this study does appear to show that the counterfactual impact of lobbying in this context is positive, it is difficult to conclude what the exact value of that counterfactual impact is more generally.

In another experimental study, Cluverius (2017) conducted a hypothetical experimental study, delivered as a survey to state legislators in seven states in the United States. The survey included a hypothetical email from imagined constituents asking legislators to take action on an issue. The survey asked legislators which action they would take in response to the hypothetical email. Cluverius found that 8.5% of legislators would introduce a bill; 19.3% would cosponsor a bill; 32.6% would vote for a bill; and 16.3% would urge fellow legislators to support a bill. These results did vary across the salience of the issue (e.g. gun control vs GMO labelling) and across the volume of emails that the survey asked legislators to imagine they had received. However, the study's design did not include a control group. This means that the counterfactual impact of lobbying cannot be inferred from the results. Also, since the survey was entirely hypothetical, there is no way to validate whether the legislators' actual behaviour would match their stated behaviour.

Given the lack of control group in the experiment by Cluverius (2017), we believe that the experiment by Bergan (2009) provides the most relevant piece of evidence for estimating the counterfactual impact of lobbying. However, Bergan's study has not been replicated or even repeated across other contexts (e.g. different states and countries, different policy issues, and different types of government officials), so it is impossible to know how accurate and generalisable the findings are. Until the study is replicated, it would be unwise to draw any conclusions on the basis of this single study in a single context.

CONCLUSION

The purpose of this report is to find out whether this academic literature contains quantitative estimates that could be useful in estimating policy success and, specifically, in assessing the counterfactual impact of lobbying.

There are numerous serious challenges that cast doubt on whether the academic literature is sufficient to make meaningful estimates of lobbying success rates. These doubts are reinforced by the surprising and strange results that we obtained in our systematic review. Even ignoring these challenges, we only identified a single empirical study that estimated the counterfactual impact of lobbying. This study was limited to a single policy issue in a single context. Until this study is repeated across a wider range of policy issues and contexts, it is insufficient to provide sound conclusions on the counterfactual impact of lobbying.

Gauging the counterfactual impact of lobbying would provide valuable information for evaluating campaign strategy and best directing limited advocacy resources. Our findings provide evidence against the value of academic literature as an option for assessing this counterfactual impact. However, even if the academic literature is ruled out as an option, several options remain:

• The intuition of individual researchers. We do not believe that individual intuition should be accepted as much more than a wild guess⁷ (Smith 2019). However, since this is the option that is most often used - usually implicitly - when evaluating campaign strategy, this provides a baseline against which other options can be compared.
• The judgement of people who are experts within a specific context and policy area. The key benefit of using these domain experts is that they often have on-the-ground expertise on particular policy issues. The principal weakness is that their judgement is unlikely to have been validated over many questions. We have also found that experts are often hesitant to assign specific probabilities, which may necessitate the use of qualitative categories (e.g. asking experts whether lobbying is likely to be very helpful, somewhat helpful, or not helpful). There are some systematic methods available for analysing expert judgement (e.g. Meyer and Booker 1987; Linstone and Turoff 1975).
• Forecasts made by paid panels of superforecasters. The key benefit of using superforecasters is that their judgments have been shown to achieve a significant degree of accuracy over many forecasts (Tetlock and Gardner 2016). However, superforecasters are unlikely to have on-the-ground expertise on particular policy issues, so they might miss important factors.

We think that it is particularly worth exploring the latter two options: expert judgement and superforecasters. Although these options suffer from weaknesses, they are both likely to represent large improvements over the intuition of individual researchers. At Animal Ask, we intend to explore these options in the future. We encourage readers to get in touch with us if they have tried (or intend to try) either of these options, or have ideas for other options to assess the counterfactual impact of lobbying.

APPENDIX 1: OUR FOLLOW-UP QUESTION

Recall that there are several faces of power: the first face of power is to do with who wins and loses on particular policy issues, while the second face is to do with who can control the agenda on which those issues appear or do not appear (Lukes 2004).

In this report, we found that there are serious challenges with measuring the counterfactual impact of lobbying. This work was focused on who wins or loses on particular policy issues, which is the first face of power.

But it is possible to take a step back and consider how policy issues appear in the first place. If an organisation is concerned with which pieces of power make it into a legislature to begin with, then that organisation is focused on the second face of power.

Introducing new pieces of legislation (the second face of power) is indeed a focus for some animal advocacy organisations. These organisations might campaign to submit a new Bill into Parliament or Congress, or they might seek to add a new regulation that is not already under consideration.

For these organisations, it is most useful to know the base rates of success of different types of policies. For example, an organisation might already be in contact with Members of Parliament and thus have the ability to introduce a new Bill. Here, the organisation might just want to know what type of policies are most likely to succeed - is it better to introduce a narrow, obscure Bill that limits the expansion of new insect farms, or is it better to introduce a broader, more popular Bill that restricts pollution from chicken farms?

In this case, the counterfactual impact of additional lobbying, as we have investigated in this report, is less relevant. What really matters is the baseline rate of success of a policy. Building on the previous example, it might be possible to examine all Bills introduced to Parliament in a particular time-frame. Hypothetically, we might observe that Bills that are narrower, more obscure, and focused on smaller industries tend to pass into law more than Bills that are broader, more popular, and focused on larger industries. In this case, it would be wiser for animal advocacy organisations to focus their efforts on the types of Bills that tend to pass into law more often⁸. Or, we might observe that Bills related to one species pass into law more often than Bills related to another species - that would be one reason to focus an organisation's efforts on the former species for the time being.

It would be useful for the animal advocacy movement to learn which types of Bills tend to pass into law.

Our pilot study

We conducted a pilot study to determine whether it is possible to gain insight into this question. Our pilot study involved an analysis of pro-animal Bills. For the pilot study, we chose to analyse Bills in Australia, which is a country that has a political system very similar to countries that are high-priority for animal advocacy (namely, the UK). The lead author is also familiar with the Australian context. Specifically, we analysed Bills in one Australian federal Parliament (42nd Parliament, 2008-2010) and one state Parliament (New South Wales, 56th Parliament, 2015-2019).

Our goal was to uncover whether it was possible to link the success of Bills to particular characteristics. For example, were more successful Bills generally focused on a particular species? Or did they have a particular scope, or a particular level of public engagement?

We quickly found that Bills introduced by the government had all been successful, and Bills introduced by other Members of Parliament (private members) were never successful. We expect that this is because Australia has a two-party system and a strong party whip, which is also the case in the UK. This trend may be less strong in other countries. For example, US legislatures have weaker party whips, and many European countries have multi-party systems. However, it is clear that the success or failure of Bills depends strongly on the context of a particular political system. So, we think that analysing Bills in a particular country might produce useful insights for organisations working in that country, but these insights would be unlikely to generalise to other countries.

APPENDIX 2: METHODS AND RESULTS OF SYSTEMATIC REVIEW

Methods of systematic review

For our systematic review, our main information sources were the databases Web of Science and HeinOnline. We conducted less formal searches on Google Scholar and Google. Once we had conducted these searches and identified all relevant review papers, we also looked through those review papers' reference lists, citing papers on Google Scholar, and similar papers on ConnectedPapers. Our search terms are given below (Box 1). Studies were evaluated by a single researcher (the primary author of this report).

Note that our systematic review differs from many traditional reviews in that we are mindful of the diminishing returns to extra research effort. Given this perspective, which is unconventional among researchers conducting systematic reviews, we made some simplifying assumptions in our inclusion and exclusion criteria.

Box 1: Search terms in our systematic review.

We adopted the following inclusion and exclusion criteria:

• We sought to include any study that gives a quantitative estimate of the success rate of policies or lobbying in legislatures or other government policymaking bodies.
• We only considered estimates measured at the level of the organisational campaign. This is the specific combination of a lobbyist campaigning for a particular policy, sometimes referred to as the 'policy-lobbyist dyad'. This is because we were interested in the impact of lobbying on policy adoption. We did not consider estimates at the level of the policy (which would measure policy adoption) or at the level of the lobbyist (which would measure the success of lobbying over time). 
• We only considered studies that gave a binary (dichotomous) measure of success. There are many good reasons to prefer measuring success along a continuous scale for lobbying research in general (see above section, 'Studying influence: Methodological challenges'). But we intend to apply our results in modelling potential lobbying campaigns, and a continuous scale is not suited to that context.
• We only included studies that gave useful information on the probability of success given some lobbying effort (the output of lobbying). This meant that we excluded other questions, like access to politicians or the reasons why groups decide to lobby (the input of lobbying).
• We only included studies that analyse democratic countries. While research on lobbying in non-democratic countries is an intriguing and useful line of inquiry (Weil 2018; Popović 2020), our focus as an organisation has generally been in democratic countries. There are likely to be different political mechanisms for animal advocacy in non-democratic countries (Chung, Anderson, and Li 2021; Wulderk et al. 2022), and different contexts and levels of success for lobbying in general (Uba 2009; Weymouth 2012). We classified a country as democratic if it was categorised as 'full democracy' or 'flawed democracy' in the 2021 Democracy Index (Economist Intelligence Unit 2022).
• We only included studies published after the year 2000. This is an arbitrary choice, but one that saved substantial time by reducing the number of database results by 20%. We suspect that the results that this excluded are generally the least relevant, particularly as quantitative methods in evaluating lobbying have evolved over the last couple of decades.
• We only include publications with full-texts in English. We believe that studies in languages other than English are likely to be useful. However, we were constrained by the time and language skills to find and evaluate such publications.
• We did not include purely qualitative case studies on specific instances of lobbying.
• We did not include theoretical modelling papers, like game theory studies or simulation studies.
• We did not count direct payments of money (e.g. campaign contributions) as lobbying, following the definition of lobbying in de Figueiredo and Richter (2014).
• We did not include studies where the person or group lobbying was a government or a government official. This mostly occurred where a local government lobbies a state or national government - this phenomenon is not relevant to our focus on lobbying by interest groups or grassroots campaigns. 
• We did not include judicial lobbying. The context of judicial lobbying is inherently different from legislative and regulatory lobbying, and we rarely focus on judicial lobbying in our work.
• We did not include studies where corporations lobby governments for contracts.
• We only included lobbying within a country, not lobbying across borders.
• We automatically obtained the full-text of any paper we encountered on the topic of animal advocacy, even if it did not meet our criteria. None of these ended up being useful for this study, even for qualitative information.

There were also many studies on the determinants of lobbying, lobbying strategies, lobbying ethics and regulations, and related topics. None of these are within the scope of this report, despite being commonly researched topics (A. Bunea and Baumgartner 2014). Also, as our focus was on measuring the success of lobbying, we did not include any studies looking at how to lobby.

Results of systematic review

Our initial search resulted in 3,997 results from Web of Science and 579 results from HeinOnline (including duplicates). After scanning the titles and abstracts of these, we obtained 182 papers. We also obtained a further 58 papers through other search methods or that were citing, cited by, or connected to the review papers we had obtained. After scanning the full-text, we retained 74 papers. Of these, there were 16 studies with quantitative estimates of rates of lobbying success and 24 review articles (or similar). These results are summarised in the flow chart below (Figure 3).

The 16 studies in our final sample mostly focused on the US and Europe across a range of lobbying contexts. Fourteen of these studies were observational, while two were experimental. The details of the 16 studies are summarised in the table below (Table 2).

Figure 3: Flowchart summarising the number of papers identified and obtained throughout our systematic review.

Table 2: Our systematic review identified 16 papers that make an estimate of rates of policy success. The estimates made by each study were closely distributed around 50% and were associated with very high variances. We have not listed these estimates in the table, as we do not want to encourage the reader to update their beliefs based on this information (see section 'Success of Lobbying'). The two experimental studies, which are particularly relevant and considered separately in this report, are denoted by *.

NOTES

1. In this report, we compare success and failure. It is also possible for policies to partially succeed. Many academic studies use methods that capture partial success (see footnote on binary vs continuous methods below). We think that modelling partial success is even more difficult than modelling success vs failure, and so we ignore partial success here.

2. We use the terms 'binary' and 'continuous' to avoid confusion. In the literature, these are often called 'qualitative' and 'quantitative' respectively. However, these terms carry a different meaning in this report, so we avoid them in this instance.

3. These 'informal meta-analysis' papers rely on vote counting, which is a very problematic way to integrate information across multiple studies (Borenstein et al. 2021). However, we agree with the authors of these papers that the ideal method, formal meta-analysis, would be poorly suited to studies on lobbying and interest groups due to the inconsistent methods used in this field.

4. This problem is less severe for this report, as we deliberately limit our focus to democratic countries (see Appendix 2).

5. We do not think that these two components literally sum together in a simple way, since lobbying involves complex interactions within a dynamic system. This is just a useful way to think about the two components of policy success.

6. Tabling the bill would have defeated the bill. In the vote on tabling the bill, to vote in favour of the bill means to vote against tabling the bill.

7. There might be an exception if individual researchers track these guesses over time as a way of validating their intuition.

8. Ideally, this information would be considered along with the many other pieces of information that make an ask impactful, like the number of animals affected, the amount of suffering prevented, and so on.

REFERENCES

Animal Charity Evaluators. 2022. “What Is the Effect of Legal Work on Improving Animal Welfare Standards?” Animal Charity Evaluators. 2022. https://animalcharityevaluators.org/research/research-briefs/what-is-the-effect-of-legal-work-on-improving-animal-welfare-standards.

Anzia, Sarah F. 2019. “Looking for Influence in All the Wrong Places: How Studying Subnational Policy Can Revive Research on Interest Groups.” The Journal of Politics 81 (1): 343–51.

Banfield, Edward C. 1961. Political Influence: A New Theory of Urban Politics. New York: Free Press.

Baumgartner, Frank R., Jeffrey M. Berry, Marie Hojnacki, Beth L. Leech, and David C. Kimball. 2009. Lobbying and Policy Change. University of Chicago Press.

Bergan, Daniel E. 2009. “Does Grassroots Lobbying Work?: A Field Experiment Measuring the Effects of an E-Mail Lobbying Campaign on Legislative Behavior.” American Politics Research 37 (2): 327–52.

Bernhagen, Patrick, Andreas Dür, and David Marshall. 2014. “Measuring Lobbying Success Spatially.” Interest Groups & Advocacy 3 (2). https://doi.org/10.1057/iga.2014.13.

Beyers, Jan, Rainer Eising, and William Maloney. 2008. “Researching Interest Group Politics in Europe and Elsewhere: Much We Study, Little We Know?” West European Politics 31 (6): 1103–28.

Binderkrantz, Anne Skorkjær, and Anne Rasmussen. 2015. “Comparing the Domestic and the EU Lobbying Context: Perceived Agenda-Setting Influence in the Multi-Level System of the European Union.” Journal of European Public Policy 22 (4): 552–69.

Borenstein, Michael, Larry V. Hedges, Julian P. T. Higgins, and Hannah R. Rothstein. 2021. Introduction to Meta-Analysis. John Wiley & Sons.

Bunea, A., and F. R. Baumgartner. 2014. “The State of the Discipline: Authorship, Research Designs, and Citation Patterns in Studies of EU Interest Groups and Lobbying.” Journal of European Public Policy. https://www.tandfonline.com/doi/abs/10.1080/13501763.2014.936483.

Bunea, Adriana, and Raimondas Ibenskas. 2015. “Quantitative Text Analysis and the Study of EU Lobbying and Interest Groups.” European Union Politics 16 (3): 429–55.

Burstein, Paul. 2011. “How Our Understanding of Policy Change Is Affected by the Measurement of Public Policy, Hypothetical Determinants of Change, and the Relationship Between Them.” APSA 2011 Annual Meeting Paper. https://papers.ssrn.com/abstract=1901006.

———. 2019. “The Influence of Organizations on Policy: Theories, Findings, Conclusions.” Interest Groups & Advocacy 8 (1): 1–22.

Burstein, Paul, and April Linton. 2002. “The Impact of Political Parties, Interest Groups, and Social Movement Organizations on Public Policy: Some Recent Evidence and Theoretical Concerns.” Social Forces; a Scientific Medium of Social Study and Interpretation 81 (2): 380–408.

Chung, Jah Ying, Jo Anderson, and Congcong Li. 2021. “Growing A Community: How To Support Farmed Animal Protection In China.” Faunalytics. https://faunalytics.org/growing-a-community-farmed-animal-protection-in-china/.

Cluverius, John. 2017. “How the Flattened Costs of Grassroots Lobbying Affect Legislator Responsiveness.” Political Research Quarterly 70 (2): 279–90.

Dür, Andreas. 2008a. “Interest Groups in the European Union: How Powerful Are They?” West European Politics 31 (6): 1212–30.

———. 2008b. “Measuring Interest Group Influence in the EU: A Note on Methodology.” European Union Politics 9 (4): 559–76.

Economist Intelligence Unit. 2022. “Democracy Index 2021: The China Challenge.” Economist Intelligence Unit. https://www.eiu.com/n/campaigns/democracy-index-2021/.

European Commission. 2021. “Questions and Answers: Commission’s Response to the European Citizens' Initiative on ‘End the Cage Age.’” European Commission. 2021. https://ec.europa.eu/commission/presscorner/detail/en/QANDA_21_3298.

Fellowes, Matthew, Virginia Gray, and David Lowery. 2006. “What’s on the Table? The Content of State Policy Agendas.” Party Politics 12 (1): 35–55.

Figueiredo, John M. de, and Brian Kelleher Richter. 2014. “Advancing the Empirical Research on Lobbying.” Annual Review of Political Science (Palo Alto, Calif.) 17 (1): 163–85.

Finger, Leslie K. 2019. “Interest Group Influence and the Two Faces of Power.” American Politics Research 47 (4): 852–86.

Gamboa, Ricardo, Carolina Segovia, and Octavio Avendaño. 2016. “Interest Groups and Policymaking: Evidence from Chile, 2006–2014.” Interest Groups & Advocacy 5 (2): 141–64.

Garlick, Alexander. 2016. “Interest Groups, Lobbying and Polarization in the United States.” Edited by Marc N. Meredith. Ann Arbor, United States: University of Pennsylvania. 

Heinz, John P., Edward O. Laumann, Robert L. Nelson, and Robert H. Salisbury. 1993. The Hollow Core: Private Interests in National Policy Making. Harvard University Press.

Hojnacki, Marie, and David C. Kimball. 1998. “Organized Interests and the Decision of Whom to Lobby in Congress.” The American Political Science Review 92 (4): 775–90.

Hojnacki, M., and D. C. Kimball. 2012. “Studying Organizational Advocacy and Influence: Reexamining Interest Group Research.” Annual Review of Political Science. https://www.annualreviews.org/doi/abs/10.1146/annurev-polisci-070910-104051.

Leech, Beth L. 2010. “Lobbying and Influence.” In The Oxford Handbook of American Political Parties and Interest Groups, edited by L. Sandy Maisel, Jeffrey M. Berry, and George C. Edwards III. Oxford University Press.

Lerner, Matt. 2020. “Informational Lobbying: Theory and Effectiveness.” Effective Altruism Forum. 2020. https://forum.effectivealtruism.org/posts/K638s9L2wCEW78DEF/informational-lobbying-theory-and-effectiveness.

Linstone, Harold A., and Murray Turoff. 1975. The Delphi Method: Techniques and Applications. Addison-Wesley Publishing Company, Advanced Book Program.

Lowery, David. 2013. “Lobbying Influence: Meaning, Measurement and Missing.” Interest Groups & Advocacy 2 (1): 1–26.

Lukes, Steven. 2004. Power: A Radical View. Macmillan International Higher Education.

Lyons, Benjamin A., Amy Melissa McKay, and Jason Reifler. 2020. “High-Status Lobbyists Are Most Likely to Overrate Their Success.” Nature Human Behaviour 4 (2): 153–59.

Mahoney, Christine. 2008. Brussels Versus the Beltway: Advocacy in the United States and the European Union. Georgetown University Press.

McKay, Amy. 2012a. “Negative Lobbying and Policy Outcomes.” American Politics Research 40 (1): 116–46.

———. 2012b. “Buying Policy? The Effects of Lobbyists’ Resources on Their Policy Success.” Political Research Quarterly 65 (4): 908–23.

Meyer, Mary A., and Jane M. Booker. 1987. Eliciting and Analyzing Expert Judgment: A Practical Guide. Reprint edition. Society for Industrial and Applied Mathematics.

Oeri, Fintan, Adrian Rinscheid, and Aya Kachi. 2021. “Lobbying Influence - The Role of Money, Strategies and Measurements.” arXiv [econ.GN]. arXiv. http://arxiv.org/abs/2109.13928.

Pedersen, Helene Helboe, and Anne Skorkjær Binderkrantz. 2014. “Are You Influential? Interest Group Influence on Political Agendas and Decisions.” In ECPR General Conference.

Popović, Emina. 2020. “Advocacy Groups in China’s Environmental Policymaking: Pathways to Influence.” Journal of Environmental Management 261 (May): 109928.

Robinson, Gregory, John E. McNulty, and Jonathan S. Krasno. 2009. “Observing the Counterfactual? The Search for Political Experiments in Nature.” Political Analysis: An Annual Publication of the Methodology Section of the American Political Science Association 17 (4): 341–57.

Sekar, Samantha. 2020. “Understanding Misrepresentation: How Policymakers Perceive Signals from Constituents on Environmental Policies.” Unpublished Manuscript.  https://osf.io/bhzd4/download.

Selling, Niels. 2021. “The Long Shadow of Lobbying: Ideational Power of Lobbying as Illustrated by Welfare Profits in Sweden.” Interest Groups & Advocacy 10 (1): 47–67.

Sentience Institute. 2020. “The Sentience Institute Podcast: Frank Baumgartner of UNC-Chapel Hill on Policy Dynamics, Lobbying, and Issue Framing.” Sentience Institute. 2020. https://www.sentienceinstitute.org/podcast/episode-10.html.

Smith, C. 2019. “The Optimizer’s Curse & Wrong-Way Reductions.” ChristianSmith.cc. 2019. https://christiansmith.cc/2019/04/03/the-optimizers-curse-wrong-way-reductions/.

Tetlock, Philip E., and Dan Gardner. 2016. Superforecasting: The Art and Science of Prediction. Random House.

Uba, Katrin. 2009. “The Contextual Dependence of Movement Outcomes: A Simplified Meta-Analysis.” Mobilization: An International Quarterly 14 (4): 433–48.

Weil, Stefanie. 2018. “The Strange Case of Pluralist Lobbying in a Corporatist Setting: Defending Western Business Interests in China.” Business and Politics 20 (1): 70–97.

Weymouth, Stephen. 2012. “Firm Lobbying and Influence in Developing Countries: A Multilevel Approach.” Business and Politics 14 (4): 1–26.

Woll, Cornelia. 2019. “Corporate Power beyond Lobbying.” American Affairs 3 (3): 38–55.

Wulderk, Zach, Sebastian Quaade, Jo Anderson, Courtney Dillard, Walter Sánchez-Suárez, and Tom Beggs. 2022. “Beliefs About Fishes and Chicken & Their Relation to Animal-Positive Behaviors in China.” Faunalytics and Mercy For Animals. https://faunalytics.org/chicken-and-fish-2-china/.

Yackee, Jason Webb, and Susan Webb Yackee. 2006. “A Bias Towards Business? Assessing Interest Group Influence on the U.S. Bureaucracy.” The Journal of Politics 68 (1): 128–39.

Yackee, Susan Webb. 2012. “The Politics of Ex Parte Lobbying: Pre-Proposal Agenda Building and Blocking during Agency Rulemaking.” Journal of Public Administration Research and Theory 22 (2): 373–93.