Thank you so much for these ideas and thoughts! (And my apologies it has taken so long to respond.) I plan to start with the list Lizka posted, but will absolutely think about whether I can pull off the strategy you mentioned if I come up empty handed from a more straightforward approach.
I would also be interested in any recordings or notes you have from this conversation! I've wondered whether protests work for years, and the answer seems complicated and I'd love some entry points or summaries!
I love this idea! I somewhat recently realized it would be helpful to try to just build an EA community within RTI, more generally. But you're right that it's very unlikely I'm alone right now, and I could really use the extra hands and brains to make progress on these initiatives more quickly.
I appreciate your questions, and I didn’t find the tone off-putting at all! Please read my frank tone as honesty and (attempted) clarity rather than a sign that I’m ungrateful for your input. :)
Your thoughts raised some new questions for me. Here are some responses, but for what it’s worth, I would not categorize all of them as “answers.”
I’d like to start with your final point because I think it will help contextualize both my original post and the rest of my thoughts that follow.
Linch re: starting small and expanding outwards into the organization --
Lauren: In short, yes! That’s my only plan for implementation. I plan to pilot a few of these ideas in small(ish) groups of willing people, and then bring the results to management. At this point, I would also enumerate all of the reasons I can think of for why implementing them more broadly would be helpful from a business perspective. (For example, there is an initiative fairly high-up in the organization centered on employee retention. This tells me RTI is nervous about our retention rates. Based on anecdotal evidence, I think better aligning our actions with our mission/vision would be really helpful in keeping our folks fulfilled.)
Now onto the rest of your questions, in the order you posed them:
Linch: You mention you're pretty junior at the org. What's your degree of actual power or institutional buy-in at RTI to change the way decisions are made?
Lauren: I don’t have a lot of power at the moment. However, I have reasons to believe my voice will be heard if I come with compelling evidence. (Some evidence for my beliefs: we have basically a “club of young, entry-level professionals” who have successfully pitched and implemented medium/large sized programs that have been enthusiastically supported by upper management.)
Linch: You say "I’m currently volunteering my time[…] " Does this mean that your manager/other higher-ups at the company are actively supportive of your work? Or is this more of a personal passion project without much/any institutional buy-in (whether official or informal?)
Lauren: My direct manager is supportive, although I’m not getting paid for this work. He’s also skeptical anything will change. That said, he’s not particularly “high-up” in the grand scheme of things, and I haven’t talked him through my actual plans. I’ve basically only told him “I’ve got problems with the way we run things,” and he’s been like “That’s fair. Go be loud about it, if you want.” That said, I’m in the process of implementing one idea designed to measure the impact of our work, and I’ve received really enthusiastic responses from high-up people who matter a lot to the success and viability of the project’s pilot.
Linch: I'm interested in this because I think it's very hard for junior people changing high-level institutional decisions unless they have an unusual degree of soft power/institutional buy-in or if they're unusually intrapreneurial.
Lauren: It makes sense to be skeptical! I’m very excited about this project, at large (and am therefore very motivated). I also think I have a decent personality for politics/building consensus. So, I’m hoping to fall into the unusually intrapreneurial bucket (given the caveat that I’m definitely starting small).
Linch: Which is not to say this is impossible, tbc, or not worth trying even if there's a low probability of success.
Lauren: I think there’s between a 5%-95% chance that all of this crumbles. (That is to say, I’m very unsure how much confidence I should have in this endeavor.) However, I think I will learn a lot about the challenges the EA community may face in mapping their ideas and metrics onto non-EA organizations, and I think I have a lot to learn from failing. (Even though success is obviously the much preferred outcome.)
Linch: Why are clients willing to pay your institution if they're not willing to use the outputs of your research?
Lauren: I ask myself this question all the time.
Linch: This part is the most bizarre to me. Given the market dynamics involved, I'm surprised that your research does not change decisions.
Lauren: I may have been misleading in my original post. My current hypothesis is something like “our research does not change most decisions.” My real answer is, “I have a lot of thoughts on why it might not be impacting decisions, but we don’t collect that data right now, so I can’t speak with any authority. But I’m currently working on piloting a way to measure the impact of our work.” Some possible reasons we’re not impacting decisions:
“bureaucratic check boxes”: e.g., maybe the CDC is required, annually, to evaluate a public relations campaign they do. While this, in theory, impacts budgeting decisions, it mostly just justifies a line item so that they can back up their spending with research if anyone asks.
No political will: maybe the agency loves the research, but they don’t have the power to pass the relevant law
Lack of institutional power: this seems to be especially problematic for some of our international work
Lack of resources: maybe a great idea just doesn’t have the funding, or folks don’t have the bandwidth to act on the science
Technical issues: sometimes we build tools and they don’t seem to get used. Perhaps this is a technical expertise problem on the client-side; perhaps this is us making bad products; or maybe it’s some of each.
Maybe my hypothesis is wrong: it’s important to note that my hypothesis is based completely on personal experience, and stories from colleagues. Perhaps we’ll measure it and find that we’re super impactful. I just don’t think that’s true.
Linch: One claim/critique I've heard about strategy or management consultancies is that they aren't hired to discover true things or help make new decisions, but to justify existing decisions (eg by giving the stamp of approval and legitimacy to help solve internal principal-agent problems, or for PR reasons).
Lauren: Yeah, this might be most similar to the “bureaucratic check boxes” issue I note above.
Your questions are making me realize the word “change” may be really important in the hypothesis I stated above. Namely, is it important research to do if we don’t change anything—if we don’t “move the needle”? Or should we just seek to “impact” or “inform” decisions? The latter options seem like a lower bar. I’m just not sure I’m not currently convinced that we should spend our resources and brain power on just checking boxes for people when we aren’t really changing decisions.
Perhaps we should be really enthusiastic about evaluating programs the first [insert reasonable number of] times to really understand the impact of an intervention. However, maybe after 5 years of fairly stable numbers, we should deprioritize that kind of work, or recommend it only be done once every two years?
Linch: But you say "Based on my experience, and the experiences of colleagues, our research is very rarely turned into practice" Which sounds like the opposite problem!
Lauren: I think a response here would be redundant, given my two previous responses.
Linch: High-quality research not changing decisions is certainly one of my larger fears about work at RP.
Lauren: I don’t think “turning knowledge into practice” is a current strength of humanity’s right now. I think there’s a ton of great research that comes out of universities that very few people read, let alone use/implement. And I think implementing research (outside of private industry) is somewhat rare.
These opinions are all based off of conversations with professors, so there might be evidence showing this is totally false. But, it seems to feel salient for professors all the way up to Harvard-level prestige. (Michael Porter of Harvard Business School expressed this sentiment in the beginning of a book he co-authored called The Politics Industry.)
“Research not impacting policy” is a larger question I think about a lot, and I don’t think RTI would be able to “fix” the problem—at least not in the short/medium term. But I think RTI may be able to better understand why knowledge is hard to turn into practice, and we may be able to come up with ways to lessen the gap between research and policy.
Linch: My proposed solution (in progress) for RP is to charge EA clients more, especially for work we're inside-view less excited about, since presumably clients are less willing to pay large sums of money for research if they don't think the research will plausibly affect their behavior.
Lauren: This point makes sense, but (my understanding) is that RTI is already quite expensive.
Your question makes me curious about what “pot of money” organizations use to pay for RTI research. For instance, does the FDA just have a “research and development” fund that they use to pay us? Is it because they have an annual “evaluate our advertisements” fund? How do the agencies justify the money they currently spend on our research? This could be an easy/medium question to answer for people more familiar with governmental funding processes and norms.
Linch: To the extent you face similar dynamics, one possible solution is for you guys to also charge much more. I can imagine many companies/government agencies being willing to pay (say) $50/hour for random research outputs that may not affect real decisions much, but to be laser-focused on questions that actually matter (to them) if you're charging >$500/hour.
Linch: (This may have unfortunate implications for job security)
Linch: (I think it's very unlikely you can pull off such a large institutional change tbc)
Lauren: All great thoughts. I think I would want to get a better understanding of why our research doesn’t really change (or “meaningfully impact”? I need to think about what word makes the most sense there…) decisions before suggesting anything like this. I don’t think I would be successful with this kind of suggestion without having a lot of solid evidence.
Linch: What institutional incentives or individual incentives does your institution have to change to be aligned with their stated mission/vision? To quote one of my favorite blog posts about management, "Real values aren't what you talk about, they're what you do when times get tough." To the extent that the real crux is that the real values of your institution are just pretty far away from the stated mission, I'm curious what your theory of change/story of winning looks like.
Lauren: The strongest evidence I have on this point is that, in the summer of 2020, RTI embarked on a long-term, evidence-based mission to pursue racial justice both within our organization, and within the research we do. We seem to be investing a fair amount of time, money, and bandwidth (especially at the upper management levels). This mission has also been pretty constantly discussed throughout the year, and management just released a new set of specific changes they’re making throughout the organization to improve on metrics they’ve set for themselves. (And they really do seem to be done in good faith, and based in evidence, rather than “just fool people into thinking we care about this.”) This makes me think that when someone or something makes RTI leadership see a gap between our mission/vision and our actions, there is substantial drive to change our behavior.
It seems to be the case at many American universities that the deadline for finalizing your schedule is usually a couple weeks into the semester. This gives you time to try out a bunch of classes, and to only keep the ones you think will be most valuable. So, I would always recommend signing up for as many as possible, and then going to all of them for the first couple of weeks to figure out which ones are the best, and then dropping down to only the best opportunities.
Also, if you expect a class to be fabulous, and then the first day is boring, I think it's usually best to drop it. The first day of the class is the professor's one opportunity to really wow students, and if they have no enthusiasm on day 1, I don't think there's hope going forward. I wasted so much time taking classes that I thought would surely "get better once we were talking about real stuff." But that never happened. For all of my favorite classes, I knew they would be great on my first day. And for all of my worst classes: they were bad on the first day.
Lastly, if you're able to sign up to pass/fail classes, I highly recommend that as a not-super-stressful-way to hold you accountable for learning a skill set that you know is important, but which you also know you won't have the dedication to learn on your own. For instance, I took two computer science courses my senior year because I knew if I graduated without having programming experience, I would never have the drive to learn it myself. It was an incredible experience because I could just do the homework well enough to understand the content at the level I was interested in learning, and then didn't have to spend the extra effort I would have needed in order to get an "A" for my GPA's sake. Those two computer science classes are probably ~20% of the reason I was able to land my research job directly out of college. A lot of students seem to "save" their pass/fail credits in case they do much worse in a class than they expected and want to be able to avoid the negative GPA impacts. But, I think for highly motivated students, they end up never really using those pass/fail credits, and waste an opportunity to take useful classes with really low-stakes.
It looks like Founders Pledge could be useful for thinking, specifically, about climate change. At the moment, I'm really unsure about whether, practically, it would make more sense for me to try to implement a general framework for evaluating the criticality of our research portfolio vs. trying to rank the criticality of potential interventions within one small sub-section of our research portfolio to give the organization an example of what ranking criticality looks like (e.g., climate change). The answer is probably that I'll need to do both. I will definitely keep Founders Pledge in mind as a resource.
Re: open access publications -- Thank you for raising this point! This touches on a larger, tangential problem I've been thinking about: namely, lack of meaningful, public access to academic research, and how that relates to the gap between research and policy. I think there is certainly room for ideas like these as I get further along in the implementation process. I will add that idea to my idea tracker which includes all the things I need to create momentum around until they're implemented. (It's amazing how long each small idea will probably take to implement.)
As an aside: I recently learned that the word "scooped" is used to refer to when someone else publishes similar results to yours first. Like, "Oh no! We got scooped!" I think it's a funny word to use, so I thought I'd share it in case it brings joy to others as well.
I will look into it! Off the top of my head, RTI may not want research published there as I think Vox is perceived as somewhat "left" leaning, and RTI fancies itself a deeply non-partisan organization.
Thank you so much, Lizka! I will take a look at these!
Thank you so much for these ideas and thoughts! (And my apologies it has taken so long to respond.) I plan to start with the list Lizka posted, but will absolutely think about whether I can pull off the strategy you mentioned if I come up empty handed from a more straightforward approach.
I would also be interested in any recordings or notes you have from this conversation! I've wondered whether protests work for years, and the answer seems complicated and I'd love some entry points or summaries!
I love this idea! I somewhat recently realized it would be helpful to try to just build an EA community within RTI, more generally. But you're right that it's very unlikely I'm alone right now, and I could really use the extra hands and brains to make progress on these initiatives more quickly.
Hi Linch,
My apologies for the delayed response.
I appreciate your questions, and I didn’t find the tone off-putting at all! Please read my frank tone as honesty and (attempted) clarity rather than a sign that I’m ungrateful for your input. :)
Your thoughts raised some new questions for me. Here are some responses, but for what it’s worth, I would not categorize all of them as “answers.”
I’d like to start with your final point because I think it will help contextualize both my original post and the rest of my thoughts that follow.
Linch re: starting small and expanding outwards into the organization --
Lauren: In short, yes! That’s my only plan for implementation. I plan to pilot a few of these ideas in small(ish) groups of willing people, and then bring the results to management. At this point, I would also enumerate all of the reasons I can think of for why implementing them more broadly would be helpful from a business perspective. (For example, there is an initiative fairly high-up in the organization centered on employee retention. This tells me RTI is nervous about our retention rates. Based on anecdotal evidence, I think better aligning our actions with our mission/vision would be really helpful in keeping our folks fulfilled.)
Now onto the rest of your questions, in the order you posed them:
I think this is excellent advice.
Some additional thoughts:
Thank you for these suggestions!
As an aside: I recently learned that the word "scooped" is used to refer to when someone else publishes similar results to yours first. Like, "Oh no! We got scooped!" I think it's a funny word to use, so I thought I'd share it in case it brings joy to others as well.
Thank you for linking this post to those other posts! Definitely interesting, and I can see some overlap.