Note: I recently published this guide on OSF - check out that link for the PDF version.

About this guide

As a research consultant, I’ve led several prioritisation projects^[1]. These projects typically involve helping organisations determine what they should focus on or recommend. They have varied in intensity and consequence (two collectively influenced approximately AUD $50 million in research funding). 

This guide outlines the systematic process I follow to do this, integrating a recently published paper I led as a case study. The process I outline draws heavily from Multi-Criteria Decision Analysis, prioritisation literature (e.g., Rudan et al., 2008), and insights from experts in the field^[2].

Who should read this guide?

People who want to make an optimal choice regarding a specific goal, based on a large set of options, with consideration of several criteria. The approach I outline may be particularly useful for those seeking to incorporate input from multiple people or groups.

Examples of those who may find it useful include:

• Organisations aiming to identify next year’s focus areas by evaluating options against criteria like importance, alignment with organisational mission, and potential for funding.
• Grant-makers with the goal of allocating funds effectively, assessing research proposals using criteria such as impact, feasibility, and alignment with funding objectives.
• University students seeking to determine how to spend summer break, choosing from a variety of activities while considering criteria like enjoyment, personal growth, and cost.

Summary of the process

Following a systematic approach to determine priorities can increase impact and transparency. 

I consider there to be three main phases in prioritisation:

1. Define your goals, criteria, and what you intend to prioritise: clarify the what, why, and how of this process.
2. Build a long list of unique options^[3]: find out what all the options are, before figuring out which are the best.
3. Prioritise the long list of unique options using criteria: apply predefined criteria to the long list of options to see which perform best.

Why conduct a systematic prioritisation process?

There are two parts to this question: 1) why prioritise and 2) why use a systematic process to do it.

Why prioritise things? We have limited resources (e.g., time, energy, money), and there are many endeavours that we could spend these resources on. Some of these endeavours may be better than others (e.g., they might produce more impact or make us happier). To maximise use of our scarce resources, we need to choose and pursue what’s most important.

Why use a systematic process to prioritise things? There are many ways to determine which of several options is a priority. We could rely on chance and draw numbers out of a hat. We could rely on convenience and do what’s easiest. We could rely on defaults and pick what’s next on the to-do list. We could rely on our desires and pursue what’s most enjoyable. So why follow a systematic process?

• It can lead to a better outcome. When we’re trying to prioritise something, there are often many possible options, considerations, stakeholders, and information sources.  If we don’t think these through, we’ll likely miss important information. Following a systematic process increases the likelihood that all relevant options and factors are considered, helping us make well-informed decisions.
• It can increase respect for, and commitment to, your selected option. Using a systematic process can make your decision-making steps clearer and more transparent to others. This can foster understanding of, respect for, and commitment to, the outcome you reach.

How to conduct a systematic prioritisation process

Below I outline the three main phases of a prioritisation process and the steps involved, integrating a case study to help demonstrate how this could look^[4].

The case study: prioritising funding calls for cardiovascular disease and diabetes

Throughout this article, I will use a project I collaborated on as a case study to illustrate the systematic prioritisation process. I recently published a peer reviewed article describing the process and results of that project. Examples from the case study will be presented in blue boxes throughout. I will also integrate other examples (either hypothetical or from other projects) in text.

Phase 1: Define your goals, criteria, and what you intend to prioritise

As with any research project, you need to decide what you’re going to do before you go do it. By the end of this phase, you want to understand the:

1. Goal of the project;
2. Scope of the project;
3. Unit of prioritisation (i.e., what you’re going to prioritise); and
4. Prioritisation criteria

What is the goal of the project?

Any prioritisation process takes time, resources, and energy – establish why you’re doing this before you get started. Your goal shouldn’t just be, ‘to know what’s the highest priority’: make it more concrete.

Prompting questions to articulate the goal of your prioritisation process.

What is the scope of the project?

Determine the scope of the project before you start so you can allocate resources accordingly, involve the right people, and not get distracted by things that aren’t in focus.

Prompting questions to articulate the scope of your prioritisation process.

What is the unit of prioritisation?

You need to articulate exactly what you will prioritise. The goal is that all options you are prioritising have the same format and level of specificity. This helps ensure: a) you are collecting all the relevant information when building the long list, and b) options in the long list are comparable.

Prompting questions to articulate the unit of prioritisation.

What are the prioritisation criteria?

In this process, you use explicit, pre-defined criteria to prioritise the options. This ensures that a) the priorities are based on what is most important for the user/outcome, and b) every option is assessed against the same criteria. As the criteria determine the priorities, they’re important to get right.

What do good criteria look like?

They consider the unit of prioritisation. E.g., if you’re prioritising problems, you might want a criterion on the impact / size of that problem. If you’re prioritising potential interventions, you might want a criterion on how feasible it is to implement that intervention.

They have minimal overlap. If you have multiple criteria assessing similar concepts (e.g., Criterion 1 = ‘Impact’ and Criterion 2 = ‘Importance’), that concept will have a greater sway over the prioritisation results. It’s better to minimise the overlap, and instead use explicit weights if some criteria are more important (see ‘Weighting criteria’).

They reflect your values, and what’s most important for the outcome. E.g., if it’s important for you to identify top-priority research questions that are not being addressed by others, this suggests you should incorporate neglectedness as a criterion.

There are probably 2-5 criteria. The more criteria you have, the longer and harder the assessment aspect will be. E.g., Participants will struggle if they have to assess each of the 30 options by 10 criteria.

They are well defined. If people can interpret the criteria in different ways, you may not get consistent or useful ratings. E.g., Some people may interpret ‘Health impacts’ as referring to physical health, while others may think of mental health. Make sure you have a clear definition that reduces the interpretation required.

They can be assessed. It should be feasible to assess the options against the criteria. Something that’s vague and hard to assess like, ‘Impact on global economy’ probably won’t work if you’re conducting this process in a short period of time. If participants are assessing the options against criteria, consider: 

• Will they be able to understand these criteria?
• Do they have the experience or expertise needed to assess these options against criteria?
• What would I struggle with if I was trying to assess options against these criteria?

What process could you follow to determine the criteria?

Ideally, work closely with the project team, advisors, and users to determine the criteria. You could do a mini prioritisation process, where you generate a list of potential criteria, then work with the team to select the top ones.

Generating potential criteria

Prompting questions to generate potential criteria.

Finalising criteria

After generating potential criteria, you need to decide which ones you’ll actually use. To do this, you could:

• If you’re working with a team or client, present the criteria you recommend (and why), and get their reactions.
  • Flag criteria that you don’t think are feasible to assess (e.g., “This will be very hard for participants to assess in a survey”).
• Vote for the criteria. A couple of options include:
  • Typical voting: each person in the team could vote for five criteria, and the top ones could be used.
  • Quadratic voting: people vote on their preferred criteria, and can use up more of their votes to express stronger preferences (this method may take a little longer to implement/explain).
• Pilot the criteria with yourself, the team, colleagues, the client, or your target audience. Try assessing made-up options against those criteria.
  • Remove criteria that are difficult to understand or assess (and can’t be improved).
• Assess whether some criteria could be used as a filter (e.g., to determine what options are included in the long list), rather than to assess options. You can think of these as exclusion/inclusion criteria, rather than prioritisation criteria.
  • E.g., ‘Relevance to behaviour change’ may be suggested as a criteria. If you only want to include options that are relevant to behaviour change in the long list, this could be a filter for what’s included.

Weighting criteria

Once you’ve finalised the criteria, decide whether some are more important than others. 

Pairwise comparison can help you determine what, if any, weights should be used. Compare two criteria at a time (either by yourself or with the team/client) and ask whether a) they are of equal importance b) one is slightly more important or c) one is much more important. You then use the responses to determine the weights.

• If you google, ‘Pairwise comparison to determine criteria weights’ you can find more comprehensive instructions (e.g., this site or this paper that uses the method).

Phase 2: Build a long list of unique options

Create a thorough list of options to avoid overlooking any possibilities. This is the ‘long list’.

Building the long list of options

The two main methods I’ve used to build long lists are 1) getting people to contribute options (e.g., through a survey) and 2) extracting options from relevant documents or literature. Your approach might involve one or both of these methods. 

Getting people to contribute options through a survey

There are many ways you could get people to contribute options, from casual brainstorming sessions to more formal surveys or seminars. Your approach depends on resourcing, the desired level of formality, and your target participants. You may only want people within the team/organisation to generate options, or a broad range of experts. I’ve typically used surveys to collect options from people, so I’ll elaborate on that method below.

A survey might be a good option if:

• You want to reach a large number of people, beyond what’s feasible through individual meetings;
• The topic hasn’t been covered much in relevant documents or literature;
• You want to reduce the work required to code options in the desired format later (this can be minimised by requesting responses in a specific format in the survey);
• Participants prefer to remain anonymous for any reason.

Developing the survey

The purpose of the survey is to get participants to contribute options to your long list. At a high level, you do this by including a survey item that elucidates options in the desired format. These are some key considerations when developing the survey:

Determine who you want to complete the survey. The ideal is to include all relevant groups when building the long list. Consider how you’ll reach participants with different backgrounds (e.g., who work in different areas, are located in different countries, or have different life experiences).

Develop the main survey item based on the unit of prioritisation. If your unit of prioritisation – the type of thing that you’re prioritising – is problems, then the main survey item should be framed to gather problems. You also need to structure questions to elicit responses in the desired format. For example, if you desire outputs in the format, ‘[Population] experiences [problem]’, then instruct participants to provide responses like this (e.g., “Please suggest options in this format…Specify both X and Y”). Providing a specific format can help clarify what’s required from participants, and reduce later coding efforts.

Keep in mind that it’s easier to filter out information than gather more. If you’re uncertain if you want to prioritise multiple things (e.g., populations AND problems), consider asking for more detail, rather than less. You may end up deciding to only prioritise problems, but the extra detail can be included in future briefings or reports.

Consider what participants need to know to ensure their responses are in scope. If you’re only interested in interventions that can be completed in less than a year, or in specific countries, then provide participants with this context. E.g., “We are trying to prioritise interventions that can be actioned in a two-month period”. Note that participants may struggle to judge whether something meets your requirements (e.g., they might not know which interventions can be actioned in two months). Encourage them to provide their suggestions anyway, even if unsure about relevance.

Suggest a minimum number of options you want participants to contribute. This can encourage participants to contribute more options. Note that if you impose a maximum limit, this could make the long list less comprehensive^[7]. 

Consider what information you need to know about participants, such as their work area, location, or professional experience. This can help gauge your sample’s representativeness. Avoid unnecessary questions that may lengthen the survey, feel intrusive, or compromise anonymity.

Build momentum with participants, if useful. Determine what connection you want to have with participants or how you want them to feel (e.g., excited about potential collaborations or being involved in the next stage of the process). Consider how you can build interest (e.g., provide detail in the survey about what’s next and how they can be involved).

Pilot the survey. Check the survey makes sense to people who aren’t in the project team, and that it seems likely you’ll get responses in the desired format.

Collecting responses

Once you’ve developed your survey, it’s time to collect responses. Here are a couple of options:

Email the survey link. This is low effort, though it can be difficult to get people to respond to surveys via emails – consider implementing the strategies below.

Strategies for boosting email survey response rates.

Organise an event where people can complete the survey. Some participants may be more motivated to go to an event (e.g., a seminar or talk) than click on an email link. This is particularly the case if participants see a personal benefit to their attendance (e.g., they want to hear what you’re working on or connect with others). An event is also a good option if you want to build credibility, connections^[8], or participant investment in the project (e.g., because you want them involved in future stages). Below are suggestions for what the event (which could be online) could cover.

Agenda items for a survey event

Note that you could combine both of the above approaches: email the survey to people and also invite them to an (optional) seminar where they can hear more about the project, ask questions, and complete the survey.

Engage a third party to handle recruitment. Depending on resourcing and your target audience, it may be useful to have a third party handle recruitment. For example, there are companies that specialise in recruiting participants from the general population (e.g., Online Research Unit in Australia) or experts in particular domains (e.g., Research Australia).

Extracting options from relevant documents or literature

Instead of, or in addition to, a survey, you might extract options from relevant documents or literature. 

In a more formal, academic process, this could involve a systematic search strategy and a rapid review. You might screen articles and extract the relevant information from those included. For example, “X paper discusses how Y can be used to help this group experiencing Z, so let’s add that intervention to the long list”. I use a spreadsheet to extract relevant information such as author, article, prioritisation units, relevant text extract, and page number. A single article could contain multiple options, especially if it’s a review paper.

In more informal settings, your search might be less systematic or focus on grey literature. For example, if you’re deciding what project to pursue next, you might extract ideas from several blog posts where individuals or organisations list projects they’d like to see happen.

The documents you find may also provide a framework you can use to start coding your options. E.g., You might see that in the literature, interventions are typically grouped under these ten categories. You could then update this based on the other intervention options you gather.

Finalising the long list of options

Once you’ve created your long list of options, you need to process them to ensure they are consistent in format and are each a unique option (i.e., there are no duplicates). This involves:

• Coding them in the same format (e.g., so they contain similar information and are at the same level of specificity). If you don’t do this, you may be comparing things that aren’t comparable.
• Removing any duplicates. It’s likely there will be multiple contributions of the same options when building the long list. Duplicates can distort prioritisation outcomes by splitting votes. This can result in an option seeming like less of a priority than it actually is.

To demonstrate the importance of the above steps, consider these options that could have been generated in a survey:

1. Interventions that target blood-borne diseases.
2. Provide families with bed nets to prevent Malaria in sub-saharan Africa.

While addressing the same topic, these options are in a very different format. Option 1 is fairly vague, and contains option 2. Option 2 specifies a specific intervention, population, and location, while Option 1 does not. To include both of these options in the prioritisation phase may be confusing and result in unhelpful outcomes (e.g., it’s possible that Option 1 may be too general to be useful to you, or Option 2 may be too specific).

Coding the long list can be the most difficult part of the whole process. It’s also hard to provide specific detail on how to do this without knowing the context (e.g., what you’re prioritising, how formal the process is, or how many options your long list contains). 

Considerations during coding

Determine the level of detail you want to capture. This is important for the outcome (e.g., what level of detail is too much or too little) and the process (e.g., how many options would be too many to prioritise). Ask yourself, would it be useful if I found out this option was the highest priority?

• Note that you could code options into multiple levels. E.g., Interventions may be coded into three levels ranging from broad (e.g., does it target the individual, environment, or systems) to specific (e.g., what is the actual intervention). This can speed up coding (e.g, you can just focus on one category at a time), and may be useful if you’re not sure which level the prioritisation will occur at.

Consider whether the options are independent. Does it matter if some of your options contain other options (like the blood-borne diseases example above)?

Make the presentation format consistent. To make things simpler and clearer during the prioritising stage, place options in similar formats. E.g., ‘[Intervention] that addresses [population]’. Hopefully, if you’ve used something like a survey, responses should be vaguely in this format already.

The coding process

You may start off with an initial framework (e.g., that has been used previously). This framework will develop alongside coding (e.g., there may be options that don’t fit into the framework, so you adjust it)

You may have two people coding the same statements for higher rigour, or one person doing a first pass with another person reviewing. If you’re working in a team, it can be useful to check in with others after you’ve coded an initial set, to confirm you’re on the same page.

You might work in Miro or Google Sheets. In Google sheets, you might have a column for the original responses, and a column for each prioritisation unit (e.g., one to code ‘population’ and one to code ‘intervention’). For each prioritisation unit, you may have multiple columns (e.g., ‘Broad intervention’, ‘Specific intervention’).

Note that some responses may be irrelevant or too vague to be coded. You could mark these as excluded.

Deduplicating the long list

Once you’ve coded all the responses/options, make a new list where you remove duplicates. If you’ve coded at multiple levels, you’ll have to decide which level to deduplicate at. E.g., Do you include every unique specific intervention, or only the unique broad-level interventions? Do you care about every unique combination of population AND intervention, or just the unique populations that are mentioned? Note that you should not literally remove duplicates from the original dataset. Avoid deleting any data as it may be useful to come back to.

Phase 3: Prioritise the long list of unique options using criteria

Now you have the long list of unique options, it’s time to use the criteria you developed earlier to prioritise that list. At a high level, you want to know how each option performs according to the criteria you care about. You then calculate how well each option does overall, which allows you to order the options and determine your top priorities. Below I outline some key considerations.

Sources of data for assessing options

There are two main sources of data for assessing options against criteria: 1) objective data and 2) people. The choice depends on the context: whether data is available or appropriate, whether you can gather the right people, what resourcing you have etc. You could also use objective data (e.g., cost of a potential program, carbon emissions estimates for different environmental problems) to assess some criteria, and people to assess others.

If you’re using people to assess some or all criteria, consider who is best placed to do this. This could include team members, experts, people who will be impacted by the outcome (e.g., users of a product or beneficiaries of an intervention), or the general public.

Methods for assessing options

It sounds simple to say ‘get people to assess the options’, but there are many ways you could do this (again depending on resourcing, expertise, context etc.). Below are just a couple of options.

Methods to assess options against criteria.

Using people to assess options

If you’re using people to assess the options, consider how you’ll do this. In my experience, running a workshop can work well. Such an event could cover:

• Overview of project aims, process, and team;
• The purpose of the session, ground rules, and housekeeping;
• Brief participant introductions;
• Questions regarding process and aims of the session;
• Introduction to, and running of, prioritisation activity;
• Break while project team calculates results;
• Presenting top priorities against each of the criteria and overall;
• Breakout rooms to discuss results;
• Reporting back, next steps, and closing.

As with previous steps, consider whether you want participants to be involved in future stages or decisions, and communicate accordingly. 

Interpreting the prioritisation results

One important element that’s listed above is discussing the prioritisation results. No matter how sophisticated your assessment process is, relying solely on the prioritised list will likely be an oversimplification. Participants can add nuance that you may want to consider when taking action. For example, they may point out that another project is already addressing the top priority, that there are issues with the coding process (e.g., “X and Y should be coded as the same thing”), or that particular voices were missing from the process.

Some questions to prompt discussion include:

• Were you surprised by the results? Why / why not?
• Are you aware of any initiatives, projects, or organisations currently addressing the priorities?
• How do you think we could make the most progress on these priorities?
• What potential risks or challenges do you foresee in addressing these priorities?
• Are there any reasons we should not pursue these priorities?

Key points from the discussion could be summarised and considered, along with the prioritisation results, when determining what action to take next.

Take action

Once the top priorities are identified, take action aligned with your initial goals. Consider the immediate decisions that can be made and gather any necessary additional information. Communicate the results to relevant stakeholders, including the general public, colleagues, or board members. Follow up with individuals involved in the process to express gratitude, share results, and outline next steps. Finally, take a moment to reflect on the prioritisation process – evaluate its utility, consider lessons learned, and determine if you would apply a similar approach in future decision-making endeavours.

Acknowledgements

Thank you to Alexander Saeri for your extremely helpful feedback on this guide.

Funding disclosure

I did not receive funding to create this guide.

The case study discussed in this guide was funded by MTPConnect through its delivery of the Targeted Translation Research Accelerator program. The Targeted Translation Research Accelerator program is an initiative of the Australian Government’s Medical Research Future Fund. 

1. ^{^}

   Most of these projects were conducted at BehaviourWorks Australia, a behaviour change research enterprise based at Monash University, Australia.

2. ^{^}

   Particular thanks to my colleague, Peter Bragge – a Professor of Evidence Based Policy and Practice at Monash Sustainable Development Institute, with extensive experience in prioritisation.

3. ^{^}

   Note about terminology: ‘Options’ refers to the things that you are prioritising. ‘Long list’ refers to the extensive (and ideally exhaustive) list of options that you gather and then prioritise.

^{^}

Artificial intelligence such as large language models (e.g., ChatGPT) could be integrated at various points throughout this process (e.g., to help generate or code options). Discussing this is out of scope for this guide.

^{^}

It’s also important to consider typical project management questions. When do you meet? Who is in charge of what? How do you make decisions? How do you communicate (e.g., Signal/Zoom)? How do you keep the timeline on track? Etc.

^{^}

To explain this further: if you’re prioritising based on both problems and populations, every unique combination of problem and population will be an option. You may end up with ten options for every one problem, because each of those specifies a different population. You may update your approach later on, depending on how big your long list is (e.g., if you have too many options in your long list, you may decide to just prioritise problems, or just prioritise populations – but not both).

^{^}

Even if you don’t impose a maximum limit, participants may essentially do this themselves. For example, it’s likely that they’ll just suggest the main things that come to mind, get bored, or be limited on time. This is a flaw of the process which should be acknowledged. In a way, there is an implicit prioritisation process already going on in the long-listing stage. If participants are only generating five options, it’s probably going to be the five most important in their mind.

^{^}

When you’re involving others in your prioritisation process, consider the benefits beyond the specific project. For example, it may be useful to update people on what your organisation is doing, build credibility, build your own connections, or foster a network by introducing various experts / like-minded people to each other.

Show all footnotes