I organise and attend a lot of conferences, so the below is correct and need not be caveated based on my experience, but I could be missing some angles here. Also on my substack.
When you imagine a session at an event going wrong, you’re probably thinking of the hapless, unlucky speaker. Maybe their slides broke, they forgot their lines, or they tripped on a cable and took the whole stage backdrop down.
This happens sometimes, but event organizers usually remember to invest the effort required to prevent this from happening (e.g., checking that the slides work, not leaving cables lying on the stage).
But there’s another big way that sessions go wrong that is sorely neglected: wasting everyone’s time, often without people noticing.
Let’s give talks a break. They often suck, but event organizers are mostly doing the right things to make them not suck. I’m going to pick on two event formats that (often) suck, why they suck, and how to run more useful content instead.
Panels
Panels. (very often). suck.
Reid Hoffman (and others) have already explained why, but this message has not yet reached a wide enough audience:
Because panelists know they'll only have limited time to speak, they tend to focus on clear and simple messages that will resonate with the broadest number of people. The result is that you get one person giving you an overly simplistic take on the subject at hand. And then the process repeats itself multiple times! Instead of going deeper or providing more nuance, the panel format ensures shallowness.
Even worse, this shallow discourse manifests as polite groupthink. After all, panelists attend conferences for the same reasons that attendees do – they want to make connections and build relationships. So panels end up heavy on positivity and agreement, and light on the sort of discourse which, through contrasting opinions and debate, could potentially be more illuminating.
The worst form of shallow discourse is the rampant and endless nodding along that often happens on panels. It goes like this:
- Panelist A: [makes a point].
- Panelist B: “Panelist A is right. [rephrases Panelist A’s point]”
- Panelist C: “I think Panelist B makes a great point, though I’d just want to add that [basically Panelist A’s point again with a caveat everyone already thought of]”
- Moderator: “It sounds like [panelist A’s point]”.
The greatest gift you can give to your audience as a panelist is the following phrase: “Nothing to add”.
There are versions of panels—or rather, getting multiple important people on stage and using time well—that do work.
- Starting with short talks: Let each panel member speak briefly (!) about the topic. This could then be followed by:
- Group Q&A: Let the audience direct questions to one panel member at a time or ask the moderator to pick one (avoid an endless meandering answer where everyone wants to chip in).
- Active, moderated disagreement. If you prepare your panelists and moderators accordingly, and pick people who are good at debating with good faith, you can set them up to disagree well and educate your audience about the topic in the process. Understanding disagreements between people can be very interesting and useful, especially if you find double-cruxes among people with fundamentally different perspectives.
The “group brainstorm”
A session format that’s increasingly common and which has received insufficient criticism is the type of group brainstorming that goes like this:
- The session lead gives a short introduction about a topic, and then divides the room into groups.
- Each group is then given a question, often about a sub-field of the broader topic, and maybe some instructions.
- Group members then discuss the question among themselves or quietly brainstorm ideas.
- One person from each group reports back what they discussed to the wider group.
- The session lead then gathers these ideas.
I’ve led sessions like this, and I will get to how I think they can be useful, but group brainstorming is often a poor use of everyone’s time.
Your session attendees do not have the answers.
Your session attendees are probably quite smart, but they’re usually missing a lot of context (unless you’ve carefully selected your attendees) and context is extremely important for coming up with good ideas. They also usually don’t feel invested; yes, they want to be a helpful and willing participant, but they probably didn’t attend this event to help you solve your problem. Unless session attendees directly work on the problem and are grouped together, your attendees are unlikely to generate something novel or actionable in 30 minutes. In fact, the first few ideas you get probably won’t be much better than if the session lead thought a bit harder themselves.
If it’s ideas you want, use AI. LLMs can generate lots of ideas much faster than a room full of people politely talking to each other. Give the LLM some context on your problem, maybe some other documents, and ask it for 100 ideas.
Ideas are easy. Bandwidth is low.
Often, solutions that are easy to generate are very ambitious or just downright unrealistic. “What if someone simply started an organization focused on this problem?”, “What if some smart and capable people spent a week working on this?”, “What if we just found someone extremely talented at solving this particular problem, and they solved it?”.
A common experience at the end of a brainstorming session is someone asking, “okay, who has capacity to do this?” The answer, dear reader, is absolutely nobody at any point in time. Solutions are hard to implement and require dedicated attention. Unless session participants have good context on things like the capacity available, the priorities of those involved, the steps it would take to reach the solution, and the appetite for the solution in the first place, their ideas are likely to be much too ambitious. This makes it all the more likely that the session lead dismisses these ideas outright, and time is wasted.
The ideas are not worth the time cost.
Unless there’s a clear plan for tracking and using the group’s ideas, a lot of brainstorming notes languish in long Google docs or are collected up in the form of colourful sticky notes and chucked into a bin. Even when a few ideas are used, session leads are probably not tracking the person-hours it took to generate those ideas. If you had 20 people in a 30-minute session, and you got a few good ideas for your project, that’s 10 hours or over a day’s work just for a few ideas. Imagine asking a colleague with the average intelligence and context of your session attendees to spend a day trying to make progress on a problem, and then they delivered 5 hastily written sticky notes onto your desk. That sounds like extremely poor returns to me!
Group brainstorming can sometimes go well. In particular, if the session lead genuinely wants a long list of ideas or to get a temperature check on several ideas at once and carefully notes down the suggestions generated, this session format can work. Another benefit is that participants are forced to reflect on an issue; a useful prompt for brainstroming, therefore, is a prompt where everyone has high context and is actually looking to take actions (e.g., “what’s your experience of this problem? what can you personally do about this issue? Tell your group what actions you plan to take”).
Added: group brainstorming is much more likely to go well when high-context people actively choose which topics to prioritize, and comment on Google docs together.
Choosing more valuable content: fidelity per person-minute
A useful heuristic for valuable content is to consider the person-minutes (length of session * attendees) going into each session and the fidelity of information each person is receiving in that time. You should look for high-fidelity information flow relative to each person-minute. Consider:
- Are the people who know what they’re talking about doing the talking? If not, are participants able to steer things until this is the case?
- What proportion of your participants are active in each minute? How many are listening and learning or contributing something useful?
- Is it easy to leave if returns are poor?
1:1s are particularly good. The person-minute cost is low (2 people * length of meeting), and the information flow is very easy to control and manage. You can ask your partner the exact thing you need from them, they can try and give you that information, and if you aren’t finding that useful, you can then talk about something else. An old EA forum post on why 1:1s are great that I strongly agree with:
- 1-on-1s offer a unique opportunity to get personalized feedback and information that is helpful for your top priority questions, uncertainties, career plans, and more, unlike talks and workshops that aren’t specifically catered to you.
- People often think better when talking to others.
- Perhaps most importantly, you get very quick feedback loops on your thinking, which are harder to get in other formats - mistakes, alternatives, and potential improvements to your ideas and their communication can be pointed out quickly
- They offer a uniquely good setting for asking and being asked/answering really good questions.
- It can be easier to stay focused when talking to someone else and being regularly required to actively contribute
- Conversations can force you to refine your thinking and communication of ideas, since your conversation partner needs to understand what you’re saying.
- You can incorporate new perspectives/information from your conversation partner quickly
- Easy accountability for ideas that come up - ideas can turn into concrete plans, which your conversation partner can provide accountability for.
- 1-on-1s can turn into long-lasting friendships, professional connections, references/referrals for job/project/internship opportunities
Talks also look pretty good on this. Well-attended talks are a lot of person-minutes, but every person in that audience except the speaker is getting information from someone who the organisers have hopefully chosen because of their ability to share useful information. On top of this, it’s easy to leave if you aren’t finding it useful. A tangential take is that more people should probably be walking out of talks they aren’t finding useful. If you’re one of three attendees, consider staying for the speaker’s wellbeing, but if you aren’t confident you’ll enjoy the talk, sit near the back and then leave when you want to.
Panels and group brainstorming are not strong on this metric.
- As covered above, panels often lead to low fidelity information being shared: panelists aren’t delivered prepared notes, they’re often riffing on a point that they just heard. On the plus side, panels are easy to leave if you aren’t finding it useful (I strongly encourage more people to leave panel sessions, maybe even the panelists themselves).
- Group brainstorming is weak on both. It involves a lot of person-minutes to brainstorm together, and participants are sharing information with each other on a topic they aren’t familiar with, with people they don’t need to talk to, and without the remit to steer the conversation to whatever would be more useful to talk about. Brainstorming groups often consist of one person with limited context on the topic sharing their poorly-informed takes with several people at once, all of whom would probably benefit more from talking one-on-one with each other or hearing more from the session lead or whoever has more context.
Events, among many other things, are about getting as much value from other people as you can, while you’re in the room with them sharing air and takes and time together. Make it useful!
Interesting. Does anyone do group brainstorming because they actually expect it to make meaningful progress towards solving a problem? At least when you're at a large event with people who are not high context on the problem, that seems pretty doomed. I assumed the main reason for doing something like that is to get people engaged and actually thinking about ideas and participating in a way that you can't in a very extremely large group. If any good ideas happen, that's a fun bonus
If I wanted to actually generate good ideas, I would do a meeting of people filtered for being high context and having relevant thoughts, which is much more likely to work.
A specific format that has worked well for me eg for running research brainstorms for my team is as follows:
This seems to work well on groups of three to nine people. If your group is larger than that, well, firstly, why on earth are you doing a brainstorming meeting of this many people? But if you do need to do this, then splitting up into groups who each think about a different question, and share the most interesting takeaways after each round, might work ok.
Panels are almost always a massive waste of time though, strongly agreed there - I suspect they generally happen as a way to get more fancy people to turn up (by inviting them to be panellists), because there's a hole in the schedule that needs to be filled, or because of copying what events "should" do
[Epistemic status: argument from authority*]
I think your suggested format is a significant upgrade on the (much more common, unfortunately) "group brainstorm" set up that Ollie is criticising, for roughly the reasons he outlines; It does much better on "fidelity per person-minute".
Individual brainstorming is obviously great for this, for the reasons you said (among others).
Commenting on a doc (rather than discussing in groups of 6-8) again allows many more people to be engaging in a high-quality/active way simultaneously.
It also seems worth saying that choosing questions well, which means they are:
is a) necessary for group discussion to be worthwhile, b) difficult, and c) significantly more difficult for a group of mixed ability and context, whom you don't know well. c, of course, applies much more strongly in the context Ollie is primarily concerned with (EAGs and similar events), to the one you're describing (research team meeting).
I think that, almost without exception, if event sessions want to incorporate some discussion, they should start with the 'individual silent thought' exercise you mention, and then expand to pairs (with some 3s to stop people needing to do a lot of rearranging odd numbers of people). There are lots of reasons that this works better than larger groupings, but again Ollie's heuristic of 'fidelity per person-minute' is one. A less obvious one is that minimising the distance between speaker and listener allows conversation volume to be much quieter, and if you think about how volume scales with distance, this more than outweighs the effect of having more people talking simultaneously.
Feeding back to the whole group from these discussions can happen (and be great), but is worth parallelising where possible, e.g. by commenting on a central gdoc as you suggest, or with a Slido, where people are encouraged to submit questions that their paired/small group discussions did not resolve, which the facilitator can then answer, or suggest steps to answer, at the end.
*I spent a ~decade as a teacher, and have facilitated many highly-reviewed workshops during and since, so I feel like I do have reasonable grounds to claim authority, but this is a joke, it just seemed like a funny epistemic status.
I don't know what motivations people usually have, but I also feel skeptical of this vague "activation" theory of change. If session leads don't know what actions they want session participants to take, I'm not optimistic about attendees generating useful actions themselves by discussing the topic for 10 minutes in a casual no-stakes, no-rigour, no-guidance setting. I'm more optimistic if the ask is "open a doc and write things that you could do".
Yep, the thing you've described here sounds promising for the reasons Alex covered :) I realise I was thinking of the conference setting in my critique here (and probably should've made that explicit), but I'm much more optimistic about brainstorming in small groups of people with shared context, shared goals and using something like the format you've described.
I've run very successful group brainstorming sessions with experts just in order to require them to actually think about a topic enough to realize what seems obvious to me. Getting people to talk through what the next decade of AI progress will look like didn't make them experts, or even get to the basic level I could have presented in a 15 minute talk - but it gives me me a chance to push them beyond their cached thoughts, without them rejecting views they see as extremes, since they are the ones thinking them!