Note: This was originally posted on my personal site (see linkpost), and lightly adapted to fit into the forum format here.
Summary
I think automated alignment evaluations, from tools like Petri, do have a place when used very carefully, and can be further improved to increase their trustworthiness and effectiveness.
The disadvantages center around variability between model responses, and risk of not detecting an alignment issue. These are addressed through introducing multiple models into the auditing and judging automations, ensuring humans remain in the loop, and using the tooling consistently.
Background
This is the first in a potential multi-part series where I think through ways to alleviate AI model evaluation bottlenecks to ensure we can stay up to date with assessing the latest and most powerful models.
Disclaimer: I'm writing this as if Petri was just announced, and I was assessing it from within a frontier lab or independent organization. I'm also writing this as a relative outsider, about a month into my AI Safety and Evals journey, after completing the BlueDot AGI Strategy course, and getting myself up to speed on the domain. I'm far from an expert on this topic, but everything I'm writing below is my original thinking and reasoning. In preparing this post, I did find a couple of posts related to automated alignment evaluations ( We cannot safely automate value alignment evaluation and research without thinking about delegation and discretion — EA Forum and Automated Alignment is Harder Than You Think — LessWrong), but haven't fully analyzed them yet, and chose not to further edit my work to respond to them. That could be a future piece though! Ultimately: I could be repeating what's been said many times, or be off-base in some areas (and would be happy to discuss and evolve my understanding).
Petri is part of a larger suite of Inspect tools supporting frontier AI evaluations.
Below are descriptions provided about Petri, from the orgs working on it. I haven't examined other similar tools yet, whether openly available, or internal only.
"Petri is a collaboration between Meridian Labs and the UK AISI Red Team, and is based on work originally done by the Alignment team at Anthropic." (source: Petri Documentation).
"Petri (Parallel Exploration Tool for Risky Interactions) is our new open-source tool that enables researchers to explore hypotheses about model behavior with ease. Petri deploys an automated agent to test a target AI system through diverse multi-turn conversations involving simulated users and tools; Petri then scores and summarizes the target’s behavior.
This automation handles a significant part of the work that one needs to do to build a broad understanding of a new model, and makes it possible to test many individual hypotheses about how a model might behave in some new circumstance with only minutes of hands-on effort.
As AI becomes more capable and is deployed across more domains and with wide-ranging affordances, we need to evaluate a broader range of behaviors. This makes it increasingly difficult for humans to properly audit each model—the sheer volume and complexity of potential behaviors far exceeds what researchers can manually test." (source: Anthropic Announcement)
From everything I can see about AI evaluations, we are indeed facing a large shortage of headcount and bandwidth to perform model evaluations. As stated in the summary, I think there is a place for automated evaluations when done carefully.
How to Use Carefully As-Is
By understanding and planning around the limitations below, Petri can be used for automating some evaluations. But this isn't a substitute for human judgment, or the need to slow down AI model development and release timelines overall so these evaluations, in all aspects, can catch up.
- As new models are readied out of training, Petri could automatically be run very early to quickly fail broken models, major regressions, etc., without needing any human testing.
- Petri is never the final decision maker for a model's release or deployment. It is an input into that process.
- There must still be humans in the loop. Ideally there is a formal playbook and process to define this, with someone overseeing the program aspects of it:
- Choosing which models are eligible to audit and judge.
- Generating new seeds or hypotheses.
- Reviewing the test scenarios, and specific prompts generated by an auditor model.
- Reviewing the judgments and scores generated by a judge model.
- Running full evaluations end-to-end, even if they are simpler, less interesting, etc.
- Auditing end-to-end automated evaluations.
- Performing meta-analysis on the effectiveness of the evaluations, if they are saturating, if the models are aware, cheating, etc.
Advantages of an Automated Evaluation
- Ability to generate and then execute a higher count and wider variety of permutations of scenarios, which could expose wrinkles missed in simpler testing that may only be able to take fewer novel approaches to each hypothesis.
- Frees up time and energy spent on simpler evaluations to be spent on bigger bets, more complex issues, developing newer evaluations, etc.
- Also reduces some of the mundane administrative tasks surrounding the core evaluation like logging the inputs, responses, scores, etc.
- When used early in pre-deployment testing, it can quickly fail a model with low human investment.
Disadvantages of an Automated Evaluation
There's a wide variety of reasons Petri's results can't be fully trusted without humans in the loop:
- Alignment issues of the auditor models - Scheming or alignment faking, especially if the auditor is aware that it's performing an evaluation. That could lead to it favoring its own family of models by asking easy questions or avoiding known gaps, or to it punishing another family of models. Or it could try to help out all models.
- Alignment issues of the judge models - Even with explicit instructions, there's a risk that the judging model doesn't agree that the audited model's behavior is actually a problem or otherwise ignores some instructions.
- Carries a similar alignment faking risk to auditors: if they know they're performing an evaluation, they might choose to protect all models, or judge certain models harshly.
- Score Reliability - Numerical scores from a judge model are noisy in two ways. The same judge on the same input can score differently between runs, and separately, any given judge carries its own blind spots and biases that show up the same way every time.
- Very powerful models being evaluated could overrun Petri:
- Cheating - Already a big problem in human evaluations, and it might be easier to trick judge models, especially when simpler models are used.
- Sandbagging - The model knows it's an evaluation and that it needs to be on good behavior so it doesn't respond how it normally would. I predict this is hard to detect automatically.
- Escaping (See Mythos and GPT-5.6 Sol testing incidents) - Test environments for powerful models need to be incredibly strong.
- Open-Source Risk - Models could cheat and look for exploits as Petri and its documentation are open source and freely available on the web.
- As automated evaluations improve over time and fewer issues are found with them, it risks humans missing the quality checks as they fall into a bit of a complacency trap.
- Currently Petri is analyzing the raw thinking logs, but these are not the complete picture of what's going on inside a model's response.
A quick note on the above: Petri does evaluate 35+ dimensions including ones like unfaithful_thinking, unprompted_leaking, or unprompted_blackmail. However, there's a risk that the automated models do not pick up on these or intentionally ignore them.
Improving and Expanding Petri's Automated Evaluations
After spending years as a product manager, I couldn't resist thinking about how to improve Petri, the Inspect suite and tools like it.
Standard warning from that same product manager: this list is not validated because I haven't had a chance to speak with actual users to understand their pains or challenges, and I have not done a thorough analysis of the ecosystem that Petri sits within.
The deeper I got into writing this section, the more these ideas started sounding like my old job: judge score variance, false positive discipline, curating a metric set that survives a large userbase, and playbooks that keep a decentralized practice from drifting. I spent 3.5 years on those and more supporting an online A/B experimentation platform and program, and they transferred over a lot more cleanly than I expected.
If I could only build one thing on this list, it would be the first one below: running more than one model against the same evaluation. I picked it because I see untrustworthy scoring as the largest potential risk from my list above, and it's likely a simpler build than the others (although it would cost more compute to run).
Putting more LLM-eyes on the Problem
It's always best practice to have reviews of proposals and implementations to check for simple mistakes, and allow for different ways of thinking to contribute, so why should automated auditing and judging be any different? Using multiple auditors or judges builds trust that differences between models and the variability of the same model on the same inputs are made visible.
- Multiple judges per evaluation - Implementing multiple judges reduces the risk of a single judge's miss causing a bad evaluation outcome. Note: There are more permutations of these, existing LLM-as-a-judge scenarios could be reviewed, and then studied in this context to find which might be most effective. I'm thinking of this type of study as a potential side project.
- Option 1 - Pairwise - one judge argues for passing the category, another judge argues for failing it, and then an arbiter judge makes the call. Likely the two debating models need to be the same to make it fair, but the arbiter could be the same or different.
- Option 2 - Multiple Independent - three (or more) independent judges independently assess the results without being instructed to argue for it, and then:
- 2.1 - Aggregate their scoring and evaluation
- 2.2 - Record each separately for meta-analysis (Note: This is already somewhat supported through --action append, but my suggestion is to automate this looping and the capture of each judge's results)
- Option 3 - Judging as an auditor post-hoc - Could potentially use an even more powerful model than the judge, and doesn't run on every evaluation. Have it assess the logic, etc., of the judge, and flag anything concerning.
- Mixing auditor and judge pairs - Similar to above, but flip the roles of auditor and judge. For example: Pass 1, Model A is the auditor and Model B is the judge; Pass 2, Model B is the auditor and Model A is the judge.
Bringing Humans in and Supporting Them
Right now Petri has a limited UI/UX, although based on open positions for two front-end developers, it looks like that might become a focus. These types of features make it easier for humans to support any part of the evaluation and do it consistently.
- Natively supports humans-in-the-loop:
- Running Evaluations End-to-End - Can take care of the mundane tasks that don't need expertise and aren't exposed to the risks above.
- Running parts of the evaluation - for example, automated scenarios run, but a human performs the evaluation instead of or in addition to a judge.
- Supports auditing and reviewing anything, and logging those results.
- Note: it looks like bits of these are supported like dimension validation, although not currently a primary feature.
- UX nudges toward and puts guardrails around scientific best practices - Providing a strong tool and a playbook is a great start, but human users may still drift in their usage, even when intending to use it in a bona fide way. A strong UX guides the user to the best practices, but also allows them to be overridden, because there may be corner cases that are true exceptions. Examples: Limiting false positive rates by warning against evaluating too many metrics, or overriding a very close LLM judgment score.
Creating Richer and Stronger Datasets
These support building stronger inputs and metrics, to get a clean reading at a single-test level, then go further to think about how to best support the meta-reporting to understand the body of work, and the meta-analysis to understand overall effectiveness and trends across many tests.
- Quantitative Metrics - Deeper instrumentation of objective events to create advanced metrics that describe exactly what happened, not just what was revealed in thinking. These go deeper than scraping the thinking logs, although that could yield some early wins too. Examples: counts of how many paths were explored, number of turns per path, how many rejected ideas occurred, latency, time spent, number of tokens spent, if sub-agents or tools were used, etc. This may be outside of purely Petri's scope, may land somewhere between a sister-tool to extract the metrics while inference is running, and needs to be coordinated with the frontier labs to expose this kind of access.
- Shared databases of seeds, prompts, responses, judge evaluations, metric definitions, metric results, etc. - Enable more consistency and broaden the horizons within an org, or even with the broader community. The database, meta-analysis, etc. don't necessarily have to be a part of Petri, but Petri does need the ability to cleanly log results to it.
- Optional sharing of inputs and definitions like seeds, prompts and metric definitions allows for sharing of best practices of measurements.
- Optional sharing of results allows meta-analysis across many tests run by many people, orgs and models.
- Support hierarchy, permutations/variations of designs/prompts - Product work is oftentimes organized under a theme or goal, and then broken down into smaller and smaller pieces until the smallest individually testable pieces are reached. I would imagine evaluation topics are similar. This creates linkage between these ideas to describe how the smallest bits of work roll up to the largest themes or goals. This can be helpful for reporting, and can also be helpful for automating meta-analysis of results. Additionally at the granular test-level this could also help automate and support creating multiple variants of the same testable idea to tease out more wrinkles than a single test might.
Hardening and a Question of How Open
Petri itself has to play its part in ensuring fair and secure testing.
- Security hardening - Petri itself has to continuously be hardened and certified against vulnerabilities that could become a vector of escape for a powerful model during evaluations.
- Gated Access? - I truly appreciate why Petri and its documentation are made easily accessible and open source, allowing anyone to analyze it, improve it and use it on any project. That openness is also a big part of why it has spread across the ecosystem, so gating anything would cost something real. However, parts or all may need to become gated so that it can become a black box to models being evaluated.
Improvements to Automated Evaluations as a Program or Practice
Petri is a tool, and although it automates a lot, ultimately it's operated by humans with different backgrounds and opinions, which can introduce variability into how testing is performed. One answer to this is to tightly centralize under a few practitioners, which can work within one organization, but struggles to scale. Typically it then starts to decentralize into groups that still work dotted-line to a core team, or can become completely decentralized. What I experienced at Expedia Group was decentralized practices that had drifted so far that they started to cause serious challenges to the usefulness of the outputs, in addition to actually slowing down testing and learning.
A quick note on tool features vs program: A lot of times the tool and the program or practice have to evolve together. I listed a few important things in both sections, but others I didn't duplicate. Further, there could be research preceding either to validate approaches like multiple judges for their efficacy and usefulness, and then to guide on how they should be implemented as a practice.
Assuming I was working within an evaluations org, if I could only implement one thing in this list, it would be developing the shared playbook. Even if not initially integrated into the UX, this sets the standards that everyone can align their work to, which is a big boost to the credibility of evaluators outputs.
Some early thoughts on this side of the work:
- Core theme: humans have to stay vigilant - build this into the programs, documentation, etc. It should always be on people's minds.
- Make it scientific! Develop shared playbooks - define things like thresholds, confidence intervals, false positive rates. Don't let users treat it like the wild west. Be wary of post-hoc theorizing and other pitfalls we commonly see in orgs with decentralized experimentation programs.
- Shared and evolved metrics - Creating good metrics is incredibly hard to do well. Petri already supports custom metrics. Create a community effort around creating, curating and deprecating the most informative metrics.
- Decision Metrics - Creating "Overall Evaluation Criterion (OEC)" metrics that aggregate multiple metrics to generate a high-level answer.
- Augment automated testing with non-researcher human testing - Add more humans into the evaluations, but they don't all need to be top-notch researchers. Skill up and enable these people to perform trustworthy evaluations. Doing so will not only increase research bandwidth, but also bring diversity of ideas and analysis. This is another topic I plan to write on soon.
What I experienced at Expedia Group, and what I suspect could happen in alignment evaluations, is that the tooling and its automations are expected to carry the full load of producing high quality outputs. That can't work without people following shared policies and processes. What we found is that the way to bridge the two is a product that guides users toward established best practices while still letting them operate outside of them, because it's impossible to account for every situation in code.
