Dangers may arise in sequential steps that are made using the agent's tools. The steps may involve such processes as planning, tool choice, parameter definition, tool execution and result acquisition.
Simplifying the problem:
Are safety issues prone to occur at particular steps of the sequence? Then maybe not all of the steps require to be carefully monitored.
Hypothesis
It seems that safety risks will only appear during a number of the agent's behavioral steps.
For instance, the agent will behave dangerously while planning or will face issues during parameters definition for the tool.
However, the critical part about my hypothesis is the fact that I am uncertain whether these issues really occur at certain steps or they could be evenly spread across the process.
How would I verify this hypothesis?
To verify this hypothesis I would divide the agent's behaviors into clearly defined steps:
planning → tool choice → parameters generation → execution → output
For each safety failure, the questions I would ask are:
How early could this particular safety violation have been detected based on the information available at the time?
Then I would determine the number of violations within the most common stages.
The basic question I am interested in is whether observing the stages provides greater coverage of safety than observing all stages.
A Problem
I developed a prototype for testing this hypothesis. However, there was an issue with the prototype that I did not consider sufficiently ...the monitor considered an intended capability failure a safety failure.
Conclusions
- This is quite unlike the case where the agent itself made an error or acted unsafely.
- When the distinction cannot be made reliably by the monitor, any metric labeled "safety failures" loses its meaning.
Thus, I do not consider the prototype as evidence of my hypothesis.
What would disprove the hypothesis?
I would have less confidence in my theory if:
- there is a relatively even distribution throughout all phases;
- there is a lack of uniformity in the level of concentration between various models or functions; or
- there is no considerable reduction in cost due to focused monitoring.
For now, I have not demonstrated that the agent’s failures in safety issues exhibit Pareto distribution. The important question is whether or not there is such a distribution.
