Ship Real Agents: Hands-On Evals for Agentic Applications — Laurie Voss, Arize

Next, we will make our first API call. My API keys were retrieved using Colab Secrets, but you can paste them directly since you’re the only one viewing the screen. The two critical lines of code are to import Phoenix.otel and call the register function. You provide a project name, which determines how it appears in Phoenix, and set auto_instrument to true. This command enables logging by accessing the internal processes. An error message may appear because it is using a span processor, which is suitable for demonstrations but not for production environments. In production, you would want to use a batch processor to handle the thousands of spans generated, but this detail is not a concern for our current demonstration.

This is what a code eval looks like. We use the create evaluator decorator to assign a name, which will be sent to Phoenix, and specify the kind as code to indicate that this is a simple deterministic eval that does not require LLM execution. The eval is implemented as a standard Python function, though TypeScript can also be used.

In this case, I employed a basic regex to search for capitalized phrases while excluding non-ticker phrases, ultimately looking for the stock ticker in the output text. This straightforward approach serves as an effective first line of defense: did it mention the company I inquired about?

To run the eval, we evaluate the data frame by providing the evaluator, which is the mention sticker function we just defined, along with the data frame of parent spans we identified. Notably, we are using suppressed tracing with Anthropic for the evals.

For instance, you can deterministically test JSON parsing, length limits, and forbidden phrases like “as an AI language model, I cannot.” A code evaluation can go beyond simple string operations; it can query a database to verify product pricing or call an API to check stock prices. Any evaluation with grading yields consistent results.

When writing a code evaluation, focus on the output produced by the agent rather than the path it took to arrive at that output. Avoid checking for all the steps you believe the agent should have taken to reach the expected answer. Instead, simply verify whether the answer is correct. Additionally, be flexible in parsing strings. For example, if you request a time estimate, the agent might respond with “two hours,” “120 minutes,” or a large number of seconds, all of which are valid representations of two hours. Your code evaluation should accommodate all these variations. Ultimately, the correctness of the answer is what matters, not the specific path the agent took to get there.

The correctness eval checks whether a response is factually accurate, complete, and logically consistent. This eval is pre-built, and you can inspect the prompt we created to ensure transparency. We also offer evals for tool selection, assessing whether the agent chose the appropriate tool, and tool invocation, verifying if it passed the correct arguments. Furthermore, we have built-in evals for document relevance and refusal detection, covering various checks needed in a real eval without requiring you to create them from scratch. Our primary goal is to determine correctness, so let’s set up the judge using the built-in correctness evaluator. First, we need to provide it with an LLM.

For our actionability judge, I’ve used a span query to filter the same examples as before, focusing only on the human actionable ones. I transformed the complicated attributes that input the value into input and attributes that output the value into output, as this aligns with what our evaluation expects. Now, we can determine whether our agents agree or disagree with the annotations regarding human actionable items. In this case, there were disagreements in two out of six instances between my human actionable label and the actionable label.

I must admit that I randomly assigned human actionable versus not actionable labels to generate real data, as I only had six examples, which is insufficient for a robust analysis. Ideally, a real set would contain 20, 50, 100, or even 200 examples. However, this exercise provides insight into how my human judgment aligns with the judgment of the LLM. This is a test of my golden data set against the LLM as a judge, allowing us to identify discrepancies.

This leads us to rubric iteration. Just as we can enhance our agent by refining the prompt, we can also improve the LLM as a judge through iteration. To achieve this, we need to consider precision and recall, which are terms often used in machine learning. They are straightforward concepts. For instance, consider a spam predictor that determines whether an item is spam. You can compare its predictions against the actual spam status. There are four possible outcomes: it can correctly identify spam (true positive), incorrectly label something as spam (false positive), correctly identify non-spam (true negative), or fail to identify spam (false negative).

The second measure is recall, defined as the percentage of true positives out of the total real positives and misses. To increase recall, you need to minimize false negatives. A relevant example is cancer screening, where it is vital to have as few misses as possible. In this case, you may tolerate a higher number of false positives to ensure that you do not overlook someone who actually has cancer. These two measures will typically move in opposite directions when optimized, so you must choose one to focus on.

A common question is how many samples are needed. I’ve mentioned numbers like 50, 100, 200, and 400 samples. You don’t need to estimate this subjectively; you can apply mathematical principles. If you aim for an agent that fails only 5% or 3% of the time, using 200 samples with a 3% defect rate will yield a 95% confidence interval, which could range from 0.6% to 5.4%.

So, who should write these evaluations? As mentioned earlier, involving non-technical stakeholders is crucial, as they have a clearer understanding of what constitutes good and bad outcomes for evaluation purposes. Product managers, customer success representatives, and salespeople can all contribute to evaluation tasks, enhancing the quality of your evaluations.

One approach to making these trade-offs concrete is cost-normalized accuracy, a term from Google. It is calculated as accuracy divided by cost. For instance, an agent that is 92% accurate at two cents per query may provide better value than one that is 95% accurate at 15 cents per query. Evaluations will help determine whether that trade-off is worthwhile.

Next is pairwise evaluation. Instead of asking the agent to rate something from one to ten, which it struggles with, you can present two examples and ask it to compare them. This method is more effective because it provides two concrete examples for comparison. It is particularly useful for A/B testing prompt versions or model upgrades.

Finally, there is reliability scoring. I mentioned pass at K, which assesses whether the agent can succeed at least once in K tries. In contrast, pass to the power of K evaluates whether it can succeed every time in K tries. As K increases, these two reliability measures diverge significantly: pass at K approaches 100%, while pass to the power of K approaches zero. The choice between them depends on your use case. For instance, a coding assistant that eventually produces a correct result is suitable for pass at K, as you can keep trying until it works.

Regarding your question, I understand you work with a construction AI company focused on architectural checks and compliance. You face challenges with numerous checks from architects and are trying to scope them to determine what needs to be built, especially given the lack of labeled data. You’ve started using Cloud Code to automate this scoping and assess consistency by checking if the same solution is produced multiple times. This consistency serves as an evaluation for the complexity of the problem. You refer to these as meta evaluations, which help in addressing ongoing problems.

Yes, we previously discussed meta evaluations. Using an LLM to evaluate another LLM’s output is a classic example of this. You can leverage an LLM to determine which of several agent configurations is superior. This approach allows for a closed-loop system where an agent generates multiple prompt variations and tests them simultaneously against the evaluation to identify which variation yields the best results without human intervention.

Does that answer your question?