Evals

A tool that works is not the same as a tool the model decides to call. DevTools and the playground let you check that by hand; evals check it in a test. @skybridge/test runs a real conversation against your app, in process, and hands you the tool calls the model made so you can assert on them.

Set Up

Add the test runtime, vitest, the AI SDK, and the provider you want to drive the conversation:

npm install -D @skybridge/test vitest@^4 ai @ai-sdk/anthropic

pnpm add -D @skybridge/test vitest@^4 ai @ai-sdk/anthropic

yarn add -D @skybridge/test vitest@^4 ai @ai-sdk/anthropic

bun add -d @skybridge/test vitest@^4 ai @ai-sdk/anthropic

Turn on evals in the Vite plugin. It registers the expect.chat matchers, picks up evals/**/*.eval.ts, raises the per-scenario timeout to two minutes, and loads your .env so the provider key is available:

vite.config.ts

import { skybridge } from "@skybridge/vite-plugin";
import react from "@vitejs/plugin-react";
import { defineConfig } from "vite";

export default defineConfig({
  plugins: [
    skybridge({ evals: {} }),
    react(),
  ],
});

Then add a script to run them:

package.json

"evals": "vitest run evals"

Write a Scenario

A scenario opens a conversation against your app, sends a prompt, and asserts on the tools the model called:

evals/search.eval.ts

import { anthropic } from "@ai-sdk/anthropic";
import { start } from "@skybridge/test";
import { expect, it } from "vitest";
import { app } from "../src/server.js";

it("reaches the search tool from a natural prompt", async () => {
  const chat = await start({ app, model: anthropic("claude-sonnet-4-5") });
  await chat.send("Find me running shoes under 100 dollars");

  expect.chat(chat).toHaveCalledToolWith("search-products", { query: "running shoes" });
});

start serves the app in process: no port, no fixture. Each send is one turn, during which the model can call several tools before it answers. The session closes when the test finishes. This is why the app lives in src/server.ts and run() in src/index.ts: importing the app starts nothing.

Assert on Tool Calls

expect.chat(chat) is typed against your app, so tool names autocomplete and arguments are checked against each tool’s input schema.

Matcher	Passes when
`toHaveCalledToolOnce(name, args?)`	exactly one successful call to `name`, optionally matching `args`
`toHaveCalledToolWith(name, args)`	some successful call to `name` matched `args`
`toNeverHaveCalledTool(name)`	`name` was never attempted
`toHaveFailedToolCall(name)`	a call to `name` was refused or threw
`toHaveSaid(text)`	an assistant turn contains `text` (string or `RegExp`)
`toHaveCalledToolsInOrder(...names)`	the named tools were called in that relative order
`toHaveCalledNoTools()`	no tool was attempted at all
`toPassJudgment(criteria)`	a judge model grades the conversation against `criteria`

toHaveCalledToolsInOrder matches a subsequence, so unrelated calls in between do not break it. Like the other positive matchers it only counts calls that succeeded, so a failed call never satisfies a step. Every matcher supports .not and sees the whole conversation, not just the last send. On failure, the message lists the calls the model actually made, arguments included. chat.toolCalls and chat.assistantTurns are also available for custom assertions.

Judge the Answer

Sometimes it’s not easy to test an agent behavior using on deterministic matchers. For example, did the assistant reply using the right tone? Was it polite? toPassJudgment sends the conversation to model, asking it to judge if the conversation meets the criteria:

evals/trip.eval.ts

it("plans a trip within budget", async () => {
  const chat = await start({ app, model: anthropic("claude-sonnet-4-5") });
  await chat.send("Plan a weekend in Lisbon under 500 euros");

  await expect.chat(chat).toPassJudgment(
    "proposes a concrete plan, stays under the budget, and relies on the tool result rather than inventing prices",
  );
});

The reasoning is the failure message:

expected the conversation to pass judgment:
  "proposes a concrete plan, stays under the budget, and relies on the tool result rather than inventing prices"

judge: FAIL
  The plan totals 640 euros. The assistant quoted hotel prices that do not appear in the search-hotels result.

By default, the judge runs on the chat’s own model. Pass model to use another one:

await expect.chat(chat).toPassJudgment("answers in one short paragraph", {
  model: anthropic("claude-haiku-4-5"),
});

Judge With Something Other Than a Model

Pass judge to grade the conversation yourself. It receives the criteria and the rendered transcript, and returns a verdict. This is how you plug in an evaluation model, which answers a typed question with a probability instead of writing text, or any scoring service you already run: Define it once in its own module, so every scenario that wants it imports the same judge:

evals/typesafe-judge.ts

import type { Judgment, Verdict } from "@skybridge/test";
import { TypeSafeClient, noul } from "@typesafe-ai/sdk";

export async function typesafeJudge({
  criteria,
  transcript,
}: Judgment): Promise<Verdict> {
  const { answers } = await new TypeSafeClient().systemOne({
    state: { conversation: transcript },
    questions: { pass: noul(criteria) },
  });

  return {
    pass: answers.pass.noul >= 0.5,
    reasoning: `P(pass) = ${answers.pass.noul.toFixed(2)}`,
  };
}

evals/trip.eval.ts

import { typesafeJudge } from "./typesafe-judge.js";

await expect
  .chat(chat)
  .toPassJudgment("stays under the budget", { judge: typesafeJudge });

Judgment and Verdict are exported from @skybridge/test, so a judge written this way is typed without repeating the shape. reasoning is optional, and the failure message drops the line when there is none. A judge that returns no prose is cheaper and faster, but a red eval then tells you only that it failed, so it is worth returning the score. By default a judgment is a live model call: it costs money, and it is not reproducible the way a tool-call assertion is. A custom judge costs and varies only as much as whatever it calls, and a local heuristic is free and deterministic. Either way, use it only when no other matcher fits, and keep the criteria narrow so a failure points at one thing. When the judge throws, the assertion throws judge unavailable: <error> instead of failing. That covers a broken provider and a bug in your own callback alike, so neither reads as an app regression.

Stub a Tool Result

A scenario that depends on today’s date or on a live catalogue passes today and fails next month, with no change to your code. Pass stubs to answer a tool from the scenario instead of the app:

evals/flights.eval.ts

const chat = await start({
  app,
  model: anthropic("claude-sonnet-4-5"),
  stubs: {
    "search-flights": ({ from, to }) => ({ flights: [{ from, to, price: 120, departsAt: "2026-04-03T08:15:00Z" }] }),
  },
});

The arguments are typed against your registry, and a stubbed call still lands in chat.toolCalls, so every matcher works on it. Return a string or any JSON value: the runner wraps it the way a real tool result reaches the model. Naming a tool your app does not expose fails the scenario rather than passing silently. Return undefined to fall through to the real server, which is how you pin one set of arguments and leave the rest live:

stubs: {
  "search-flights": ({ to }) => (to === "LIS" ? lisbonFixture : undefined),
}

A stub replaces the call: the tool never reaches your handler, so neither its schema validation nor its scope checks run. Stub the tools whose results move under you, and leave the ones the scenario is actually about.

Test Authenticated Tools

For an app behind OAuth, claim an identity for the session:

evals/checkout.eval.ts

const chat = await start({
  app,
  model: anthropic("claude-sonnet-4-5"),
  authInfo: { token: "eval", clientId: "evals", scopes: ["checkout"], extra: { email: "ada@example.com" } },
});

Only token verification is skipped. Per-tool schemes and scope checks run for real against those claims, and extra reaches your handlers as extra.http.authInfo.extra. Omit authInfo to exercise the anonymous path, auth challenges included.

setup and oauth still resolve on the first request, so an app wired to an identity provider needs its .env to run evals, the same as dev.

Tune the Run

Defaults every scenario starts from go in the plugin option, and any of them can be overridden per start:

vite.config.ts

skybridge({
  evals: {
    temperature: 0,          // unset by default: the model's own default applies
    systemPrompt: "You are a shopping assistant.",
    maxSteps: 8,             // tool-call rounds per turn, default
    timeout: 120_000,        // per scenario, default
  },
})

Evals are live model calls: they cost money and the wording varies from run to run. Set the temperature to 0 when your model supports it (some recent models reject the parameter), assert on the calls rather than on exact phrasing, and reach for toHaveSaid with a loose pattern when the answer itself matters.

Go Further

Skybridge

The app your scenarios import

Playground

Chat with a real model running your app

Register Tools

Write descriptions the model can act on

Attach Input Files

chat.send accepts AI SDK UserContent: either a string or an array of text and file parts. Use the AI SDK’s FilePart directly, and keep that same object for assertions:

evals/resize.eval.ts

import { readFile } from "node:fs/promises";
import type { FilePart } from "ai";

const portrait: FilePart = {
  type: "file",
  mediaType: "image/png",
  filename: "photo.png",
  data: await readFile(new URL("./fixtures/photo.png", import.meta.url)),
};

await chat.send([
  { type: "text", text: "Resize this photo to a vertical 1080x1920 portrait." },
  portrait,
]);

expect.chat(chat).toHaveCalledToolWith("image_crop_and_resize", {
  file: expect.inputFile(portrait),
});

expect.chat(chat).toHaveCalledToolOnce("asset_openai_file_upload", {
  files: [expect.inputFile(portrait)],
});

Declare file parameters with FileRef and list their names in _meta["openai/fileParams"]. Your handler or stub receives the attached file, and its download URL remains available until the conversation closes. expect.inputFile(part) matches the file object minted for that exact FilePart. Two parts with identical bytes do not match each other. Using it under a parameter not listed in openai/fileParams throws a declaration error before calls are compared, including nested matchers, .not assertions, and conversations with no calls. File data can be bytes, base64, a URL, or AI SDK tagged data, URL or text content. Provider file references are rejected because the app under test cannot download them.

Get Started

Resources

Set Up

Write a Scenario

Assert on Tool Calls

Judge the Answer

Judge With Something Other Than a Model

Stub a Tool Result

Test Authenticated Tools

Tune the Run

Go Further

Skybridge

Playground

Register Tools

Attach Input Files

​Set Up

​Write a Scenario

​Assert on Tool Calls

​Judge the Answer

​Judge With Something Other Than a Model

​Stub a Tool Result

​Test Authenticated Tools

​Tune the Run

​Go Further

Skybridge

Playground

Register Tools

​Attach Input Files

Set Up

Write a Scenario

Assert on Tool Calls

Judge the Answer

Judge With Something Other Than a Model

Stub a Tool Result

Test Authenticated Tools

Tune the Run

Go Further

Attach Input Files