Test a plugin with evals

A plugin can load cleanly and still not help. Evals check the part that matters: when someone asks a real question with your plugin loaded, does the answer contain what it should? syntic plugin eval runs a list of prompts against your plugin, scores each answer with simple checks, and writes a report you can open in a browser or read from CI.

How a run works

1. The suite is read from evals/evals.json inside the plugin folder, or from the file you name.
2. Each case’s prompt runs as its own one-shot session: syntic -p --plugin-dir <plugin> "<prompt>". The plugin is loaded exactly as --plugin-dir would load it, and the prompt is answered by your usual model, with your usual settings and permissions.
3. The answer is scored on your machine against the case’s checks.
4. report.json and report.html are written, a summary is printed, and the command exits with 0 if every case passed or 1 if any failed.

Cases run one after another. Each case may take up to five minutes; a case that runs longer is stopped and marked as failed with timed out after 300s.

Write your first suite

release-helper/
  .syntic-plugin/plugin.json
  skills/...
  evals/evals.json

{
  "cases": [
    {
      "name": "suggests the next version",
      "prompt": "We fixed two bugs since v1.4.0. What should the next version be?",
      "expect": { "contains": ["1.4.1"], "not_contains": ["2.0.0"] }
    },
    {
      "name": "slash command explains itself",
      "prompt": "/release-helper:ship --help",
      "expect": { "contains": ["changelog", "tag"], "min_chars": 80 }
    },
    {
      "name": "answers something",
      "prompt": "Summarise what the release helper can do."
    }
  ]
}

syntic plugin eval ./release-helper

pass  suggests the next version  (100%)
FAIL  slash command explains itself  (67%)
pass  answers something  (100%)
2 of 3 cases passed, score 89%. Reports: ./release-helper/evals/results

syntic plugin test is another name for the same command.

How a case is scored

Each check in expect passes or fails on its own:

A case’s score is the share of its checks that passed. A case passes only when all of them do. The run’s score is the average of the case scores.

If the session itself fails (for example, the prompt could not be sent), the case scores 0 and the reason is recorded instead of an answer.

Suite reference

{ "cases": [ { "name": "...", "prompt": "...", "expect": { ... } } ] }

Other fields are ignored.

Command options

syntic plugin eval <path> [--out <folder>] [--json]

Keeping several suites is easy: put them side by side, such as evals/smoke.json and evals/full.json, and name the one you want.

Read the results

HTML report

report.html needs no other files. It shows the overall score, then one section per case with each check marked pass or fail and the answer the session gave. Open it in any browser.

JSON report

report.json has the same data for scripts:

{
  "plugin": "./release-helper",
  "passed": 2,
  "total": 3,
  "score": 0.889,
  "cases": [
    {
      "name": "slash command explains itself",
      "passed": false,
      "score": 0.667,
      "checks": [
        { "check": "contains \"changelog\"", "passed": true },
        { "check": "contains \"tag\"", "passed": false },
        { "check": "at least 80 characters", "passed": true }
      ],
      "output": "…the answer, cut at 4,000 characters…",
      "error": null
    }
  ]
}

Run evals in CI

The exit code is all CI needs:

- name: Plugin evals
  run: syntic plugin eval ./plugins/release-helper --out eval-results
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: plugin-eval
    path: eval-results

The job must be signed in, like any other headless run; see Headless mode. A case that needs a tool which would normally ask for approval cannot get it in a one-shot run, so set the permission rules the cases need in the project’s settings.

Writing good cases

• Test behaviour, not wording. Check for the facts that must be there (a version number, a file name), not for whole sentences.
• Use not_contains for mistakes you have seen. If the agent once suggested the wrong package manager, add a check that it does not.
• Cover each component. One case per command or skill is a good start.
• Expect some variation. Answers differ between runs. If a case fails now and then, loosen its checks rather than retrying until it passes.

Troubleshooting

Related pages

• Create a plugin
• Release a plugin
• Plugin commands