Benchmark Provenance

Benchmark suites usually need more than a prompt and a score. They carry source pins, task patches, generated dataset rows, oracle data, setup scripts, and verification commands. AgentV represents that with existing primitives:

Put prompt matrices in prompts, tests[].vars, default_test, and assert.
Put coding-agent testbed setup in environment.
Put lifecycle callbacks and instrumentation in extensions.
Put provider/eval variables in top-level env.
Put provenance and classification in per-case metadata.
Put bulky per-case authoring inputs in optional case directories and supporting files.
Use generated run folders, not hand-authored source bundles, as the portable audit artifact.

These are documentation patterns, not special runtime schema keys. AgentV does not interpret metadata keys such as source_commit, test_patch, or question_type unless your setup command, lifecycle extension, or custom assertion reads them.

Operational Vs Informational Fields

Use this split when deciding where a benchmark key belongs:

Field area	Operational?	What AgentV does
`environment`	Yes	Prepares the host or Docker testbed, including workdir, setup argv, fixtures, services, and repository checkout scripts.
`environment: file://...`	Yes	Loads a reusable environment recipe from a field-local file reference.
`environment.setup.command`	Yes	Runs the argv command that materializes repo state, installs dependencies, builds fixtures, or prepares services.
`extensions`	Yes	Runs Promptfoo-style lifecycle callbacks after the environment is prepared.
Top-level `env`	Yes	Supplies provider/eval variables and template inputs.
`prompts`, `tests`, `vars`, `default_test`	Yes	Builds the authored matrix of prompts, cases, and row data.
`assert`	Yes	Runs deterministic, LLM, assert-set, or script graders.
Top-level `name`, `version`, `tags`, `license`, `requires`	Informational	Identifies and categorizes the suite.
`tests[].metadata`	Informational to AgentV	Passes arbitrary case data through to results, setup payloads, and extension context.

metadata can still become operational inside your own setup. For example, a beforeEach extension can read case_metadata.test_patch and apply that patch before the target starts. The distinction is that AgentV itself only passes the metadata along; your extension owns the behavior.

Extension Context

File lifecycle extensions export functions named beforeAll, beforeEach, afterEach, or afterAll. AgentV calls each function with context including the current test’s metadata as case_metadata:

{
  "workspace_path": "/home/user/.agentv/workspaces/run-123/case-01",
  "test_id": "case-01",
  "eval_run_id": "run-123",
  "case_input": "Fix the bug",
  "case_metadata": {
    "source_commit": "4f3e2d1",
    "test_patch": "cases/case-01/test.patch"
  }
}

beforeAll runs after the authored environment is prepared, so it should do suite lifecycle work only. Use beforeEach when setup depends on per-case metadata such as a patch path, source row, or selected test list.

Task Artifact Anatomy

Benchmark task packs map cleanly onto AgentV fields at authoring time:

Task artifact	AgentV pattern
Prompt or instruction	`prompts` rendered with `tests[].vars`
Source checkout	`environment.setup.command` materializes the checkout from argv inputs
Per-case setup	`extensions: ["file://scripts/setup.mjs:beforeEach"]` reading `case_metadata`
Gold answer or reference context	`expected_output` when the data is passive grader context
Active verification	`assert`, especially `script` for commands or artifact checks
Provenance	`tests[].metadata` with source pins, generator rows, and curation labels
Bulky task files	`tests: file://./cases.yaml` or directory cases with supporting files

Use this separation only when it makes the source eval easier to maintain. It is not a first-class artifact schema. After an eval runs, AgentV writes the portable audit surface into the generated run folder: each result can link from index.jsonl to a run-local test/ bundle containing EVAL.yaml, targets.yaml, and copied files/ or graders/ snapshots where applicable. Review, Dashboard files views, and rerun workflows should inspect those generated run artifacts instead of requiring authors to maintain a parallel source-side bundle layout. See Generated Test Bundles.

SWE-Style Case

A SWE-style benchmark usually needs a source repo, a commit pin, a patch that adds or selects tests, and a list of failing tests that should pass after the agent’s fix. Keep the checkout operational under environment; keep benchmark provenance and per-case test selectors in metadata.

name: swe-style-regression
description: Regression tasks against pinned source commits.

environment:
  type: host
  workdir: ./workspaces/widget
  setup:
    command:
      - bash
      - ./scripts/materialize-repo.sh
      - ./workspaces/widget
      - https://github.com/example/widget.git
      - 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
    cwd: "."
    timeout_ms: 120000

extensions:
  - file://scripts/apply-test-patch.mjs:beforeEach

assert:
  - metric: focused-tests
    type: script
    command: ["python", "./graders/run-focused-tests.py"]
    required: true

prompts:
  - "{{ task }}"

tests:
  - id: widget-1234
    vars:
      task: |
        Work in the prepared widget checkout. Fix the parser regression
        described by the failing tests. Do not change unrelated public APIs.
    metadata:
      repo_url: https://github.com/example/widget.git
      source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
      test_patch: cases/widget-1234/test.patch
      fail_to_pass_tests:
        - tests/parser.test.ts::handles-empty-widget
        - tests/parser.test.ts::preserves-widget-id

In this example, the setup argv is the operational checkout instruction. The matching metadata.source_commit is audit data that gets recorded with the case and is available to extensions. apply-test-patch.mjs can read case_metadata.test_patch and case_metadata.fail_to_pass_tests, then apply the patch and write the selected test list into the prepared workdir. The script grader can read that state through its workspace_path payload. See Environment Recipes.

Native AgentV Vs Harbor-Backed Benchmarks

Use native AgentV environment recipes for repo-backed evals where AgentV should own the run lifecycle: prepare host or Docker testbeds, run targets, execute extensions and graders, gate CI, and write AgentV result bundles. This fits custom internal suites, target comparisons, narrow regression suites, and CI checks built from AgentV primitives.

name: repo-regressions

environment: file://.agentv/environments/widget.yaml

targets:
  - id: codex-host
    provider: codex-cli
    runtime: host
  - id: claude-host
    provider: claude-cli
    runtime: host

extensions:
  - file://scripts/apply-case-fixtures.mjs:beforeEach

prompts:
  - "{{ task }}"

tests:
  - id: parser-regression
    vars:
      task: Fix the parser regression in the prepared checkout.
    assert:
      - type: script
        command: ["python", "./graders/run-tests.py"]
        required: true

Use a Harbor-backed runner for standard benchmark suites Harbor owns, such as SWE-Bench Verified, Multi-SWE-Bench, Terminal-Bench, or suites with Harbor-owned Docker and Compose adapters. In that path AgentV should stay at the orchestration boundary: launch or import the Harbor job, apply AgentV gates to the imported results, and link external traces when the runner emits them.

# Proposed runner boundary, not a current AgentV task schema.
name: swebench-verified-codex

target: codex-gpt5-mini
runner:
  type: harbor
  options:
    trace_links: true

Do not translate Harbor task.toml, verifier packaging, or suite-specific Docker/Compose adapter fields into AgentV core eval schema. If the benchmark’s runtime contract is already owned by Harbor, keep those details in Harbor and let AgentV consume the job metadata, rewards, artifacts, and trace links. Do not add a generic top-level source field just to identify Harbor. If a future Harbor adapter needs suite selection, keep that selector narrow and adapter-owned instead of making it the AgentV environment model.

Finance-Style Generated Dataset

Generated datasets often need stable row provenance more than repository setup. Keep the generated row identity in metadata, use expected_output for the gold answer, and score with rubrics or an LLM/script grader.

name: finance-research-generated
description: Generated finance research cases with row-level provenance.

prompts:
  - "{{ question }}"

assert:
  - metric: answer-quality
    type: llm-rubric
    prompt: ./graders/finance-answer.md
    required: true

tests:
  - id: finance-agent-row-0042
    vars:
      question: |
        Research the company filing and answer:
        What drove the year-over-year change in gross margin?
    expected_output:
      - role: assistant
        content: |
          Gross margin improved because product mix shifted toward higher-margin
          software revenue while fulfillment costs declined.
    metadata:
      source_repo: https://github.com/example/finance-research-dataset.git
      source_commit: 05b8b2e9f071e8d0a6f1c2b3d4e5f60718293abc
      source_file: data/generated/finance_agent.csv
      source_row: 42
      question_type: margin_analysis

Here, source_repo, source_commit, source_file, source_row, and question_type are informational metadata. They support audits, slices, and regeneration checks. If a hook or grader needs the source file at runtime, materialize it through environment.setup.command or make the generator output available as a normal fixture file.

Optional Source-Side Case Directories

Inline YAML is fine when a case has a short prompt, a short expected answer, and a few metadata fields. Move source inputs into case directories only when the benchmark starts accumulating bulky authoring resources:

The case has patches, hidden tests, oracle JSON, screenshots, reports, or fixture files.
The prompt or expected output is long enough that YAML diffs become hard to review.
Each task needs different supporting setup files.
A generator emits many rows and reviewers need to inspect individual cases.
Hook and grader scripts need stable file paths for per-case resources.

Use an external YAML or JSONL file for many simple generated rows:

name: generated-finance
tests: file://./cases.jsonl

Use case directories when each case needs supporting files:

swe-benchmark/
  EVAL.yaml
  .agentv/
    environments/
      widget.yaml
  cases/
    widget-1234/
      case.yaml
      prompt.md
      test.patch
      oracle.json

name: swe-benchmark
environment: file://.agentv/environments/widget.yaml
tests: file://./cases/

id: widget-1234
vars:
  task:
    - role: user
      content:
        - type: file
          value: cases/widget-1234/prompt.md
metadata:
  repo_url: https://github.com/example/widget.git
  source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
  test_patch: cases/widget-1234/test.patch
  oracle_file: cases/widget-1234/oracle.json

When tests points to a directory, AgentV discovers each immediate subdirectory’s case.yaml and uses the directory name as id if no id is set. File blocks still use the normal eval-file search roots, so include the case directory in paths such as cases/widget-1234/prompt.md. Metadata paths are not resolved by AgentV; resolve them in your hook or grader script.

Authoring Rules

Do not add benchmark-specific fields when metadata plus setup commands, hooks, or custom assert entries can express the need.
Do not duplicate operational checkout state only in metadata. Put the real checkout under environment.
Keep metadata snake_case because it crosses process and result boundaries.
Prefer expected_output for passive gold answers and script for active commands, file checks, or generated artifact validation.
Prefer case directories over long inline YAML only for bulky source inputs; the generated run folder remains the portable artifact contract.