hajer.yaml reference

hajer.yaml declares two things about a repository:

• Suites: the promptfoo configurations hajer eval runs when you give it no -c.
• Obligations: the behavioural requirements a test may claim to cover under metadata.hajer.obligationIds.

hajer eval reads it locally. The Hajer platform reads the same file from the repository's default branch through the GitHub App, so your suites, their tests and your obligations are visible on the Tests page before a run is uploaded.

Full example

hajer.yaml

version: 1

suites:
  - id: support
    path: evals/support/promptfooconfig.yaml
    description: Customer support regression suite
  - id: billing
    path: evals/billing/promptfooconfig.yaml

obligations:
  - id: obl_refund_status_disclosed
    title: A refund's status is disclosed as it is
    workflow: wf_support
    description: The agent reports a pending refund as pending and never claims it completed.
  - id: obl_no_card_numbers
    title: Card numbers are never repeated back
    workflow: wf_billing

Discovery

hajer eval looks in the working directory, then in each parent directory up to the filesystem root. In each directory it checks hajer.yaml first, then hajer.yml, and uses the first file it finds. You can therefore run hajer eval from a service's subdirectory and get the same result as from the root.

Each declared suite runs with the manifest's directory as the working directory.

warning

The platform reads only hajer.yaml, at the repository root. Use that name and location if you want the suites and obligations to appear in the app. A hajer.yml or a manifest in a subdirectory works for local runs only.

Top-level keys

Key	Type	Required	Rules
version	integer	yes	Must be 1. A quoted "1" is refused.
suites	list	no	At most 100 entries. Suite ids are unique.
obligations	list	no	At most 500 entries. Obligation ids are unique.

The file must be a YAML mapping and at most 262,144 bytes (256 KiB).

A manifest with no suites is valid, but hajer eval with no -c then exits with code 2 and asks you to add a suite or pass a configuration.

suites[]

Key	Type	Required	Rules
id	string	yes	Matches ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$: a letter or digit first, then letters, digits, ., _ or -; at most 128 characters.
path	string	yes	The promptfoo configuration, relative to hajer.yaml. See below.
description	string	no	At most 4,096 characters.

path rules:

• Relative to the directory holding hajer.yaml, written with forward slashes.
• No leading /, no .. segment, no backslashes, no leading or trailing whitespace.
• At most 512 characters.
• The file must exist when hajer eval runs.

The platform's scan reads suite files ending in .yaml, .yml or .json. A JavaScript or TypeScript configuration runs locally, but the app shows it as unsupported and cannot list its tests.

obligations[]

Key	Type	Required	Rules
id	string	yes	Same pattern as a suite id.
title	string	yes	1 to 512 characters.
workflow	string	yes	The workflow id the obligation is about, same pattern as a suite id. This is the id your code passes to hajer.workflow(...).
description	string	no	At most 4,096 characters.

When a manifest is in force, hajer eval refuses any test whose metadata.hajer.obligationIds names an obligation the manifest does not declare (E_OBLIGATION_UNDECLARED), and refuses an --obligation flag that names one. This check applies even when you pass -c, as long as a manifest is found.

Strict validation

The schema is strict. Each of these stops hajer eval with exit code 2 before the engine starts:

• an unknown key at any level (suites.0.descripton is not a known key)
• a missing required key
• a value of the wrong type, including a string where an integer is expected
• a duplicate suite id or obligation id
• an id that does not match the pattern
• a suite path that is absolute, leaves the repository, or names a file that does not exist
• a file over 256 KiB, or one that is not valid YAML

The message names the file and the key:

hajer eval: HAJER_INVALID_MANIFEST: /repo/hajer.yaml names a suite file that does not exist: evals/support/promptfooconfig.yaml

The platform validates the file with the same limits, so a manifest that hajer eval accepts is one the platform stores.

How the platform reads it

Once the repository is linked in Settings → GitHub, the platform reads hajer.yaml and every suite file it names from the repository's default branch:

• when you link the repository,
• on every push to the default branch,
• when you click Scan repository on the Tests page.

Pushes to other branches do not trigger a scan. The Tests page shows the scanned commit and the manifest's state: found and valid, missing, invalid (with the reason), or unavailable (GitHub could not be read, or the App is not connected).

From each suite file, the scan reads the tests listed inline under tests and their metadata.hajer, using the same rules as hajer eval (see Test metadata). The scan does not follow a tests: value that points at another file, and it reads at most the first 1,000 tests of a suite. Where hajer eval would stop on a malformed metadata.hajer, the scan still lists the test and shows the problem on its row.