Running in CI

Run hajer eval --upload in CI the way you run your tests. A failing test fails the job, and every run, with its commit, branch and pull request, appears on the repository's Tests page.

GitHub Actions

1. Add the secrets

In the GitHub repository, open Settings → Secrets and variables → Actions and add:

• HAJER_API_KEY: a team API key from Settings → API keys in the Hajer app.
• HAJER_TEAM_ID: the team id shown with the key.

Add any keys your providers or graders need (for example OPENAI_API_KEY) the same way.

2. Add the workflow

.github/workflows/evals.yml

name: evals

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7

      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"

      - uses: actions/setup-node@v7
        with:
          node-version: "22"            # the engine needs Node.js 22.22.0 or newer

      - name: Install hajer
        run: pip install "hajer[evals]"

      - name: Read the hajer version
        id: hajer
        run: echo "version=$(python -c 'import hajer; print(hajer.__version__)')" >> "$GITHUB_OUTPUT"

      - name: Cache the eval engine
        uses: actions/cache@v6
        with:
          path: ~/.cache/hajer/engine
          key: hajer-engine-${{ runner.os }}-${{ runner.arch }}-${{ steps.hajer.outputs.version }}

      - name: Install the eval engine
        run: hajer eval --install-only

      - name: Run evals
        run: hajer eval --upload
        env:
          HAJER_API_KEY: ${{ secrets.HAJER_API_KEY }}
          HAJER_TEAM_ID: ${{ secrets.HAJER_TEAM_ID }}
          # OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Pin hajer in your requirements (for example hajer[evals]==0.2.0) so the engine version only changes when you choose to upgrade.

How the pieces fit

The engine cache. Each hajer release pins one promptfoo version and lockfile, and installs it under ~/.cache/hajer/engine/<lockfile digest>/. Keying the cache on the installed hajer version restores the right install, and a new hajer version gets a new key. With a cache hit, the run does not contact the npm registry. See The engine.

The warm-up step. hajer eval --install-only is optional; hajer eval installs the engine on its own when it is missing. A separate step keeps install problems (exit 3 for Node, 4 for the install) apart from test failures in the job log. It also prints the engine's status as JSON.

Failing the job. hajer eval exits 100 when any test fails or errors, and GitHub Actions fails the step on any non-zero exit code. To report results without blocking merges, add continue-on-error: true to the step. See exit codes.

The upload never fails the job. A refused or unreachable upload is reported on the summary line and stderr, but the exit code stays the engine's. Pull requests from forks do not receive repository secrets. Their evals still run and gate the pull request; the upload is reported skipped (INERT).

Pull request context is automatic. hajer eval reads GITHUB_SHA, GITHUB_REF, GITHUB_HEAD_REF, GITHUB_BASE_REF, GITHUB_RUN_ID and GITHUB_REPOSITORY, among others, and records the commit, the branch, the pull request number, the base branch and the workflow run. On a pull_request event, the commit is the merge commit GitHub checked out and the branch is the pull request's head branch. GITHUB_REPOSITORY is how the platform matches the run to your linked repository. See Git and CI context.

Pushes to main update the Tests page. A push to the default branch also makes the platform rescan hajer.yaml, so new suites and obligations appear on their own. Runs on the default branch are marked as such.

Monorepos

hajer eval finds hajer.yaml by walking up from the working directory, so you can run it from any subdirectory. To run one suite per job, use --suite:

    strategy:
      matrix:
        suite: [support, billing]
    steps:
      # ... the setup steps above ...
      - run: hajer eval --suite ${{ matrix.suite }} --upload
        env:
          HAJER_API_KEY: ${{ secrets.HAJER_API_KEY }}
          HAJER_TEAM_ID: ${{ secrets.HAJER_TEAM_ID }}

Parallel jobs on one machine that start with a cold cache do not corrupt each other: the engine install takes a file lock, and the second job waits and reuses the first job's install.

Other CI providers

The same three commands work anywhere with Python 3.11 or newer and Node.js 22.22.0 or newer:

pip install "hajer[evals]"
hajer eval --install-only
hajer eval --upload

Cache ~/.cache/hajer/engine (or $HAJER_CACHE_DIR/engine) keyed on the installed hajer version, and provide HAJER_API_KEY and HAJER_TEAM_ID as masked or secret variables.

Provider	Detected by	Context captured from
GitLab CI	GITLAB_CI	CI_COMMIT_SHA, CI_COMMIT_REF_NAME, CI_MERGE_REQUEST_IID, CI_MERGE_REQUEST_SOURCE_BRANCH_NAME, CI_MERGE_REQUEST_TARGET_BRANCH_NAME, CI_PIPELINE_ID, CI_PROJECT_URL
CircleCI	CIRCLECI	CIRCLE_SHA1, CIRCLE_BRANCH, CIRCLE_PULL_REQUEST, CIRCLE_BUILD_NUM, CIRCLE_REPOSITORY_URL
Buildkite	BUILDKITE	BUILDKITE_COMMIT, BUILDKITE_BRANCH, BUILDKITE_PULL_REQUEST, BUILDKITE_PULL_REQUEST_BASE_BRANCH, BUILDKITE_BUILD_ID, BUILDKITE_REPO
anything else	CI	the local git checkout only

Runs are matched to repositories linked through the Hajer GitHub App. On CircleCI or Buildkite building a GitHub repository, the checkout's github.com origin remote identifies it. A repository hosted on GitLab cannot be linked, so its runs are stored as unlinked runs of the team.