Smart Agent TeamsTesting
Every SAT test suite, how to run it locally, what CI runs on each pull request, and where to add tests for a change.
SAT has five layers of tests, from fast in-process unit tests to an end-to-end run of the CLI and runner against a real PostgreSQL. Each catches a different class of bug, and CI runs all but the browser suite on every pull request.
| Suite | Tool | Where | Database | In CI |
|---|---|---|---|---|
| API tests | pytest | apps/api/tests | In-memory SQLite | api job |
| CLI and runner unit tests | bun test | apps/cli/test | None (fakes) | web job |
| Web unit tests | Vitest (jsdom) | apps/web/src/**/*.test.ts | None | web job |
| CLI smoke test | Bash | scripts/03-testing/cli-smoke.sh | PostgreSQL 16 | cli-e2e job |
| Browser end-to-end | Playwright | tests/e2e | PostgreSQL (through a running API) | Not in CI |
API tests (pytest)
cd apps/api
.venv/bin/python -m pytest tests -q
.venv/bin/python -m pytest tests/test_runner_api.py -q -k claim| File | Covers |
|---|---|
test_company_api.py | Companies, agents and the org chart, tasks, goals, projects, approvals, budgets, routines, dashboard, costs, activity, and the event bus |
test_runner_api.py | API keys, run claims, heartbeats, leases and expiry, run reports, agent attribution with run_id |
test_password_reset.py | Forgot and reset password, token expiry and reuse, session revocation |
test_me_preferences.py | PATCH /api/me and preference merging |
test_api_comprehensive.py | Register, login, refresh, logout, error formats, rate limiting, /health (healthy, migrations failed, database unreachable), and the legacy endpoints |
How they work:
- Each test file creates an in-memory SQLite engine (
sqlite://withStaticPool), builds tables withBase.metadata.create_all, and overrides theget_dbdependency. Tables are dropped after each test. - Tests call the real app through FastAPI's
TestClient, so routing, auth, validation and error handlers are exercised. - Startup still tries
init_db()againstDATABASE_URL; when no PostgreSQL is reachable that fails harmlessly because the tests never use that connection.
What SQLite cannot catch
SQLite returns naive datetimes, has no FOR UPDATE SKIP LOCKED, and cannot run migration 002. Timezone bugs, claim concurrency and migrations are only tested by the CLI smoke test against PostgreSQL. A refresh-token bug that signed everyone out after 30 minutes on PostgreSQL passed every pytest test; the smoke test now forces a refresh to catch it.
CLI and runner unit tests (bun test)
bun run --cwd apps/cli test
bun run --cwd apps/cli typecheck| File | Covers |
|---|---|
runner.test.ts | The runner engine against an in-memory FakeApi: claiming, concurrency, transcript flushing, heartbeats, lease loss, cancellation, the BLOCKED: convention, task moves, failure reporting |
adapters.test.ts | Parsing Claude Code stream-json and Codex --json events into transcript, usage and results; clipping; Codex cost only when prices are set; echo and unsupported adapters; system, task and heartbeat prompts |
scheduler.test.ts | Which routines and agents get schedules, firing and stale-run expiry, the duplicate window |
session.test.ts | Profile and credential resolution, SAT_API_KEY precedence, a single shared refresh on 401, the SSE frame parser |
helpers.test.ts | Money and budget input, option parsing, name and prefix resolution (never guessing on ambiguity), exit codes, table and time output |
fixtures.ts provides fakeContext(), a complete RunContext for runner tests.
Web unit tests (Vitest)
bun run --cwd apps/web testVitest runs with jsdom and Testing Library (src/test/setup.ts). src/test/units.test.ts covers formatting, the status vocabulary, the command palette filter, live-event cache refresh, preference merging, dashboard personalisation, recent items and saved views, and locale and time zone handling. libs/api-client and libs/shared run Vitest with --passWithNoTests.
All JavaScript tests, type checks and builds run together with:
bunx turbo run typecheck test buildCLI smoke test (PostgreSQL)
scripts/03-testing/cli-smoke.sh drives the real sat CLI and runner against a running API with the demo company seeded. It uses a temporary config directory and file credential store, so it never touches your own credentials.
# PostgreSQL 16 running, DATABASE_URL exported
cd apps/api && .venv/bin/uvicorn main:app --port 8080 &
.venv/bin/python scripts/seed_demo.py && cd ../..
SAT_API_URL=http://localhost:8080 scripts/03-testing/cli-smoke.shIt checks, in order: sign-in and company selection; transparent refresh of an expired access token; exit codes (4 not found, 3 signed out, 2 bad input); API key creation and authentication; a simulated runner (sat runner start --simulate --once) executing an assigned task for Toni, which must end in_review with Toni's comment and a succeeded run; deciding an approval; and revoking the key, after which it must stop working.
Variables: SAT_API_URL (default http://localhost:8080), SAT_BIN (default bun apps/cli/src/main.ts; set it to apps/cli/dist/sat to test the binary), SAT_DEMO_EMAIL, SAT_DEMO_PASSWORD.
Browser end-to-end (Playwright)
The suite needs the API and the web app running.
# Terminal 1: API on :8080 against PostgreSQL
cd apps/api && DATABASE_URL=postgresql://... .venv/bin/uvicorn main:app --port 8080
# Terminal 2: web on :4200
bun run dev
# Terminal 3
bunx playwright install chromium # once
bun run test:e2eglobal-setup.tsregisters a fresh user (e2e-<timestamp>@example.com) and creates a company "E2E Robotics" (prefixE2E) with Ada, Linus, Grace, a pending hire, a goal and a task, through the API. Tests never depend on seeded data.- Projects:
desktop(Desktop Chrome, 1440x900) runs everything;mobile(Pixel 7) runsnavigation.spec.ts. navigation.spec.tsvisits every screen and runs axe accessibility checks.- Specs:
home,navigation,password-reset,preferences,work. - Point it at any deployed environment with
E2E_BASE_URLandE2E_API_URL.
Pages hold an SSE connection open, so never wait for networkidle; wait for a visible element.
The root package.json also defines test:api* and test:integration* scripts. They run older suites under tests/integration written for the legacy API, and CI does not run them.
CI
.github/workflows/ci.yml runs on every pull request, on pushes to main, and on demand. A newer push cancels the running workflow for the same ref.
| Job | Steps |
|---|---|
web | Bun 1.4.0; bun install --frozen-lockfile (with a read-only npm token from a CI secret for @yarlisai/ui); bun audit --audit-level=high; bun run lint (Biome); bunx turbo run typecheck test build (web, CLI, docs and libraries); docs link check (bun run --cwd apps/docs check-links) and CLI reference check (bun run --cwd apps/cli docs:check); CLI binary smoke (sat --version, --help, completion bash). |
api | Python from .python-version; pip install -r requirements-production.txt pytest pytest-asyncio; pytest tests -q in apps/api. |
cli-e2e | A postgres:16 service; installs both toolchains; starts the API with ENVIRONMENT=development and waits for /health; seeds the demo company; runs cli-smoke.sh. Prints the last 200 lines of the API log on failure. |
.github/workflows/tollgate-conformance.yml runs on every pull request and push to main and fails if a declared deploy target has a second deployer.
build.yml (image build) runs only on pushes to main. security-scan-scheduled.yml runs only when started by hand.
Adding tests
| You changed | Add |
|---|---|
| An API endpoint or rule | A pytest test in the matching file in apps/api/tests. Use the client fixture and register a user through /api/register. Assert the error code, not only the status. |
| Anything timezone-, lock- or migration-sensitive | A step in cli-smoke.sh, because only it runs on PostgreSQL. |
| A CLI command | A bun test case, and a smoke step if it talks to the API in a new way. Regenerate the CLI reference (bun run --cwd apps/cli docs). |
| Runner behaviour | A runner.test.ts case against FakeApi; adapter parsing goes in adapters.test.ts with a recorded event line. |
| A web helper | A Vitest case in apps/web/src/test. |
| A screen or flow | A Playwright spec in tests/e2e, using the signedIn fixture and role-based locators. Create the data you need through the API in the test or in global-setup.ts. |