Monitoring

SAT exposes one health endpoint, writes a log line for every request, and runs on Cloud Run, which supplies request and instance metrics. There is no metrics endpoint and no tracing. This page tells you what each signal means and what to alert on.

Health endpointCopy Anchor Link

GET /health is public and needs no token. It runs SELECT 1 against the database on every call and also reports whether startup migrations succeeded.

curl -s -w '\n%{http_code}\n' https://sat.example.com/health

{
  "status": "healthy",
  "service": "SAT API",
  "version": "2.1.0",
  "environment": "production",
  "timestamp": 1791178030.84
}

Who uses it:

• Your deploy pipeline should probe /health on a new revision before it takes traffic, and roll back on failure.
• The Docker image declares HEALTHCHECK every 30 seconds (curl -fsS http://localhost:8080/health). Cloud Run ignores Docker HEALTHCHECK; it applies only when you run the image with Docker.
• sat doctor calls it first.

/health is limited to 100 requests per minute per client address. A monitor polling every few seconds is well within that.

LogsCopy Anchor Link

The API writes to stdout and stderr, which Cloud Run sends to Cloud Logging.

Request logsCopy Anchor Link

A middleware in api/main_api.py logs every request through structlog with a JSON payload:

2026-10-05 05:27:10,839 - api.main_api - INFO - {"method": "GET", "url": "https://sat.example.com/api/companies/.../tasks", "status_code": 200, "client_ip": "...", "user_agent": "...", "process_time": 0.0151, "event": "HTTP request processed", "logger": "api.main_api", "level": "info", "timestamp": "2026-10-05T05:27:10.839672Z"}

Each line is the standard logging prefix (time - logger - LEVEL - ) followed by the JSON object. Cloud Logging stores such lines as text (textPayload), not as structured jsonPayload, so use substring or regular-expression filters rather than JSON field filters, and severity is not set from level.

Other log lines use the plain prefix format: authentication events (User logged in successfully: ..., Login failed: ..., Account locked due to failed login attempts: ...), password reset requests, and SMTP warnings.

Useful queriesCopy Anchor Link

On Cloud Run, in the Logs Explorer for your project, with resource cloud_run_revision and your API service:

resource.type="cloud_run_revision"
resource.labels.service_name="<service>"
textPayload:"\"status_code\": 5"

Startup writes Starting SAT API and either Database initialized successfully or Database initialization failed, followed by Alembic's Running upgrade 003 -> 004 style lines when migrations apply.

Error responsesCopy Anchor Link

Unhandled exceptions are logged with a stack trace as Unhandled exception and answered with 500:

{"error": {"code": "INTERNAL_ERROR", "message": "An unexpected error occurred", "details": {}}}

When ENVIRONMENT is not exactly production, details.error contains the exception text.

Cloud Run metricsCopy Anchor Link

Cloud Run records these for the API service without any code in SAT. Find them in Cloud Monitoring under Cloud Run Revision.

For the database, watch CPU, memory, disk utilisation and active connections. The API opens a new connection per request (NullPool), so connection count tracks request concurrency.

What to alert onCopy Anchor Link

Runner and lease signalsCopy Anchor Link

The API cannot see runners directly; it sees their claims, heartbeats and reports. These are the signals that work is flowing.

Lease expiryCopy Anchor Link

A runner heartbeats every 30 seconds and reports transcript output every 2 seconds. A running run whose last heartbeat (or start) is older than RUN_LEASE_SECONDS (default 300) is failed by the next POST /runs/claim, POST /runs/expire-stale or sat scheduler sweep, with:

• runs.error set to Runner <runner_id> stopped reporting (lease expired) (or The process running it stopped reporting (lease expired) for runs with no runner id),
• the agent set to error if it has no other running run,
• an audit event with verb failed and actor type system.

Occasional expiries mean a runner host was shut down mid-run. A steady rate means runner hosts are crashing, losing network, or being killed (for example by an out-of-memory killer). Find them in the activity log or:

sat runs list --status failed --limit 100 --json | jq '.[] | select(.error // "" | test("lease expired"))'

Queue depthCopy Anchor Link

Queued runs that nobody claims mean no runner serves those agents.

sat runner status          # runs in flight by runner, and the queue
sat runs list --status queued

A run stays queued when no runner is running for the company, no runner serves the agent's adapter, the agent is paused, awaiting approval or terminated, or the agent is busy with another run. See Troubleshooting.

Budget pausesCopy Anchor Link

When an agent reaches its monthly budget the API pauses it and opens a budget_override approval. A paused agent's runs are not claimed. Watch pending_approvals on the dashboard (GET /dashboard) or the inbox.

sat doctorCopy Anchor Link

sat doctor checks a profile end to end from wherever you run it. It exits 1 if any check fails.

sat doctor --profile mine

✓ profile              mine → https://sat.example.com (config ~/.config/sat/config.json, credentials in keychain)
✓ api                  healthy · production · v2.1.0
✓ auth                 you@example.com via api key (keychain)
✓ company              NoteFlow (NF), role owner
✓ live stream          connected via https://<your Cloud Run service URL>
✓ adapter claude_code  /usr/local/bin/claude 2.0.0 (Claude Code)
– adapter codex        `codex` not found on PATH (npm i -g @openai/codex)
✓ git                  /usr/bin/git

Add --json for machine-readable output.

RelatedCopy Anchor Link