Timeouts
timeout: 300 sends SIGTERM to the process group, waits about five seconds, then SIGKILL. The run is marked timed_out and the retry policy applies.
Otter/Docs
Docs
You do not need this website's repository to use Otter. Install the binary and work in your own directory. The authoritative reference lives in the project docs; this page is the shortest path to something running.
On this page
01 · Quickstart
# macOS — a Homebrew cask
brew install tkoizumi/tap/otter
# Ubuntu / Debian — a .deb per architecture, named with the version
VERSION=0.5.3; ARCH=amd64 # amd64 or arm64
curl -LO "https://github.com/tkoizumi/otter/releases/download/v${VERSION}/otter_${VERSION}_linux_${ARCH}.deb"
sudo apt install "./otter_${VERSION}_linux_${ARCH}.deb"
Both packages install otter (the client) and otterd (the same daemon with its own flag defaults). Python 3 is required on the machine that runs jobs, unless a job opts into python.mode: managed.
otter init my-project # a workspace and one job
cd my-project
otter validate my-project # check the manifest without a daemon
otter release my-project # stage and activate an immutable snapshot
otter start --detach # free loopback port; records where it listens
otter run my-project # queue a run and follow the outcome
otter state get my-project count
otter stop
Release before you run. A run executes the job's active release rather than its source tree, so otter run refuses until the job has been released once. After an edit, release again or the run keeps executing the previous snapshot.
02 · Anatomy of a job
my-project/
├── jobs/
│ └── counter/
│ ├── otter.yaml # what the runtime needs to know
│ ├── main.py # the program
│ ├── logic.py # pure logic, captured into the release
│ └── test_logic.py # ordinary unit test
└── otter.env # optional; loaded by `otter start`
Every otter.yaml under the jobs root is discovered recursively, validated and registered. A broken manifest is reported and marked invalid — it never takes the daemon down.
03 · Manifest
version: 1
name: salesforce-to-netsuite
description: Sync changed accounts into NetSuite.
entrypoint: main.py
trigger:
cron: "*/5 * * * *"
webhook:
enabled: true
timeout: 300
concurrency: 1
retry:
attempts: 5 # total attempts: one initial plus up to four retries
backoff: exponential # none | linear | exponential
initial_delay: 2s
max_delay: 60s
env:
NETSUITE_ACCOUNT: ${NETSUITE_ACCOUNT}
secrets:
- SALESFORCE_TOKEN
- NETSUITE_TOKEN
| Key | What it controls |
|---|---|
| name | How every CLI verb refers to the job. The directory name does not have to match. |
| entrypoint | The Python file executed for each run. |
| trigger.cron | Five-field schedule, re-registered from the manifest at every start. |
| trigger.webhook | Opt in to a token-guarded POST /v1/hooks/<job>. |
| timeout | Seconds before SIGTERM, then SIGKILL about five seconds later. |
| concurrency | Simultaneous runs of this job. Extra triggers queue instead of racing. |
| retry | Attempts, backoff shape, first delay and the cap. |
| secrets | Variables read from the daemon's environment and injected into the child. |
| python.mode | host (default) or managed for a prepared interpreter with locked dependencies. |
| capture | full, metadata or off for this job's HTTP recording. |
The manifest schema is versioned by version: and covered by the manifest reference.
04 · Python SDK
On first start the SDK is extracted to <data>/sdk/python and prepended to the child's PYTHONPATH, so import otter works with no pip install step.
from otter import run
@run
def main(ctx):
cursor = ctx.state.get("cursor")
records = get_changed_accounts(cursor)
for record in records:
upsert_customer(record)
if records:
ctx.state.set("cursor", records[-1]["last_modified"])
from otter import Context
ctx = Context.from_environment()
count = ctx.state.get("count") or 0
count += 1
ctx.log.info("Counter executed", count=count)
ctx.state.set("count", count)
| Member | Description |
|---|---|
| ctx.state.get(key, default=None) | Read a JSON value; default when unset. |
| ctx.state.set(key, value) | Persist any JSON-serialisable value. |
| ctx.state.delete(key) | Remove a key; returns True if it existed. |
| ctx.state.all() | Every key/value pair for this job. |
| ctx.log.debug/info/warning/error(msg, **fields) | Structured log line, stored against the run. |
| ctx.run_id · ctx.job_id | Identifiers for the current run. |
| ctx.trigger.type | manual, cron or webhook. |
| ctx.trigger.body / headers / header(name) | The webhook payload, when there is one. |
The authoritative state lives in the daemon; the SDK is a thin HTTP client, so a crashed or killed process can never corrupt it.
05 · Triggers
trigger:
cron: "*/5 * * * *"
Standard five-field expressions, re-registered from manifests on every start, so a restart never loses a future schedule. Occurrences missed while the daemon was offline are not replayed — that behaviour is documented rather than implied.
otter run salesforce-to-netsuite
Always available, regardless of manifest configuration. Use it for the first run, a backfill, or a repair — it goes through the same queue and policy as everything else.
POST /v1/hooks/<job>
X-Otter-Token: …
Opt in per job. Otter generates a static token on first start and persists it. A call without the token gets a 401 — unauthenticated execution is never exposed.
$ otter inspect shopify-to-erp
webhook: enabled
webhook url: POST http://127.0.0.1:7337/v1/hooks/shopify-to-erp
webhook token: 5cde21dc53f207cb621b98df53d4caa06d1c15ffb98a084165f2094bdee757c1
06 · Reliability
timeout: 300 sends SIGTERM to the process group, waits about five seconds, then SIGKILL. The run is marked timed_out and the retry policy applies.
retry.attempts is the total number of attempts. Configuration failures — a missing secret, a missing entrypoint, an invalid manifest — are never retried, because the next attempt would fail identically.
On start, any run left running is marked failed with otter daemon restarted during execution, and a fresh attempt is enqueued when policy allows. Recovery is fail-closed.
On SIGTERM/SIGINT Otter stops claiming work, gives running jobs --shutdown-grace (default 15s), terminates what is left, records those runs, and closes SQLite cleanly.
A job saves progress in ctx.state and reads it on the next run. Otter does not restore Python execution at the point of failure; your code decides where to resume.
Each attempt is its own run record linked by parent_run_id, so the whole retry history is auditable from one run id.
run id: afd1b271-7e83-4bbc-a202-a7592fc0320d
job: flaky
status: failed
attempt: 1
error: process exited with code 1
attempts:
RUN ID ATTEMPT STATUS STARTED DURATION
afd1b271-7e83-4bbc-a202-a7592fc0320d 1 failed 13:16:22 150ms
b69229bd-96af-4c4e-a9cd-77d5190651a4 2 failed 13:16:23 120ms
8a6c3c9e-8cbe-4a23-a287-1f7a6cd5fa18 3 succeeded 13:16:25 103ms
07 · HTTP capture
Runs record their outgoing HTTP by default — sanitized headers and bounded JSON bodies included — because capture observes live traffic and cannot be turned on after an unattended failure. otter trace <run-id> merges lifecycle lines, captured output and HTTP exchanges into one chronological page.
job: orders-sync status: failed attempt: 1
release: 8c1d4f0a9b3e trigger: manual duration: 41ms
error: process exited with code 1
retry context: otter run-status 3f2a91c4-7d18-4a6e-8b21-5c0d9e4a17bb
TIME KIND DETAIL
03:30:00 · lifecycle run queued (trigger manual)
03:30:00 · lifecycle run started (attempt 1 of 1, trigger manual)
03:30:00 ✓ http POST api.example.test/v2/cursor -> 200, 6ms [1]
03:30:00 ! http POST api.example.test/v2/records?access_token=REDACTED -> 400, 9ms [2]
03:30:00 × log [stderr] RuntimeError: upstream rejected the batch
03:30:00 × lifecycle run failed (attempt 1, 41ms), exit code 1
└1. main.py:12; completed; payloads: full
└2. main.py:26; completed; payloads: full
08 · CLI
otter status # queue age and depth, run counts, freshness, storage
otter jobs [--all] # registered jobs; --all adds invalid and retired
otter inspect <job> # manifest view, triggers, recent runs
otter reload # re-read jobs without restarting running work
otter pause [<job>|.] # suspend cron and webhook; manual runs still work
otter resume [<job>|.] # re-arm what a pause suspended
otter run [<job>] # queue a manual run; prints the run id
otter runs [<job>] [--all] # one job, or every job with --all
otter cancel <run-id> # stop a queued or running run; never retried
otter run-status <run-id> # one run plus its retry attempts
otter logs <run-id> [--follow] # captured output
otter trace <run-id> # context, output and HTTP in time order
otter requests <run-id> # captured outgoing requests
otter request <request-id> # one exchange, run resolved for you
otter state get|set|delete <job> <key>
otter init [name] # scaffold a workspace and one job
otter validate <otter.yaml|dir|name> # validate without a running daemon
otter start [--detach] / otter stop # this project's runtime
otter deploy --host <user@host> # install or update a runtime over SSH
otter prepare [<job>] # prepare opt-in managed Python environments
otter release [<job>|.] [--all] # stage and activate an immutable release
otter release --list <job> # staged releases, newest first
otter release --activate <digest> <job> # roll back to a staged release
Global flags: --api <url>, --token <token>, --json, --version. Without --api, the daemon URL is read from the nearest .otter/serve/listen.url, walking up from the working directory.
09 · HTTP API
GET /v1/version and GET /health are public, so a client can establish what shape and schema it is talking to before it holds a credential. Loopback binding needs no token; binding elsewhere requires OTTER_API_TOKEN, and the daemon refuses to start without it.
GET /health
GET /v1/version
GET /v1/jobs
GET /v1/jobs/{id}
POST /v1/jobs/{id}/runs ?capture=off|metadata|full
GET /v1/runs ?job_id=&status=&limit=&offset=
GET /v1/runs/{id}
GET /v1/runs/{id}/logs ?after_id=&limit=
GET /v1/runs/{id}/requests ?after_id=&limit=
GET /v1/runs/{id}/requests/{request_id}
POST /v1/runs/{id}/cancel
GET /v1/jobs/{id}/state
GET /v1/jobs/{id}/state/{key}
PUT /v1/jobs/{id}/state/{key}
DELETE /v1/jobs/{id}/state/{key}
POST /v1/hooks/{job}
curl -s http://127.0.0.1:7337/v1/version
{
"schema_version": 1,
"contract_version": 3,
"product_version": "0.5.3",
"manifest_schema": 1,
"sdk_version": "0.1.0",
"supported_platforms": ["linux/amd64", "linux/arm64", "darwin/amd64", "darwin/arm64"]
}
10 · Deploying
otter deploy --host deploy@10.0.0.4
Cross-compiles locally, pushes the binaries, creates an unprivileged service account and installs the systemd unit. Each job is staged under <remote>/jobs/<name> with every declared tree at the relative depth the manifest names, so a release digest means the same layout everywhere. otter deploy --status reports what this project last deployed.
otter login
otter deploy --cloud
Promotes an existing local release onto a pooled tenant we operate — --build restores packaging if you want it. Credentials are org-bound, and jobs can be deleted. A deploy prunes inactive snapshots to --keep (default 3) while always keeping the previous release as the rollback target.
otterd \
--jobs /opt/otter/jobs \
--data /var/lib/otter \
--listen 127.0.0.1:7337 \
--workers 8
| Flag / env | Default | Purpose |
|---|---|---|
| --jobs · OTTER_JOBS_DIR | ./jobs | Root scanned recursively for otter.yaml. |
| --data · OTTER_DATA_DIR | ./tmp | Holds otter.db and the extracted SDK. |
| --listen · OTTER_LISTEN | 127.0.0.1:7337 | HTTP API address. |
| --workers · OTTER_WORKERS | CPU cores, max 8 | Maximum concurrent runs across all jobs. |
| --api-token · OTTER_API_TOKEN | none | Bearer token; required for non-loopback binding. |
| --shutdown-grace | 15s | Time running jobs get on shutdown. |
| --capture-default | full | HTTP capture for runs that do not choose one. |
| --capture-retention | 168h | How long captured payloads are kept; 0 disables expiry. |
| --run-retention · --log-retention | forever | Retention for terminal run history and captured output. |
| --log-format | json | Structured daemon logs; pretty during development. |
11 · Limits
Each of these is stated in the published runtime contract as a promise or a non-promise. If a non-promise is load-bearing for you, that is important to know before you install rather than after.
The full documentation lives with the source, and issues are read.