Two engineers, forty pipelines
You do not have a platform team and you do not want one. You want the pipelines to run, recover and explain themselves without a cluster to operate.
Otter/Solutions/Run data pipelines
Solution
Extraction, loading and transformation, written in plain Python with the libraries your team already uses. Otter decides when a pipeline starts, which version runs, how it recovers, and where its progress lives.
On this page
What it is
A pipeline in Otter is a job: a directory with an otter.yaml and an entrypoint. The manifest declares the schedule, the timeout, the retry policy, the concurrency limit and the secrets. The Python does the work — call an API, land Parquet, run a transformation, write to the warehouse.
Otter adds what the Python does not have: a durable queue in front of your code, an immutable release behind it, a place to checkpoint progress, and a run record you can read afterwards. Control flow stays in code rather than in a graphical editor, so it reviews like the rest of your repository.
orders-sync/
├── otter.yaml # schedule, retry, secrets, concurrency
├── main.py # extract → transform → load
├── transform.py # shared module, captured into the release
└── requirements.txt # optional in managed Python mode
You do not have a platform team and you do not want one. You want the pipelines to run, recover and explain themselves without a cluster to operate.
Your transformations already live in Python modules and notebooks. Otter runs them as they are instead of asking for a rewrite in someone's DSL.
Batch inference, document parsing and agent-feeding jobs can share the same runtime, secrets and run history as the ETL that feeds them.
Failure modes
They fail because of everything around the Python: the schedule, the version, the overlap, the checkpoint, and the fact that nobody can see what happened. Here is how each one is handled.
Otter: a transient failure exits non-zero, and the retry policy applies — exponential backoff from initial_delay, capped at max_delay, up to attempts total. Each attempt is its own run record linked to the previous one, so the retry chain is auditable rather than invisible.
Otter: a run executes the active release, an immutable snapshot with a digest, never your working tree. Editing a file is not a deploy. Rolling back is otter release --activate <digest>, and the previous release is always retained as the target.
Otter: concurrency: 1 means the second trigger queues durably instead of racing the first. --workers caps the whole host so a busy morning cannot starve the machine.
Otter: persist a cursor in ctx.state as you go. After a failure Otter starts a fresh process and your code resumes from the last checkpoint. State lives in the daemon behind an HTTP client, so a killed process cannot leave it half-written.
Otter: anything left running is marked failed on the way back up and re-enqueued when policy allows; queued work stays queued. Recovery is fail-closed — the daemon refuses to start rather than serve with stranded work.
Otter: outgoing HTTP is recorded during the run — status, timing, bounded bodies, redacted headers — and otter trace interleaves it with your output and lifecycle lines. The evidence exists because it was captured while the run happened, not because someone remembered to log.
A complete pipeline
This is the whole thing: a manifest, an entrypoint, and a cursor in state. No framework classes, no task decorators, no graph.
Swap the libraries freely. Polars, pandas, DataFusion, dltHub, Sling, a warehouse driver — Otter runs the process and never inspects how you transform data.
version: 1
name: s3-to-warehouse
description: Load new objects from S3 into the warehouse.
entrypoint: main.py
trigger:
cron: "*/5 * * * *"
timeout: 900
concurrency: 1
retry:
attempts: 5
backoff: exponential
initial_delay: 2s
max_delay: 60s
secrets:
- S3_ACCESS_KEY
- WAREHOUSE_DSN
from otter import run
from transform import clean
@run
def main(ctx):
cursor = ctx.state.get("cursor") # last object we loaded
objects = list_new_objects(after=cursor)
if not objects:
ctx.log.info("Nothing new", cursor=cursor)
return
df = clean(load_parquet(objects))
warehouse.write(df, table="events", mode="append")
# Only advance once the write committed.
ctx.state.set("cursor", objects[-1]["key"])
ctx.log.info("Loaded batch", rows=len(df), objects=len(objects))
otter runs s3-to-warehouse --limit 5
otter run-status afd1b271-7e83-4bbc-a202-a7592fc0320d
otter logs afd1b271 --follow
otter state get s3-to-warehouse cursor
Position
A pipeline is a program. It belongs in version control, it should review like code, and it should run the same way on a laptop as it does on a server.
The runtime is one binary. Run the job on your laptop with otter start, then deploy the same release digest to a host or to Otter Cloud. No “works locally, fails in the scheduler” gap caused by a different executor.
Branches, loops and early returns live in Python, so a reviewer can read them in a diff. There is no second representation of the pipeline to keep in sync with the code.
Manifest schema, Python SDK, CLI JSON and HTTP API are frozen behind a compatibility and deprecation policy, so the automation you build on top stops moving underneath you.
Where it runs
otterd --jobs /opt/otter/jobs --data /var/lib/otter discovers and validates every manifest under the root. A broken manifest is reported and marked invalid; it never takes the daemon down.
otter login then otter deploy --cloud moves a release you already validated into a pooled tenant we operate. Same manifest, same SDK, same run semantics.
Adjacent work
Anything that is “Python plus a schedule plus a reason to be reliable” fits the same model.
Score a morning's worth of records, checkpoint between batches, and retry the ones the model server dropped.
Parse, extract and file documents on a schedule, with the run history proving which batch handled which file.
Generate the report, upload it, and get a run record that says whether the upload actually succeeded.
Keep a retrieval index or feature store current, with a webhook trigger when the upstream source pushes.
Where Otter stops. It is not a distributed compute engine and it does not schedule across a fleet. It runs jobs on a host — reliably, observably, on a schedule — which is what most pipelines actually need. If you have outgrown a single host, Otter is not the answer to that problem, and we would rather tell you now.
Install, scaffold, drop in your existing script, and release it.