Otter/Solutions/Run data pipelines

Solution

Run data pipelines without owning a scheduler.

Extraction, loading and transformation, written in plain Python with the libraries your team already uses. Otter decides when a pipeline starts, which version runs, how it recovers, and where its progress lives.

What it is

A durable execution layer for pipelines you already wrote.

A pipeline in Otter is a job: a directory with an otter.yaml and an entrypoint. The manifest declares the schedule, the timeout, the retry policy, the concurrency limit and the secrets. The Python does the work — call an API, land Parquet, run a transformation, write to the warehouse.

Otter adds what the Python does not have: a durable queue in front of your code, an immutable release behind it, a place to checkpoint progress, and a run record you can read afterwards. Control flow stays in code rather than in a graphical editor, so it reviews like the rest of your repository.

repository shape
orders-sync/
├── otter.yaml        # schedule, retry, secrets, concurrency
├── main.py           # extract → transform → load
├── transform.py      # shared module, captured into the release
└── requirements.txt  # optional in managed Python mode
Small data teams

Two engineers, forty pipelines

You do not have a platform team and you do not want one. You want the pipelines to run, recover and explain themselves without a cluster to operate.

Python-first shops

Polars, DataFusion, dltHub, Sling

Your transformations already live in Python modules and notebooks. Otter runs them as they are instead of asking for a rewrite in someone's DSL.

Mixed workloads

Pipelines plus the things next to them

Batch inference, document parsing and agent-feeding jobs can share the same runtime, secrets and run history as the ETL that feeds them.

Failure modes

Pipelines rarely fail because the Python is wrong.

They fail because of everything around the Python: the schedule, the version, the overlap, the checkpoint, and the fact that nobody can see what happened. Here is how each one is handled.

Failure 01

The upstream API rate-limited you at 03:00

Otter: a transient failure exits non-zero, and the retry policy applies — exponential backoff from initial_delay, capped at max_delay, up to attempts total. Each attempt is its own run record linked to the previous one, so the retry chain is auditable rather than invisible.

Failure 02

Someone fixed a typo and the job changed in production

Otter: a run executes the active release, an immutable snapshot with a digest, never your working tree. Editing a file is not a deploy. Rolling back is otter release --activate <digest>, and the previous release is always retained as the target.

Failure 03

A slow run was still going when the next one started

Otter: concurrency: 1 means the second trigger queues durably instead of racing the first. --workers caps the whole host so a busy morning cannot starve the machine.

Failure 04

It died halfway and replayed 40,000 rows

Otter: persist a cursor in ctx.state as you go. After a failure Otter starts a fresh process and your code resumes from the last checkpoint. State lives in the daemon behind an HTTP client, so a killed process cannot leave it half-written.

Failure 05

The host rebooted during a batch

Otter: anything left running is marked failed on the way back up and re-enqueued when policy allows; queued work stays queued. Recovery is fail-closed — the daemon refuses to start rather than serve with stranded work.

Failure 06

The warehouse rejected one batch and you can't see why

Otter: outgoing HTTP is recorded during the run — status, timing, bounded bodies, redacted headers — and otter trace interleaves it with your output and lifecycle lines. The evidence exists because it was captured while the run happened, not because someone remembered to log.

A complete pipeline

S3 to a warehouse, with a cursor you can trust.

This is the whole thing: a manifest, an entrypoint, and a cursor in state. No framework classes, no task decorators, no graph.

  • Runs every five minutes, one at a time
  • Five total attempts with exponential backoff
  • Two secrets injected from the daemon's environment
  • Resumes from the last loaded batch after any failure

Swap the libraries freely. Polars, pandas, DataFusion, dltHub, Sling, a warehouse driver — Otter runs the process and never inspects how you transform data.

otter.yaml
version: 1

name: s3-to-warehouse
description: Load new objects from S3 into the warehouse.

entrypoint: main.py

trigger:
  cron: "*/5 * * * *"

timeout: 900
concurrency: 1

retry:
  attempts: 5
  backoff: exponential
  initial_delay: 2s
  max_delay: 60s

secrets:
  - S3_ACCESS_KEY
  - WAREHOUSE_DSN
main.py
from otter import run
from transform import clean

@run
def main(ctx):
    cursor = ctx.state.get("cursor")          # last object we loaded
    objects = list_new_objects(after=cursor)

    if not objects:
        ctx.log.info("Nothing new", cursor=cursor)
        return

    df = clean(load_parquet(objects))
    warehouse.write(df, table="events", mode="append")

    # Only advance once the write committed.
    ctx.state.set("cursor", objects[-1]["key"])
    ctx.log.info("Loaded batch", rows=len(df), objects=len(objects))
reading the run back
otter runs s3-to-warehouse --limit 5
otter run-status afd1b271-7e83-4bbc-a202-a7592fc0320d
otter logs afd1b271 --follow
otter state get s3-to-warehouse cursor

Position

A Python-first experience for data teams.

A pipeline is a program. It belongs in version control, it should review like code, and it should run the same way on a laptop as it does on a server.

Local and production agree

The runtime is one binary. Run the job on your laptop with otter start, then deploy the same release digest to a host or to Otter Cloud. No “works locally, fails in the scheduler” gap caused by a different executor.

Code review is the control flow

Branches, loops and early returns live in Python, so a reviewer can read them in a diff. There is no second representation of the pipeline to keep in sync with the code.

One interface, published

Manifest schema, Python SDK, CLI JSON and HTTP API are frozen behind a compatibility and deprecation policy, so the automation you build on top stops moving underneath you.

Where it runs

On your infrastructure, or on ours.

Self-hosted

One host, two directories

otterd --jobs /opt/otter/jobs --data /var/lib/otter discovers and validates every manifest under the root. A broken manifest is reported and marked invalid; it never takes the daemon down.

  • Runs under systemd with an unprivileged service account
  • otter deploy --host user@host installs or updates over SSH
  • Dead-man alarms and health checks for unattended operation
Otter Cloud · alpha

The same runtime, promoted

otter login then otter deploy --cloud moves a release you already validated into a pooled tenant we operate. Same manifest, same SDK, same run semantics.

  • Promote the digest you tested — no cloud-side rebuild
  • Org-bound credentials, per-tenant isolation
  • Alpha: interfaces and pricing are still moving

Adjacent work

The same runtime, other jobs.

Anything that is “Python plus a schedule plus a reason to be reliable” fits the same model.

Batch inference

Score a morning's worth of records, checkpoint between batches, and retry the ones the model server dropped.

Document processing

Parse, extract and file documents on a schedule, with the run history proving which batch handled which file.

Reports and exports

Generate the report, upload it, and get a run record that says whether the upload actually succeeded.

Feeding agents

Keep a retrieval index or feature store current, with a webhook trigger when the upstream source pushes.

Where Otter stops. It is not a distributed compute engine and it does not schedule across a fleet. It runs jobs on a host — reliably, observably, on a schedule — which is what most pipelines actually need. If you have outgrown a single host, Otter is not the answer to that problem, and we would rather tell you now.

Point it at the pipeline you already have.

Install, scaffold, drop in your existing script, and release it.