Why build durable automations without a queue
Not every internal automation needs a heavyweight message broker or a full eventing stack. Many “long-running jobs” in internal tooling are really orchestrations: a sequence of API calls, human approvals, data pulls, transformations, and writes—often with minutes or hours between steps. What makes them hard is not throughput; it’s durability. You need each step’s state to survive deploys, worker restarts, and partial failures, and you need recovery to be predictable without an operator manually piecing things together.
Designing stateful internal automations without a queue means treating the workflow engine (or your app) as the source of truth for step state. The core ingredients are durable step state, checkpointing, and crash-only recovery. This article lays out a pragmatic design that scales from “one script that sometimes runs long” to production-grade workflows with auditable execution history.
Model the job as a state machine, not a process
Start by defining a job as a record in durable storage (typically Postgres) whose lifecycle is described by explicit states. A worker process is allowed to die at any time; it should never be the only place state exists. The job record contains metadata like job_id, created_at, requested_by, and a status field (RUNNING, WAITING, FAILED, SUCCEEDED, CANCELED). Then break the job into steps, each with its own state.
A useful mental model is: the database is your queue—not for high-throughput fanout, but as a durable coordination layer. Workers poll for runnable steps or are triggered by schedules/webhooks. Importantly, “runnable” is a function of persisted step state, not in-memory progress.
Step state as first-class data
For each step, persist:
- step_name and step_version (so you can evolve logic safely)
- attempt count and timestamps
- inputs (or an input hash plus a pointer to stored payload)
- outputs (or output pointer) and a summarized “result”
- status (PENDING, RUNNING, WAITING, RETRYABLE_FAILED, TERMINAL_FAILED, DONE)
- checkpoint data (explained below)
- idempotency key for any external side-effect
This turns debugging from “what happened on worker #12?” into “what does the durable step log say?” If you already build event-driven frontends, many of these concepts mirror idempotency and retries, even if you’re not using a broker. (See idempotency keys, retries, and dead-letter queues for a complementary perspective.)
Checkpointing that survives crashes and deploys
Checkpointing is the difference between “restart the whole job” and “resume from the last safe boundary.” The key is to checkpoint at boundaries where re-execution is safe and deterministic.
Choose checkpoint boundaries deliberately
Good checkpoint boundaries typically align with:
- Before an external side-effect (charging a card, sending an email, creating a ticket)
- After a durable write (you’ve committed the result to your DB)
- After an expensive read where re-fetching would be slow or inconsistent
- Before waiting on human approval or a long external process
A checkpoint can be as small as “cursor position in a paginated API” or “the last processed row ID.” For long loops, store incremental progress (e.g., every 100 items). The checkpoint should be sufficient to make the next execution attempt either skip already-completed work or continue with minimal duplication.
Keep checkpoint payloads stable
Checkpoint data is durable API between versions of your own code. Prefer simple, explicit structures: cursors, item IDs, and state enums. If you store complex objects, you’ll eventually hit a “can’t deserialize after refactor” failure mode. A versioned checkpoint schema (step_version + payload_version) is usually enough.
Crash-only recovery as the default operating mode
“Crash-only” means you do not rely on graceful shutdown, in-process cleanup, or ephemeral locks to maintain correctness. Every step must be restartable. When a worker dies mid-step, the next worker should be able to re-acquire the step, rehydrate inputs from storage, consult checkpoint state, and proceed safely.
Leases instead of locks
Use a lease model for steps in RUNNING state: store lease_owner and lease_expires_at. A worker can claim work by atomically setting these fields if the lease is empty or expired. If the worker crashes, the lease naturally expires and another worker retries. This avoids “stuck forever” locks and makes recovery routine.
At-least-once execution requires idempotency
Without a queue, you are still effectively at-least-once: a step may run twice after a crash, timeout, or lease expiration. Correctness comes from idempotent side-effects. For each external call that changes state, include an idempotency key derived from (job_id, step_name, logical action). Persist it before the call, and store the external reference you get back (payment_intent_id, ticket_id, etc.). If you retry, you reuse the same key and reconcile against the stored reference.
Classify failures: retryable vs terminal
Durable automations need a failure taxonomy that is reflected in persisted state:
- Retryable: timeouts, 429s, transient 5xx, temporary network errors
- Terminal: invalid inputs, permission errors, “not found” when it can’t be recovered, schema validation errors
Persist the classification and the last error summary (plus a pointer to full logs). This is how you avoid an endless loop of retries that never succeed and gives operators a clear decision point: fix inputs, re-run from a checkpoint, or cancel.
Durable orchestration patterns without a broker
Once you have step state, checkpointing, and leases, you can implement the patterns most teams associate with queues—without the operational overhead.
Wait states for humans and external systems
Many internal automations pause for approvals or asynchronous external processing. Model this explicitly as WAITING with a wakeup_condition: a webhook token, a schedule time, or a database flag. The job resumes when a trigger flips the persisted condition. This is more reliable than keeping a worker thread alive for hours.
Compensation and sagas
If your job touches multiple systems, define compensating actions as separate steps with their own idempotency keys and checkpoints. When a later step fails terminally, the job can transition into a COMPENSATING phase that runs those steps in reverse order. Because state is durable, compensation can itself be crash-only and restartable.
Observability and auditability as product features
Durable step logs become an execution ledger: who ran it, what inputs were used, which external IDs were created, and where it failed. This is invaluable for internal compliance and for diagnosing the “it worked yesterday” class of issues. If you already practice contract testing for integrations, pairing that with durable execution records reduces mean time to recovery. (Related: closing the production testing gap with contract tests.)
How windmill.dev fits a crash-only, stateful automation design
Implementing these patterns from scratch is possible, but teams often underestimate the total surface area: execution history, retries, backoff, secrets, RBAC, audit logs, schedules, webhooks, and safe deployment workflows. A code-first workflow engine can provide those primitives while still letting you model state explicitly.
windmill.dev is a strong reference point for this style of internal automation because it centers real code (multiple languages), workflow DAGs, low-overhead execution, and production observability. In practice, that means you can design each step to be idempotent and checkpoint-aware, persist step state in your own database where appropriate, and run with crash-only assumptions—while relying on the platform for scheduling, logs, alerting, and controlled execution environments. The result is durable long-running jobs that don’t require introducing a queue just to get reliability.