wfe/README.md

WFE

A persistent, embeddable workflow engine for Rust. Trait-based, pluggable, built for real infrastructure.

Rust port of workflow-core, rebuilt from scratch with async/await, pluggable persistence, and a YAML frontend with shell and Deno executors.

---

What is WFE?

WFE is a workflow engine you embed directly into your Rust application. Define workflows as code using a fluent builder API, or as YAML files with shell and JavaScript steps. Workflows persist across restarts, support event-driven pausing, parallel execution, saga compensation, and distributed locking.

Built for:

• Persistent workflows — steps survive process restarts. Pick up where you left off.
• Embeddable CLIs — drop it into a binary, no external orchestrator required.
• Portable CI pipelines — YAML workflows with shell and Deno steps, variable interpolation, structured outputs.

---

Architecture

wfe/
├── wfe-core          Traits, models, builder, executor, primitives
├── wfe               Umbrella crate — WorkflowHost, WorkflowHostBuilder
├── wfe-yaml          YAML workflow loader, shell executor, Deno executor
├── wfe-sqlite        SQLite persistence + queue + lock provider
├── wfe-postgres      PostgreSQL persistence + queue + lock provider
├── wfe-valkey        Valkey (Redis) distributed lock + queue provider
└── wfe-opensearch    OpenSearch search index provider

wfe-core defines the traits. Provider crates implement them. wfe wires everything together through WorkflowHost. wfe-yaml adds a YAML frontend with built-in executors.

---

Quick start — Rust builder API

Define steps by implementing StepBody, then chain them with the builder:

use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use wfe::builder::WorkflowBuilder;
use wfe::models::*;
use wfe::traits::step::{StepBody, StepExecutionContext};

#[derive(Debug, Clone, Default, Serialize, Deserialize)]
struct MyData {
    message: String,
}

#[derive(Default)]
struct FetchData;

#[async_trait]
impl StepBody for FetchData {
    async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe::Result<ExecutionResult> {
        println!("Fetching data...");
        Ok(ExecutionResult::next())
    }
}

#[derive(Default)]
struct Transform;

#[async_trait]
impl StepBody for Transform {
    async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe::Result<ExecutionResult> {
        println!("Transforming...");
        Ok(ExecutionResult::next())
    }
}

#[derive(Default)]
struct Publish;

#[async_trait]
impl StepBody for Publish {
    async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe::Result<ExecutionResult> {
        println!("Publishing.");
        Ok(ExecutionResult::next())
    }
}

let definition = WorkflowBuilder::<MyData>::new()
    .start_with::<FetchData>()
        .name("Fetch")
    .then::<Transform>()
        .name("Transform")
        .on_error(ErrorBehavior::Retry {
            interval: std::time::Duration::from_secs(5),
            max_retries: 3,
        })
    .then::<Publish>()
        .name("Publish")
    .end_workflow()
    .build("etl-pipeline", 1);

The builder supports .then(), .parallel(), .if_do(), .while_do(), .for_each(), .saga(), .compensate_with(), .wait_for(), .delay(), and .then_fn() for inline closures.

See wfe/examples/pizza.rs for a full example using every feature.

---

Quick start — YAML

workflow:
  id: deploy-pipeline
  version: 1
  steps:
    - name: Lint
      config:
        run: cargo clippy --all-targets -- -D warnings
        timeout: "120s"

    - name: Test
      config:
        run: cargo test --workspace
        timeout: "300s"

    - name: Build
      config:
        run: cargo build --release
        timeout: "600s"

    - name: Notify
      type: deno
      config:
        script: |
          const result = await fetch("https://hooks.slack.com/...", {
            method: "POST",
            body: JSON.stringify({ text: "Deploy complete" }),
          });
          Wfe.setOutput("status", result.status.toString());
        permissions:
          net: ["hooks.slack.com"]
        timeout: "10s"

Load and run:

use std::collections::HashMap;
use std::path::Path;

let config = HashMap::new();
let compiled = wfe_yaml::load_workflow(Path::new("deploy.yaml"), &config)?;

Variables use ${{ var.name }} interpolation syntax. Outputs from earlier steps are available as workflow data in later steps.

---

Providers

Concern	Provider	Crate	Connection
Persistence	SQLite	wfe-sqlite	File path or :memory:
Persistence	PostgreSQL	wfe-postgres	postgres://user:pass@host/db
Distributed lock	Valkey / Redis	wfe-valkey	redis://host:6379
Queue	Valkey / Redis	wfe-valkey	Same connection
Search index	OpenSearch	wfe-opensearch	http://host:9200

All providers implement traits from wfe-core. SQLite and PostgreSQL crates include their own lock and queue implementations for single-node deployments. Use Valkey when you need distributed coordination across multiple hosts.

In-memory implementations of every trait ship with wfe-core (behind the test-support feature) for testing and prototyping.

---

The Deno executor

The deno step type embeds a V8 runtime for running JavaScript or TypeScript inside your workflow. Scripts run in a sandboxed environment with fine-grained permissions.

- name: Process webhook
  type: deno
  config:
    script: |
      const data = Wfe.getData();
      const response = await fetch(`https://api.example.com/v1/${data.id}`);
      const result = await response.json();
      Wfe.setOutput("processed", JSON.stringify(result));
    permissions:
      net: ["api.example.com"]
      read: []
      write: []
      env: []
      run: false
    timeout: "30s"

Permission	Type	Default	What it controls
net	string[]	[]	Allowed network hosts
read	string[]	[]	Allowed filesystem read paths
write	string[]	[]	Allowed filesystem write paths
env	string[]	[]	Allowed environment variable names
run	bool	false	Whether subprocess spawning is allowed
dynamic_import	bool	false	Whether dynamic import() is allowed

Everything is denied by default. You allowlist what each step needs. The V8 isolate is terminated hard on timeout — no infinite loops surviving on your watch.

Enable with the deno feature flag on wfe-yaml.

---

Feature flags

Crate	Flag	What it enables
wfe	otel	OpenTelemetry tracing (spans for every step execution)
wfe-core	otel	OTel span attributes on the executor
wfe-core	test-support	In-memory persistence, lock, and queue providers
wfe-yaml	deno	Deno JavaScript/TypeScript executor

---

Testing

Unit tests run without any external dependencies:

cargo test --workspace

Integration tests for PostgreSQL, Valkey, and OpenSearch need their backing services. A Docker Compose file is included:

docker compose up -d
cargo test --workspace
docker compose down

The compose file starts:

• PostgreSQL 17 on port 5433
• Valkey 8 on port 6379
• OpenSearch 2 on port 9200

SQLite tests use temporary files and run everywhere.

---

License

MIT

Built by Sunbeam Studios. We run this in production. It works.