37 Commits

Author SHA1 Message Date
a7c2eb1d9b chore: add sunbeam registry annotations for crate publishing 2026-03-27 00:35:42 +00:00
496a192198 chore: bump version to 1.4.0 2026-03-26 23:52:50 +00:00
d9e2c485f4 fix: pipeline coverage step produces valid JSON, deno reads it with readFile() 2026-03-26 23:37:34 +00:00
ed9c97ca32 fix: add host_context field to container executor test contexts 2026-03-26 23:37:24 +00:00
31a46ecbbd feat(wfe-yaml): add readFile() op to deno runtime with permission checking 2026-03-26 23:29:11 +00:00
d3426e5d82 feat(wfe-yaml): auto-convert ##wfe[output] values to typed JSON (bool, number) 2026-03-26 23:28:10 +00:00
ed38caecec fix(wfe-core): resolve .outputs. paths flat and pass empty object to child workflows 2026-03-26 23:18:48 +00:00
f0cc531ada docs: update README with condition system and task file include documentation 2026-03-26 17:26:11 +00:00
b1a1098fbc test(wfe-yaml): add condition schema, compiler, validation, and include tests 2026-03-26 17:25:26 +00:00
04c52c8158 feat(wfe-yaml): add task file includes with cycle detection and config override 2026-03-26 17:22:02 +00:00
1f14c9ac9a feat(wfe-yaml): add condition field path validation, type checking, and unused output detection 2026-03-26 17:21:50 +00:00
6c11473999 feat(wfe-yaml): compile YAML conditions into StepCondition with all operators 2026-03-26 17:21:28 +00:00
ced1916def feat(wfe-yaml): add YamlCondition types with combinator and comparison deserialization 2026-03-26 17:21:20 +00:00
57d4bdfb79 fix(wfe-postgres): add Skipped status to pointer status conversion 2026-03-26 17:20:28 +00:00
dd724e0a3c feat(wfe-core): integrate condition check into executor before step execution 2026-03-26 17:11:37 +00:00
ab1dbea329 feat(wfe-core): add condition evaluator with field path resolution and cascade skip 2026-03-26 17:10:05 +00:00
9c90f0a477 feat(wfe-core): add when condition field to WorkflowStep 2026-03-26 17:05:30 +00:00
aff3df6fcf feat(wfe-core): add StepCondition types and PointerStatus::Skipped 2026-03-26 17:05:14 +00:00
a71fa531f9 docs: add self-hosting CI pipeline section to README
Documents pipeline architecture, how to run it, WFE features
demonstrated, preflight tool checks, and graceful infrastructure
skipping. Adds nextest cover profile for llvm-cov integration.
2026-03-26 16:03:14 +00:00
aeb51614cb feat: self-hosting CI pipeline with 12 composable workflows
workflows.yaml defines the canonical CI pipeline: preflight → lint →
test (unit + integration + containers) → cover → package → tag →
publish → release, orchestrated by the ci workflow.

Demonstrates: nested workflows, typed I/O schemas, shell + deno executors,
YAML anchors with merge keys, variable interpolation, error handling with
retry, on_failure hooks, ensure hooks, infrastructure detection (docker/lima).

run_pipeline example loads and executes the pipeline with InMemory providers.
2026-03-26 16:01:51 +00:00
39b3daf57c feat(wfe-yaml): add YAML 1.1 merge key support via yaml-merge-keys
Preprocesses <<: *anchor merge keys before serde_yaml 0.9 deserialization.
serde_yaml implements YAML 1.2 which dropped merge keys; the yaml-merge-keys
crate resolves them as a preprocessing step, giving full anchor + merge
support for DRY pipeline definitions.
2026-03-26 15:59:28 +00:00
fe65d2debc fix(wfe-yaml): replace SubWorkflow placeholder with real implementation
The YAML compiler was using SubWorkflowPlaceholderStep that returned
next() immediately. Replaced with real SubWorkflowStep from wfe-core
that starts child workflows and waits for completion events.

Added regression test verifying the compiled factory produces a step
that calls host_context.start_workflow() and returns wait_for_event.
2026-03-26 15:58:47 +00:00
20f32531b7 chore: add nextest cover profile, update backward-compat imports
Nextest cover profile for cargo llvm-cov integration.
Update existing test imports from load_workflow_from_str to
load_single_workflow_from_str for backward compatibility.
2026-03-26 14:15:50 +00:00
856edbd22e feat(wfe): implement HostContext for nested workflow execution
HostContextImpl delegates start_workflow to persistence/registry/queue.
Background consumer passes host_context to executor so SubWorkflowStep
can start child workflows. SubWorkflowStep auto-registered as primitive.

E2E tests: parent-child workflow, typed inputs/outputs, child failure
propagation, nonexistent child definition. 90% line coverage.
2026-03-26 14:15:19 +00:00
bf252c51f0 feat(wfe-yaml): add workflow step type, cross-ref validation, cycle detection
Compiler dispatches type: workflow to SubWorkflowStep. Validation
detects circular workflow references via DFS with coloring. Cross-
workflow reference checking for multi-workflow files. Duplicate
workflow ID detection. 28 edge case tests for validation paths.
2026-03-26 14:14:39 +00:00
821ef2f570 feat(wfe-yaml): add multi-workflow YAML and typed input/output schemas
YamlWorkflowFile supports both single (workflow:) and multi (workflows:)
formats. WorkflowSpec gains typed inputs/outputs declarations.
Type string parser for inline types ("string?", "list<number>", etc.).
load_workflow_from_str returns Vec<CompiledWorkflow>.
Backward-compatible load_single_workflow_from_str convenience function.
2026-03-26 14:14:15 +00:00
a3211552a5 feat(wfe-core): add typed workflow schema system
SchemaType enum with inline syntax parsing: "string", "string?",
"list<number>", "map<string>", nested generics. WorkflowSchema
validates inputs/outputs against type declarations at both compile
time and runtime. 39 tests for parse and validate paths.
2026-03-26 14:12:51 +00:00
0317c6adea feat(wfe-buildkit): rewrite to use own generated protos (tonic 0.14)
Replaced third-party buildkit-client git dependency with
wfe-buildkit-protos generated from official moby/buildkit protos.

Direct ControlClient gRPC calls: SolveRequest with frontend attrs,
exporters, cache options. Daemon-local context paths for builds
(session protocol for remote transfer is TODO).

Both proto crates now use tonic 0.14 / prost 0.14 — no transitive
dependency conflicts. 95 combined tests, 85.6% region coverage.
2026-03-26 12:43:02 +00:00
2f861a9192 feat(wfe-buildkit-protos): generate full BuildKit gRPC API (tonic 0.14)
New crate generating Rust gRPC stubs from the official BuildKit
proto files (git submodule from moby/buildkit). Control service,
LLB definitions, session protocols, and source policy.
tonic 0.14 / prost 0.14.
2026-03-26 12:29:00 +00:00
27ce28e2ea feat(wfe-containerd): rewrite to use generated containerd gRPC protos
Replaced nerdctl CLI shell-out with direct gRPC communication via
wfe-containerd-protos (tonic 0.14). Connects to containerd daemon
over Unix socket.

Implementation:
- connect() with tonic Unix socket connector
- ensure_image() via ImagesClient (full pull is TODO)
- build_oci_spec() constructing OCI runtime spec with process args,
  env, user, cwd, mounts, and linux namespaces
- Container lifecycle: create → snapshot → task create → start →
  wait → read FIFOs → cleanup
- containerd-namespace header injection on every request

FIFO-based stdout/stderr capture using named pipes.
40 tests, 88% line coverage (cargo-llvm-cov).
2026-03-26 12:11:28 +00:00
d71f86a38b feat(wfe-containerd-protos): generate full containerd gRPC API (tonic 0.14)
New crate generating Rust gRPC stubs from the official containerd
proto files (vendored as git submodule). Full client-facing API surface
using tonic 0.14 / prost 0.14. No transitive dependency conflicts.

Services: containers, content, diff, events, images, introspection,
leases, mounts, namespaces, sandbox, snapshots, streaming, tasks,
transfer, version.
2026-03-26 12:00:46 +00:00
b02da21aac feat(wfe-buildkit): rewrite to use buildkit-client gRPC instead of CLI
Replaced buildctl CLI shell-out with direct gRPC communication via
buildkit-client crate. Connects to buildkitd daemon over Unix socket
or TCP with optional TLS.

Implementation:
- connect() with custom tonic UnixStream connector
- execute_build() implementing the solve protocol directly against
  ControlClient (session setup, file sync, frontend attributes)
- Extracts digest from containerimage.digest in solve response

Added custom lima template (test/lima/wfe-test.yaml) that provides
both buildkitd and containerd with host-forwarded Unix sockets for
reproducible integration testing.

E2E tests against real buildkitd daemon via WFE_BUILDKIT_ADDR env var.
54 tests total. 89% line coverage (cargo-llvm-cov with E2E).
2026-03-26 11:18:22 +00:00
30b26ca5f0 feat(wfe-buildkit, wfe-containerd): add container executor crates
Standalone workspace crates for BuildKit image building and containerd
container execution. Config types, YAML schema integration, compiler
dispatch, validation rules, and mock-based unit tests.

Current implementation shells out to buildctl/nerdctl — will be
replaced with proper gRPC clients (buildkit-client, containerd protos)
in a follow-up. Config types, YAML integration, and test infrastructure
are stable and reusable.

wfe-buildkit: 60 tests, 97.9% library coverage
wfe-containerd: 61 tests, 97.8% library coverage
447 total workspace tests.
2026-03-26 10:28:53 +00:00
d4519e862f feat(wfe-buildkit): add BuildKit image builder executor
Standalone crate implementing StepBody for building container images
via buildctl CLI. Supports Dockerfiles, multi-stage targets, tags,
build args, cache import/export, push to registry.

Security: TLS client certs for buildkitd connections, per-registry
authentication for push operations.

Testable without daemon via build_command() and parse_digest().
20 tests, 85%+ coverage.
2026-03-26 10:00:42 +00:00
4fc16646eb chore: add versions and sunbeam registry config for publishing 2026-03-26 01:02:34 +00:00
a26a088c69 chore: add versions to workspace path dependencies for crates.io 2026-03-26 01:00:19 +00:00
71d9821c4c chore: bump version to 1.0.0 and add repository metadata 2026-03-26 00:59:20 +00:00
74 changed files with 11613 additions and 139 deletions

2
.cargo/config.toml Normal file
View File

@@ -0,0 +1,2 @@
[registries.sunbeam]
index = "sparse+https://src.sunbeam.pt/api/packages/studio/cargo/"

View File

@@ -34,6 +34,17 @@ retries = 2
[profile.ci.junit]
path = "target/nextest/ci/junit.xml"
[profile.cover]
# Coverage profile — used with cargo llvm-cov nextest
fail-fast = false
test-threads = "num-cpus"
failure-output = "immediate-final"
success-output = "never"
slow-timeout = { period = "60s", terminate-after = 2 }
[profile.cover.junit]
path = "target/nextest/cover/junit.xml"
# Postgres tests must run serially (shared database state)
[[profile.default.overrides]]
filter = "package(wfe-postgres)"

6
.gitmodules vendored Normal file
View File

@@ -0,0 +1,6 @@
[submodule "wfe-containerd-protos/vendor/containerd"]
path = wfe-containerd-protos/vendor/containerd
url = https://github.com/containerd/containerd.git
[submodule "wfe-buildkit-protos/vendor/buildkit"]
path = wfe-buildkit-protos/vendor/buildkit
url = https://github.com/moby/buildkit.git

View File

@@ -1,11 +1,13 @@
[workspace]
members = ["wfe-core", "wfe-sqlite", "wfe-postgres", "wfe-opensearch", "wfe-valkey", "wfe", "wfe-yaml"]
members = ["wfe-core", "wfe-sqlite", "wfe-postgres", "wfe-opensearch", "wfe-valkey", "wfe", "wfe-yaml", "wfe-buildkit", "wfe-containerd", "wfe-containerd-protos", "wfe-buildkit-protos"]
resolver = "2"
[workspace.package]
version = "0.1.0"
version = "1.4.0"
edition = "2024"
license = "MIT"
repository = "https://src.sunbeam.pt/studio/wfe"
homepage = "https://src.sunbeam.pt/studio/wfe"
[workspace.dependencies]
# Core
@@ -36,15 +38,18 @@ redis = { version = "0.27", features = ["tokio-comp", "connection-manager"] }
opensearch = "2"
# Internal crates
wfe-core = { path = "wfe-core" }
wfe-sqlite = { path = "wfe-sqlite" }
wfe-postgres = { path = "wfe-postgres" }
wfe-opensearch = { path = "wfe-opensearch" }
wfe-valkey = { path = "wfe-valkey" }
wfe-yaml = { path = "wfe-yaml" }
wfe-core = { version = "1.4.0", path = "wfe-core", registry = "sunbeam" }
wfe-sqlite = { version = "1.4.0", path = "wfe-sqlite", registry = "sunbeam" }
wfe-postgres = { version = "1.4.0", path = "wfe-postgres", registry = "sunbeam" }
wfe-opensearch = { version = "1.4.0", path = "wfe-opensearch", registry = "sunbeam" }
wfe-valkey = { version = "1.4.0", path = "wfe-valkey", registry = "sunbeam" }
wfe-yaml = { version = "1.4.0", path = "wfe-yaml", registry = "sunbeam" }
wfe-buildkit = { version = "1.4.0", path = "wfe-buildkit", registry = "sunbeam" }
wfe-containerd = { version = "1.4.0", path = "wfe-containerd", registry = "sunbeam" }
# YAML
serde_yaml = "0.9"
yaml-merge-keys = { version = "0.8", features = ["serde_yaml"] }
regex = "1"
# Deno runtime

View File

@@ -1,20 +1,24 @@
# WFE
A persistent, embeddable workflow engine for Rust. Trait-based, pluggable, built for real infrastructure.
> Rust port of [workflow-core](https://github.com/danielgerlag/workflow-core), rebuilt from scratch with async/await, pluggable persistence, and a YAML frontend with shell and Deno executors.
A persistent, embeddable workflow engine for Rust. Trait-based, pluggable, built for large, highly complex build, test, deployment, and release pipelines of the highest levels of complexity.
---
## What is WFE?
WFE is a workflow engine you embed directly into your Rust application. Define workflows as code using a fluent builder API, or as YAML files with shell and JavaScript steps. Workflows persist across restarts, support event-driven pausing, parallel execution, saga compensation, and distributed locking.
WFE is a technical love letter from a former [VMware](https://www.vmware.com) and [Pivotal](https://pivotal.io) engineer (@siennathesane). Its internal workflow architecture is based on the amazing [workflow-core](https://github.com/danielgerlag/workflow-core) library by Daniel Gerlag, and the YAML structure and support is based on [Concourse CI](https://concourse-ci.org). WFE is a pluggable, extendable library that can be used to design embedded business workflows, CLI applications, and CI/CD pipelines. It is designed to be embedded into your application, and can scale open-ended with pluggable architectures. You can deploy Cloud Foundry, Kubernetes, or even bootstrap a public cloud with WFE.
Built for:
You can define workflows as code using a fluent builder API, or as YAML files with shell and JavaScript steps. Workflows persist across restarts, support event-driven pausing, parallel execution, saga compensation, and distributed locking. It also comes with native support for containerd and buildkitd embedded into the application.
- **Persistent workflows** — steps survive process restarts. Pick up where you left off.
- **Embeddable CLIs** — drop it into a binary, no external orchestrator required.
- **Portable CI pipelines** — YAML workflows with shell and Deno steps, variable interpolation, structured outputs.
The only thing not included is a server and a web UI (open to contributions that have been agreed upon and discussed ahead of time).
## Why?
Every CI/CD system [I've](https://src.sunbeam.pt/siennathesane) ever used has been either lovely or terrible. I wanted something that was lovely, but also flexible enough to be used in a variety of contexts. The list of CI systems I've used over the years has been extensive, and they all have their _thing_ that makes them stand out in their own ways. I wanted something that could meet every single requirement I have: a distributed workflow engine that can be embedded into any application, has pluggable executors, a statically-verifiable workflow definition language, a simple, easy to use API, YAML 1.1 merge anchors, YAML 1.2 support, multi-file workflows, can be used as a library, can be used as a CLI, and can be used to write servers.
With that, I wanted the user experience to be essentially identical for embedded application developers, systems engineers, and CI/CD engineers. Whether you write your workflows in code, YAML, or you have written a web server endpoint that accepts gRPC workflow definitions and you're relying on this library as a hosted service, it should feel like the same piece of software in every context.
Hell, I'm so dedicated to this being the most useful, most pragmatic, most embeddable workflow engine library ever that I'm willing to accept a C ABI contribution to make it embeddable in any language.
---
@@ -245,6 +249,76 @@ SQLite tests use temporary files and run everywhere.
---
## Self-hosting CI pipeline
WFE includes a self-hosting CI pipeline defined in `workflows.yaml` at the repository root. The pipeline uses WFE's own YAML workflow engine to build, test, and publish WFE itself.
### Pipeline architecture
```
ci (orchestrator)
|
+-------------------+--------------------+
| | |
preflight lint test (fan-out)
(tool check) (fmt + clippy) |
+----------+----------+
| | |
test-unit test-integration test-containers
| (docker compose) (lima VM)
| | |
+----------+----------+
|
+---------+---------+
| | |
cover package tag
| | |
+---------+---------+
|
+---------+---------+
| |
publish release
(crates.io) (git tags + notes)
```
### Running the pipeline
```sh
# Default — uses current directory as workspace
cargo run --example run_pipeline -p wfe -- workflows.yaml
# With explicit configuration
WFE_CONFIG='{"workspace_dir":"/path/to/wfe","registry":"sunbeam","git_remote":"origin","coverage_threshold":85}' \
cargo run --example run_pipeline -p wfe -- workflows.yaml
```
### WFE features demonstrated
The pipeline exercises every major WFE feature:
- **Workflow composition** — the `ci` orchestrator invokes child workflows (`lint`, `test`, `cover`, `package`, `tag`, `publish`, `release`) using the `workflow` step type.
- **Shell executor** — most steps run bash commands with configurable timeouts.
- **Deno executor** — the `cover` workflow uses a Deno step to parse coverage JSON; the `release` workflow uses Deno to generate release notes.
- **YAML anchors/templates** — `_templates` defines `shell_defaults` and `long_running` anchors, reused across steps via `<<: *shell_defaults`.
- **Structured outputs** — steps emit `##wfe[output key=value]` markers to pass data between steps and workflows.
- **Variable interpolation** — `((workspace_dir))` syntax passes inputs through workflow composition.
- **Error handling** — `on_failure` handlers, `error_behavior` with retry policies, and `ensure` blocks for cleanup (e.g., `docker-down`, `lima-down`).
### Preflight tool check
The `preflight` workflow runs first and checks for all required tools: `cargo`, `cargo-nextest`, `cargo-llvm-cov`, `docker`, `limactl`, `buildctl`, and `git`. Essential tools (cargo, nextest, git) cause a hard failure if missing. Optional tools (docker, lima, buildctl, llvm-cov) are reported but do not block the pipeline.
### Graceful infrastructure skipping
Integration and container tests handle missing infrastructure without failing:
- **test-integration**: The `docker-up` step checks if Docker is available. If `docker info` fails, it sets `docker_started=false` and exits cleanly. Subsequent steps (`postgres-tests`, `valkey-tests`, `opensearch-tests`) check this flag and skip if Docker is not running.
- **test-containers**: The `lima-up` step checks if `limactl` is installed. If missing, it sets `lima_started=false` and exits cleanly. The `buildkit-tests` and `containerd-tests` steps check this flag and skip accordingly.
This means the pipeline runs successfully on any machine with the essential Rust toolchain, reporting which optional tests were skipped rather than failing outright.
---
## License
[MIT](LICENSE)

50
test/lima/wfe-test.yaml Normal file
View File

@@ -0,0 +1,50 @@
# WFE Test VM — BuildKit + containerd with host-accessible sockets
#
# Provides both buildkitd and containerd daemons with Unix sockets
# forwarded to the host for integration testing.
#
# Usage:
# limactl start ./test/lima/wfe-test.yaml
#
# Sockets (on host after start):
# BuildKit: unix://$HOME/.lima/wfe-test/sock/buildkitd.sock
# containerd: unix://$HOME/.lima/wfe-test/sock/containerd.sock
#
# Verify:
# BUILDKIT_HOST="unix://$HOME/.lima/wfe-test/sock/buildkitd.sock" buildctl debug workers
# # containerd accessible via gRPC at unix://$HOME/.lima/wfe-test/sock/containerd.sock
#
# Teardown:
# limactl stop wfe-test
# limactl delete wfe-test
message: |
WFE integration test VM is ready.
BuildKit socket: unix://{{.Dir}}/sock/buildkitd.sock
containerd socket: unix://{{.Dir}}/sock/containerd.sock
Verify BuildKit:
BUILDKIT_HOST="unix://{{.Dir}}/sock/buildkitd.sock" buildctl debug workers
Run tests:
WFE_BUILDKIT_ADDR="unix://{{.Dir}}/sock/buildkitd.sock" \
WFE_CONTAINERD_ADDR="unix://{{.Dir}}/sock/containerd.sock" \
cargo nextest run -p wfe-buildkit -p wfe-containerd
minimumLimaVersion: 2.0.0
base: template:_images/ubuntu-lts
containerd:
system: false
user: true
portForwards:
# BuildKit daemon socket
- guestSocket: "/run/user/{{.UID}}/buildkit-default/buildkitd.sock"
hostSocket: "{{.Dir}}/sock/buildkitd.sock"
# containerd daemon socket (rootless)
- guestSocket: "/run/user/{{.UID}}/containerd/containerd.sock"
hostSocket: "{{.Dir}}/sock/containerd.sock"

View File

@@ -0,0 +1,19 @@
[package]
name = "wfe-buildkit-protos"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Generated gRPC stubs for the full BuildKit API"
[dependencies]
tonic = "0.14"
tonic-prost = "0.14"
prost = "0.14"
prost-types = "0.14"
[build-dependencies]
tonic-build = "0.14"
tonic-prost-build = "0.14"
prost-build = "0.14"

View File

@@ -0,0 +1,56 @@
use std::path::PathBuf;
fn main() -> Result<(), Box<dyn std::error::Error>> {
// Use Go-style import paths so protoc sees each file only once
let proto_dir = PathBuf::from("proto");
let go_prefix = "github.com/moby/buildkit";
let proto_files: Vec<PathBuf> = vec![
// Core control service (Solve, Status, ListWorkers, etc.)
"api/services/control/control.proto",
// Types
"api/types/worker.proto",
// Solver / LLB definitions
"solver/pb/ops.proto",
// Source policy
"sourcepolicy/pb/policy.proto",
// Session protocols
"session/auth/auth.proto",
"session/filesync/filesync.proto",
"session/secrets/secrets.proto",
"session/sshforward/ssh.proto",
"session/upload/upload.proto",
"session/exporter/exporter.proto",
// Utilities
"util/apicaps/pb/caps.proto",
"util/stack/stack.proto",
]
.into_iter()
.map(|p| proto_dir.join(go_prefix).join(p))
.collect();
println!(
"cargo:warning=Compiling {} buildkit proto files",
proto_files.len()
);
let mut prost_config = prost_build::Config::new();
prost_config.include_file("mod.rs");
tonic_prost_build::configure()
.build_server(false)
.compile_with_config(
prost_config,
&proto_files,
// Include paths for import resolution:
// 1. The vendor dir inside buildkit (for Go-style github.com/... imports)
// 2. The buildkit root itself (for relative imports)
// 3. Our proto/ dir (for google/rpc/status.proto)
&[
// proto/ has symlinks that resolve Go-style github.com/... imports
PathBuf::from("proto"),
],
)?;
Ok(())
}

View File

@@ -0,0 +1 @@
/Users/sienna/Development/sunbeam/wfe/wfe-buildkit-protos/vendor/buildkit

View File

@@ -0,0 +1,7 @@
// Stub for vtprotobuf extensions — not needed for Rust codegen
syntax = "proto3";
package vtproto;
import "google/protobuf/descriptor.proto";
extend google.protobuf.MessageOptions {
bool mempool = 64101;
}

View File

@@ -0,0 +1 @@
/Users/sienna/Development/sunbeam/wfe/wfe-buildkit-protos/vendor/buildkit/vendor/github.com/tonistiigi/fsutil

View File

@@ -0,0 +1,49 @@
// Copyright 2025 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
syntax = "proto3";
package google.rpc;
import "google/protobuf/any.proto";
option cc_enable_arenas = true;
option go_package = "google.golang.org/genproto/googleapis/rpc/status;status";
option java_multiple_files = true;
option java_outer_classname = "StatusProto";
option java_package = "com.google.rpc";
option objc_class_prefix = "RPC";
// The `Status` type defines a logical error model that is suitable for
// different programming environments, including REST APIs and RPC APIs. It is
// used by [gRPC](https://github.com/grpc). Each `Status` message contains
// three pieces of data: error code, error message, and error details.
//
// You can find out more about this error model and how to work with it in the
// [API Design Guide](https://cloud.google.com/apis/design/errors).
message Status {
// The status code, which should be an enum value of
// [google.rpc.Code][google.rpc.Code].
int32 code = 1;
// A developer-facing error message, which should be in English. Any
// user-facing error message should be localized and sent in the
// [google.rpc.Status.details][google.rpc.Status.details] field, or localized
// by the client.
string message = 2;
// A list of messages that carry the error details. There is a common set of
// message types for APIs to use.
repeated google.protobuf.Any details = 3;
}

View File

@@ -0,0 +1,19 @@
//! Generated gRPC stubs for the full BuildKit API.
//!
//! Built from the official BuildKit proto files at
//! <https://github.com/moby/buildkit>.
//!
//! ```rust,ignore
//! use wfe_buildkit_protos::moby::buildkit::v1::control_client::ControlClient;
//! use wfe_buildkit_protos::moby::buildkit::v1::StatusResponse;
//! ```
#![allow(clippy::all)]
#![allow(warnings)]
include!(concat!(env!("OUT_DIR"), "/mod.rs"));
/// Re-export tonic and prost for downstream convenience.
pub use prost;
pub use prost_types;
pub use tonic;

30
wfe-buildkit/Cargo.toml Normal file
View File

@@ -0,0 +1,30 @@
[package]
name = "wfe-buildkit"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "BuildKit image builder executor for WFE"
[dependencies]
wfe-core = { workspace = true }
tokio = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
async-trait = { workspace = true }
tracing = { workspace = true }
thiserror = { workspace = true }
regex = { workspace = true }
wfe-buildkit-protos = { version = "1.4.0", path = "../wfe-buildkit-protos", registry = "sunbeam" }
tonic = "0.14"
tower = { version = "0.4", features = ["util"] }
hyper-util = { version = "0.1", features = ["tokio"] }
uuid = { version = "1", features = ["v4"] }
tokio-stream = "0.1"
[dev-dependencies]
pretty_assertions = { workspace = true }
tempfile = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
tokio-util = "0.7"

95
wfe-buildkit/README.md Normal file
View File

@@ -0,0 +1,95 @@
# wfe-buildkit
BuildKit image builder executor for WFE.
## What it does
`wfe-buildkit` provides a `BuildkitStep` that implements the `StepBody` trait from `wfe-core`. It shells out to the `buildctl` CLI to build container images using BuildKit, capturing stdout/stderr and parsing image digests from the output.
## Quick start
Use it standalone:
```rust
use wfe_buildkit::{BuildkitConfig, BuildkitStep};
let config = BuildkitConfig {
dockerfile: "Dockerfile".to_string(),
context: ".".to_string(),
tags: vec!["myapp:latest".to_string()],
push: true,
..Default::default()
};
let step = BuildkitStep::new(config);
// Inspect the command that would be executed.
let args = step.build_command();
println!("{}", args.join(" "));
```
Or use it through `wfe-yaml` with the `buildkit` feature:
```yaml
workflow:
id: build-image
version: 1
steps:
- name: build
type: buildkit
config:
dockerfile: Dockerfile
context: .
tags:
- myapp:latest
- myapp:v1.0
push: true
build_args:
RUST_VERSION: "1.78"
cache_from:
- type=registry,ref=myapp:cache
cache_to:
- type=registry,ref=myapp:cache,mode=max
timeout: 10m
```
## Configuration
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `dockerfile` | String | Yes | - | Path to the Dockerfile |
| `context` | String | Yes | - | Build context directory |
| `target` | String | No | - | Multi-stage build target |
| `tags` | Vec\<String\> | No | [] | Image tags |
| `build_args` | Map\<String, String\> | No | {} | Build arguments |
| `cache_from` | Vec\<String\> | No | [] | Cache import sources |
| `cache_to` | Vec\<String\> | No | [] | Cache export destinations |
| `push` | bool | No | false | Push image after build |
| `output_type` | String | No | "image" | Output type: image, local, tar |
| `buildkit_addr` | String | No | unix:///run/buildkit/buildkitd.sock | BuildKit daemon address |
| `tls` | TlsConfig | No | - | TLS certificate paths |
| `registry_auth` | Map\<String, RegistryAuth\> | No | {} | Registry credentials |
| `timeout_ms` | u64 | No | - | Execution timeout in milliseconds |
## Output data
After execution, the step writes the following keys into `output_data`:
| Key | Description |
|---|---|
| `{step_name}.digest` | Image digest (sha256:...), if found in output |
| `{step_name}.tags` | Array of tags applied to the image |
| `{step_name}.stdout` | Full stdout from buildctl |
| `{step_name}.stderr` | Full stderr from buildctl |
## Testing
```sh
cargo test -p wfe-buildkit
```
The `build_command()` method returns the full argument list without executing, making it possible to test command construction without a running BuildKit daemon.
## License
MIT

358
wfe-buildkit/src/config.rs Normal file
View File

@@ -0,0 +1,358 @@
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
/// Configuration for a BuildKit image build step.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct BuildkitConfig {
/// Path to the Dockerfile (or directory containing it).
pub dockerfile: String,
/// Build context directory.
pub context: String,
/// Multi-stage build target.
pub target: Option<String>,
/// Image tags to apply.
#[serde(default)]
pub tags: Vec<String>,
/// Build arguments passed as `--opt build-arg:KEY=VALUE`.
#[serde(default)]
pub build_args: HashMap<String, String>,
/// Cache import sources.
#[serde(default)]
pub cache_from: Vec<String>,
/// Cache export destinations.
#[serde(default)]
pub cache_to: Vec<String>,
/// Whether to push the built image.
#[serde(default)]
pub push: bool,
/// Output type: "image", "local", "tar".
pub output_type: Option<String>,
/// BuildKit daemon address.
#[serde(default = "default_buildkit_addr")]
pub buildkit_addr: String,
/// TLS configuration for the BuildKit connection.
#[serde(default)]
pub tls: TlsConfig,
/// Registry authentication credentials keyed by registry host.
#[serde(default)]
pub registry_auth: HashMap<String, RegistryAuth>,
/// Execution timeout in milliseconds.
pub timeout_ms: Option<u64>,
}
/// TLS certificate paths for securing the BuildKit connection.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct TlsConfig {
/// Path to the CA certificate.
pub ca: Option<String>,
/// Path to the client certificate.
pub cert: Option<String>,
/// Path to the client private key.
pub key: Option<String>,
}
/// Credentials for authenticating with a container registry.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RegistryAuth {
pub username: String,
pub password: String,
}
fn default_buildkit_addr() -> String {
"unix:///run/buildkit/buildkitd.sock".to_string()
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
#[test]
fn serde_round_trip_full_config() {
let mut build_args = HashMap::new();
build_args.insert("RUST_VERSION".to_string(), "1.78".to_string());
let mut registry_auth = HashMap::new();
registry_auth.insert(
"ghcr.io".to_string(),
RegistryAuth {
username: "user".to_string(),
password: "pass".to_string(),
},
);
let config = BuildkitConfig {
dockerfile: "./Dockerfile".to_string(),
context: ".".to_string(),
target: Some("runtime".to_string()),
tags: vec!["myapp:latest".to_string(), "myapp:v1.0".to_string()],
build_args,
cache_from: vec!["type=registry,ref=myapp:cache".to_string()],
cache_to: vec!["type=registry,ref=myapp:cache,mode=max".to_string()],
push: true,
output_type: Some("image".to_string()),
buildkit_addr: "tcp://buildkitd:1234".to_string(),
tls: TlsConfig {
ca: Some("/certs/ca.pem".to_string()),
cert: Some("/certs/cert.pem".to_string()),
key: Some("/certs/key.pem".to_string()),
},
registry_auth,
timeout_ms: Some(300_000),
};
let json = serde_json::to_string(&config).unwrap();
let deserialized: BuildkitConfig = serde_json::from_str(&json).unwrap();
assert_eq!(config.dockerfile, deserialized.dockerfile);
assert_eq!(config.context, deserialized.context);
assert_eq!(config.target, deserialized.target);
assert_eq!(config.tags, deserialized.tags);
assert_eq!(config.build_args, deserialized.build_args);
assert_eq!(config.cache_from, deserialized.cache_from);
assert_eq!(config.cache_to, deserialized.cache_to);
assert_eq!(config.push, deserialized.push);
assert_eq!(config.output_type, deserialized.output_type);
assert_eq!(config.buildkit_addr, deserialized.buildkit_addr);
assert_eq!(config.tls.ca, deserialized.tls.ca);
assert_eq!(config.tls.cert, deserialized.tls.cert);
assert_eq!(config.tls.key, deserialized.tls.key);
assert_eq!(config.timeout_ms, deserialized.timeout_ms);
}
#[test]
fn serde_round_trip_minimal_config() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": "."
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.dockerfile, "Dockerfile");
assert_eq!(config.context, ".");
assert_eq!(config.target, None);
assert!(config.tags.is_empty());
assert!(config.build_args.is_empty());
assert!(config.cache_from.is_empty());
assert!(config.cache_to.is_empty());
assert!(!config.push);
assert_eq!(config.output_type, None);
assert_eq!(config.buildkit_addr, "unix:///run/buildkit/buildkitd.sock");
assert_eq!(config.timeout_ms, None);
// Round-trip
let serialized = serde_json::to_string(&config).unwrap();
let deserialized: BuildkitConfig = serde_json::from_str(&serialized).unwrap();
assert_eq!(config.dockerfile, deserialized.dockerfile);
assert_eq!(config.context, deserialized.context);
}
#[test]
fn default_buildkit_addr_value() {
let addr = default_buildkit_addr();
assert_eq!(addr, "unix:///run/buildkit/buildkitd.sock");
}
#[test]
fn tls_config_defaults_to_none() {
let tls = TlsConfig::default();
assert_eq!(tls.ca, None);
assert_eq!(tls.cert, None);
assert_eq!(tls.key, None);
}
#[test]
fn registry_auth_serde() {
let auth = RegistryAuth {
username: "admin".to_string(),
password: "secret123".to_string(),
};
let json = serde_json::to_string(&auth).unwrap();
let deserialized: RegistryAuth = serde_json::from_str(&json).unwrap();
assert_eq!(auth.username, deserialized.username);
assert_eq!(auth.password, deserialized.password);
}
#[test]
fn serde_custom_addr() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"buildkit_addr": "tcp://remote:1234"
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.buildkit_addr, "tcp://remote:1234");
}
#[test]
fn serde_with_timeout() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"timeout_ms": 60000
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.timeout_ms, Some(60000));
}
#[test]
fn serde_with_tags_and_push() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"tags": ["myapp:latest", "myapp:v1.0"],
"push": true
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.tags, vec!["myapp:latest", "myapp:v1.0"]);
assert!(config.push);
}
#[test]
fn serde_with_build_args() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"build_args": {"VERSION": "1.0", "DEBUG": "false"}
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.build_args.len(), 2);
assert_eq!(config.build_args["VERSION"], "1.0");
assert_eq!(config.build_args["DEBUG"], "false");
}
#[test]
fn serde_with_cache_config() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"cache_from": ["type=registry,ref=cache:latest"],
"cache_to": ["type=registry,ref=cache:latest,mode=max"]
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.cache_from.len(), 1);
assert_eq!(config.cache_to.len(), 1);
}
#[test]
fn serde_with_output_type() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"output_type": "tar"
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.output_type, Some("tar".to_string()));
}
#[test]
fn serde_with_registry_auth() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"registry_auth": {
"ghcr.io": {"username": "bot", "password": "tok"},
"docker.io": {"username": "u", "password": "p"}
}
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.registry_auth.len(), 2);
assert_eq!(config.registry_auth["ghcr.io"].username, "bot");
assert_eq!(config.registry_auth["docker.io"].password, "p");
}
#[test]
fn serde_with_tls() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"tls": {
"ca": "/certs/ca.pem",
"cert": "/certs/cert.pem",
"key": "/certs/key.pem"
}
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.tls.ca, Some("/certs/ca.pem".to_string()));
assert_eq!(config.tls.cert, Some("/certs/cert.pem".to_string()));
assert_eq!(config.tls.key, Some("/certs/key.pem".to_string()));
}
#[test]
fn serde_partial_tls() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"tls": {"ca": "/certs/ca.pem"}
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.tls.ca, Some("/certs/ca.pem".to_string()));
assert_eq!(config.tls.cert, None);
assert_eq!(config.tls.key, None);
}
#[test]
fn serde_empty_tls_object() {
let json = r#"{
"dockerfile": "Dockerfile",
"context": ".",
"tls": {}
}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.tls.ca, None);
assert_eq!(config.tls.cert, None);
assert_eq!(config.tls.key, None);
}
#[test]
fn tls_config_clone() {
let tls = TlsConfig {
ca: Some("ca".to_string()),
cert: Some("cert".to_string()),
key: Some("key".to_string()),
};
let cloned = tls.clone();
assert_eq!(tls.ca, cloned.ca);
assert_eq!(tls.cert, cloned.cert);
assert_eq!(tls.key, cloned.key);
}
#[test]
fn tls_config_debug() {
let tls = TlsConfig::default();
let debug = format!("{:?}", tls);
assert!(debug.contains("TlsConfig"));
}
#[test]
fn buildkit_config_debug() {
let json = r#"{"dockerfile": "Dockerfile", "context": "."}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
let debug = format!("{:?}", config);
assert!(debug.contains("BuildkitConfig"));
}
#[test]
fn registry_auth_clone() {
let auth = RegistryAuth {
username: "u".to_string(),
password: "p".to_string(),
};
let cloned = auth.clone();
assert_eq!(auth.username, cloned.username);
assert_eq!(auth.password, cloned.password);
}
#[test]
fn buildkit_config_clone() {
let json = r#"{"dockerfile": "Dockerfile", "context": "."}"#;
let config: BuildkitConfig = serde_json::from_str(json).unwrap();
let cloned = config.clone();
assert_eq!(config.dockerfile, cloned.dockerfile);
assert_eq!(config.context, cloned.context);
assert_eq!(config.buildkit_addr, cloned.buildkit_addr);
}
}

5
wfe-buildkit/src/lib.rs Normal file
View File

@@ -0,0 +1,5 @@
pub mod config;
pub mod step;
pub use config::{BuildkitConfig, RegistryAuth, TlsConfig};
pub use step::{build_output_data, parse_digest, BuildkitStep};

905
wfe-buildkit/src/step.rs Normal file
View File

@@ -0,0 +1,905 @@
use std::collections::HashMap;
use std::path::Path;
use async_trait::async_trait;
use regex::Regex;
use tokio_stream::StreamExt;
use tonic::transport::{Channel, Endpoint, Uri};
use wfe_buildkit_protos::moby::buildkit::v1::control_client::ControlClient;
use wfe_buildkit_protos::moby::buildkit::v1::{
CacheOptions, CacheOptionsEntry, Exporter, SolveRequest, StatusRequest,
};
use wfe_core::models::ExecutionResult;
use wfe_core::traits::step::{StepBody, StepExecutionContext};
use wfe_core::WfeError;
use crate::config::BuildkitConfig;
/// Result of a BuildKit solve operation.
#[derive(Debug, Clone)]
pub(crate) struct BuildResult {
/// Image digest produced by the build, if any.
pub digest: Option<String>,
/// Full exporter response metadata from the daemon.
#[allow(dead_code)]
pub metadata: HashMap<String, String>,
}
/// A workflow step that builds container images via the BuildKit gRPC API.
pub struct BuildkitStep {
config: BuildkitConfig,
}
impl BuildkitStep {
/// Create a new BuildKit step from configuration.
pub fn new(config: BuildkitConfig) -> Self {
Self { config }
}
/// Connect to the BuildKit daemon and return a raw `ControlClient`.
///
/// Supports Unix socket (`unix://`), TCP (`tcp://`), and HTTP (`http://`)
/// endpoints.
async fn connect(&self) -> Result<ControlClient<Channel>, WfeError> {
let addr = &self.config.buildkit_addr;
tracing::info!(addr = %addr, "connecting to BuildKit daemon");
let channel = if addr.starts_with("unix://") {
let socket_path = addr
.strip_prefix("unix://")
.unwrap()
.to_string();
// Verify the socket exists before attempting connection.
if !Path::new(&socket_path).exists() {
return Err(WfeError::StepExecution(format!(
"BuildKit socket not found: {socket_path}"
)));
}
// tonic requires a dummy URI for Unix sockets; the actual path
// is provided via the connector.
Endpoint::try_from("http://[::]:50051")
.map_err(|e| {
WfeError::StepExecution(format!("failed to create endpoint: {e}"))
})?
.connect_with_connector(tower::service_fn(move |_: Uri| {
let path = socket_path.clone();
async move {
tokio::net::UnixStream::connect(path)
.await
.map(hyper_util::rt::TokioIo::new)
}
}))
.await
.map_err(|e| {
WfeError::StepExecution(format!(
"failed to connect to buildkitd via Unix socket at {addr}: {e}"
))
})?
} else {
// TCP or HTTP endpoint.
let connect_addr = if addr.starts_with("tcp://") {
addr.replacen("tcp://", "http://", 1)
} else {
addr.clone()
};
Endpoint::from_shared(connect_addr.clone())
.map_err(|e| {
WfeError::StepExecution(format!(
"invalid BuildKit endpoint {connect_addr}: {e}"
))
})?
.timeout(std::time::Duration::from_secs(30))
.connect()
.await
.map_err(|e| {
WfeError::StepExecution(format!(
"failed to connect to buildkitd at {connect_addr}: {e}"
))
})?
};
Ok(ControlClient::new(channel))
}
/// Build frontend attributes from the current configuration.
///
/// These attributes tell the BuildKit dockerfile frontend how to process
/// the build: which file to use, which target stage, build arguments, etc.
fn build_frontend_attrs(&self) -> HashMap<String, String> {
let mut attrs = HashMap::new();
// Dockerfile filename (relative to context).
if self.config.dockerfile != "Dockerfile" {
attrs.insert("filename".to_string(), self.config.dockerfile.clone());
}
// Target stage for multi-stage builds.
if let Some(ref target) = self.config.target {
attrs.insert("target".to_string(), target.clone());
}
// Build arguments (sorted for determinism).
let mut sorted_args: Vec<_> = self.config.build_args.iter().collect();
sorted_args.sort_by_key(|(k, _)| (*k).clone());
for (key, value) in &sorted_args {
attrs.insert(format!("build-arg:{key}"), value.to_string());
}
attrs
}
/// Build exporter configuration for image output.
fn build_exporters(&self) -> Vec<Exporter> {
if self.config.tags.is_empty() {
return vec![];
}
let mut export_attrs = HashMap::new();
export_attrs.insert("name".to_string(), self.config.tags.join(","));
if self.config.push {
export_attrs.insert("push".to_string(), "true".to_string());
}
vec![Exporter {
r#type: "image".to_string(),
attrs: export_attrs,
}]
}
/// Build cache options from the configuration.
fn build_cache_options(&self) -> Option<CacheOptions> {
if self.config.cache_from.is_empty() && self.config.cache_to.is_empty() {
return None;
}
let imports = self
.config
.cache_from
.iter()
.map(|source| {
let mut attrs = HashMap::new();
attrs.insert("ref".to_string(), source.clone());
CacheOptionsEntry {
r#type: "registry".to_string(),
attrs,
}
})
.collect();
let exports = self
.config
.cache_to
.iter()
.map(|dest| {
let mut attrs = HashMap::new();
attrs.insert("ref".to_string(), dest.clone());
attrs.insert("mode".to_string(), "max".to_string());
CacheOptionsEntry {
r#type: "registry".to_string(),
attrs,
}
})
.collect();
Some(CacheOptions {
export_ref_deprecated: String::new(),
import_refs_deprecated: vec![],
export_attrs_deprecated: HashMap::new(),
exports,
imports,
})
}
/// Execute the build against a connected BuildKit daemon.
///
/// Constructs a `SolveRequest` using the dockerfile.v0 frontend and
/// sends it via the Control gRPC service. The build context must be
/// accessible to the daemon on its local filesystem (shared mount or
/// same machine).
///
/// # Session protocol
///
/// TODO: For remote daemons where the build context is not on the same
/// filesystem, a full session protocol implementation is needed to
/// transfer files. Currently we rely on the context directory being
/// available to buildkitd (e.g., via a shared mount in Lima/colima).
async fn execute_build(
&self,
control: &mut ControlClient<Channel>,
) -> Result<BuildResult, WfeError> {
let build_ref = format!("wfe-build-{}", uuid::Uuid::new_v4());
let session_id = uuid::Uuid::new_v4().to_string();
// Resolve the absolute context path.
let abs_context = std::fs::canonicalize(&self.config.context).map_err(|e| {
WfeError::StepExecution(format!(
"failed to resolve context path {}: {e}",
self.config.context
))
})?;
// Build frontend attributes with local context references.
let mut frontend_attrs = self.build_frontend_attrs();
// Point the frontend at the daemon-local context directory.
// The "context" attr tells the dockerfile frontend where to find
// the build context. For local builds we use the local source type
// with a shared-key reference.
let context_name = "context";
let dockerfile_name = "dockerfile";
frontend_attrs.insert(
"context".to_string(),
format!("local://{context_name}"),
);
frontend_attrs.insert(
format!("local-sessionid:{context_name}"),
session_id.clone(),
);
// Also provide the dockerfile source as a local reference.
frontend_attrs.insert(
"dockerfilekey".to_string(),
format!("local://{dockerfile_name}"),
);
frontend_attrs.insert(
format!("local-sessionid:{dockerfile_name}"),
session_id.clone(),
);
let request = SolveRequest {
r#ref: build_ref.clone(),
definition: None,
exporter_deprecated: String::new(),
exporter_attrs_deprecated: HashMap::new(),
session: session_id.clone(),
frontend: "dockerfile.v0".to_string(),
frontend_attrs,
cache: self.build_cache_options(),
entitlements: vec![],
frontend_inputs: HashMap::new(),
internal: false,
source_policy: None,
exporters: self.build_exporters(),
enable_session_exporter: false,
source_policy_session: String::new(),
};
// Attach session metadata headers so buildkitd knows which
// session provides the local source content.
let mut grpc_request = tonic::Request::new(request);
let metadata = grpc_request.metadata_mut();
// The x-docker-expose-session-uuid header tells buildkitd which
// session owns the local sources. The x-docker-expose-session-grpc-method
// header lists the gRPC methods the session implements.
if let Ok(key) =
"x-docker-expose-session-uuid"
.parse::<tonic::metadata::MetadataKey<tonic::metadata::Ascii>>()
&& let Ok(val) = session_id
.parse::<tonic::metadata::MetadataValue<tonic::metadata::Ascii>>()
{
metadata.insert(key, val);
}
// Advertise the filesync method so the daemon knows it can request
// local file content from our session.
if let Ok(key) =
"x-docker-expose-session-grpc-method"
.parse::<tonic::metadata::MetadataKey<tonic::metadata::Ascii>>()
{
if let Ok(val) = "/moby.filesync.v1.FileSync/DiffCopy"
.parse::<tonic::metadata::MetadataValue<tonic::metadata::Ascii>>()
{
metadata.append(key.clone(), val);
}
if let Ok(val) = "/moby.filesync.v1.Auth/Credentials"
.parse::<tonic::metadata::MetadataValue<tonic::metadata::Ascii>>()
{
metadata.append(key, val);
}
}
tracing::info!(
context = %abs_context.display(),
session_id = %session_id,
"sending solve request to BuildKit"
);
let response = control
.solve(grpc_request)
.await
.map_err(|e| WfeError::StepExecution(format!("BuildKit solve failed: {e}")))?;
let solve_response = response.into_inner();
// Monitor progress (non-blocking, best effort).
let status_request = StatusRequest {
r#ref: build_ref.clone(),
};
if let Ok(stream_resp) = control.status(status_request).await {
let mut stream = stream_resp.into_inner();
while let Some(Ok(status)) = stream.next().await {
for vertex in &status.vertexes {
if !vertex.name.is_empty() {
tracing::debug!(vertex = %vertex.name, "build progress");
}
}
}
}
// Extract digest.
let digest = solve_response
.exporter_response
.get("containerimage.digest")
.cloned();
tracing::info!(digest = ?digest, "build completed");
Ok(BuildResult {
digest,
metadata: solve_response.exporter_response,
})
}
/// Build environment variables for registry authentication.
///
/// This is still useful when the BuildKit daemon reads credentials from
/// environment variables rather than session-based auth.
pub fn build_registry_env(&self) -> HashMap<String, String> {
let mut env = HashMap::new();
for (host, auth) in &self.config.registry_auth {
let sanitized_host = host.replace(['.', '-'], "_").to_uppercase();
env.insert(
format!("BUILDKIT_HOST_{sanitized_host}_USERNAME"),
auth.username.clone(),
);
env.insert(
format!("BUILDKIT_HOST_{sanitized_host}_PASSWORD"),
auth.password.clone(),
);
}
env
}
}
/// Parse the image digest from buildctl or BuildKit progress output.
///
/// Looks for patterns like `exporting manifest sha256:<hex>` or
/// `digest: sha256:<hex>` or the raw `containerimage.digest` value.
pub fn parse_digest(output: &str) -> Option<String> {
let re = Regex::new(r"(?:exporting manifest |digest: )sha256:([a-f0-9]{64})").unwrap();
re.captures(output)
.map(|caps| format!("sha256:{}", &caps[1]))
}
/// Build the output data JSON object from step execution results.
///
/// Assembles a `serde_json::Value::Object` containing the step's stdout,
/// stderr, digest (if found), and tags (if any).
pub fn build_output_data(
step_name: &str,
stdout: &str,
stderr: &str,
digest: Option<&str>,
tags: &[String],
) -> serde_json::Value {
let mut outputs = serde_json::Map::new();
if let Some(digest) = digest {
outputs.insert(
format!("{step_name}.digest"),
serde_json::Value::String(digest.to_string()),
);
}
if !tags.is_empty() {
outputs.insert(
format!("{step_name}.tags"),
serde_json::Value::Array(
tags.iter()
.map(|t| serde_json::Value::String(t.clone()))
.collect(),
),
);
}
outputs.insert(
format!("{step_name}.stdout"),
serde_json::Value::String(stdout.to_string()),
);
outputs.insert(
format!("{step_name}.stderr"),
serde_json::Value::String(stderr.to_string()),
);
serde_json::Value::Object(outputs)
}
#[async_trait]
impl StepBody for BuildkitStep {
async fn run(
&mut self,
context: &StepExecutionContext<'_>,
) -> wfe_core::Result<ExecutionResult> {
let step_name = context.step.name.as_deref().unwrap_or("unknown");
// Connect to the BuildKit daemon.
let mut control = self.connect().await?;
tracing::info!(step = step_name, "submitting build to BuildKit");
// Execute the build with optional timeout.
let result = if let Some(timeout_ms) = self.config.timeout_ms {
let duration = std::time::Duration::from_millis(timeout_ms);
match tokio::time::timeout(duration, self.execute_build(&mut control)).await {
Ok(Ok(result)) => result,
Ok(Err(e)) => return Err(e),
Err(_) => {
return Err(WfeError::StepExecution(format!(
"BuildKit build timed out after {timeout_ms}ms"
)));
}
}
} else {
self.execute_build(&mut control).await?
};
// Extract digest from BuildResult.
let digest = result.digest.clone();
tracing::info!(
step = step_name,
digest = ?digest,
"build completed"
);
let output_data = build_output_data(
step_name,
"", // gRPC builds don't produce traditional stdout
"", // gRPC builds don't produce traditional stderr
digest.as_deref(),
&self.config.tags,
);
Ok(ExecutionResult {
proceed: true,
output_data: Some(output_data),
..Default::default()
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
use std::collections::HashMap;
use crate::config::{BuildkitConfig, RegistryAuth, TlsConfig};
fn minimal_config() -> BuildkitConfig {
BuildkitConfig {
dockerfile: "Dockerfile".to_string(),
context: ".".to_string(),
target: None,
tags: vec![],
build_args: HashMap::new(),
cache_from: vec![],
cache_to: vec![],
push: false,
output_type: None,
buildkit_addr: "unix:///run/buildkit/buildkitd.sock".to_string(),
tls: TlsConfig::default(),
registry_auth: HashMap::new(),
timeout_ms: None,
}
}
// ---------------------------------------------------------------
// build_registry_env tests
// ---------------------------------------------------------------
#[test]
fn build_registry_env_with_auth() {
let mut config = minimal_config();
config.registry_auth.insert(
"ghcr.io".to_string(),
RegistryAuth {
username: "user".to_string(),
password: "token".to_string(),
},
);
let step = BuildkitStep::new(config);
let env = step.build_registry_env();
assert_eq!(
env.get("BUILDKIT_HOST_GHCR_IO_USERNAME"),
Some(&"user".to_string())
);
assert_eq!(
env.get("BUILDKIT_HOST_GHCR_IO_PASSWORD"),
Some(&"token".to_string())
);
}
#[test]
fn build_registry_env_sanitizes_host() {
let mut config = minimal_config();
config.registry_auth.insert(
"my-registry.example.com".to_string(),
RegistryAuth {
username: "u".to_string(),
password: "p".to_string(),
},
);
let step = BuildkitStep::new(config);
let env = step.build_registry_env();
assert!(env.contains_key("BUILDKIT_HOST_MY_REGISTRY_EXAMPLE_COM_USERNAME"));
assert!(env.contains_key("BUILDKIT_HOST_MY_REGISTRY_EXAMPLE_COM_PASSWORD"));
}
#[test]
fn build_registry_env_empty_when_no_auth() {
let step = BuildkitStep::new(minimal_config());
let env = step.build_registry_env();
assert!(env.is_empty());
}
#[test]
fn build_registry_env_multiple_registries() {
let mut config = minimal_config();
config.registry_auth.insert(
"ghcr.io".to_string(),
RegistryAuth {
username: "gh_user".to_string(),
password: "gh_pass".to_string(),
},
);
config.registry_auth.insert(
"docker.io".to_string(),
RegistryAuth {
username: "dh_user".to_string(),
password: "dh_pass".to_string(),
},
);
let step = BuildkitStep::new(config);
let env = step.build_registry_env();
assert_eq!(env.len(), 4);
assert_eq!(env["BUILDKIT_HOST_GHCR_IO_USERNAME"], "gh_user");
assert_eq!(env["BUILDKIT_HOST_GHCR_IO_PASSWORD"], "gh_pass");
assert_eq!(env["BUILDKIT_HOST_DOCKER_IO_USERNAME"], "dh_user");
assert_eq!(env["BUILDKIT_HOST_DOCKER_IO_PASSWORD"], "dh_pass");
}
// ---------------------------------------------------------------
// parse_digest tests
// ---------------------------------------------------------------
#[test]
fn parse_digest_from_output() {
let output = "some build output\nexporting manifest sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789\ndone";
let digest = parse_digest(output);
assert_eq!(
digest,
Some(
"sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"
.to_string()
)
);
}
#[test]
fn parse_digest_with_digest_prefix() {
let output = "digest: sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef\n";
let digest = parse_digest(output);
assert_eq!(
digest,
Some(
"sha256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef"
.to_string()
)
);
}
#[test]
fn parse_digest_missing_returns_none() {
let output = "building image...\nall done!";
let digest = parse_digest(output);
assert_eq!(digest, None);
}
#[test]
fn parse_digest_partial_hash_returns_none() {
let output = "exporting manifest sha256:abcdef";
let digest = parse_digest(output);
assert_eq!(digest, None);
}
#[test]
fn parse_digest_empty_input() {
assert_eq!(parse_digest(""), None);
}
#[test]
fn parse_digest_wrong_prefix() {
let output =
"sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789";
assert_eq!(parse_digest(output), None);
}
#[test]
fn parse_digest_uppercase_hex_returns_none() {
let output = "exporting manifest sha256:ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789ABCDEF0123456789";
assert_eq!(parse_digest(output), None);
}
#[test]
fn parse_digest_multiline_with_noise() {
let output = r#"
[+] Building 12.3s (8/8) FINISHED
=> exporting to image
=> exporting manifest sha256:aabbccdd0011223344556677aabbccdd0011223344556677aabbccdd00112233
=> done
"#;
assert_eq!(
parse_digest(output),
Some("sha256:aabbccdd0011223344556677aabbccdd0011223344556677aabbccdd00112233".to_string())
);
}
#[test]
fn parse_digest_first_match_wins() {
let hash1 = "a".repeat(64);
let hash2 = "b".repeat(64);
let output = format!(
"exporting manifest sha256:{hash1}\ndigest: sha256:{hash2}"
);
let digest = parse_digest(&output).unwrap();
assert_eq!(digest, format!("sha256:{hash1}"));
}
// ---------------------------------------------------------------
// build_output_data tests
// ---------------------------------------------------------------
#[test]
fn build_output_data_with_digest_and_tags() {
let digest = "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789";
let tags = vec!["myapp:latest".to_string(), "myapp:v1".to_string()];
let result = build_output_data("build", "out", "err", Some(digest), &tags);
let obj = result.as_object().unwrap();
assert_eq!(obj["build.digest"], digest);
assert_eq!(
obj["build.tags"],
serde_json::json!(["myapp:latest", "myapp:v1"])
);
assert_eq!(obj["build.stdout"], "out");
assert_eq!(obj["build.stderr"], "err");
}
#[test]
fn build_output_data_without_digest() {
let result = build_output_data("step1", "hello", "", None, &[]);
let obj = result.as_object().unwrap();
assert!(!obj.contains_key("step1.digest"));
assert!(!obj.contains_key("step1.tags"));
assert_eq!(obj["step1.stdout"], "hello");
assert_eq!(obj["step1.stderr"], "");
}
#[test]
fn build_output_data_with_digest_no_tags() {
let digest = "sha256:0000000000000000000000000000000000000000000000000000000000000000";
let result = build_output_data("img", "ok", "warn", Some(digest), &[]);
let obj = result.as_object().unwrap();
assert_eq!(obj["img.digest"], digest);
assert!(!obj.contains_key("img.tags"));
assert_eq!(obj["img.stdout"], "ok");
assert_eq!(obj["img.stderr"], "warn");
}
#[test]
fn build_output_data_no_digest_with_tags() {
let tags = vec!["app:v2".to_string()];
let result = build_output_data("s", "", "", None, &tags);
let obj = result.as_object().unwrap();
assert!(!obj.contains_key("s.digest"));
assert_eq!(obj["s.tags"], serde_json::json!(["app:v2"]));
}
#[test]
fn build_output_data_empty_strings() {
let result = build_output_data("x", "", "", None, &[]);
let obj = result.as_object().unwrap();
assert_eq!(obj["x.stdout"], "");
assert_eq!(obj["x.stderr"], "");
assert_eq!(obj.len(), 2);
}
// ---------------------------------------------------------------
// build_frontend_attrs tests
// ---------------------------------------------------------------
#[test]
fn frontend_attrs_minimal() {
let step = BuildkitStep::new(minimal_config());
let attrs = step.build_frontend_attrs();
// Default Dockerfile name is not included (only non-default).
assert!(!attrs.contains_key("filename"));
assert!(!attrs.contains_key("target"));
}
#[test]
fn frontend_attrs_with_target() {
let mut config = minimal_config();
config.target = Some("runtime".to_string());
let step = BuildkitStep::new(config);
let attrs = step.build_frontend_attrs();
assert_eq!(attrs.get("target"), Some(&"runtime".to_string()));
}
#[test]
fn frontend_attrs_with_custom_dockerfile() {
let mut config = minimal_config();
config.dockerfile = "docker/Dockerfile.prod".to_string();
let step = BuildkitStep::new(config);
let attrs = step.build_frontend_attrs();
assert_eq!(
attrs.get("filename"),
Some(&"docker/Dockerfile.prod".to_string())
);
}
#[test]
fn frontend_attrs_with_build_args() {
let mut config = minimal_config();
config
.build_args
.insert("RUST_VERSION".to_string(), "1.78".to_string());
config
.build_args
.insert("BUILD_MODE".to_string(), "release".to_string());
let step = BuildkitStep::new(config);
let attrs = step.build_frontend_attrs();
assert_eq!(
attrs.get("build-arg:BUILD_MODE"),
Some(&"release".to_string())
);
assert_eq!(
attrs.get("build-arg:RUST_VERSION"),
Some(&"1.78".to_string())
);
}
// ---------------------------------------------------------------
// build_exporters tests
// ---------------------------------------------------------------
#[test]
fn exporters_empty_when_no_tags() {
let step = BuildkitStep::new(minimal_config());
assert!(step.build_exporters().is_empty());
}
#[test]
fn exporters_with_tags_and_push() {
let mut config = minimal_config();
config.tags = vec!["myapp:latest".to_string(), "myapp:v1.0".to_string()];
config.push = true;
let step = BuildkitStep::new(config);
let exporters = step.build_exporters();
assert_eq!(exporters.len(), 1);
assert_eq!(exporters[0].r#type, "image");
assert_eq!(
exporters[0].attrs.get("name"),
Some(&"myapp:latest,myapp:v1.0".to_string())
);
assert_eq!(
exporters[0].attrs.get("push"),
Some(&"true".to_string())
);
}
#[test]
fn exporters_with_tags_no_push() {
let mut config = minimal_config();
config.tags = vec!["myapp:latest".to_string()];
config.push = false;
let step = BuildkitStep::new(config);
let exporters = step.build_exporters();
assert_eq!(exporters.len(), 1);
assert!(!exporters[0].attrs.contains_key("push"));
}
// ---------------------------------------------------------------
// build_cache_options tests
// ---------------------------------------------------------------
#[test]
fn cache_options_none_when_empty() {
let step = BuildkitStep::new(minimal_config());
assert!(step.build_cache_options().is_none());
}
#[test]
fn cache_options_with_imports_and_exports() {
let mut config = minimal_config();
config.cache_from = vec!["type=registry,ref=myapp:cache".to_string()];
config.cache_to = vec!["type=registry,ref=myapp:cache,mode=max".to_string()];
let step = BuildkitStep::new(config);
let opts = step.build_cache_options().unwrap();
assert_eq!(opts.imports.len(), 1);
assert_eq!(opts.exports.len(), 1);
assert_eq!(opts.imports[0].r#type, "registry");
assert_eq!(opts.exports[0].r#type, "registry");
}
// ---------------------------------------------------------------
// connect helper tests
// ---------------------------------------------------------------
#[test]
fn tcp_addr_converted_to_http() {
let mut config = minimal_config();
config.buildkit_addr = "tcp://buildkitd:1234".to_string();
let step = BuildkitStep::new(config);
assert_eq!(step.config.buildkit_addr, "tcp://buildkitd:1234");
}
#[test]
fn unix_addr_preserved() {
let config = minimal_config();
let step = BuildkitStep::new(config);
assert!(step.config.buildkit_addr.starts_with("unix://"));
}
#[tokio::test]
async fn connect_to_missing_unix_socket_returns_error() {
let mut config = minimal_config();
config.buildkit_addr = "unix:///tmp/nonexistent-wfe-test.sock".to_string();
let step = BuildkitStep::new(config);
let err = step.connect().await.unwrap_err();
let msg = format!("{err}");
assert!(
msg.contains("socket not found"),
"expected 'socket not found' error, got: {msg}"
);
}
#[tokio::test]
async fn connect_to_invalid_tcp_returns_error() {
let mut config = minimal_config();
config.buildkit_addr = "tcp://127.0.0.1:1".to_string();
let step = BuildkitStep::new(config);
let err = step.connect().await.unwrap_err();
let msg = format!("{err}");
assert!(
msg.contains("failed to connect"),
"expected connection error, got: {msg}"
);
}
// ---------------------------------------------------------------
// BuildkitStep construction tests
// ---------------------------------------------------------------
#[test]
fn new_step_stores_config() {
let config = minimal_config();
let step = BuildkitStep::new(config.clone());
assert_eq!(step.config.dockerfile, "Dockerfile");
assert_eq!(step.config.context, ".");
}
}

View File

@@ -0,0 +1,234 @@
//! Integration tests for wfe-buildkit using a real BuildKit daemon.
//!
//! These tests require a running BuildKit daemon. The socket path is read
//! from `WFE_BUILDKIT_ADDR`, falling back to
//! `unix:///Users/sienna/.lima/wfe-test/sock/buildkitd.sock`.
//!
//! If the daemon is not available, the tests are skipped gracefully.
use std::collections::HashMap;
use std::path::Path;
use wfe_buildkit::config::{BuildkitConfig, TlsConfig};
use wfe_buildkit::BuildkitStep;
use wfe_core::models::{ExecutionPointer, WorkflowInstance, WorkflowStep};
use wfe_core::traits::step::{StepBody, StepExecutionContext};
/// Get the BuildKit daemon address from the environment or use the default.
fn buildkit_addr() -> String {
std::env::var("WFE_BUILDKIT_ADDR").unwrap_or_else(|_| {
"unix:///Users/sienna/.lima/wfe-test/sock/buildkitd.sock".to_string()
})
}
/// Check whether the BuildKit daemon socket is reachable.
fn buildkitd_available() -> bool {
let addr = buildkit_addr();
if let Some(path) = addr.strip_prefix("unix://") {
Path::new(path).exists()
} else {
// For TCP endpoints, optimistically assume available.
true
}
}
fn make_test_context(
step_name: &str,
) -> (
WorkflowStep,
ExecutionPointer,
WorkflowInstance,
) {
let mut step = WorkflowStep::new(0, "buildkit");
step.name = Some(step_name.to_string());
let pointer = ExecutionPointer::new(0);
let instance = WorkflowInstance::new("test-wf", 1, serde_json::json!({}));
(step, pointer, instance)
}
#[tokio::test]
async fn build_simple_dockerfile_via_grpc() {
if !buildkitd_available() {
eprintln!(
"SKIP: BuildKit daemon not available at {}",
buildkit_addr()
);
return;
}
// Create a temp directory with a trivial Dockerfile.
let tmp = tempfile::tempdir().unwrap();
let dockerfile = tmp.path().join("Dockerfile");
std::fs::write(
&dockerfile,
"FROM alpine:latest\nRUN echo built\n",
)
.unwrap();
let config = BuildkitConfig {
dockerfile: "Dockerfile".to_string(),
context: tmp.path().to_string_lossy().to_string(),
target: None,
tags: vec![],
build_args: HashMap::new(),
cache_from: vec![],
cache_to: vec![],
push: false,
output_type: None,
buildkit_addr: buildkit_addr(),
tls: TlsConfig::default(),
registry_auth: HashMap::new(),
timeout_ms: Some(120_000), // 2 minutes
};
let mut step = BuildkitStep::new(config);
let (ws, pointer, instance) = make_test_context("integration-build");
let cancel = tokio_util::sync::CancellationToken::new();
let ctx = StepExecutionContext {
item: None,
execution_pointer: &pointer,
persistence_data: None,
step: &ws,
workflow: &instance,
cancellation_token: cancel,
host_context: None,
};
let result = step.run(&ctx).await.expect("build should succeed");
assert!(result.proceed);
let data = result.output_data.expect("should have output_data");
let obj = data.as_object().expect("output_data should be an object");
// Without tags/push, BuildKit does not produce a digest in the exporter
// response. The build succeeds but the digest is absent.
assert!(
obj.contains_key("integration-build.stdout"),
"expected stdout key, got: {:?}",
obj.keys().collect::<Vec<_>>()
);
assert!(
obj.contains_key("integration-build.stderr"),
"expected stderr key, got: {:?}",
obj.keys().collect::<Vec<_>>()
);
// If a digest IS present (e.g., newer buildkitd versions), validate its format.
if let Some(digest_val) = obj.get("integration-build.digest") {
let digest = digest_val.as_str().unwrap();
assert!(
digest.starts_with("sha256:"),
"digest should start with sha256:, got: {digest}"
);
assert_eq!(
digest.len(),
7 + 64,
"digest should be sha256:<64hex>, got: {digest}"
);
}
}
#[tokio::test]
async fn build_with_build_args() {
if !buildkitd_available() {
eprintln!(
"SKIP: BuildKit daemon not available at {}",
buildkit_addr()
);
return;
}
let tmp = tempfile::tempdir().unwrap();
let dockerfile = tmp.path().join("Dockerfile");
std::fs::write(
&dockerfile,
"FROM alpine:latest\nARG MY_VAR=default\nRUN echo \"value=$MY_VAR\"\n",
)
.unwrap();
let mut build_args = HashMap::new();
build_args.insert("MY_VAR".to_string(), "custom_value".to_string());
let config = BuildkitConfig {
dockerfile: "Dockerfile".to_string(),
context: tmp.path().to_string_lossy().to_string(),
target: None,
tags: vec![],
build_args,
cache_from: vec![],
cache_to: vec![],
push: false,
output_type: None,
buildkit_addr: buildkit_addr(),
tls: TlsConfig::default(),
registry_auth: HashMap::new(),
timeout_ms: Some(120_000),
};
let mut step = BuildkitStep::new(config);
let (ws, pointer, instance) = make_test_context("build-args-test");
let cancel = tokio_util::sync::CancellationToken::new();
let ctx = StepExecutionContext {
item: None,
execution_pointer: &pointer,
persistence_data: None,
step: &ws,
workflow: &instance,
cancellation_token: cancel,
host_context: None,
};
let result = step.run(&ctx).await.expect("build with args should succeed");
assert!(result.proceed);
let data = result.output_data.expect("should have output_data");
let obj = data.as_object().unwrap();
// Build should complete and produce output data entries.
assert!(
obj.contains_key("build-args-test.stdout"),
"expected stdout key, got: {:?}",
obj.keys().collect::<Vec<_>>()
);
}
#[tokio::test]
async fn connect_to_unavailable_daemon_returns_error() {
// Use a deliberately wrong address to test error handling.
let config = BuildkitConfig {
dockerfile: "Dockerfile".to_string(),
context: ".".to_string(),
target: None,
tags: vec![],
build_args: HashMap::new(),
cache_from: vec![],
cache_to: vec![],
push: false,
output_type: None,
buildkit_addr: "unix:///tmp/nonexistent-buildkitd.sock".to_string(),
tls: TlsConfig::default(),
registry_auth: HashMap::new(),
timeout_ms: Some(5_000),
};
let mut step = BuildkitStep::new(config);
let (ws, pointer, instance) = make_test_context("error-test");
let cancel = tokio_util::sync::CancellationToken::new();
let ctx = StepExecutionContext {
item: None,
execution_pointer: &pointer,
persistence_data: None,
step: &ws,
workflow: &instance,
cancellation_token: cancel,
host_context: None,
};
let err = step.run(&ctx).await;
assert!(err.is_err(), "should fail when daemon is unavailable");
}

View File

@@ -0,0 +1,19 @@
[package]
name = "wfe-containerd-protos"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Generated gRPC stubs for the full containerd API"
[dependencies]
tonic = "0.14"
tonic-prost = "0.14"
prost = "0.14"
prost-types = "0.14"
[build-dependencies]
tonic-build = "0.14"
tonic-prost-build = "0.14"
prost-build = "0.14"

View File

@@ -0,0 +1,50 @@
use std::path::PathBuf;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let api_dir = PathBuf::from("vendor/containerd/api");
// Collect all .proto files, excluding internal runtime shim protos
let proto_files: Vec<PathBuf> = walkdir(&api_dir)?
.into_iter()
.filter(|p| {
let s = p.to_string_lossy();
!s.contains("/runtime/task/") && !s.contains("/runtime/sandbox/")
})
.collect();
println!(
"cargo:warning=Compiling {} containerd proto files",
proto_files.len()
);
let _out_dir = PathBuf::from(std::env::var("OUT_DIR")?);
// Use tonic-prost-build (the tonic 0.14 way)
let mut prost_config = prost_build::Config::new();
prost_config.include_file("mod.rs");
tonic_prost_build::configure()
.build_server(false)
.compile_with_config(
prost_config,
&proto_files,
&[api_dir, PathBuf::from("proto")],
)?;
Ok(())
}
/// Recursively collect all .proto files under a directory.
fn walkdir(dir: &PathBuf) -> Result<Vec<PathBuf>, Box<dyn std::error::Error>> {
let mut protos = Vec::new();
for entry in std::fs::read_dir(dir)? {
let entry = entry?;
let path = entry.path();
if path.is_dir() {
protos.extend(walkdir(&path)?);
} else if path.extension().is_some_and(|ext| ext == "proto") {
protos.push(path);
}
}
Ok(protos)
}

View File

@@ -0,0 +1,49 @@
// Copyright 2025 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
syntax = "proto3";
package google.rpc;
import "google/protobuf/any.proto";
option cc_enable_arenas = true;
option go_package = "google.golang.org/genproto/googleapis/rpc/status;status";
option java_multiple_files = true;
option java_outer_classname = "StatusProto";
option java_package = "com.google.rpc";
option objc_class_prefix = "RPC";
// The `Status` type defines a logical error model that is suitable for
// different programming environments, including REST APIs and RPC APIs. It is
// used by [gRPC](https://github.com/grpc). Each `Status` message contains
// three pieces of data: error code, error message, and error details.
//
// You can find out more about this error model and how to work with it in the
// [API Design Guide](https://cloud.google.com/apis/design/errors).
message Status {
// The status code, which should be an enum value of
// [google.rpc.Code][google.rpc.Code].
int32 code = 1;
// A developer-facing error message, which should be in English. Any
// user-facing error message should be localized and sent in the
// [google.rpc.Status.details][google.rpc.Status.details] field, or localized
// by the client.
string message = 2;
// A list of messages that carry the error details. There is a common set of
// message types for APIs to use.
repeated google.protobuf.Any details = 3;
}

View File

@@ -0,0 +1,27 @@
//! Generated gRPC stubs for the full containerd API.
//!
//! Built from the official containerd proto files at
//! <https://github.com/containerd/containerd/tree/main/api>.
//!
//! The module structure mirrors the protobuf package hierarchy:
//!
//! ```rust,ignore
//! use wfe_containerd_protos::containerd::services::containers::v1::containers_client::ContainersClient;
//! use wfe_containerd_protos::containerd::services::tasks::v1::tasks_client::TasksClient;
//! use wfe_containerd_protos::containerd::services::images::v1::images_client::ImagesClient;
//! use wfe_containerd_protos::containerd::services::version::v1::version_client::VersionClient;
//! use wfe_containerd_protos::containerd::types::Mount;
//! use wfe_containerd_protos::containerd::types::Descriptor;
//! ```
#![allow(clippy::all)]
#![allow(warnings)]
// tonic-build generates a mod.rs that defines the full module tree
// matching the protobuf package structure.
include!(concat!(env!("OUT_DIR"), "/mod.rs"));
/// Re-export tonic and prost for downstream convenience.
pub use prost;
pub use prost_types;
pub use tonic;

30
wfe-containerd/Cargo.toml Normal file
View File

@@ -0,0 +1,30 @@
[package]
name = "wfe-containerd"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "containerd container runner executor for WFE"
[dependencies]
wfe-core = { workspace = true }
wfe-containerd-protos = { version = "1.4.0", path = "../wfe-containerd-protos", registry = "sunbeam" }
tokio = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
async-trait = { workspace = true }
tracing = { workspace = true }
thiserror = { workspace = true }
tonic = "0.14"
tower = "0.5"
hyper-util = { version = "0.1", features = ["tokio"] }
prost-types = "0.14"
uuid = { version = "1", features = ["v4"] }
libc = "0.2"
[dev-dependencies]
pretty_assertions = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
tempfile = { workspace = true }
tokio-util = "0.7"

70
wfe-containerd/README.md Normal file
View File

@@ -0,0 +1,70 @@
# wfe-containerd
Containerd container runner executor for WFE.
## What it does
`wfe-containerd` runs containers via `nerdctl` as workflow steps. It pulls images, manages registry authentication, and executes containers with configurable networking, resource limits, volume mounts, and TLS settings. Output is captured and parsed for `##wfe[output key=value]` directives, following the same convention as the shell executor.
## Quick start
Add a containerd step to your YAML workflow:
```yaml
workflow:
id: container-pipeline
version: 1
steps:
- name: run-tests
type: containerd
config:
image: node:20-alpine
run: npm test
network: none
memory: 512m
cpu: "1.0"
timeout: 5m
env:
NODE_ENV: test
volumes:
- source: /workspace
target: /app
readonly: true
```
Enable the feature in `wfe-yaml`:
```toml
[dependencies]
wfe-yaml = { version = "1.0.0", features = ["containerd"] }
```
## Configuration
| Field | Type | Default | Description |
|---|---|---|---|
| `image` | `String` | required | Container image to run |
| `run` | `String` | - | Shell command (uses `sh -c`) |
| `command` | `Vec<String>` | - | Command array (mutually exclusive with `run`) |
| `env` | `HashMap` | `{}` | Environment variables |
| `volumes` | `Vec<VolumeMount>` | `[]` | Volume mounts |
| `working_dir` | `String` | - | Working directory inside container |
| `user` | `String` | `65534:65534` | User/group to run as (nobody by default) |
| `network` | `String` | `none` | Network mode: `none`, `host`, or `bridge` |
| `memory` | `String` | - | Memory limit (e.g. `512m`, `1g`) |
| `cpu` | `String` | - | CPU limit (e.g. `1.0`, `0.5`) |
| `pull` | `String` | `if-not-present` | Pull policy: `always`, `if-not-present`, `never` |
| `containerd_addr` | `String` | `/run/containerd/containerd.sock` | Containerd socket address |
| `tls` | `TlsConfig` | - | TLS configuration for containerd connection |
| `registry_auth` | `HashMap` | `{}` | Registry authentication per registry hostname |
| `timeout` | `String` | - | Execution timeout (e.g. `30s`, `5m`) |
## Output parsing
The step captures stdout and stderr. Lines matching `##wfe[output key=value]` are extracted as workflow outputs. Raw stdout, stderr, and exit code are also available under `{step_name}.stdout`, `{step_name}.stderr`, and `{step_name}.exit_code`.
## Security defaults
- Runs as nobody (`65534:65534`) by default
- Network disabled (`none`) by default
- Containers are always `--rm` (removed after execution)

View File

@@ -0,0 +1,226 @@
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ContainerdConfig {
pub image: String,
pub command: Option<Vec<String>>,
pub run: Option<String>,
#[serde(default)]
pub env: HashMap<String, String>,
#[serde(default)]
pub volumes: Vec<VolumeMountConfig>,
pub working_dir: Option<String>,
#[serde(default = "default_user")]
pub user: String,
#[serde(default = "default_network")]
pub network: String,
pub memory: Option<String>,
pub cpu: Option<String>,
#[serde(default = "default_pull")]
pub pull: String,
#[serde(default = "default_containerd_addr")]
pub containerd_addr: String,
/// CLI binary name: "nerdctl" (default) or "docker".
#[serde(default = "default_cli")]
pub cli: String,
#[serde(default)]
pub tls: TlsConfig,
#[serde(default)]
pub registry_auth: HashMap<String, RegistryAuth>,
pub timeout_ms: Option<u64>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VolumeMountConfig {
pub source: String,
pub target: String,
#[serde(default)]
pub readonly: bool,
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct TlsConfig {
pub ca: Option<String>,
pub cert: Option<String>,
pub key: Option<String>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RegistryAuth {
pub username: String,
pub password: String,
}
fn default_user() -> String {
"65534:65534".to_string()
}
fn default_network() -> String {
"none".to_string()
}
fn default_pull() -> String {
"if-not-present".to_string()
}
fn default_containerd_addr() -> String {
"/run/containerd/containerd.sock".to_string()
}
fn default_cli() -> String {
"nerdctl".to_string()
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
#[test]
fn serde_round_trip_full_config() {
let config = ContainerdConfig {
image: "alpine:3.18".to_string(),
command: Some(vec!["echo".to_string(), "hello".to_string()]),
run: None,
env: HashMap::from([("FOO".to_string(), "bar".to_string())]),
volumes: vec![VolumeMountConfig {
source: "/host/path".to_string(),
target: "/container/path".to_string(),
readonly: true,
}],
working_dir: Some("/app".to_string()),
user: "1000:1000".to_string(),
network: "host".to_string(),
memory: Some("512m".to_string()),
cpu: Some("1.0".to_string()),
pull: "always".to_string(),
containerd_addr: "/custom/containerd.sock".to_string(),
cli: "nerdctl".to_string(),
tls: TlsConfig {
ca: Some("/ca.pem".to_string()),
cert: Some("/cert.pem".to_string()),
key: Some("/key.pem".to_string()),
},
registry_auth: HashMap::from([(
"registry.example.com".to_string(),
RegistryAuth {
username: "user".to_string(),
password: "pass".to_string(),
},
)]),
timeout_ms: Some(30000),
};
let json = serde_json::to_string(&config).unwrap();
let deserialized: ContainerdConfig = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.image, config.image);
assert_eq!(deserialized.command, config.command);
assert_eq!(deserialized.run, config.run);
assert_eq!(deserialized.env, config.env);
assert_eq!(deserialized.volumes.len(), 1);
assert_eq!(deserialized.volumes[0].source, "/host/path");
assert_eq!(deserialized.volumes[0].readonly, true);
assert_eq!(deserialized.working_dir, Some("/app".to_string()));
assert_eq!(deserialized.user, "1000:1000");
assert_eq!(deserialized.network, "host");
assert_eq!(deserialized.memory, Some("512m".to_string()));
assert_eq!(deserialized.cpu, Some("1.0".to_string()));
assert_eq!(deserialized.pull, "always");
assert_eq!(deserialized.containerd_addr, "/custom/containerd.sock");
assert_eq!(deserialized.tls.ca, Some("/ca.pem".to_string()));
assert_eq!(deserialized.tls.cert, Some("/cert.pem".to_string()));
assert_eq!(deserialized.tls.key, Some("/key.pem".to_string()));
assert!(deserialized.registry_auth.contains_key("registry.example.com"));
assert_eq!(deserialized.timeout_ms, Some(30000));
}
#[test]
fn serde_round_trip_minimal_config() {
let json = r#"{"image": "alpine:latest"}"#;
let config: ContainerdConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.image, "alpine:latest");
assert_eq!(config.command, None);
assert_eq!(config.run, None);
assert!(config.env.is_empty());
assert!(config.volumes.is_empty());
assert_eq!(config.working_dir, None);
assert_eq!(config.user, "65534:65534");
assert_eq!(config.network, "none");
assert_eq!(config.memory, None);
assert_eq!(config.cpu, None);
assert_eq!(config.pull, "if-not-present");
assert_eq!(config.containerd_addr, "/run/containerd/containerd.sock");
assert_eq!(config.timeout_ms, None);
// Round-trip
let serialized = serde_json::to_string(&config).unwrap();
let deserialized: ContainerdConfig = serde_json::from_str(&serialized).unwrap();
assert_eq!(deserialized.image, "alpine:latest");
assert_eq!(deserialized.user, "65534:65534");
}
#[test]
fn default_values() {
let json = r#"{"image": "busybox"}"#;
let config: ContainerdConfig = serde_json::from_str(json).unwrap();
assert_eq!(config.user, "65534:65534");
assert_eq!(config.network, "none");
assert_eq!(config.pull, "if-not-present");
assert_eq!(config.containerd_addr, "/run/containerd/containerd.sock");
}
#[test]
fn volume_mount_serde() {
let vol = VolumeMountConfig {
source: "/data".to_string(),
target: "/mnt/data".to_string(),
readonly: false,
};
let json = serde_json::to_string(&vol).unwrap();
let deserialized: VolumeMountConfig = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.source, "/data");
assert_eq!(deserialized.target, "/mnt/data");
assert_eq!(deserialized.readonly, false);
// With readonly=true
let vol_ro = VolumeMountConfig {
source: "/src".to_string(),
target: "/dest".to_string(),
readonly: true,
};
let json_ro = serde_json::to_string(&vol_ro).unwrap();
let deserialized_ro: VolumeMountConfig = serde_json::from_str(&json_ro).unwrap();
assert_eq!(deserialized_ro.readonly, true);
}
#[test]
fn tls_config_defaults() {
let tls = TlsConfig::default();
assert_eq!(tls.ca, None);
assert_eq!(tls.cert, None);
assert_eq!(tls.key, None);
let json = r#"{}"#;
let deserialized: TlsConfig = serde_json::from_str(json).unwrap();
assert_eq!(deserialized.ca, None);
assert_eq!(deserialized.cert, None);
assert_eq!(deserialized.key, None);
}
#[test]
fn registry_auth_serde() {
let auth = RegistryAuth {
username: "admin".to_string(),
password: "secret123".to_string(),
};
let json = serde_json::to_string(&auth).unwrap();
let deserialized: RegistryAuth = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.username, "admin");
assert_eq!(deserialized.password, "secret123");
}
}

View File

@@ -0,0 +1,5 @@
pub mod config;
pub mod step;
pub use config::{ContainerdConfig, RegistryAuth, TlsConfig, VolumeMountConfig};
pub use step::ContainerdStep;

1097
wfe-containerd/src/step.rs Normal file

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,166 @@
//! Integration tests for the containerd gRPC-based runner.
//!
//! These tests require a live containerd daemon. They are skipped when the
//! socket is not available. Set `WFE_CONTAINERD_ADDR` to point to a custom
//! socket, or use the default `~/.lima/wfe-test/sock/containerd.sock`.
//!
//! Before running, ensure the test image is pre-pulled:
//! ctr -n default image pull docker.io/library/alpine:3.18
use std::collections::HashMap;
use std::path::Path;
use wfe_containerd::config::{ContainerdConfig, TlsConfig};
use wfe_containerd::ContainerdStep;
use wfe_core::models::{ExecutionPointer, WorkflowInstance, WorkflowStep};
use wfe_core::traits::step::{StepBody, StepExecutionContext};
/// Returns the containerd socket address if available, or None.
fn containerd_addr() -> Option<String> {
let addr = std::env::var("WFE_CONTAINERD_ADDR").unwrap_or_else(|_| {
format!(
"unix://{}/.lima/wfe-test/sock/containerd.sock",
std::env::var("HOME").unwrap_or_else(|_| "/root".to_string())
)
});
let socket_path = addr.strip_prefix("unix://").unwrap_or(addr.as_str());
if Path::new(socket_path).exists() {
Some(addr)
} else {
None
}
}
fn minimal_config(addr: &str) -> ContainerdConfig {
ContainerdConfig {
image: "docker.io/library/alpine:3.18".to_string(),
command: None,
run: Some("echo hello".to_string()),
env: HashMap::new(),
volumes: vec![],
working_dir: None,
user: "0:0".to_string(),
network: "none".to_string(),
memory: None,
cpu: None,
pull: "never".to_string(),
containerd_addr: addr.to_string(),
cli: "nerdctl".to_string(),
tls: TlsConfig::default(),
registry_auth: HashMap::new(),
timeout_ms: None,
}
}
fn make_context<'a>(
step: &'a WorkflowStep,
workflow: &'a WorkflowInstance,
pointer: &'a ExecutionPointer,
) -> StepExecutionContext<'a> {
StepExecutionContext {
item: None,
execution_pointer: pointer,
persistence_data: None,
step,
workflow,
cancellation_token: tokio_util::sync::CancellationToken::new(),
host_context: None,
}
}
// ── Connection error for missing socket ──────────────────────────────
#[tokio::test]
async fn connect_error_for_missing_socket() {
let config = minimal_config("/tmp/nonexistent-wfe-containerd-integ.sock");
let mut step = ContainerdStep::new(config);
let wf_step = WorkflowStep::new(0, "containerd");
let workflow = WorkflowInstance::new("test-wf", 1, serde_json::json!({}));
let pointer = ExecutionPointer::new(0);
let ctx = make_context(&wf_step, &workflow, &pointer);
let result = step.run(&ctx).await;
let err = result.expect_err("should fail with socket not found");
let msg = format!("{err}");
assert!(
msg.contains("socket not found"),
"expected 'socket not found' error, got: {msg}"
);
}
// ── Image check failure for non-existent image ──────────────────────
#[tokio::test]
async fn image_not_found_error() {
let Some(addr) = containerd_addr() else {
eprintln!("SKIP: containerd socket not available");
return;
};
let mut config = minimal_config(&addr);
config.image = "nonexistent-image-wfe-test:latest".to_string();
config.pull = "if-not-present".to_string();
let mut step = ContainerdStep::new(config);
let wf_step = WorkflowStep::new(0, "containerd");
let workflow = WorkflowInstance::new("test-wf", 1, serde_json::json!({}));
let pointer = ExecutionPointer::new(0);
let ctx = make_context(&wf_step, &workflow, &pointer);
let result = step.run(&ctx).await;
let err = result.expect_err("should fail with image not found");
let msg = format!("{err}");
assert!(
msg.contains("not found"),
"expected 'not found' error, got: {msg}"
);
}
// ── pull=never skips image check ─────────────────────────────────────
#[tokio::test]
async fn skip_image_check_when_pull_never() {
let Some(addr) = containerd_addr() else {
eprintln!("SKIP: containerd socket not available");
return;
};
// Using a non-existent image but pull=never should skip the check.
// The step will fail later at container creation, but the image check is skipped.
let mut config = minimal_config(&addr);
config.image = "nonexistent-image-wfe-test-never:latest".to_string();
config.pull = "never".to_string();
let mut step = ContainerdStep::new(config);
let wf_step = WorkflowStep::new(0, "containerd");
let workflow = WorkflowInstance::new("test-wf", 1, serde_json::json!({}));
let pointer = ExecutionPointer::new(0);
let ctx = make_context(&wf_step, &workflow, &pointer);
let result = step.run(&ctx).await;
// It should fail, but NOT with "not found in containerd" (image check).
// It should fail later (container creation, snapshot, etc.).
let err = result.expect_err("should fail at container or task creation");
let msg = format!("{err}");
assert!(
!msg.contains("Pre-pull it with"),
"image check should have been skipped for pull=never, got: {msg}"
);
}
// ── Step name defaults to "unknown" when None ────────────────────────
#[tokio::test]
async fn unnamed_step_uses_unknown_in_output_keys() {
// This test only verifies build_output_data behavior — no socket needed.
let parsed = HashMap::from([("result".to_string(), "ok".to_string())]);
let data = ContainerdStep::build_output_data("unknown", "out", "err", 0, &parsed);
let obj = data.as_object().unwrap();
assert!(obj.contains_key("unknown.stdout"));
assert!(obj.contains_key("unknown.stderr"));
assert!(obj.contains_key("unknown.exit_code"));
assert_eq!(obj.get("result").unwrap(), "ok");
}

View File

@@ -3,6 +3,8 @@ name = "wfe-core"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Core traits, models, builder, and executor for the WFE workflow engine"
[features]

View File

@@ -0,0 +1,761 @@
use crate::models::condition::{ComparisonOp, FieldComparison, StepCondition};
use crate::WfeError;
/// Evaluate a step condition against workflow data.
///
/// Returns `Ok(true)` if the step should run, `Ok(false)` if it should be skipped.
/// Missing field paths return `Ok(false)` (cascade skip behavior).
pub fn evaluate(
condition: &StepCondition,
workflow_data: &serde_json::Value,
) -> Result<bool, WfeError> {
match evaluate_inner(condition, workflow_data) {
Ok(result) => Ok(result),
Err(EvalError::FieldNotPresent) => Ok(false), // cascade skip
Err(EvalError::Wfe(e)) => Err(e),
}
}
/// Internal error type that distinguishes missing-field from real errors.
#[derive(Debug)]
enum EvalError {
FieldNotPresent,
Wfe(WfeError),
}
impl From<WfeError> for EvalError {
fn from(e: WfeError) -> Self {
EvalError::Wfe(e)
}
}
fn evaluate_inner(
condition: &StepCondition,
data: &serde_json::Value,
) -> Result<bool, EvalError> {
match condition {
StepCondition::All(conditions) => {
for c in conditions {
if !evaluate_inner(c, data)? {
return Ok(false);
}
}
Ok(true)
}
StepCondition::Any(conditions) => {
for c in conditions {
if evaluate_inner(c, data)? {
return Ok(true);
}
}
Ok(false)
}
StepCondition::None(conditions) => {
for c in conditions {
if evaluate_inner(c, data)? {
return Ok(false);
}
}
Ok(true)
}
StepCondition::OneOf(conditions) => {
let mut count = 0;
for c in conditions {
if evaluate_inner(c, data)? {
count += 1;
if count > 1 {
return Ok(false);
}
}
}
Ok(count == 1)
}
StepCondition::Not(inner) => {
let result = evaluate_inner(inner, data)?;
Ok(!result)
}
StepCondition::Comparison(comp) => evaluate_comparison(comp, data),
}
}
/// Resolve a dot-separated field path against JSON data.
///
/// Path starts with `.` which is stripped, then split by `.`.
/// Segments that parse as `usize` are treated as array indices.
fn resolve_field_path<'a>(
path: &str,
data: &'a serde_json::Value,
) -> Result<&'a serde_json::Value, EvalError> {
let path = path.strip_prefix('.').unwrap_or(path);
if path.is_empty() {
return Ok(data);
}
let segments: Vec<&str> = path.split('.').collect();
// Try resolving the full path first (for nested data like {"outputs": {"x": 1}}).
// If the first segment is "outputs"/"inputs" and doesn't exist as a key,
// strip it and resolve flat (for workflow data where outputs merge flat).
if segments.len() >= 2
&& (segments[0] == "outputs" || segments[0] == "inputs")
&& data.get(segments[0]).is_none()
{
return walk_segments(&segments[1..], data);
}
walk_segments(&segments, data)
}
fn walk_segments<'a>(
segments: &[&str],
data: &'a serde_json::Value,
) -> Result<&'a serde_json::Value, EvalError> {
let mut current = data;
for segment in segments {
if let Ok(idx) = segment.parse::<usize>() {
match current.as_array() {
Some(arr) => {
current = arr.get(idx).ok_or(EvalError::FieldNotPresent)?;
}
None => {
return Err(EvalError::FieldNotPresent);
}
}
} else {
match current.as_object() {
Some(obj) => {
current = obj.get(*segment).ok_or(EvalError::FieldNotPresent)?;
}
None => {
return Err(EvalError::FieldNotPresent);
}
}
}
}
Ok(current)
}
fn evaluate_comparison(
comp: &FieldComparison,
data: &serde_json::Value,
) -> Result<bool, EvalError> {
let resolved = resolve_field_path(&comp.field, data)?;
match &comp.operator {
ComparisonOp::IsNull => Ok(resolved.is_null()),
ComparisonOp::IsNotNull => Ok(!resolved.is_null()),
ComparisonOp::Equals => {
let expected = comp.value.as_ref().ok_or_else(|| {
EvalError::Wfe(WfeError::StepExecution(
"Equals operator requires a value".into(),
))
})?;
Ok(resolved == expected)
}
ComparisonOp::NotEquals => {
let expected = comp.value.as_ref().ok_or_else(|| {
EvalError::Wfe(WfeError::StepExecution(
"NotEquals operator requires a value".into(),
))
})?;
Ok(resolved != expected)
}
ComparisonOp::Gt => compare_numeric(resolved, comp, |a, b| a > b),
ComparisonOp::Gte => compare_numeric(resolved, comp, |a, b| a >= b),
ComparisonOp::Lt => compare_numeric(resolved, comp, |a, b| a < b),
ComparisonOp::Lte => compare_numeric(resolved, comp, |a, b| a <= b),
ComparisonOp::Contains => evaluate_contains(resolved, comp),
}
}
fn compare_numeric(
resolved: &serde_json::Value,
comp: &FieldComparison,
cmp_fn: fn(f64, f64) -> bool,
) -> Result<bool, EvalError> {
let expected = comp.value.as_ref().ok_or_else(|| {
EvalError::Wfe(WfeError::StepExecution(format!(
"{:?} operator requires a value",
comp.operator
)))
})?;
let a = resolved.as_f64().ok_or_else(|| {
EvalError::Wfe(WfeError::StepExecution(format!(
"cannot compare non-numeric field value: {}",
resolved
)))
})?;
let b = expected.as_f64().ok_or_else(|| {
EvalError::Wfe(WfeError::StepExecution(format!(
"cannot compare with non-numeric value: {}",
expected
)))
})?;
Ok(cmp_fn(a, b))
}
fn evaluate_contains(
resolved: &serde_json::Value,
comp: &FieldComparison,
) -> Result<bool, EvalError> {
let expected = comp.value.as_ref().ok_or_else(|| {
EvalError::Wfe(WfeError::StepExecution(
"Contains operator requires a value".into(),
))
})?;
// String contains substring.
if let Some(s) = resolved.as_str()
&& let Some(substr) = expected.as_str()
{
return Ok(s.contains(substr));
}
// Array contains element.
if let Some(arr) = resolved.as_array() {
return Ok(arr.contains(expected));
}
Err(EvalError::Wfe(WfeError::StepExecution(format!(
"Contains requires a string or array field, got {}",
value_type_name(resolved)
))))
}
fn value_type_name(value: &serde_json::Value) -> &'static str {
match value {
serde_json::Value::Null => "null",
serde_json::Value::Bool(_) => "bool",
serde_json::Value::Number(_) => "number",
serde_json::Value::String(_) => "string",
serde_json::Value::Array(_) => "array",
serde_json::Value::Object(_) => "object",
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::models::condition::{ComparisonOp, FieldComparison, StepCondition};
use serde_json::json;
// -- resolve_field_path tests --
#[test]
fn resolve_simple_field() {
let data = json!({"name": "alice"});
let result = resolve_field_path(".name", &data).unwrap();
assert_eq!(result, &json!("alice"));
}
#[test]
fn resolve_nested_field() {
let data = json!({"outputs": {"status": "success"}});
let result = resolve_field_path(".outputs.status", &data).unwrap();
assert_eq!(result, &json!("success"));
}
#[test]
fn resolve_missing_field() {
let data = json!({"name": "alice"});
let result = resolve_field_path(".missing", &data);
assert!(matches!(result, Err(EvalError::FieldNotPresent)));
}
#[test]
fn resolve_array_index() {
let data = json!({"items": [10, 20, 30]});
let result = resolve_field_path(".items.1", &data).unwrap();
assert_eq!(result, &json!(20));
}
#[test]
fn resolve_array_index_out_of_bounds() {
let data = json!({"items": [10, 20]});
let result = resolve_field_path(".items.5", &data);
assert!(matches!(result, Err(EvalError::FieldNotPresent)));
}
#[test]
fn resolve_deeply_nested() {
let data = json!({"a": {"b": {"c": {"d": 42}}}});
let result = resolve_field_path(".a.b.c.d", &data).unwrap();
assert_eq!(result, &json!(42));
}
#[test]
fn resolve_empty_path_returns_root() {
let data = json!({"x": 1});
let result = resolve_field_path(".", &data).unwrap();
assert_eq!(result, &data);
}
#[test]
fn resolve_field_on_non_object() {
let data = json!({"x": 42});
let result = resolve_field_path(".x.y", &data);
assert!(matches!(result, Err(EvalError::FieldNotPresent)));
}
// -- Comparison operator tests --
fn comp(field: &str, op: ComparisonOp, value: Option<serde_json::Value>) -> StepCondition {
StepCondition::Comparison(FieldComparison {
field: field.to_string(),
operator: op,
value,
})
}
#[test]
fn equals_match() {
let data = json!({"status": "ok"});
let cond = comp(".status", ComparisonOp::Equals, Some(json!("ok")));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn equals_mismatch() {
let data = json!({"status": "fail"});
let cond = comp(".status", ComparisonOp::Equals, Some(json!("ok")));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn equals_numeric() {
let data = json!({"count": 5});
let cond = comp(".count", ComparisonOp::Equals, Some(json!(5)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn not_equals_match() {
let data = json!({"status": "fail"});
let cond = comp(".status", ComparisonOp::NotEquals, Some(json!("ok")));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn not_equals_mismatch() {
let data = json!({"status": "ok"});
let cond = comp(".status", ComparisonOp::NotEquals, Some(json!("ok")));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn gt_match() {
let data = json!({"count": 10});
let cond = comp(".count", ComparisonOp::Gt, Some(json!(5)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn gt_mismatch() {
let data = json!({"count": 3});
let cond = comp(".count", ComparisonOp::Gt, Some(json!(5)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn gt_equal_is_false() {
let data = json!({"count": 5});
let cond = comp(".count", ComparisonOp::Gt, Some(json!(5)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn gte_match() {
let data = json!({"count": 5});
let cond = comp(".count", ComparisonOp::Gte, Some(json!(5)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn gte_mismatch() {
let data = json!({"count": 4});
let cond = comp(".count", ComparisonOp::Gte, Some(json!(5)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn lt_match() {
let data = json!({"count": 3});
let cond = comp(".count", ComparisonOp::Lt, Some(json!(5)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn lt_mismatch() {
let data = json!({"count": 7});
let cond = comp(".count", ComparisonOp::Lt, Some(json!(5)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn lte_match() {
let data = json!({"count": 5});
let cond = comp(".count", ComparisonOp::Lte, Some(json!(5)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn lte_mismatch() {
let data = json!({"count": 6});
let cond = comp(".count", ComparisonOp::Lte, Some(json!(5)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn contains_string_match() {
let data = json!({"msg": "hello world"});
let cond = comp(".msg", ComparisonOp::Contains, Some(json!("world")));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn contains_string_mismatch() {
let data = json!({"msg": "hello world"});
let cond = comp(".msg", ComparisonOp::Contains, Some(json!("xyz")));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn contains_array_match() {
let data = json!({"tags": ["a", "b", "c"]});
let cond = comp(".tags", ComparisonOp::Contains, Some(json!("b")));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn contains_array_mismatch() {
let data = json!({"tags": ["a", "b", "c"]});
let cond = comp(".tags", ComparisonOp::Contains, Some(json!("z")));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn is_null_true() {
let data = json!({"val": null});
let cond = comp(".val", ComparisonOp::IsNull, None);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn is_null_false() {
let data = json!({"val": 42});
let cond = comp(".val", ComparisonOp::IsNull, None);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn is_not_null_true() {
let data = json!({"val": 42});
let cond = comp(".val", ComparisonOp::IsNotNull, None);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn is_not_null_false() {
let data = json!({"val": null});
let cond = comp(".val", ComparisonOp::IsNotNull, None);
assert!(!evaluate(&cond, &data).unwrap());
}
// -- Combinator tests --
#[test]
fn all_both_true() {
let data = json!({"a": 1, "b": 2});
let cond = StepCondition::All(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn all_one_false() {
let data = json!({"a": 1, "b": 99});
let cond = StepCondition::All(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn all_empty_is_true() {
let data = json!({});
let cond = StepCondition::All(vec![]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn any_one_true() {
let data = json!({"a": 1, "b": 99});
let cond = StepCondition::Any(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn any_none_true() {
let data = json!({"a": 99, "b": 99});
let cond = StepCondition::Any(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn any_empty_is_false() {
let data = json!({});
let cond = StepCondition::Any(vec![]);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn none_all_false() {
let data = json!({"a": 99, "b": 99});
let cond = StepCondition::None(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn none_one_true() {
let data = json!({"a": 1, "b": 99});
let cond = StepCondition::None(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn none_empty_is_true() {
let data = json!({});
let cond = StepCondition::None(vec![]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn one_of_exactly_one_true() {
let data = json!({"a": 1, "b": 99});
let cond = StepCondition::OneOf(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn one_of_both_true() {
let data = json!({"a": 1, "b": 2});
let cond = StepCondition::OneOf(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn one_of_none_true() {
let data = json!({"a": 99, "b": 99});
let cond = StepCondition::OneOf(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".b", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn not_true_becomes_false() {
let data = json!({"a": 1});
let cond = StepCondition::Not(Box::new(comp(
".a",
ComparisonOp::Equals,
Some(json!(1)),
)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn not_false_becomes_true() {
let data = json!({"a": 99});
let cond = StepCondition::Not(Box::new(comp(
".a",
ComparisonOp::Equals,
Some(json!(1)),
)));
assert!(evaluate(&cond, &data).unwrap());
}
// -- Cascade skip tests --
#[test]
fn missing_field_returns_false_cascade_skip() {
let data = json!({"other": 1});
let cond = comp(".missing", ComparisonOp::Equals, Some(json!(1)));
// Missing field -> cascade skip -> Ok(false)
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn missing_nested_field_returns_false() {
let data = json!({"a": {"b": 1}});
let cond = comp(".a.c", ComparisonOp::Equals, Some(json!(1)));
assert!(!evaluate(&cond, &data).unwrap());
}
#[test]
fn missing_field_in_all_returns_false() {
let data = json!({"a": 1});
let cond = StepCondition::All(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".missing", ComparisonOp::Equals, Some(json!(2))),
]);
assert!(!evaluate(&cond, &data).unwrap());
}
// -- Nested combinator tests --
#[test]
fn nested_all_any_not() {
let data = json!({"a": 1, "b": 2, "c": 3});
// All(Any(a==1, a==99), Not(c==99))
let cond = StepCondition::All(vec![
StepCondition::Any(vec![
comp(".a", ComparisonOp::Equals, Some(json!(1))),
comp(".a", ComparisonOp::Equals, Some(json!(99))),
]),
StepCondition::Not(Box::new(comp(
".c",
ComparisonOp::Equals,
Some(json!(99)),
))),
]);
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn nested_any_of_alls() {
let data = json!({"x": 10, "y": 20});
// Any(All(x>5, y>25), All(x>5, y>15))
let cond = StepCondition::Any(vec![
StepCondition::All(vec![
comp(".x", ComparisonOp::Gt, Some(json!(5))),
comp(".y", ComparisonOp::Gt, Some(json!(25))),
]),
StepCondition::All(vec![
comp(".x", ComparisonOp::Gt, Some(json!(5))),
comp(".y", ComparisonOp::Gt, Some(json!(15))),
]),
]);
assert!(evaluate(&cond, &data).unwrap());
}
// -- Edge cases / error cases --
#[test]
fn gt_on_string_errors() {
let data = json!({"name": "alice"});
let cond = comp(".name", ComparisonOp::Gt, Some(json!(5)));
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn gt_with_string_value_errors() {
let data = json!({"count": 5});
let cond = comp(".count", ComparisonOp::Gt, Some(json!("not a number")));
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn contains_on_number_errors() {
let data = json!({"count": 42});
let cond = comp(".count", ComparisonOp::Contains, Some(json!("4")));
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn equals_without_value_errors() {
let data = json!({"a": 1});
let cond = comp(".a", ComparisonOp::Equals, None);
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn not_equals_without_value_errors() {
let data = json!({"a": 1});
let cond = comp(".a", ComparisonOp::NotEquals, None);
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn gt_without_value_errors() {
let data = json!({"a": 1});
let cond = comp(".a", ComparisonOp::Gt, None);
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn contains_without_value_errors() {
let data = json!({"msg": "hello"});
let cond = comp(".msg", ComparisonOp::Contains, None);
let result = evaluate(&cond, &data);
assert!(result.is_err());
}
#[test]
fn equals_bool_values() {
let data = json!({"active": true});
let cond = comp(".active", ComparisonOp::Equals, Some(json!(true)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn equals_null_value() {
let data = json!({"val": null});
let cond = comp(".val", ComparisonOp::Equals, Some(json!(null)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn float_comparison() {
let data = json!({"score": 3.14});
assert!(evaluate(&comp(".score", ComparisonOp::Gt, Some(json!(3.0))), &data).unwrap());
assert!(evaluate(&comp(".score", ComparisonOp::Lt, Some(json!(4.0))), &data).unwrap());
assert!(!evaluate(&comp(".score", ComparisonOp::Equals, Some(json!(3.0))), &data).unwrap());
}
#[test]
fn contains_array_numeric_element() {
let data = json!({"nums": [1, 2, 3]});
let cond = comp(".nums", ComparisonOp::Contains, Some(json!(2)));
assert!(evaluate(&cond, &data).unwrap());
}
#[test]
fn one_of_empty_is_false() {
let data = json!({});
let cond = StepCondition::OneOf(vec![]);
assert!(!evaluate(&cond, &data).unwrap());
}
}

View File

@@ -1,3 +1,4 @@
pub mod condition;
mod error_handler;
mod result_processor;
mod step_registry;

View File

@@ -3,11 +3,12 @@ use std::sync::Arc;
use chrono::Utc;
use tracing::{debug, error, info, warn};
use super::condition;
use super::error_handler;
use super::result_processor;
use super::step_registry::StepRegistry;
use crate::models::{
ExecutionError, PointerStatus, QueueType, WorkflowDefinition, WorkflowStatus,
Event, ExecutionError, PointerStatus, QueueType, WorkflowDefinition, WorkflowStatus,
};
use crate::traits::{
DistributedLockProvider, LifecyclePublisher, PersistenceProvider, QueueProvider, SearchIndex,
@@ -61,7 +62,7 @@ impl WorkflowExecutor {
/// 8. Release lock
#[tracing::instrument(
name = "workflow.execute",
skip(self, definition, step_registry),
skip(self, definition, step_registry, host_context),
fields(
workflow.id = %workflow_id,
workflow.definition_id,
@@ -73,6 +74,7 @@ impl WorkflowExecutor {
workflow_id: &str,
definition: &WorkflowDefinition,
step_registry: &StepRegistry,
host_context: Option<&dyn crate::traits::HostContext>,
) -> Result<()> {
// 1. Acquire distributed lock.
let acquired = self.lock_provider.acquire_lock(workflow_id).await?;
@@ -82,7 +84,7 @@ impl WorkflowExecutor {
}
let result = self
.execute_inner(workflow_id, definition, step_registry)
.execute_inner(workflow_id, definition, step_registry, host_context)
.await;
// 7. Release lock (always).
@@ -98,6 +100,7 @@ impl WorkflowExecutor {
workflow_id: &str,
definition: &WorkflowDefinition,
step_registry: &StepRegistry,
host_context: Option<&dyn crate::traits::HostContext>,
) -> Result<()> {
// 2. Load workflow instance.
let mut workflow = self
@@ -142,6 +145,41 @@ impl WorkflowExecutor {
.find(|s| s.id == step_id)
.ok_or(WfeError::StepNotFound(step_id))?;
// Check step condition before executing.
if let Some(ref when) = step.when {
match condition::evaluate(when, &workflow.data) {
Ok(true) => { /* condition met, proceed */ }
Ok(false) => {
info!(
workflow_id,
step_id,
step_name = step.name.as_deref().unwrap_or("(unnamed)"),
"Step skipped (condition not met)"
);
workflow.execution_pointers[idx].status = PointerStatus::Skipped;
workflow.execution_pointers[idx].active = false;
workflow.execution_pointers[idx].end_time = Some(Utc::now());
// Activate next step via outcomes (same as Complete).
let next_step_id = step.outcomes.first().map(|o| o.next_step);
if let Some(next_id) = next_step_id {
let mut next_pointer =
crate::models::ExecutionPointer::new(next_id);
next_pointer.predecessor_id =
Some(workflow.execution_pointers[idx].id.clone());
next_pointer.scope =
workflow.execution_pointers[idx].scope.clone();
workflow.execution_pointers.push(next_pointer);
}
continue;
}
Err(e) => {
return Err(e);
}
}
}
info!(
workflow_id,
step_id,
@@ -173,6 +211,7 @@ impl WorkflowExecutor {
step,
workflow: &workflow,
cancellation_token,
host_context,
};
// d. Call step.run(context).
@@ -270,6 +309,7 @@ impl WorkflowExecutor {
matches!(
p.status,
PointerStatus::Complete
| PointerStatus::Skipped
| PointerStatus::Compensated
| PointerStatus::Cancelled
| PointerStatus::Failed
@@ -280,6 +320,18 @@ impl WorkflowExecutor {
info!(workflow_id, "All pointers complete, workflow finished");
workflow.status = WorkflowStatus::Complete;
workflow.complete_time = Some(Utc::now());
// Publish completion event for SubWorkflow parents.
let completion_event = Event::new(
"wfe.workflow.completed",
workflow_id,
serde_json::json!({ "status": "Complete", "data": workflow.data }),
);
let _ = self.persistence.create_event(&completion_event).await;
let _ = self
.queue_provider
.queue_work(&completion_event.id, QueueType::Event)
.await;
}
tracing::Span::current().record("workflow.status", tracing::field::debug(&workflow.status));
@@ -573,7 +625,7 @@ mod tests {
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Complete);
@@ -604,7 +656,7 @@ mod tests {
persistence.create_new_workflow(&instance).await.unwrap();
// First execution: step 0 completes, step 1 pointer created.
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers.len(), 2);
@@ -613,7 +665,7 @@ mod tests {
assert_eq!(updated.execution_pointers[1].step_id, 1);
// Second execution: step 1 completes.
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Complete);
@@ -644,7 +696,7 @@ mod tests {
// Execute three times for three steps.
for _ in 0..3 {
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
}
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
@@ -684,7 +736,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers.len(), 2);
@@ -707,7 +759,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Runnable);
@@ -733,7 +785,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers[0].status, PointerStatus::Sleeping);
@@ -756,7 +808,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(
@@ -796,7 +848,7 @@ mod tests {
instance.execution_pointers.push(pointer);
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers[0].status, PointerStatus::Complete);
@@ -822,7 +874,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
// 1 original + 3 children.
@@ -858,7 +910,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers[0].retry_count, 1);
@@ -884,7 +936,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Suspended);
@@ -908,7 +960,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Terminated);
@@ -936,7 +988,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers[0].status, PointerStatus::Failed);
@@ -964,7 +1016,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(1));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Complete);
@@ -999,7 +1051,7 @@ mod tests {
persistence.create_new_workflow(&instance).await.unwrap();
// Should not error on a completed workflow.
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Complete);
@@ -1024,7 +1076,7 @@ mod tests {
instance.execution_pointers.push(pointer);
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
// Should still be sleeping since sleep_until is in the future.
@@ -1048,7 +1100,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let errors = persistence.get_errors().await;
assert_eq!(errors.len(), 1);
@@ -1072,7 +1124,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
// Executor itself doesn't publish lifecycle events in the current implementation,
// but the with_lifecycle builder works correctly.
@@ -1097,7 +1149,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Terminated);
@@ -1118,7 +1170,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert!(updated.execution_pointers[0].start_time.is_some());
@@ -1148,7 +1200,7 @@ mod tests {
instance.execution_pointers.push(ExecutionPointer::new(0));
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(
@@ -1203,13 +1255,13 @@ mod tests {
persistence.create_new_workflow(&instance).await.unwrap();
// First execution: fails, retry scheduled.
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers[0].retry_count, 1);
assert_eq!(updated.execution_pointers[0].status, PointerStatus::Sleeping);
// Second execution: succeeds (sleep_until is in the past with 0ms interval).
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.execution_pointers[0].status, PointerStatus::Complete);
assert_eq!(updated.status, WorkflowStatus::Complete);
@@ -1227,7 +1279,7 @@ mod tests {
// No execution pointers at all.
persistence.create_new_workflow(&instance).await.unwrap();
executor.execute(&instance.id, &def, &registry).await.unwrap();
executor.execute(&instance.id, &def, &registry, None).await.unwrap();
let updated = persistence.get_workflow_instance(&instance.id).await.unwrap();
assert_eq!(updated.status, WorkflowStatus::Runnable);

View File

@@ -0,0 +1,209 @@
use serde::{Deserialize, Serialize};
/// A condition that determines whether a workflow step should execute.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum StepCondition {
/// All sub-conditions must be true (AND).
All(Vec<StepCondition>),
/// At least one sub-condition must be true (OR).
Any(Vec<StepCondition>),
/// No sub-conditions may be true (NOR).
None(Vec<StepCondition>),
/// Exactly one sub-condition must be true (XOR).
OneOf(Vec<StepCondition>),
/// Negation of a single condition (NOT).
Not(Box<StepCondition>),
/// A leaf comparison against a field in workflow data.
Comparison(FieldComparison),
}
/// A comparison of a workflow data field against an expected value.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub struct FieldComparison {
/// Dot-separated field path, e.g. ".outputs.docker_started".
pub field: String,
/// The comparison operator.
pub operator: ComparisonOp,
/// The value to compare against. Required for all operators except IsNull/IsNotNull.
pub value: Option<serde_json::Value>,
}
/// Comparison operators for field conditions.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum ComparisonOp {
Equals,
NotEquals,
Gt,
Gte,
Lt,
Lte,
Contains,
IsNull,
IsNotNull,
}
#[cfg(test)]
mod tests {
use super::*;
use pretty_assertions::assert_eq;
use serde_json::json;
#[test]
fn comparison_op_serde_round_trip() {
for op in [
ComparisonOp::Equals,
ComparisonOp::NotEquals,
ComparisonOp::Gt,
ComparisonOp::Gte,
ComparisonOp::Lt,
ComparisonOp::Lte,
ComparisonOp::Contains,
ComparisonOp::IsNull,
ComparisonOp::IsNotNull,
] {
let json_str = serde_json::to_string(&op).unwrap();
let deserialized: ComparisonOp = serde_json::from_str(&json_str).unwrap();
assert_eq!(op, deserialized);
}
}
#[test]
fn field_comparison_serde_round_trip() {
let comp = FieldComparison {
field: ".outputs.status".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!("success")),
};
let json_str = serde_json::to_string(&comp).unwrap();
let deserialized: FieldComparison = serde_json::from_str(&json_str).unwrap();
assert_eq!(comp, deserialized);
}
#[test]
fn field_comparison_without_value_serde_round_trip() {
let comp = FieldComparison {
field: ".outputs.result".to_string(),
operator: ComparisonOp::IsNull,
value: None,
};
let json_str = serde_json::to_string(&comp).unwrap();
let deserialized: FieldComparison = serde_json::from_str(&json_str).unwrap();
assert_eq!(comp, deserialized);
}
#[test]
fn step_condition_comparison_serde_round_trip() {
let condition = StepCondition::Comparison(FieldComparison {
field: ".count".to_string(),
operator: ComparisonOp::Gt,
value: Some(json!(5)),
});
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
#[test]
fn step_condition_not_serde_round_trip() {
let condition = StepCondition::Not(Box::new(StepCondition::Comparison(FieldComparison {
field: ".active".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!(false)),
})));
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
#[test]
fn step_condition_all_serde_round_trip() {
let condition = StepCondition::All(vec![
StepCondition::Comparison(FieldComparison {
field: ".a".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!(1)),
}),
StepCondition::Comparison(FieldComparison {
field: ".b".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!(2)),
}),
]);
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
#[test]
fn step_condition_any_serde_round_trip() {
let condition = StepCondition::Any(vec![
StepCondition::Comparison(FieldComparison {
field: ".x".to_string(),
operator: ComparisonOp::IsNull,
value: None,
}),
]);
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
#[test]
fn step_condition_none_serde_round_trip() {
let condition = StepCondition::None(vec![
StepCondition::Comparison(FieldComparison {
field: ".err".to_string(),
operator: ComparisonOp::IsNotNull,
value: None,
}),
]);
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
#[test]
fn step_condition_one_of_serde_round_trip() {
let condition = StepCondition::OneOf(vec![
StepCondition::Comparison(FieldComparison {
field: ".mode".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!("fast")),
}),
StepCondition::Comparison(FieldComparison {
field: ".mode".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!("slow")),
}),
]);
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
#[test]
fn nested_combinator_serde_round_trip() {
let condition = StepCondition::All(vec![
StepCondition::Any(vec![
StepCondition::Comparison(FieldComparison {
field: ".a".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!(1)),
}),
StepCondition::Comparison(FieldComparison {
field: ".b".to_string(),
operator: ComparisonOp::Equals,
value: Some(json!(2)),
}),
]),
StepCondition::Not(Box::new(StepCondition::Comparison(FieldComparison {
field: ".c".to_string(),
operator: ComparisonOp::IsNull,
value: None,
}))),
]);
let json_str = serde_json::to_string(&condition).unwrap();
let deserialized: StepCondition = serde_json::from_str(&json_str).unwrap();
assert_eq!(condition, deserialized);
}
}

View File

@@ -1,3 +1,4 @@
pub mod condition;
pub mod error_behavior;
pub mod event;
pub mod execution_error;
@@ -7,10 +8,12 @@ pub mod lifecycle;
pub mod poll_config;
pub mod queue_type;
pub mod scheduled_command;
pub mod schema;
pub mod status;
pub mod workflow_definition;
pub mod workflow_instance;
pub use condition::{ComparisonOp, FieldComparison, StepCondition};
pub use error_behavior::ErrorBehavior;
pub use event::{Event, EventSubscription};
pub use execution_error::ExecutionError;
@@ -20,6 +23,7 @@ pub use lifecycle::{LifecycleEvent, LifecycleEventType};
pub use poll_config::{HttpMethod, PollCondition, PollEndpointConfig};
pub use queue_type::QueueType;
pub use scheduled_command::{CommandName, ScheduledCommand};
pub use schema::{SchemaType, WorkflowSchema};
pub use status::{PointerStatus, WorkflowStatus};
pub use workflow_definition::{StepOutcome, WorkflowDefinition, WorkflowStep};
pub use workflow_instance::WorkflowInstance;

View File

@@ -0,0 +1,483 @@
use std::collections::HashMap;
use serde::{Deserialize, Serialize};
/// Describes a single type in the workflow schema type system.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum SchemaType {
String,
Number,
Integer,
Bool,
Optional(Box<SchemaType>),
List(Box<SchemaType>),
Map(Box<SchemaType>),
Any,
}
/// Defines expected input and output schemas for a workflow.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct WorkflowSchema {
#[serde(default)]
pub inputs: HashMap<String, SchemaType>,
#[serde(default)]
pub outputs: HashMap<String, SchemaType>,
}
/// Parse a type string into a [`SchemaType`].
///
/// Supported formats:
/// - `"string"`, `"number"`, `"integer"`, `"bool"`, `"any"`
/// - `"string?"` (optional)
/// - `"list<string>"`, `"map<number>"` (generic containers)
/// - Nested: `"list<list<string>>"`
pub fn parse_type(s: &str) -> crate::Result<SchemaType> {
let s = s.trim();
// Handle optional suffix.
if let Some(inner) = s.strip_suffix('?') {
let inner_type = parse_type(inner)?;
return Ok(SchemaType::Optional(Box::new(inner_type)));
}
// Handle generic containers: list<...> and map<...>.
if let Some(rest) = s.strip_prefix("list<") {
let inner = rest
.strip_suffix('>')
.ok_or_else(|| crate::WfeError::StepExecution(format!("Invalid type syntax: {s}")))?;
let inner_type = parse_type(inner)?;
return Ok(SchemaType::List(Box::new(inner_type)));
}
if let Some(rest) = s.strip_prefix("map<") {
let inner = rest
.strip_suffix('>')
.ok_or_else(|| crate::WfeError::StepExecution(format!("Invalid type syntax: {s}")))?;
let inner_type = parse_type(inner)?;
return Ok(SchemaType::Map(Box::new(inner_type)));
}
// Primitive types.
match s {
"string" => Ok(SchemaType::String),
"number" => Ok(SchemaType::Number),
"integer" => Ok(SchemaType::Integer),
"bool" => Ok(SchemaType::Bool),
"any" => Ok(SchemaType::Any),
_ => Err(crate::WfeError::StepExecution(format!(
"Unknown type: {s}"
))),
}
}
/// Validate that a JSON value matches the expected [`SchemaType`].
pub fn validate_value(value: &serde_json::Value, expected: &SchemaType) -> Result<(), String> {
match expected {
SchemaType::String => {
if value.is_string() {
Ok(())
} else {
Err(format!("expected string, got {}", value_type_name(value)))
}
}
SchemaType::Number => {
if value.is_number() {
Ok(())
} else {
Err(format!("expected number, got {}", value_type_name(value)))
}
}
SchemaType::Integer => {
if value.is_i64() || value.is_u64() {
Ok(())
} else {
Err(format!("expected integer, got {}", value_type_name(value)))
}
}
SchemaType::Bool => {
if value.is_boolean() {
Ok(())
} else {
Err(format!("expected bool, got {}", value_type_name(value)))
}
}
SchemaType::Optional(inner) => {
if value.is_null() {
Ok(())
} else {
validate_value(value, inner)
}
}
SchemaType::List(inner) => {
if let Some(arr) = value.as_array() {
for (i, item) in arr.iter().enumerate() {
validate_value(item, inner)
.map_err(|e| format!("list element [{i}]: {e}"))?;
}
Ok(())
} else {
Err(format!("expected list, got {}", value_type_name(value)))
}
}
SchemaType::Map(inner) => {
if let Some(obj) = value.as_object() {
for (key, val) in obj {
validate_value(val, inner)
.map_err(|e| format!("map key \"{key}\": {e}"))?;
}
Ok(())
} else {
Err(format!("expected map, got {}", value_type_name(value)))
}
}
SchemaType::Any => Ok(()),
}
}
fn value_type_name(value: &serde_json::Value) -> &'static str {
match value {
serde_json::Value::Null => "null",
serde_json::Value::Bool(_) => "bool",
serde_json::Value::Number(_) => "number",
serde_json::Value::String(_) => "string",
serde_json::Value::Array(_) => "array",
serde_json::Value::Object(_) => "object",
}
}
impl WorkflowSchema {
/// Validate that the given data satisfies all input field requirements.
pub fn validate_inputs(&self, data: &serde_json::Value) -> Result<(), Vec<String>> {
self.validate_fields(&self.inputs, data)
}
/// Validate that the given data satisfies all output field requirements.
pub fn validate_outputs(&self, data: &serde_json::Value) -> Result<(), Vec<String>> {
self.validate_fields(&self.outputs, data)
}
fn validate_fields(
&self,
fields: &HashMap<String, SchemaType>,
data: &serde_json::Value,
) -> Result<(), Vec<String>> {
let obj = match data.as_object() {
Some(o) => o,
None => {
return Err(vec!["expected an object".to_string()]);
}
};
let mut errors = Vec::new();
for (name, schema_type) in fields {
match obj.get(name) {
Some(value) => {
if let Err(e) = validate_value(value, schema_type) {
errors.push(format!("field \"{name}\": {e}"));
}
}
None => {
// Missing field is OK for optional types (null is acceptable).
if !matches!(schema_type, SchemaType::Optional(_)) {
errors.push(format!("missing required field: \"{name}\""));
}
}
}
}
if errors.is_empty() {
Ok(())
} else {
Err(errors)
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
// -- parse_type tests --
#[test]
fn parse_type_string() {
assert_eq!(parse_type("string").unwrap(), SchemaType::String);
}
#[test]
fn parse_type_number() {
assert_eq!(parse_type("number").unwrap(), SchemaType::Number);
}
#[test]
fn parse_type_integer() {
assert_eq!(parse_type("integer").unwrap(), SchemaType::Integer);
}
#[test]
fn parse_type_bool() {
assert_eq!(parse_type("bool").unwrap(), SchemaType::Bool);
}
#[test]
fn parse_type_any() {
assert_eq!(parse_type("any").unwrap(), SchemaType::Any);
}
#[test]
fn parse_type_optional_string() {
assert_eq!(
parse_type("string?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::String))
);
}
#[test]
fn parse_type_optional_number() {
assert_eq!(
parse_type("number?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::Number))
);
}
#[test]
fn parse_type_list_string() {
assert_eq!(
parse_type("list<string>").unwrap(),
SchemaType::List(Box::new(SchemaType::String))
);
}
#[test]
fn parse_type_list_number() {
assert_eq!(
parse_type("list<number>").unwrap(),
SchemaType::List(Box::new(SchemaType::Number))
);
}
#[test]
fn parse_type_map_string() {
assert_eq!(
parse_type("map<string>").unwrap(),
SchemaType::Map(Box::new(SchemaType::String))
);
}
#[test]
fn parse_type_map_number() {
assert_eq!(
parse_type("map<number>").unwrap(),
SchemaType::Map(Box::new(SchemaType::Number))
);
}
#[test]
fn parse_type_nested_list() {
assert_eq!(
parse_type("list<list<string>>").unwrap(),
SchemaType::List(Box::new(SchemaType::List(Box::new(SchemaType::String))))
);
}
#[test]
fn parse_type_unknown_errors() {
assert!(parse_type("foobar").is_err());
}
#[test]
fn parse_type_trims_whitespace() {
assert_eq!(parse_type(" string ").unwrap(), SchemaType::String);
}
// -- validate_value tests --
#[test]
fn validate_string_match() {
assert!(validate_value(&json!("hello"), &SchemaType::String).is_ok());
}
#[test]
fn validate_string_mismatch() {
assert!(validate_value(&json!(42), &SchemaType::String).is_err());
}
#[test]
fn validate_number_match() {
assert!(validate_value(&json!(2.78), &SchemaType::Number).is_ok());
}
#[test]
fn validate_number_mismatch() {
assert!(validate_value(&json!("not a number"), &SchemaType::Number).is_err());
}
#[test]
fn validate_integer_match() {
assert!(validate_value(&json!(42), &SchemaType::Integer).is_ok());
}
#[test]
fn validate_integer_mismatch_float() {
assert!(validate_value(&json!(2.78), &SchemaType::Integer).is_err());
}
#[test]
fn validate_bool_match() {
assert!(validate_value(&json!(true), &SchemaType::Bool).is_ok());
}
#[test]
fn validate_bool_mismatch() {
assert!(validate_value(&json!(1), &SchemaType::Bool).is_err());
}
#[test]
fn validate_optional_null_passes() {
let ty = SchemaType::Optional(Box::new(SchemaType::String));
assert!(validate_value(&json!(null), &ty).is_ok());
}
#[test]
fn validate_optional_correct_inner_passes() {
let ty = SchemaType::Optional(Box::new(SchemaType::String));
assert!(validate_value(&json!("hello"), &ty).is_ok());
}
#[test]
fn validate_optional_wrong_inner_fails() {
let ty = SchemaType::Optional(Box::new(SchemaType::String));
assert!(validate_value(&json!(42), &ty).is_err());
}
#[test]
fn validate_list_match() {
let ty = SchemaType::List(Box::new(SchemaType::Number));
assert!(validate_value(&json!([1, 2, 3]), &ty).is_ok());
}
#[test]
fn validate_list_mismatch_element() {
let ty = SchemaType::List(Box::new(SchemaType::Number));
assert!(validate_value(&json!([1, "two", 3]), &ty).is_err());
}
#[test]
fn validate_list_not_array() {
let ty = SchemaType::List(Box::new(SchemaType::Number));
assert!(validate_value(&json!("not a list"), &ty).is_err());
}
#[test]
fn validate_map_match() {
let ty = SchemaType::Map(Box::new(SchemaType::Number));
assert!(validate_value(&json!({"a": 1, "b": 2}), &ty).is_ok());
}
#[test]
fn validate_map_mismatch_value() {
let ty = SchemaType::Map(Box::new(SchemaType::Number));
assert!(validate_value(&json!({"a": 1, "b": "two"}), &ty).is_err());
}
#[test]
fn validate_map_not_object() {
let ty = SchemaType::Map(Box::new(SchemaType::Number));
assert!(validate_value(&json!([1, 2]), &ty).is_err());
}
#[test]
fn validate_any_always_passes() {
assert!(validate_value(&json!(null), &SchemaType::Any).is_ok());
assert!(validate_value(&json!("str"), &SchemaType::Any).is_ok());
assert!(validate_value(&json!(42), &SchemaType::Any).is_ok());
assert!(validate_value(&json!([1, 2]), &SchemaType::Any).is_ok());
}
// -- WorkflowSchema validate_inputs / validate_outputs tests --
#[test]
fn validate_inputs_all_present() {
let schema = WorkflowSchema {
inputs: HashMap::from([
("name".into(), SchemaType::String),
("age".into(), SchemaType::Integer),
]),
outputs: HashMap::new(),
};
let data = json!({"name": "Alice", "age": 30});
assert!(schema.validate_inputs(&data).is_ok());
}
#[test]
fn validate_inputs_missing_required_field() {
let schema = WorkflowSchema {
inputs: HashMap::from([
("name".into(), SchemaType::String),
("age".into(), SchemaType::Integer),
]),
outputs: HashMap::new(),
};
let data = json!({"name": "Alice"});
let errs = schema.validate_inputs(&data).unwrap_err();
assert!(errs.iter().any(|e| e.contains("age")));
}
#[test]
fn validate_inputs_wrong_type() {
let schema = WorkflowSchema {
inputs: HashMap::from([("count".into(), SchemaType::Integer)]),
outputs: HashMap::new(),
};
let data = json!({"count": "not-a-number"});
let errs = schema.validate_inputs(&data).unwrap_err();
assert!(!errs.is_empty());
}
#[test]
fn validate_outputs_missing_field() {
let schema = WorkflowSchema {
inputs: HashMap::new(),
outputs: HashMap::from([("result".into(), SchemaType::String)]),
};
let data = json!({});
let errs = schema.validate_outputs(&data).unwrap_err();
assert!(errs.iter().any(|e| e.contains("result")));
}
#[test]
fn validate_inputs_optional_field_missing_is_ok() {
let schema = WorkflowSchema {
inputs: HashMap::from([(
"nickname".into(),
SchemaType::Optional(Box::new(SchemaType::String)),
)]),
outputs: HashMap::new(),
};
let data = json!({});
assert!(schema.validate_inputs(&data).is_ok());
}
#[test]
fn validate_not_object_errors() {
let schema = WorkflowSchema {
inputs: HashMap::from([("x".into(), SchemaType::String)]),
outputs: HashMap::new(),
};
let errs = schema.validate_inputs(&json!("not an object")).unwrap_err();
assert!(errs[0].contains("expected an object"));
}
#[test]
fn schema_serde_round_trip() {
let schema = WorkflowSchema {
inputs: HashMap::from([("name".into(), SchemaType::String)]),
outputs: HashMap::from([("result".into(), SchemaType::Bool)]),
};
let json_str = serde_json::to_string(&schema).unwrap();
let deserialized: WorkflowSchema = serde_json::from_str(&json_str).unwrap();
assert_eq!(deserialized.inputs["name"], SchemaType::String);
assert_eq!(deserialized.outputs["result"], SchemaType::Bool);
}
}

View File

@@ -15,6 +15,7 @@ pub enum PointerStatus {
Pending,
Running,
Complete,
Skipped,
Sleeping,
WaitingForEvent,
Failed,
@@ -58,6 +59,7 @@ mod tests {
PointerStatus::Pending,
PointerStatus::Running,
PointerStatus::Complete,
PointerStatus::Skipped,
PointerStatus::Sleeping,
PointerStatus::WaitingForEvent,
PointerStatus::Failed,

View File

@@ -2,6 +2,7 @@ use std::time::Duration;
use serde::{Deserialize, Serialize};
use super::condition::StepCondition;
use super::error_behavior::ErrorBehavior;
/// A compiled workflow definition ready for execution.
@@ -46,6 +47,9 @@ pub struct WorkflowStep {
/// Serializable configuration for primitive steps (e.g. event_name, duration).
#[serde(default, skip_serializing_if = "Option::is_none")]
pub step_config: Option<serde_json::Value>,
/// Optional condition that must evaluate to true for this step to execute.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub when: Option<StepCondition>,
}
impl WorkflowStep {
@@ -62,6 +66,7 @@ impl WorkflowStep {
do_compensate: false,
saga: false,
step_config: None,
when: None,
}
}
}

View File

@@ -45,6 +45,7 @@ impl WorkflowInstance {
matches!(
p.status,
PointerStatus::Complete
| PointerStatus::Skipped
| PointerStatus::Compensated
| PointerStatus::Cancelled
| PointerStatus::Failed

View File

@@ -8,6 +8,7 @@ pub mod recur;
pub mod saga_container;
pub mod schedule;
pub mod sequence;
pub mod sub_workflow;
pub mod wait_for;
pub mod while_step;
@@ -21,6 +22,7 @@ pub use recur::RecurStep;
pub use saga_container::SagaContainerStep;
pub use schedule::ScheduleStep;
pub use sequence::SequenceStep;
pub use sub_workflow::SubWorkflowStep;
pub use wait_for::WaitForStep;
pub use while_step::WhileStep;
@@ -42,6 +44,7 @@ mod test_helpers {
step,
workflow,
cancellation_token: CancellationToken::new(),
host_context: None,
}
}

View File

@@ -0,0 +1,444 @@
use async_trait::async_trait;
use chrono::Utc;
use crate::models::schema::WorkflowSchema;
use crate::models::ExecutionResult;
use crate::traits::step::{StepBody, StepExecutionContext};
/// A step that starts a child workflow and waits for its completion.
///
/// On first invocation, it validates inputs against `input_schema`, starts the
/// child workflow via the host context, and returns a "wait for event" result.
///
/// When the child workflow completes, the event data arrives, output keys are
/// extracted, and the step proceeds.
#[derive(Default)]
pub struct SubWorkflowStep {
/// The definition ID of the child workflow to start.
pub workflow_id: String,
/// The version of the child workflow definition.
pub version: u32,
/// Input data to pass to the child workflow.
pub inputs: serde_json::Value,
/// Keys to extract from the child workflow's completion event data.
pub output_keys: Vec<String>,
/// Optional schema to validate inputs before starting the child.
pub input_schema: Option<WorkflowSchema>,
/// Optional schema to validate outputs from the child.
pub output_schema: Option<WorkflowSchema>,
}
#[async_trait]
impl StepBody for SubWorkflowStep {
async fn run(&mut self, context: &StepExecutionContext<'_>) -> crate::Result<ExecutionResult> {
// If event data has arrived, the child workflow completed.
if let Some(event_data) = &context.execution_pointer.event_data {
// Extract output_keys from event data.
let mut output = serde_json::Map::new();
// The event data contains { "status": "...", "data": { ... } }.
let child_data = event_data
.get("data")
.cloned()
.unwrap_or(serde_json::Value::Null);
if self.output_keys.is_empty() {
// If no specific keys requested, pass all child data through.
if let serde_json::Value::Object(map) = child_data {
output = map;
}
} else {
// Extract only the requested keys.
for key in &self.output_keys {
if let Some(val) = child_data.get(key) {
output.insert(key.clone(), val.clone());
}
}
}
let output_value = serde_json::Value::Object(output);
// Validate against output schema if present.
if let Some(ref schema) = self.output_schema
&& let Err(errors) = schema.validate_outputs(&output_value)
{
return Err(crate::WfeError::StepExecution(format!(
"SubWorkflow output validation failed: {}",
errors.join("; ")
)));
}
let mut result = ExecutionResult::next();
result.output_data = Some(output_value);
return Ok(result);
}
// Hydrate from step_config if our fields are empty (created via Default).
if self.workflow_id.is_empty()
&& let Some(config) = &context.step.step_config
{
if let Some(wf_id) = config.get("workflow_id").and_then(|v| v.as_str()) {
self.workflow_id = wf_id.to_string();
}
if let Some(ver) = config.get("version").and_then(|v| v.as_u64()) {
self.version = ver as u32;
}
if let Some(inputs) = config.get("inputs") {
self.inputs = inputs.clone();
}
if let Some(keys) = config.get("output_keys").and_then(|v| v.as_array()) {
self.output_keys = keys
.iter()
.filter_map(|v| v.as_str().map(|s| s.to_string()))
.collect();
}
}
// First call: validate inputs and start child workflow.
if let Some(ref schema) = self.input_schema
&& let Err(errors) = schema.validate_inputs(&self.inputs)
{
return Err(crate::WfeError::StepExecution(format!(
"SubWorkflow input validation failed: {}",
errors.join("; ")
)));
}
let host = context.host_context.ok_or_else(|| {
crate::WfeError::StepExecution(
"SubWorkflowStep requires a host context to start child workflows".to_string(),
)
})?;
// Use inputs if set, otherwise pass an empty object so the child
// workflow has a valid JSON object for storing step outputs.
let child_data = if self.inputs.is_null() {
serde_json::json!({})
} else {
self.inputs.clone()
};
let child_instance_id = host
.start_workflow(&self.workflow_id, self.version, child_data)
.await?;
Ok(ExecutionResult::wait_for_event(
"wfe.workflow.completed",
child_instance_id,
Utc::now(),
))
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::models::schema::SchemaType;
use crate::models::ExecutionPointer;
use crate::primitives::test_helpers::*;
use crate::traits::step::HostContext;
use serde_json::json;
use std::collections::HashMap;
use std::sync::Mutex;
/// A mock HostContext that records calls and returns a fixed instance ID.
struct MockHostContext {
started: Mutex<Vec<(String, u32, serde_json::Value)>>,
result_id: String,
}
impl MockHostContext {
fn new(result_id: &str) -> Self {
Self {
started: Mutex::new(Vec::new()),
result_id: result_id.to_string(),
}
}
fn calls(&self) -> Vec<(String, u32, serde_json::Value)> {
self.started.lock().unwrap().clone()
}
}
impl HostContext for MockHostContext {
fn start_workflow(
&self,
definition_id: &str,
version: u32,
data: serde_json::Value,
) -> std::pin::Pin<Box<dyn std::future::Future<Output = crate::Result<String>> + Send + '_>>
{
let def_id = definition_id.to_string();
let result_id = self.result_id.clone();
Box::pin(async move {
self.started
.lock()
.unwrap()
.push((def_id, version, data));
Ok(result_id)
})
}
}
/// A mock HostContext that returns an error.
struct FailingHostContext;
impl HostContext for FailingHostContext {
fn start_workflow(
&self,
_definition_id: &str,
_version: u32,
_data: serde_json::Value,
) -> std::pin::Pin<Box<dyn std::future::Future<Output = crate::Result<String>> + Send + '_>>
{
Box::pin(async {
Err(crate::WfeError::StepExecution(
"failed to start child".to_string(),
))
})
}
}
fn make_context_with_host<'a>(
pointer: &'a ExecutionPointer,
step: &'a crate::models::WorkflowStep,
workflow: &'a crate::models::WorkflowInstance,
host: &'a dyn HostContext,
) -> StepExecutionContext<'a> {
StepExecutionContext {
item: None,
execution_pointer: pointer,
persistence_data: pointer.persistence_data.as_ref(),
step,
workflow,
cancellation_token: tokio_util::sync::CancellationToken::new(),
host_context: Some(host),
}
}
#[tokio::test]
async fn first_call_starts_child_and_waits() {
let host = MockHostContext::new("child-123");
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({"x": 10}),
..Default::default()
};
let pointer = ExecutionPointer::new(0);
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context_with_host(&pointer, &wf_step, &workflow, &host);
let result = step.run(&ctx).await.unwrap();
assert!(!result.proceed);
assert_eq!(result.event_name.as_deref(), Some("wfe.workflow.completed"));
assert_eq!(result.event_key.as_deref(), Some("child-123"));
assert!(result.event_as_of.is_some());
let calls = host.calls();
assert_eq!(calls.len(), 1);
assert_eq!(calls[0].0, "child-def");
assert_eq!(calls[0].1, 1);
assert_eq!(calls[0].2, json!({"x": 10}));
}
#[tokio::test]
async fn child_completed_proceeds_with_output() {
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({}),
output_keys: vec!["result".into()],
..Default::default()
};
let mut pointer = ExecutionPointer::new(0);
pointer.event_data = Some(json!({
"status": "Complete",
"data": {"result": "success", "extra": "ignored"}
}));
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context(&pointer, &wf_step, &workflow);
let result = step.run(&ctx).await.unwrap();
assert!(result.proceed);
assert_eq!(
result.output_data,
Some(json!({"result": "success"}))
);
}
#[tokio::test]
async fn child_completed_no_output_keys_passes_all() {
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({}),
output_keys: vec![],
..Default::default()
};
let mut pointer = ExecutionPointer::new(0);
pointer.event_data = Some(json!({
"status": "Complete",
"data": {"a": 1, "b": 2}
}));
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context(&pointer, &wf_step, &workflow);
let result = step.run(&ctx).await.unwrap();
assert!(result.proceed);
assert_eq!(
result.output_data,
Some(json!({"a": 1, "b": 2}))
);
}
#[tokio::test]
async fn no_host_context_errors() {
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({}),
..Default::default()
};
let pointer = ExecutionPointer::new(0);
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context(&pointer, &wf_step, &workflow);
let err = step.run(&ctx).await.unwrap_err();
assert!(err.to_string().contains("host context"));
}
#[tokio::test]
async fn input_validation_failure() {
let host = MockHostContext::new("child-123");
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({"name": 42}), // wrong type
input_schema: Some(WorkflowSchema {
inputs: HashMap::from([("name".into(), SchemaType::String)]),
outputs: HashMap::new(),
}),
..Default::default()
};
let pointer = ExecutionPointer::new(0);
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context_with_host(&pointer, &wf_step, &workflow, &host);
let err = step.run(&ctx).await.unwrap_err();
assert!(err.to_string().contains("input validation failed"));
assert!(host.calls().is_empty());
}
#[tokio::test]
async fn output_validation_failure() {
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({}),
output_keys: vec![],
output_schema: Some(WorkflowSchema {
inputs: HashMap::new(),
outputs: HashMap::from([("result".into(), SchemaType::String)]),
}),
..Default::default()
};
let mut pointer = ExecutionPointer::new(0);
pointer.event_data = Some(json!({
"status": "Complete",
"data": {"result": 42}
}));
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context(&pointer, &wf_step, &workflow);
let err = step.run(&ctx).await.unwrap_err();
assert!(err.to_string().contains("output validation failed"));
}
#[tokio::test]
async fn input_validation_passes_then_starts_child() {
let host = MockHostContext::new("child-456");
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 2,
inputs: json!({"name": "Alice"}),
input_schema: Some(WorkflowSchema {
inputs: HashMap::from([("name".into(), SchemaType::String)]),
outputs: HashMap::new(),
}),
..Default::default()
};
let pointer = ExecutionPointer::new(0);
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context_with_host(&pointer, &wf_step, &workflow, &host);
let result = step.run(&ctx).await.unwrap();
assert!(!result.proceed);
assert_eq!(result.event_key.as_deref(), Some("child-456"));
assert_eq!(host.calls().len(), 1);
}
#[tokio::test]
async fn host_start_workflow_error_propagates() {
let host = FailingHostContext;
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({}),
..Default::default()
};
let pointer = ExecutionPointer::new(0);
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context_with_host(&pointer, &wf_step, &workflow, &host);
let err = step.run(&ctx).await.unwrap_err();
assert!(err.to_string().contains("failed to start child"));
}
#[tokio::test]
async fn event_data_without_data_field_returns_empty_output() {
let mut step = SubWorkflowStep {
workflow_id: "child-def".into(),
version: 1,
inputs: json!({}),
output_keys: vec!["foo".into()],
..Default::default()
};
let mut pointer = ExecutionPointer::new(0);
pointer.event_data = Some(json!({"status": "Complete"}));
let wf_step = default_step();
let workflow = default_workflow();
let ctx = make_context(&pointer, &wf_step, &workflow);
let result = step.run(&ctx).await.unwrap();
assert!(result.proceed);
assert_eq!(result.output_data, Some(json!({})));
}
#[tokio::test]
async fn default_step_has_empty_fields() {
let step = SubWorkflowStep::default();
assert!(step.workflow_id.is_empty());
assert_eq!(step.version, 0);
assert_eq!(step.inputs, json!(null));
assert!(step.output_keys.is_empty());
assert!(step.input_schema.is_none());
assert!(step.output_schema.is_none());
}
}

View File

@@ -68,6 +68,7 @@ mod tests {
step: &step,
workflow: &instance,
cancellation_token: tokio_util::sync::CancellationToken::new(),
host_context: None,
};
mw.pre_step(&ctx).await.unwrap();
}
@@ -86,6 +87,7 @@ mod tests {
step: &step,
workflow: &instance,
cancellation_token: tokio_util::sync::CancellationToken::new(),
host_context: None,
};
let result = ExecutionResult::next();
mw.post_step(&ctx, &result).await.unwrap();

View File

@@ -17,4 +17,4 @@ pub use persistence::{
pub use queue::QueueProvider;
pub use registry::WorkflowRegistry;
pub use search::{Page, SearchFilter, SearchIndex, WorkflowSearchResult};
pub use step::{StepBody, StepExecutionContext, WorkflowData};
pub use step::{HostContext, StepBody, StepExecutionContext, WorkflowData};

View File

@@ -11,8 +11,18 @@ pub trait WorkflowData: Serialize + DeserializeOwned + Send + Sync + Clone + 'st
/// Blanket implementation: any type satisfying the bounds is WorkflowData.
impl<T> WorkflowData for T where T: Serialize + DeserializeOwned + Send + Sync + Clone + 'static {}
/// Context for steps that need to interact with the workflow host.
/// Implemented by WorkflowHost to allow steps like SubWorkflow to start child workflows.
pub trait HostContext: Send + Sync {
fn start_workflow(
&self,
definition_id: &str,
version: u32,
data: serde_json::Value,
) -> std::pin::Pin<Box<dyn std::future::Future<Output = crate::Result<String>> + Send + '_>>;
}
/// Context available to a step during execution.
#[derive(Debug)]
pub struct StepExecutionContext<'a> {
/// The current item when iterating (ForEach).
pub item: Option<&'a serde_json::Value>,
@@ -26,6 +36,22 @@ pub struct StepExecutionContext<'a> {
pub workflow: &'a WorkflowInstance,
/// Cancellation token.
pub cancellation_token: tokio_util::sync::CancellationToken,
/// Host context for starting child workflows. None if not available.
pub host_context: Option<&'a dyn HostContext>,
}
// Manual Debug impl since dyn HostContext is not Debug.
impl<'a> std::fmt::Debug for StepExecutionContext<'a> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("StepExecutionContext")
.field("item", &self.item)
.field("execution_pointer", &self.execution_pointer)
.field("persistence_data", &self.persistence_data)
.field("step", &self.step)
.field("workflow", &self.workflow)
.field("host_context", &self.host_context.is_some())
.finish()
}
}
/// The core unit of work in a workflow. Each step implements this trait.

View File

@@ -3,6 +3,8 @@ name = "wfe-opensearch"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "OpenSearch index provider for WFE"
[dependencies]

View File

@@ -3,6 +3,8 @@ name = "wfe-postgres"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "PostgreSQL persistence provider for WFE"
[dependencies]

View File

@@ -66,6 +66,7 @@ impl PostgresPersistenceProvider {
PointerStatus::Pending => "Pending",
PointerStatus::Running => "Running",
PointerStatus::Complete => "Complete",
PointerStatus::Skipped => "Skipped",
PointerStatus::Sleeping => "Sleeping",
PointerStatus::WaitingForEvent => "WaitingForEvent",
PointerStatus::Failed => "Failed",
@@ -80,6 +81,7 @@ impl PostgresPersistenceProvider {
"Pending" => Ok(PointerStatus::Pending),
"Running" => Ok(PointerStatus::Running),
"Complete" => Ok(PointerStatus::Complete),
"Skipped" => Ok(PointerStatus::Skipped),
"Sleeping" => Ok(PointerStatus::Sleeping),
"WaitingForEvent" => Ok(PointerStatus::WaitingForEvent),
"Failed" => Ok(PointerStatus::Failed),

View File

@@ -3,6 +3,8 @@ name = "wfe-sqlite"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "SQLite persistence provider for WFE"
[dependencies]

View File

@@ -3,6 +3,8 @@ name = "wfe-valkey"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "Valkey/Redis provider for distributed locking, queues, and lifecycle events in WFE"
[dependencies]

View File

@@ -7,12 +7,15 @@ description = "YAML workflow definitions for WFE"
[features]
default = []
deno = ["deno_core", "deno_error", "url", "reqwest"]
buildkit = ["wfe-buildkit"]
containerd = ["wfe-containerd"]
[dependencies]
wfe-core = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
serde_yaml = { workspace = true }
yaml-merge-keys = { workspace = true }
async-trait = { workspace = true }
tokio = { workspace = true }
thiserror = { workspace = true }
@@ -22,6 +25,8 @@ deno_core = { workspace = true, optional = true }
deno_error = { workspace = true, optional = true }
url = { workspace = true, optional = true }
reqwest = { workspace = true, optional = true }
wfe-buildkit = { workspace = true, optional = true }
wfe-containerd = { workspace = true, optional = true }
[dev-dependencies]
pretty_assertions = { workspace = true }

View File

@@ -1,5 +1,6 @@
use std::time::Duration;
use serde::Serialize;
use wfe_core::models::error_behavior::ErrorBehavior;
use wfe_core::models::workflow_definition::{StepOutcome, WorkflowDefinition, WorkflowStep};
use wfe_core::traits::StepBody;
@@ -8,7 +9,22 @@ use crate::error::YamlWorkflowError;
use crate::executors::shell::{ShellConfig, ShellStep};
#[cfg(feature = "deno")]
use crate::executors::deno::{DenoConfig, DenoPermissions, DenoStep};
use crate::schema::{WorkflowSpec, YamlErrorBehavior, YamlStep};
#[cfg(feature = "buildkit")]
use wfe_buildkit::{BuildkitConfig, BuildkitStep};
#[cfg(feature = "containerd")]
use wfe_containerd::{ContainerdConfig, ContainerdStep};
use wfe_core::primitives::sub_workflow::SubWorkflowStep;
use wfe_core::models::condition::{ComparisonOp, FieldComparison, StepCondition};
use crate::schema::{WorkflowSpec, YamlCombinator, YamlComparison, YamlCondition, YamlErrorBehavior, YamlStep};
/// Configuration for a sub-workflow step.
#[derive(Debug, Clone, Serialize)]
pub struct SubWorkflowConfig {
pub workflow_id: String,
pub version: u32,
pub output_keys: Vec<String>,
}
/// Factory type alias for step creation closures.
pub type StepFactory = Box<dyn Fn() -> Box<dyn StepBody> + Send + Sync>;
@@ -68,6 +84,11 @@ fn compile_steps(
compile_steps(parallel_children, definition, factories, next_id)?;
container.children = child_ids;
// Compile condition if present.
if let Some(ref yaml_cond) = yaml_step.when {
container.when = Some(compile_condition(yaml_cond)?);
}
definition.steps.push(container);
main_step_ids.push(container_id);
} else {
@@ -94,6 +115,11 @@ fn compile_steps(
wf_step.error_behavior = Some(map_error_behavior(eb)?);
}
// Compile condition if present.
if let Some(ref yaml_cond) = yaml_step.when {
wf_step.when = Some(compile_condition(yaml_cond)?);
}
// Handle on_failure: create compensation step.
if let Some(ref on_failure) = yaml_step.on_failure {
let comp_id = *next_id;
@@ -216,6 +242,154 @@ fn compile_steps(
Ok(main_step_ids)
}
/// Convert a YAML condition tree into a `StepCondition` tree.
pub fn compile_condition(yaml_cond: &YamlCondition) -> Result<StepCondition, YamlWorkflowError> {
match yaml_cond {
YamlCondition::Comparison(cmp) => compile_comparison(cmp.as_ref()),
YamlCondition::Combinator(combinator) => compile_combinator(combinator),
}
}
fn compile_combinator(c: &YamlCombinator) -> Result<StepCondition, YamlWorkflowError> {
// Count how many combinator keys are set to detect ambiguity.
let mut count = 0;
if c.all.is_some() {
count += 1;
}
if c.any.is_some() {
count += 1;
}
if c.none.is_some() {
count += 1;
}
if c.one_of.is_some() {
count += 1;
}
if c.not.is_some() {
count += 1;
}
if count == 0 {
return Err(YamlWorkflowError::Compilation(
"Condition combinator must have at least one of: all, any, none, one_of, not"
.to_string(),
));
}
if count > 1 {
return Err(YamlWorkflowError::Compilation(
"Condition combinator must have exactly one of: all, any, none, one_of, not"
.to_string(),
));
}
if let Some(ref children) = c.all {
let compiled: Result<Vec<_>, _> = children.iter().map(compile_condition).collect();
Ok(StepCondition::All(compiled?))
} else if let Some(ref children) = c.any {
let compiled: Result<Vec<_>, _> = children.iter().map(compile_condition).collect();
Ok(StepCondition::Any(compiled?))
} else if let Some(ref children) = c.none {
let compiled: Result<Vec<_>, _> = children.iter().map(compile_condition).collect();
Ok(StepCondition::None(compiled?))
} else if let Some(ref children) = c.one_of {
let compiled: Result<Vec<_>, _> = children.iter().map(compile_condition).collect();
Ok(StepCondition::OneOf(compiled?))
} else if let Some(ref inner) = c.not {
Ok(StepCondition::Not(Box::new(compile_condition(inner)?)))
} else {
unreachable!()
}
}
fn compile_comparison(cmp: &YamlComparison) -> Result<StepCondition, YamlWorkflowError> {
// Determine which operator is specified. Exactly one must be present.
let mut ops: Vec<(ComparisonOp, Option<serde_json::Value>)> = Vec::new();
if let Some(ref v) = cmp.equals {
ops.push((ComparisonOp::Equals, Some(yaml_value_to_json(v))));
}
if let Some(ref v) = cmp.not_equals {
ops.push((ComparisonOp::NotEquals, Some(yaml_value_to_json(v))));
}
if let Some(ref v) = cmp.gt {
ops.push((ComparisonOp::Gt, Some(yaml_value_to_json(v))));
}
if let Some(ref v) = cmp.gte {
ops.push((ComparisonOp::Gte, Some(yaml_value_to_json(v))));
}
if let Some(ref v) = cmp.lt {
ops.push((ComparisonOp::Lt, Some(yaml_value_to_json(v))));
}
if let Some(ref v) = cmp.lte {
ops.push((ComparisonOp::Lte, Some(yaml_value_to_json(v))));
}
if let Some(ref v) = cmp.contains {
ops.push((ComparisonOp::Contains, Some(yaml_value_to_json(v))));
}
if let Some(true) = cmp.is_null {
ops.push((ComparisonOp::IsNull, None));
}
if let Some(true) = cmp.is_not_null {
ops.push((ComparisonOp::IsNotNull, None));
}
if ops.is_empty() {
return Err(YamlWorkflowError::Compilation(format!(
"Comparison on field '{}' must specify an operator (equals, gt, etc.)",
cmp.field
)));
}
if ops.len() > 1 {
return Err(YamlWorkflowError::Compilation(format!(
"Comparison on field '{}' must specify exactly one operator, found {}",
cmp.field,
ops.len()
)));
}
let (operator, value) = ops.remove(0);
Ok(StepCondition::Comparison(FieldComparison {
field: cmp.field.clone(),
operator,
value,
}))
}
/// Convert a serde_yaml::Value to serde_json::Value.
fn yaml_value_to_json(v: &serde_yaml::Value) -> serde_json::Value {
match v {
serde_yaml::Value::Null => serde_json::Value::Null,
serde_yaml::Value::Bool(b) => serde_json::Value::Bool(*b),
serde_yaml::Value::Number(n) => {
if let Some(i) = n.as_i64() {
serde_json::Value::Number(serde_json::Number::from(i))
} else if let Some(u) = n.as_u64() {
serde_json::Value::Number(serde_json::Number::from(u))
} else if let Some(f) = n.as_f64() {
serde_json::Number::from_f64(f)
.map(serde_json::Value::Number)
.unwrap_or(serde_json::Value::Null)
} else {
serde_json::Value::Null
}
}
serde_yaml::Value::String(s) => serde_json::Value::String(s.clone()),
serde_yaml::Value::Sequence(seq) => {
serde_json::Value::Array(seq.iter().map(yaml_value_to_json).collect())
}
serde_yaml::Value::Mapping(map) => {
let mut obj = serde_json::Map::new();
for (k, val) in map {
if let serde_yaml::Value::String(key) = k {
obj.insert(key.clone(), yaml_value_to_json(val));
}
}
serde_json::Value::Object(obj)
}
serde_yaml::Value::Tagged(tagged) => yaml_value_to_json(&tagged.value),
}
}
fn build_step_config_and_factory(
step: &YamlStep,
step_type: &str,
@@ -250,6 +424,76 @@ fn build_step_config_and_factory(
});
Ok((key, value, factory))
}
#[cfg(feature = "buildkit")]
"buildkit" => {
let config = build_buildkit_config(step)?;
let key = format!("wfe_yaml::buildkit::{}", step.name);
let value = serde_json::to_value(&config).map_err(|e| {
YamlWorkflowError::Compilation(format!(
"Failed to serialize buildkit config: {e}"
))
})?;
let config_clone = config.clone();
let factory: StepFactory = Box::new(move || {
Box::new(BuildkitStep::new(config_clone.clone())) as Box<dyn StepBody>
});
Ok((key, value, factory))
}
#[cfg(feature = "containerd")]
"containerd" => {
let config = build_containerd_config(step)?;
let key = format!("wfe_yaml::containerd::{}", step.name);
let value = serde_json::to_value(&config).map_err(|e| {
YamlWorkflowError::Compilation(format!(
"Failed to serialize containerd config: {e}"
))
})?;
let config_clone = config.clone();
let factory: StepFactory = Box::new(move || {
Box::new(ContainerdStep::new(config_clone.clone())) as Box<dyn StepBody>
});
Ok((key, value, factory))
}
"workflow" => {
let config = step.config.as_ref().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"Workflow step '{}' is missing 'config' section",
step.name
))
})?;
let child_workflow_id = config.child_workflow.as_ref().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"Workflow step '{}' must have 'config.workflow'",
step.name
))
})?;
let child_version = config.child_version.unwrap_or(1);
let sub_config = SubWorkflowConfig {
workflow_id: child_workflow_id.clone(),
version: child_version,
output_keys: step.outputs.iter().map(|o| o.name.clone()).collect(),
};
let key = format!("wfe_yaml::workflow::{}", step.name);
let value = serde_json::to_value(&sub_config).map_err(|e| {
YamlWorkflowError::Compilation(format!(
"Failed to serialize workflow config: {e}"
))
})?;
let config_clone = sub_config.clone();
let factory: StepFactory = Box::new(move || {
Box::new(SubWorkflowStep {
workflow_id: config_clone.workflow_id.clone(),
version: config_clone.version,
output_keys: config_clone.output_keys.clone(),
inputs: serde_json::Value::Null,
input_schema: None,
output_schema: None,
}) as Box<dyn StepBody>
});
Ok((key, value, factory))
}
other => Err(YamlWorkflowError::Compilation(format!(
"Unknown step type: '{other}'"
))),
@@ -346,6 +590,162 @@ fn parse_duration_ms(s: &str) -> Option<u64> {
}
}
#[cfg(feature = "buildkit")]
fn build_buildkit_config(
step: &YamlStep,
) -> Result<BuildkitConfig, YamlWorkflowError> {
let config = step.config.as_ref().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"BuildKit step '{}' is missing 'config' section",
step.name
))
})?;
let dockerfile = config.dockerfile.clone().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"BuildKit step '{}' must have 'config.dockerfile'",
step.name
))
})?;
let context = config.context.clone().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"BuildKit step '{}' must have 'config.context'",
step.name
))
})?;
let timeout_ms = config.timeout.as_ref().and_then(|t| parse_duration_ms(t));
let tls = config
.tls
.as_ref()
.map(|t| wfe_buildkit::TlsConfig {
ca: t.ca.clone(),
cert: t.cert.clone(),
key: t.key.clone(),
})
.unwrap_or_default();
let registry_auth = config
.registry_auth
.as_ref()
.map(|ra| {
ra.iter()
.map(|(k, v)| {
(
k.clone(),
wfe_buildkit::RegistryAuth {
username: v.username.clone(),
password: v.password.clone(),
},
)
})
.collect()
})
.unwrap_or_default();
Ok(BuildkitConfig {
dockerfile,
context,
target: config.target.clone(),
tags: config.tags.clone(),
build_args: config.build_args.clone(),
cache_from: config.cache_from.clone(),
cache_to: config.cache_to.clone(),
push: config.push.unwrap_or(false),
output_type: None,
buildkit_addr: config
.buildkit_addr
.clone()
.unwrap_or_else(|| "unix:///run/buildkit/buildkitd.sock".to_string()),
tls,
registry_auth,
timeout_ms,
})
}
#[cfg(feature = "containerd")]
fn build_containerd_config(
step: &YamlStep,
) -> Result<ContainerdConfig, YamlWorkflowError> {
let config = step.config.as_ref().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"Containerd step '{}' is missing 'config' section",
step.name
))
})?;
let image = config.image.clone().ok_or_else(|| {
YamlWorkflowError::Compilation(format!(
"Containerd step '{}' must have 'config.image'",
step.name
))
})?;
let timeout_ms = config.timeout.as_ref().and_then(|t| parse_duration_ms(t));
let tls = config
.tls
.as_ref()
.map(|t| wfe_containerd::TlsConfig {
ca: t.ca.clone(),
cert: t.cert.clone(),
key: t.key.clone(),
})
.unwrap_or_default();
let registry_auth = config
.registry_auth
.as_ref()
.map(|ra| {
ra.iter()
.map(|(k, v)| {
(
k.clone(),
wfe_containerd::RegistryAuth {
username: v.username.clone(),
password: v.password.clone(),
},
)
})
.collect()
})
.unwrap_or_default();
let volumes = config
.volumes
.iter()
.map(|v| wfe_containerd::VolumeMountConfig {
source: v.source.clone(),
target: v.target.clone(),
readonly: v.readonly,
})
.collect();
Ok(ContainerdConfig {
image,
command: config.command.clone(),
run: config.run.clone(),
env: config.env.clone(),
volumes,
working_dir: config.working_dir.clone(),
user: config.user.clone().unwrap_or_else(|| "65534:65534".to_string()),
network: config.network.clone().unwrap_or_else(|| "none".to_string()),
memory: config.memory.clone(),
cpu: config.cpu.clone(),
pull: config.pull.clone().unwrap_or_else(|| "if-not-present".to_string()),
containerd_addr: config
.containerd_addr
.clone()
.unwrap_or_else(|| "/run/containerd/containerd.sock".to_string()),
cli: config.cli.clone().unwrap_or_else(|| "nerdctl".to_string()),
tls,
registry_auth,
timeout_ms,
})
}
fn map_error_behavior(eb: &YamlErrorBehavior) -> Result<ErrorBehavior, YamlWorkflowError> {
match eb.behavior_type.as_str() {
"retry" => {

View File

@@ -2,6 +2,10 @@ globalThis.inputs = () => Deno.core.ops.op_inputs();
globalThis.output = (key, value) => Deno.core.ops.op_output(key, value);
globalThis.log = (msg) => Deno.core.ops.op_log(msg);
globalThis.readFile = async (path) => {
return await Deno.core.ops.op_read_file(path);
};
globalThis.fetch = async (url, options) => {
const resp = await Deno.core.ops.op_fetch(url, options || null);
return {

View File

@@ -44,9 +44,29 @@ pub fn op_log(state: &mut OpState, #[string] msg: String) {
tracing::info!(step = %name, "{}", msg);
}
/// Reads a file from the filesystem and returns its contents as a string.
/// Permission-checked against the read allowlist.
#[op2]
#[string]
pub async fn op_read_file(
state: std::rc::Rc<std::cell::RefCell<OpState>>,
#[string] path: String,
) -> Result<String, deno_error::JsErrorBox> {
// Check read permission
{
let s = state.borrow();
let checker = s.borrow::<super::super::permissions::PermissionChecker>();
checker.check_read(&path)
.map_err(|e| deno_error::JsErrorBox::new("PermissionError", e.to_string()))?;
}
tokio::fs::read_to_string(&path)
.await
.map_err(|e| deno_error::JsErrorBox::generic(format!("Failed to read file '{path}': {e}")))
}
deno_core::extension!(
wfe_ops,
ops = [op_inputs, op_output, op_log, super::http::op_fetch],
ops = [op_inputs, op_output, op_log, op_read_file, super::http::op_fetch],
esm_entry_point = "ext:wfe/bootstrap.js",
esm = ["ext:wfe/bootstrap.js" = "src/executors/deno/js/bootstrap.js"],
);

View File

@@ -92,8 +92,21 @@ impl StepBody for ShellStep {
&& let Some(eq_pos) = rest.find('=')
{
let name = rest[..eq_pos].trim().to_string();
let value = rest[eq_pos + 1..].to_string();
outputs.insert(name, serde_json::Value::String(value));
let raw_value = rest[eq_pos + 1..].to_string();
// Auto-convert typed values from string annotations
let value = match raw_value.as_str() {
"true" => serde_json::Value::Bool(true),
"false" => serde_json::Value::Bool(false),
"null" => serde_json::Value::Null,
s if s.parse::<i64>().is_ok() => {
serde_json::Value::Number(s.parse::<i64>().unwrap().into())
}
s if s.parse::<f64>().is_ok() => {
serde_json::json!(s.parse::<f64>().unwrap())
}
_ => serde_json::Value::String(raw_value),
};
outputs.insert(name, value);
}
}

View File

@@ -3,36 +3,219 @@ pub mod error;
pub mod executors;
pub mod interpolation;
pub mod schema;
pub mod types;
pub mod validation;
use std::collections::HashMap;
use std::collections::{HashMap, HashSet};
use std::path::Path;
use serde::de::Error as _;
use serde::Deserialize;
use crate::compiler::CompiledWorkflow;
use crate::error::YamlWorkflowError;
/// Load a workflow from a YAML file path, applying variable interpolation.
/// Top-level YAML file with optional includes.
#[derive(Deserialize)]
pub struct YamlWorkflowFileWithIncludes {
#[serde(default)]
pub include: Vec<String>,
#[serde(flatten)]
pub file: schema::YamlWorkflowFile,
}
/// Load workflows from a YAML file path, applying variable interpolation.
/// Returns a Vec of compiled workflows (supports multi-workflow files).
pub fn load_workflow(
path: &std::path::Path,
config: &HashMap<String, serde_json::Value>,
) -> Result<CompiledWorkflow, YamlWorkflowError> {
let yaml = std::fs::read_to_string(path)?;
load_workflow_from_str(&yaml, config)
load_single_workflow_from_str(&yaml, config)
}
/// Load a workflow from a YAML string, applying variable interpolation.
/// Load workflows from a YAML string, applying variable interpolation.
/// Returns a Vec of compiled workflows (supports multi-workflow files).
///
/// Supports YAML 1.1 merge keys (`<<: *anchor`) via the `yaml-merge-keys`
/// crate. serde_yaml 0.9 implements YAML 1.2 which dropped merge keys;
/// we preprocess the YAML to resolve them before deserialization.
pub fn load_workflow_from_str(
yaml: &str,
config: &HashMap<String, serde_json::Value>,
) -> Result<CompiledWorkflow, YamlWorkflowError> {
) -> Result<Vec<CompiledWorkflow>, YamlWorkflowError> {
// Interpolate variables.
let interpolated = interpolation::interpolate(yaml, config)?;
// Parse YAML.
let workflow: schema::YamlWorkflow = serde_yaml::from_str(&interpolated)?;
// Parse to a generic YAML value first, then resolve merge keys (<<:).
// This adds YAML 1.1 merge key support on top of serde_yaml 0.9's YAML 1.2 parser.
let raw_value: serde_yaml::Value = serde_yaml::from_str(&interpolated)?;
let merged_value = yaml_merge_keys::merge_keys_serde(raw_value)
.map_err(|e| YamlWorkflowError::Parse(serde_yaml::Error::custom(format!("merge key resolution failed: {e}"))))?;
// Validate.
validation::validate(&workflow.workflow)?;
// Deserialize the merge-resolved value into our schema.
let file: schema::YamlWorkflowFile = serde_yaml::from_value(merged_value)?;
// Compile.
compiler::compile(&workflow.workflow)
let specs = resolve_workflow_specs(file)?;
// Validate (multi-workflow validation includes per-workflow + cross-references).
validation::validate_multi(&specs)?;
// Compile each workflow.
let mut results = Vec::with_capacity(specs.len());
for spec in &specs {
results.push(compiler::compile(spec)?);
}
Ok(results)
}
/// Load a single workflow from a YAML string. Returns an error if the file
/// contains more than one workflow. This is a backward-compatible convenience
/// function.
pub fn load_single_workflow_from_str(
yaml: &str,
config: &HashMap<String, serde_json::Value>,
) -> Result<CompiledWorkflow, YamlWorkflowError> {
let mut workflows = load_workflow_from_str(yaml, config)?;
if workflows.len() != 1 {
return Err(YamlWorkflowError::Validation(format!(
"Expected single workflow, got {}",
workflows.len()
)));
}
Ok(workflows.remove(0))
}
/// Load workflows from a YAML string, resolving `include:` paths relative to `base_path`.
///
/// Processing:
/// 1. Parse the main YAML to get `include:` paths.
/// 2. For each include path, load and parse that YAML file.
/// 3. Merge workflow specs from included files into the main file's specs.
/// 4. Main file's workflows take precedence over included ones (by ID).
/// 5. Proceed with normal validation + compilation.
pub fn load_workflow_with_includes(
yaml: &str,
base_path: &Path,
config: &HashMap<String, serde_json::Value>,
) -> Result<Vec<CompiledWorkflow>, YamlWorkflowError> {
let mut visited = HashSet::new();
let canonical = base_path
.canonicalize()
.unwrap_or_else(|_| base_path.to_path_buf());
visited.insert(canonical.to_string_lossy().to_string());
let interpolated = interpolation::interpolate(yaml, config)?;
let raw_value: serde_yaml::Value = serde_yaml::from_str(&interpolated)?;
let merged_value = yaml_merge_keys::merge_keys_serde(raw_value)
.map_err(|e| {
YamlWorkflowError::Parse(serde_yaml::Error::custom(format!(
"merge key resolution failed: {e}"
)))
})?;
let with_includes: YamlWorkflowFileWithIncludes = serde_yaml::from_value(merged_value)?;
let mut main_specs = resolve_workflow_specs(with_includes.file)?;
// Process includes.
for include_path_str in &with_includes.include {
let include_path = base_path.parent().unwrap_or(base_path).join(include_path_str);
load_includes_recursive(
&include_path,
config,
&mut main_specs,
&mut visited,
)?;
}
// Main file takes precedence: included specs are only added if their ID
// isn't already present. This is handled by load_includes_recursive.
validation::validate_multi(&main_specs)?;
let mut results = Vec::with_capacity(main_specs.len());
for spec in &main_specs {
results.push(compiler::compile(spec)?);
}
Ok(results)
}
fn load_includes_recursive(
path: &Path,
config: &HashMap<String, serde_json::Value>,
specs: &mut Vec<schema::WorkflowSpec>,
visited: &mut HashSet<String>,
) -> Result<(), YamlWorkflowError> {
let canonical = path
.canonicalize()
.map_err(|e| {
YamlWorkflowError::Io(std::io::Error::new(
std::io::ErrorKind::NotFound,
format!("Include file not found: {}: {e}", path.display()),
))
})?;
let canonical_str = canonical.to_string_lossy().to_string();
if !visited.insert(canonical_str.clone()) {
return Err(YamlWorkflowError::Validation(format!(
"Circular include detected: '{}'",
path.display()
)));
}
let yaml = std::fs::read_to_string(&canonical)?;
let interpolated = interpolation::interpolate(&yaml, config)?;
let raw_value: serde_yaml::Value = serde_yaml::from_str(&interpolated)?;
let merged_value = yaml_merge_keys::merge_keys_serde(raw_value)
.map_err(|e| {
YamlWorkflowError::Parse(serde_yaml::Error::custom(format!(
"merge key resolution failed: {e}"
)))
})?;
let with_includes: YamlWorkflowFileWithIncludes = serde_yaml::from_value(merged_value)?;
let included_specs = resolve_workflow_specs(with_includes.file)?;
// Existing IDs in main specs take precedence.
let existing_ids: HashSet<String> = specs.iter().map(|s| s.id.clone()).collect();
for spec in included_specs {
if !existing_ids.contains(&spec.id) {
specs.push(spec);
}
}
// Recurse into nested includes.
for nested_include in &with_includes.include {
let nested_path = canonical.parent().unwrap_or(&canonical).join(nested_include);
load_includes_recursive(&nested_path, config, specs, visited)?;
}
Ok(())
}
/// Resolve a YamlWorkflowFile into a list of WorkflowSpecs.
fn resolve_workflow_specs(
file: schema::YamlWorkflowFile,
) -> Result<Vec<schema::WorkflowSpec>, YamlWorkflowError> {
match (file.workflow, file.workflows) {
(Some(single), None) => Ok(vec![single]),
(None, Some(multi)) => {
if multi.is_empty() {
return Err(YamlWorkflowError::Validation(
"workflows list is empty".to_string(),
));
}
Ok(multi)
}
(Some(_), Some(_)) => Err(YamlWorkflowError::Validation(
"Cannot specify both 'workflow' and 'workflows' in the same file".to_string(),
)),
(None, None) => Err(YamlWorkflowError::Validation(
"Must specify either 'workflow' or 'workflows'".to_string(),
)),
}
}

View File

@@ -2,6 +2,70 @@ use std::collections::HashMap;
use serde::Deserialize;
/// A condition in YAML that determines whether a step executes.
///
/// Uses `#[serde(untagged)]` so serde tries each variant in order.
/// A comparison has a `field:` key; a combinator has `all:/any:/none:/one_of:/not:`.
/// Comparison is listed first because it is more specific (requires `field`).
#[derive(Debug, Deserialize, Clone)]
#[serde(untagged)]
pub enum YamlCondition {
/// Leaf comparison (has a `field:` key).
Comparison(Box<YamlComparison>),
/// Combinator with sub-conditions.
Combinator(YamlCombinator),
}
/// A combinator condition containing sub-conditions.
#[derive(Debug, Deserialize, Clone)]
pub struct YamlCombinator {
#[serde(default)]
pub all: Option<Vec<YamlCondition>>,
#[serde(default)]
pub any: Option<Vec<YamlCondition>>,
#[serde(default)]
pub none: Option<Vec<YamlCondition>>,
#[serde(default)]
pub one_of: Option<Vec<YamlCondition>>,
#[serde(default)]
pub not: Option<Box<YamlCondition>>,
}
/// A leaf comparison condition that compares a field value.
#[derive(Debug, Deserialize, Clone)]
pub struct YamlComparison {
pub field: String,
#[serde(default)]
pub equals: Option<serde_yaml::Value>,
#[serde(default)]
pub not_equals: Option<serde_yaml::Value>,
#[serde(default)]
pub gt: Option<serde_yaml::Value>,
#[serde(default)]
pub gte: Option<serde_yaml::Value>,
#[serde(default)]
pub lt: Option<serde_yaml::Value>,
#[serde(default)]
pub lte: Option<serde_yaml::Value>,
#[serde(default)]
pub contains: Option<serde_yaml::Value>,
#[serde(default)]
pub is_null: Option<bool>,
#[serde(default)]
pub is_not_null: Option<bool>,
}
/// Top-level YAML file structure supporting both single and multi-workflow files.
#[derive(Debug, Deserialize)]
pub struct YamlWorkflowFile {
/// Single workflow (backward compatible).
pub workflow: Option<WorkflowSpec>,
/// Multiple workflows in one file.
pub workflows: Option<Vec<WorkflowSpec>>,
}
/// Legacy single-workflow top-level structure. Kept for backward compatibility
/// with code that deserializes `YamlWorkflow` directly.
#[derive(Debug, Deserialize)]
pub struct YamlWorkflow {
pub workflow: WorkflowSpec,
@@ -16,6 +80,13 @@ pub struct WorkflowSpec {
#[serde(default)]
pub error_behavior: Option<YamlErrorBehavior>,
pub steps: Vec<YamlStep>,
/// Typed input schema: { field_name: type_string }.
/// Example: `"repo_url": "string"`, `"tags": "list<string>"`.
#[serde(default)]
pub inputs: HashMap<String, String>,
/// Typed output schema: { field_name: type_string }.
#[serde(default)]
pub outputs: HashMap<String, String>,
/// Allow unknown top-level keys (e.g. `_templates`) for YAML anchors.
#[serde(flatten)]
pub _extra: HashMap<String, serde_yaml::Value>,
@@ -42,6 +113,9 @@ pub struct YamlStep {
pub on_failure: Option<Box<YamlStep>>,
#[serde(default)]
pub ensure: Option<Box<YamlStep>>,
/// Optional condition that must be true for this step to execute.
#[serde(default)]
pub when: Option<YamlCondition>,
}
#[derive(Debug, Deserialize, Clone)]
@@ -58,6 +132,45 @@ pub struct StepConfig {
pub permissions: Option<DenoPermissionsYaml>,
#[serde(default)]
pub modules: Vec<String>,
// BuildKit fields
pub dockerfile: Option<String>,
pub context: Option<String>,
pub target: Option<String>,
#[serde(default)]
pub tags: Vec<String>,
#[serde(default)]
pub build_args: HashMap<String, String>,
#[serde(default)]
pub cache_from: Vec<String>,
#[serde(default)]
pub cache_to: Vec<String>,
pub push: Option<bool>,
pub buildkit_addr: Option<String>,
#[serde(default)]
pub tls: Option<TlsConfigYaml>,
#[serde(default)]
pub registry_auth: Option<HashMap<String, RegistryAuthYaml>>,
// Containerd fields
pub image: Option<String>,
#[serde(default)]
pub command: Option<Vec<String>>,
#[serde(default)]
pub volumes: Vec<VolumeMountYaml>,
pub user: Option<String>,
pub network: Option<String>,
pub memory: Option<String>,
pub cpu: Option<String>,
pub pull: Option<String>,
pub containerd_addr: Option<String>,
/// CLI binary name for containerd steps: "nerdctl" (default) or "docker".
pub cli: Option<String>,
// Workflow (sub-workflow) fields
/// Child workflow ID (for `type: workflow` steps).
#[serde(rename = "workflow")]
pub child_workflow: Option<String>,
/// Child workflow version (for `type: workflow` steps).
#[serde(rename = "workflow_version")]
pub child_version: Option<u32>,
}
/// YAML-level permission configuration for Deno steps.
@@ -84,6 +197,30 @@ pub struct DataRef {
pub json_path: Option<String>,
}
/// YAML-level TLS configuration for BuildKit steps.
#[derive(Debug, Deserialize, Clone)]
pub struct TlsConfigYaml {
pub ca: Option<String>,
pub cert: Option<String>,
pub key: Option<String>,
}
/// YAML-level registry auth configuration for BuildKit steps.
#[derive(Debug, Deserialize, Clone)]
pub struct RegistryAuthYaml {
pub username: String,
pub password: String,
}
/// YAML-level volume mount configuration for containerd steps.
#[derive(Debug, Deserialize, Clone)]
pub struct VolumeMountYaml {
pub source: String,
pub target: String,
#[serde(default)]
pub readonly: bool,
}
#[derive(Debug, Deserialize)]
pub struct YamlErrorBehavior {
#[serde(rename = "type")]

252
wfe-yaml/src/types.rs Normal file
View File

@@ -0,0 +1,252 @@
/// Parsed type representation for workflow input/output schemas.
///
/// This mirrors what wfe-core's `SchemaType` will provide, but is self-contained
/// so wfe-yaml can parse type strings without depending on wfe-core's schema module.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SchemaType {
String,
Number,
Integer,
Bool,
Any,
Optional(Box<SchemaType>),
List(Box<SchemaType>),
Map(Box<SchemaType>),
}
impl std::fmt::Display for SchemaType {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
SchemaType::String => write!(f, "string"),
SchemaType::Number => write!(f, "number"),
SchemaType::Integer => write!(f, "integer"),
SchemaType::Bool => write!(f, "bool"),
SchemaType::Any => write!(f, "any"),
SchemaType::Optional(inner) => write!(f, "{inner}?"),
SchemaType::List(inner) => write!(f, "list<{inner}>"),
SchemaType::Map(inner) => write!(f, "map<{inner}>"),
}
}
}
/// Parse a type string like `"string"`, `"string?"`, `"list<number>"`, `"map<string>"`.
///
/// Supports:
/// - Primitives: `"string"`, `"number"`, `"integer"`, `"bool"`, `"any"`
/// - Optional: `"string?"` -> `Optional(String)`
/// - List: `"list<string>"` -> `List(String)`
/// - Map: `"map<number>"` -> `Map(Number)`
/// - Nested generics: `"list<list<string>>"` -> `List(List(String))`
pub fn parse_type_string(s: &str) -> Result<SchemaType, String> {
let s = s.trim();
if s.is_empty() {
return Err("Empty type string".to_string());
}
// Check for optional suffix (but not inside generics).
if s.ends_with('?') && !s.ends_with(">?") {
// Simple optional like "string?"
let inner = parse_type_string(&s[..s.len() - 1])?;
return Ok(SchemaType::Optional(Box::new(inner)));
}
// Handle optional on generic types like "list<string>?"
if s.ends_with(">?") {
let inner = parse_type_string(&s[..s.len() - 1])?;
return Ok(SchemaType::Optional(Box::new(inner)));
}
// Check for generic types: list<...> or map<...>
if let Some(inner_start) = s.find('<') {
if !s.ends_with('>') {
return Err(format!("Malformed generic type: '{s}' (missing closing '>')"));
}
let container = &s[..inner_start];
let inner_str = &s[inner_start + 1..s.len() - 1];
let inner_type = parse_type_string(inner_str)?;
match container {
"list" => Ok(SchemaType::List(Box::new(inner_type))),
"map" => Ok(SchemaType::Map(Box::new(inner_type))),
other => Err(format!("Unknown generic type: '{other}'")),
}
} else {
// Primitive types.
match s {
"string" => Ok(SchemaType::String),
"number" => Ok(SchemaType::Number),
"integer" => Ok(SchemaType::Integer),
"bool" => Ok(SchemaType::Bool),
"any" => Ok(SchemaType::Any),
other => Err(format!("Unknown type: '{other}'")),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_primitive_string() {
assert_eq!(parse_type_string("string").unwrap(), SchemaType::String);
}
#[test]
fn parse_primitive_number() {
assert_eq!(parse_type_string("number").unwrap(), SchemaType::Number);
}
#[test]
fn parse_primitive_integer() {
assert_eq!(parse_type_string("integer").unwrap(), SchemaType::Integer);
}
#[test]
fn parse_primitive_bool() {
assert_eq!(parse_type_string("bool").unwrap(), SchemaType::Bool);
}
#[test]
fn parse_primitive_any() {
assert_eq!(parse_type_string("any").unwrap(), SchemaType::Any);
}
#[test]
fn parse_optional_string() {
assert_eq!(
parse_type_string("string?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::String))
);
}
#[test]
fn parse_optional_number() {
assert_eq!(
parse_type_string("number?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::Number))
);
}
#[test]
fn parse_list_string() {
assert_eq!(
parse_type_string("list<string>").unwrap(),
SchemaType::List(Box::new(SchemaType::String))
);
}
#[test]
fn parse_map_number() {
assert_eq!(
parse_type_string("map<number>").unwrap(),
SchemaType::Map(Box::new(SchemaType::Number))
);
}
#[test]
fn parse_nested_list() {
assert_eq!(
parse_type_string("list<list<string>>").unwrap(),
SchemaType::List(Box::new(SchemaType::List(Box::new(SchemaType::String))))
);
}
#[test]
fn parse_nested_map_in_list() {
assert_eq!(
parse_type_string("list<map<integer>>").unwrap(),
SchemaType::List(Box::new(SchemaType::Map(Box::new(SchemaType::Integer))))
);
}
#[test]
fn parse_optional_list() {
assert_eq!(
parse_type_string("list<string>?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::List(Box::new(SchemaType::String))))
);
}
#[test]
fn parse_unknown_type_error() {
let result = parse_type_string("foobar");
assert!(result.is_err());
assert!(result.unwrap_err().contains("Unknown type"));
}
#[test]
fn parse_unknown_generic_error() {
let result = parse_type_string("set<string>");
assert!(result.is_err());
assert!(result.unwrap_err().contains("Unknown generic type"));
}
#[test]
fn parse_empty_string_error() {
let result = parse_type_string("");
assert!(result.is_err());
assert!(result.unwrap_err().contains("Empty type string"));
}
#[test]
fn parse_malformed_generic_error() {
let result = parse_type_string("list<string");
assert!(result.is_err());
assert!(result.unwrap_err().contains("Malformed generic type"));
}
#[test]
fn parse_whitespace_trimmed() {
assert_eq!(parse_type_string(" string ").unwrap(), SchemaType::String);
}
#[test]
fn parse_deeply_nested() {
assert_eq!(
parse_type_string("list<list<list<bool>>>").unwrap(),
SchemaType::List(Box::new(SchemaType::List(Box::new(SchemaType::List(
Box::new(SchemaType::Bool)
)))))
);
}
#[test]
fn display_roundtrip_primitives() {
for type_str in &["string", "number", "integer", "bool", "any"] {
let parsed = parse_type_string(type_str).unwrap();
assert_eq!(parsed.to_string(), *type_str);
}
}
#[test]
fn display_roundtrip_generics() {
for type_str in &["list<string>", "map<number>", "list<list<string>>"] {
let parsed = parse_type_string(type_str).unwrap();
assert_eq!(parsed.to_string(), *type_str);
}
}
#[test]
fn display_optional() {
let t = SchemaType::Optional(Box::new(SchemaType::String));
assert_eq!(t.to_string(), "string?");
}
#[test]
fn parse_map_any() {
assert_eq!(
parse_type_string("map<any>").unwrap(),
SchemaType::Map(Box::new(SchemaType::Any))
);
}
#[test]
fn parse_optional_bool() {
assert_eq!(
parse_type_string("bool?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::Bool))
);
}
}

View File

@@ -1,7 +1,8 @@
use std::collections::HashSet;
use std::collections::{HashMap, HashSet};
use crate::error::YamlWorkflowError;
use crate::schema::{WorkflowSpec, YamlStep};
use crate::schema::{WorkflowSpec, YamlCombinator, YamlComparison, YamlCondition, YamlStep};
use crate::types::{parse_type_string, SchemaType};
/// Validate a parsed workflow spec.
pub fn validate(spec: &WorkflowSpec) -> Result<(), YamlWorkflowError> {
@@ -19,6 +20,149 @@ pub fn validate(spec: &WorkflowSpec) -> Result<(), YamlWorkflowError> {
validate_error_behavior_type(&eb.behavior_type)?;
}
// Collect known outputs (from step output data refs).
let known_outputs: HashSet<String> = collect_step_outputs(&spec.steps);
// Validate condition fields and types on all steps.
validate_step_conditions(&spec.steps, spec, &known_outputs)?;
// Detect unused declared outputs.
detect_unused_outputs(spec, &known_outputs)?;
Ok(())
}
/// Validate multiple workflow specs from a multi-workflow file.
/// Checks cross-workflow references and cycles in addition to per-workflow validation.
pub fn validate_multi(specs: &[WorkflowSpec]) -> Result<(), YamlWorkflowError> {
// Validate each workflow individually.
for spec in specs {
validate(spec)?;
}
// Check for duplicate workflow IDs.
let mut seen_ids = HashSet::new();
for spec in specs {
if !seen_ids.insert(&spec.id) {
return Err(YamlWorkflowError::Validation(format!(
"Duplicate workflow ID: '{}'",
spec.id
)));
}
}
// Validate cross-workflow references and detect cycles.
validate_workflow_references(specs)?;
Ok(())
}
/// Validate that workflow step references point to known workflows
/// and detect circular dependencies.
fn validate_workflow_references(specs: &[WorkflowSpec]) -> Result<(), YamlWorkflowError> {
let known_ids: HashSet<&str> = specs.iter().map(|s| s.id.as_str()).collect();
// Build a dependency graph: workflow_id -> set of referenced workflow_ids.
let mut deps: HashMap<&str, HashSet<&str>> = HashMap::new();
for spec in specs {
let mut spec_deps = HashSet::new();
collect_workflow_refs(&spec.steps, &mut spec_deps);
deps.insert(spec.id.as_str(), spec_deps);
}
// Detect cycles using DFS with coloring.
detect_cycles(&known_ids, &deps)?;
Ok(())
}
/// Collect all workflow IDs referenced by `type: workflow` steps.
fn collect_workflow_refs<'a>(steps: &'a [YamlStep], refs: &mut HashSet<&'a str>) {
for step in steps {
if step.step_type.as_deref() == Some("workflow")
&& let Some(ref config) = step.config
&& let Some(ref wf_id) = config.child_workflow
{
refs.insert(wf_id.as_str());
}
if let Some(ref children) = step.parallel {
collect_workflow_refs(children, refs);
}
if let Some(ref hook) = step.on_success {
collect_workflow_refs(std::slice::from_ref(hook.as_ref()), refs);
}
if let Some(ref hook) = step.on_failure {
collect_workflow_refs(std::slice::from_ref(hook.as_ref()), refs);
}
if let Some(ref hook) = step.ensure {
collect_workflow_refs(std::slice::from_ref(hook.as_ref()), refs);
}
}
}
/// Detect circular references in the workflow dependency graph.
fn detect_cycles(
known_ids: &HashSet<&str>,
deps: &HashMap<&str, HashSet<&str>>,
) -> Result<(), YamlWorkflowError> {
#[derive(Clone, Copy, PartialEq)]
enum Color {
White,
Gray,
Black,
}
let mut colors: HashMap<&str, Color> = known_ids.iter().map(|id| (*id, Color::White)).collect();
fn dfs<'a>(
node: &'a str,
deps: &HashMap<&str, HashSet<&'a str>>,
colors: &mut HashMap<&'a str, Color>,
path: &mut Vec<&'a str>,
) -> Result<(), YamlWorkflowError> {
colors.insert(node, Color::Gray);
path.push(node);
if let Some(neighbors) = deps.get(node) {
for &neighbor in neighbors {
match colors.get(neighbor) {
Some(Color::Gray) => {
// Found a cycle. Build the cycle path for the error message.
let cycle_start = path.iter().position(|&n| n == neighbor).unwrap();
let cycle: Vec<&str> = path[cycle_start..].to_vec();
return Err(YamlWorkflowError::Validation(format!(
"Circular workflow reference detected: {} -> {}",
cycle.join(" -> "),
neighbor
)));
}
Some(Color::White) | None => {
// Only recurse into nodes that are in our known set.
if colors.contains_key(neighbor) {
dfs(neighbor, deps, colors, path)?;
}
}
Some(Color::Black) => {
// Already fully processed, skip.
}
}
}
}
path.pop();
colors.insert(node, Color::Black);
Ok(())
}
let nodes: Vec<&str> = known_ids.iter().copied().collect();
for node in nodes {
if colors.get(node) == Some(&Color::White) {
let mut path = Vec::new();
dfs(node, deps, &mut colors, &mut path)?;
}
}
Ok(())
}
@@ -89,6 +233,108 @@ fn validate_steps(
}
}
// BuildKit steps must have config with dockerfile and context.
if let Some(ref step_type) = step.step_type
&& step_type == "buildkit"
{
let config = step.config.as_ref().ok_or_else(|| {
YamlWorkflowError::Validation(format!(
"BuildKit step '{}' must have a 'config' section",
step.name
))
})?;
if config.dockerfile.is_none() {
return Err(YamlWorkflowError::Validation(format!(
"BuildKit step '{}' must have 'config.dockerfile'",
step.name
)));
}
if config.context.is_none() {
return Err(YamlWorkflowError::Validation(format!(
"BuildKit step '{}' must have 'config.context'",
step.name
)));
}
if config.push.unwrap_or(false) && config.tags.is_empty() {
return Err(YamlWorkflowError::Validation(format!(
"BuildKit step '{}' has push=true but no tags specified",
step.name
)));
}
}
// Containerd steps must have config with image and exactly one of run or command.
if let Some(ref step_type) = step.step_type
&& step_type == "containerd"
{
let config = step.config.as_ref().ok_or_else(|| {
YamlWorkflowError::Validation(format!(
"Containerd step '{}' must have a 'config' section",
step.name
))
})?;
if config.image.is_none() {
return Err(YamlWorkflowError::Validation(format!(
"Containerd step '{}' must have 'config.image'",
step.name
)));
}
let has_run = config.run.is_some();
let has_command = config.command.is_some();
if !has_run && !has_command {
return Err(YamlWorkflowError::Validation(format!(
"Containerd step '{}' must have 'config.run' or 'config.command'",
step.name
)));
}
if has_run && has_command {
return Err(YamlWorkflowError::Validation(format!(
"Containerd step '{}' cannot have both 'config.run' and 'config.command'",
step.name
)));
}
if let Some(ref network) = config.network {
match network.as_str() {
"none" | "host" | "bridge" => {}
other => {
return Err(YamlWorkflowError::Validation(format!(
"Containerd step '{}' has invalid network '{}'. Must be none, host, or bridge",
step.name, other
)));
}
}
}
if let Some(ref pull) = config.pull {
match pull.as_str() {
"always" | "if-not-present" | "never" => {}
other => {
return Err(YamlWorkflowError::Validation(format!(
"Containerd step '{}' has invalid pull policy '{}'. Must be always, if-not-present, or never",
step.name, other
)));
}
}
}
}
// Workflow steps must have config.workflow.
if let Some(ref step_type) = step.step_type
&& step_type == "workflow"
{
let config = step.config.as_ref().ok_or_else(|| {
YamlWorkflowError::Validation(format!(
"Workflow step '{}' must have a 'config' section",
step.name
))
})?;
if config.child_workflow.is_none() {
return Err(YamlWorkflowError::Validation(format!(
"Workflow step '{}' must have 'config.workflow'",
step.name
)));
}
}
// Validate step-level error behavior.
if let Some(ref eb) = step.error_behavior {
validate_error_behavior_type(&eb.behavior_type)?;
@@ -122,3 +368,300 @@ fn validate_error_behavior_type(behavior_type: &str) -> Result<(), YamlWorkflowE
))),
}
}
// --- Condition validation ---
/// Collect all output field names produced by steps (via their `outputs:` list).
fn collect_step_outputs(steps: &[YamlStep]) -> HashSet<String> {
let mut outputs = HashSet::new();
for step in steps {
for out in &step.outputs {
outputs.insert(out.name.clone());
}
if let Some(ref children) = step.parallel {
outputs.extend(collect_step_outputs(children));
}
if let Some(ref hook) = step.on_success {
outputs.extend(collect_step_outputs(std::slice::from_ref(hook.as_ref())));
}
if let Some(ref hook) = step.on_failure {
outputs.extend(collect_step_outputs(std::slice::from_ref(hook.as_ref())));
}
if let Some(ref hook) = step.ensure {
outputs.extend(collect_step_outputs(std::slice::from_ref(hook.as_ref())));
}
}
outputs
}
/// Walk all steps and validate their `when` conditions.
fn validate_step_conditions(
steps: &[YamlStep],
spec: &WorkflowSpec,
known_outputs: &HashSet<String>,
) -> Result<(), YamlWorkflowError> {
for step in steps {
if let Some(ref cond) = step.when {
validate_condition_fields(cond, spec, known_outputs)?;
validate_condition_types(cond, spec)?;
}
if let Some(ref children) = step.parallel {
validate_step_conditions(children, spec, known_outputs)?;
}
if let Some(ref hook) = step.on_success {
validate_step_conditions(std::slice::from_ref(hook.as_ref()), spec, known_outputs)?;
}
if let Some(ref hook) = step.on_failure {
validate_step_conditions(std::slice::from_ref(hook.as_ref()), spec, known_outputs)?;
}
if let Some(ref hook) = step.ensure {
validate_step_conditions(std::slice::from_ref(hook.as_ref()), spec, known_outputs)?;
}
}
Ok(())
}
/// Validate that all field paths in a condition tree resolve to known schema fields.
pub fn validate_condition_fields(
condition: &YamlCondition,
spec: &WorkflowSpec,
known_outputs: &HashSet<String>,
) -> Result<(), YamlWorkflowError> {
match condition {
YamlCondition::Comparison(cmp) => {
validate_field_path(&cmp.as_ref().field, spec, known_outputs)?;
}
YamlCondition::Combinator(c) => {
validate_combinator_fields(c, spec, known_outputs)?;
}
}
Ok(())
}
fn validate_combinator_fields(
c: &YamlCombinator,
spec: &WorkflowSpec,
known_outputs: &HashSet<String>,
) -> Result<(), YamlWorkflowError> {
let all_children = c
.all
.iter()
.flatten()
.chain(c.any.iter().flatten())
.chain(c.none.iter().flatten())
.chain(c.one_of.iter().flatten());
for child in all_children {
validate_condition_fields(child, spec, known_outputs)?;
}
if let Some(ref inner) = c.not {
validate_condition_fields(inner, spec, known_outputs)?;
}
Ok(())
}
/// Resolve a field path like `.inputs.foo` or `.outputs.bar` against the workflow schema.
fn validate_field_path(
field: &str,
spec: &WorkflowSpec,
known_outputs: &HashSet<String>,
) -> Result<(), YamlWorkflowError> {
// If the spec has no inputs and no outputs schema, skip field validation
// (schema-less workflow).
if spec.inputs.is_empty() && spec.outputs.is_empty() {
return Ok(());
}
let parts: Vec<&str> = field.split('.').collect();
// Expect paths like ".inputs.x" or ".outputs.x" (leading dot is optional).
let parts = if parts.first() == Some(&"") {
&parts[1..] // skip leading empty from "."
} else {
&parts[..]
};
if parts.len() < 2 {
return Err(YamlWorkflowError::Validation(format!(
"Condition field path '{field}' must have at least two segments (e.g. '.inputs.name')"
)));
}
match parts[0] {
"inputs" => {
let field_name = parts[1];
if !spec.inputs.contains_key(field_name) {
return Err(YamlWorkflowError::Validation(format!(
"Condition references unknown input field '{field_name}'. \
Available inputs: [{}]",
spec.inputs
.keys()
.cloned()
.collect::<Vec<_>>()
.join(", ")
)));
}
}
"outputs" => {
let field_name = parts[1];
// Check both the declared output schema and step-produced outputs.
if !spec.outputs.contains_key(field_name) && !known_outputs.contains(field_name) {
return Err(YamlWorkflowError::Validation(format!(
"Condition references unknown output field '{field_name}'. \
Available outputs: [{}]",
spec.outputs
.keys()
.cloned()
.collect::<Vec<_>>()
.join(", ")
)));
}
}
other => {
return Err(YamlWorkflowError::Validation(format!(
"Condition field path '{field}' must start with 'inputs' or 'outputs', got '{other}'"
)));
}
}
Ok(())
}
/// Validate operator type compatibility for condition comparisons.
pub fn validate_condition_types(
condition: &YamlCondition,
spec: &WorkflowSpec,
) -> Result<(), YamlWorkflowError> {
match condition {
YamlCondition::Comparison(cmp) => {
validate_comparison_type(cmp.as_ref(), spec)?;
}
YamlCondition::Combinator(c) => {
let all_children = c
.all
.iter()
.flatten()
.chain(c.any.iter().flatten())
.chain(c.none.iter().flatten())
.chain(c.one_of.iter().flatten());
for child in all_children {
validate_condition_types(child, spec)?;
}
if let Some(ref inner) = c.not {
validate_condition_types(inner, spec)?;
}
}
}
Ok(())
}
/// Check that the operator used in a comparison is compatible with the field type.
fn validate_comparison_type(
cmp: &YamlComparison,
spec: &WorkflowSpec,
) -> Result<(), YamlWorkflowError> {
// Resolve the field type from the schema.
let field_type = resolve_field_type(&cmp.field, spec);
let field_type = match field_type {
Some(t) => t,
// If we can't resolve the type (no schema), skip type checking.
None => return Ok(()),
};
// Check operator compatibility.
let has_gt = cmp.gt.is_some();
let has_gte = cmp.gte.is_some();
let has_lt = cmp.lt.is_some();
let has_lte = cmp.lte.is_some();
let has_contains = cmp.contains.is_some();
let has_is_null = cmp.is_null == Some(true);
let has_is_not_null = cmp.is_not_null == Some(true);
// gt/gte/lt/lte only valid for number/integer types.
if (has_gt || has_gte || has_lt || has_lte) && !is_numeric_type(&field_type) {
return Err(YamlWorkflowError::Validation(format!(
"Comparison operators gt/gte/lt/lte are only valid for number/integer types, \
but field '{}' has type '{}'",
cmp.field, field_type
)));
}
// contains only valid for string/list types.
if has_contains && !is_containable_type(&field_type) {
return Err(YamlWorkflowError::Validation(format!(
"Comparison operator 'contains' is only valid for string/list types, \
but field '{}' has type '{}'",
cmp.field, field_type
)));
}
// is_null/is_not_null only valid for optional types.
if (has_is_null || has_is_not_null) && !is_optional_type(&field_type) {
return Err(YamlWorkflowError::Validation(format!(
"Comparison operators is_null/is_not_null are only valid for optional types, \
but field '{}' has type '{}'",
cmp.field, field_type
)));
}
Ok(())
}
/// Resolve a field's SchemaType from the workflow spec.
fn resolve_field_type(field: &str, spec: &WorkflowSpec) -> Option<SchemaType> {
let parts: Vec<&str> = field.split('.').collect();
let parts = if parts.first() == Some(&"") {
&parts[1..]
} else {
&parts[..]
};
if parts.len() < 2 {
return None;
}
let type_str = match parts[0] {
"inputs" => spec.inputs.get(parts[1]),
"outputs" => spec.outputs.get(parts[1]),
_ => None,
}?;
parse_type_string(type_str).ok()
}
fn is_numeric_type(t: &SchemaType) -> bool {
match t {
SchemaType::Number | SchemaType::Integer | SchemaType::Any => true,
SchemaType::Optional(inner) => is_numeric_type(inner),
_ => false,
}
}
fn is_containable_type(t: &SchemaType) -> bool {
match t {
SchemaType::String | SchemaType::List(_) | SchemaType::Any => true,
SchemaType::Optional(inner) => is_containable_type(inner),
_ => false,
}
}
fn is_optional_type(t: &SchemaType) -> bool {
matches!(t, SchemaType::Optional(_) | SchemaType::Any)
}
/// Detect output fields declared in `spec.outputs` that no step produces.
pub fn detect_unused_outputs(
spec: &WorkflowSpec,
known_outputs: &HashSet<String>,
) -> Result<(), YamlWorkflowError> {
for output_name in spec.outputs.keys() {
if !known_outputs.contains(output_name) {
return Err(YamlWorkflowError::Validation(format!(
"Declared output '{output_name}' is never produced by any step. \
Add an output data ref with name '{output_name}' to a step."
)));
}
}
Ok(())
}

View File

@@ -2,7 +2,7 @@ use std::collections::HashMap;
use std::time::Duration;
use wfe_core::models::error_behavior::ErrorBehavior;
use wfe_yaml::load_workflow_from_str;
use wfe_yaml::{load_single_workflow_from_str, load_workflow_from_str};
#[test]
fn single_step_produces_one_workflow_step() {
@@ -16,7 +16,7 @@ workflow:
config:
run: echo hello
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
// The definition should have exactly 1 main step.
let main_steps: Vec<_> = compiled
.definition
@@ -44,7 +44,7 @@ workflow:
config:
run: echo b
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step_a = compiled
.definition
@@ -82,7 +82,7 @@ workflow:
config:
run: echo b
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let container = compiled
.definition
@@ -116,7 +116,7 @@ workflow:
config:
run: rollback.sh
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let deploy = compiled
.definition
@@ -156,7 +156,7 @@ workflow:
error_behavior:
type: suspend
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.default_error_behavior,
@@ -193,7 +193,7 @@ workflow:
type: shell
config: *default_config
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
// Should have 2 main steps + factories.
let build_step = compiled
@@ -241,7 +241,7 @@ workflow:
config:
run: echo "build succeeded"
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let build = compiled
.definition
@@ -279,7 +279,7 @@ workflow:
config:
run: cleanup.sh
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let deploy = compiled
.definition
@@ -322,7 +322,7 @@ workflow:
config:
run: cleanup.sh
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let deploy = compiled
.definition
@@ -351,7 +351,7 @@ workflow:
config:
run: echo hi
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.default_error_behavior,
ErrorBehavior::Terminate
@@ -372,7 +372,7 @@ workflow:
config:
run: echo hi
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.default_error_behavior,
ErrorBehavior::Compensate
@@ -394,7 +394,7 @@ workflow:
config:
run: echo hi
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.default_error_behavior,
ErrorBehavior::Retry {
@@ -420,7 +420,7 @@ workflow:
config:
run: echo hi
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.default_error_behavior,
ErrorBehavior::Retry {
@@ -447,7 +447,7 @@ workflow:
config:
run: echo hi
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.default_error_behavior,
ErrorBehavior::Retry {
@@ -471,7 +471,7 @@ workflow:
config:
run: echo hi
"#;
let result = load_workflow_from_str(yaml, &HashMap::new());
let result = load_single_workflow_from_str(yaml, &HashMap::new());
assert!(result.is_err());
let err = match result { Err(e) => e.to_string(), Ok(_) => panic!("expected error") };
assert!(err.contains("explode"), "Error should mention the invalid type, got: {err}");
@@ -499,7 +499,7 @@ workflow:
config:
run: echo c
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let container = compiled
.definition
@@ -541,7 +541,7 @@ workflow:
RUST_LOG: debug
working_dir: /tmp
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
@@ -571,7 +571,7 @@ workflow:
config:
file: my_script.sh
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
@@ -596,7 +596,7 @@ workflow:
config:
run: echo hello
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
@@ -626,7 +626,7 @@ workflow:
config:
run: rollback.sh
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
// Should have factories for both deploy and rollback.
let has_deploy = compiled
@@ -658,7 +658,7 @@ workflow:
config:
run: echo ok
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let has_notify = compiled
.step_factories
@@ -684,7 +684,7 @@ workflow:
config:
run: cleanup.sh
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let has_cleanup = compiled
.step_factories
@@ -713,7 +713,7 @@ workflow:
config:
run: echo b
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let container = compiled
.definition
@@ -746,7 +746,7 @@ workflow:
config:
run: echo b
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step_a = compiled
.definition
@@ -787,7 +787,7 @@ workflow:
config:
run: echo hi
"#;
let compiled = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(
compiled.definition.description.as_deref(),
Some("A test workflow")
@@ -804,7 +804,7 @@ workflow:
- name: bad-step
type: shell
"#;
let result = load_workflow_from_str(yaml, &HashMap::new());
let result = load_single_workflow_from_str(yaml, &HashMap::new());
assert!(result.is_err());
let err = match result { Err(e) => e.to_string(), Ok(_) => panic!("expected error") };
assert!(
@@ -812,3 +812,494 @@ workflow:
"Error should mention missing config, got: {err}"
);
}
// --- Workflow step compilation tests ---
#[test]
fn workflow_step_compiles_correctly() {
let yaml = r#"
workflow:
id: parent-wf
version: 1
steps:
- name: run-child
type: workflow
config:
workflow: child-wf
workflow_version: 3
outputs:
- name: result
- name: status
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("run-child"))
.unwrap();
assert!(step.step_type.contains("workflow"));
assert!(step.step_config.is_some());
// Verify the serialized config contains the workflow_id and version.
let config: serde_json::Value = step.step_config.clone().unwrap();
assert_eq!(config["workflow_id"].as_str(), Some("child-wf"));
assert_eq!(config["version"].as_u64(), Some(3));
assert_eq!(config["output_keys"].as_array().unwrap().len(), 2);
}
#[test]
fn workflow_step_version_defaults_to_1() {
let yaml = r#"
workflow:
id: parent-wf
version: 1
steps:
- name: run-child
type: workflow
config:
workflow: child-wf
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("run-child"))
.unwrap();
let config: serde_json::Value = step.step_config.clone().unwrap();
assert_eq!(config["version"].as_u64(), Some(1));
}
#[test]
fn workflow_step_factory_is_registered() {
let yaml = r#"
workflow:
id: parent-wf
version: 1
steps:
- name: run-child
type: workflow
config:
workflow: child-wf
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let has_workflow_factory = compiled
.step_factories
.iter()
.any(|(key, _)| key.contains("workflow") && key.contains("run-child"));
assert!(
has_workflow_factory,
"Should have factory for workflow step"
);
}
#[test]
fn compile_multi_workflow_file() {
let yaml = r#"
workflows:
- id: build
version: 1
steps:
- name: compile
type: shell
config:
run: cargo build
- id: test
version: 1
steps:
- name: run-tests
type: shell
config:
run: cargo test
"#;
let workflows = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(workflows.len(), 2);
assert_eq!(workflows[0].definition.id, "build");
assert_eq!(workflows[1].definition.id, "test");
}
#[test]
fn compile_multi_workflow_with_cross_references() {
let yaml = r#"
workflows:
- id: pipeline
version: 1
steps:
- name: run-build
type: workflow
config:
workflow: build
- id: build
version: 1
steps:
- name: compile
type: shell
config:
run: cargo build
"#;
let workflows = load_workflow_from_str(yaml, &HashMap::new()).unwrap();
assert_eq!(workflows.len(), 2);
// The pipeline workflow should have a workflow step.
let pipeline = &workflows[0];
let step = pipeline
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("run-build"))
.unwrap();
assert!(step.step_type.contains("workflow"));
}
#[test]
fn workflow_step_with_mixed_steps() {
let yaml = r#"
workflow:
id: mixed-wf
version: 1
steps:
- name: setup
type: shell
config:
run: echo setup
- name: run-child
type: workflow
config:
workflow: child-wf
- name: cleanup
type: shell
config:
run: echo cleanup
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
// Should have 3 main steps.
let step_names: Vec<_> = compiled
.definition
.steps
.iter()
.filter_map(|s| s.name.as_deref())
.collect();
assert!(step_names.contains(&"setup"));
assert!(step_names.contains(&"run-child"));
assert!(step_names.contains(&"cleanup"));
// setup -> run-child -> cleanup wiring.
let setup = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("setup"))
.unwrap();
let run_child = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("run-child"))
.unwrap();
let cleanup = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("cleanup"))
.unwrap();
assert_eq!(setup.outcomes[0].next_step, run_child.id);
assert_eq!(run_child.outcomes[0].next_step, cleanup.id);
}
/// Regression test: SubWorkflowStep must actually wait for child completion,
/// not return next() immediately. The compiled factory must produce a real
/// SubWorkflowStep (from wfe-core), not a placeholder.
#[tokio::test]
async fn workflow_step_factory_produces_real_sub_workflow_step() {
use wfe_core::models::{ExecutionPointer, WorkflowInstance, WorkflowStep as WfStep};
use wfe_core::traits::step::{HostContext, StepExecutionContext};
use std::pin::Pin;
use std::future::Future;
use std::sync::Mutex;
let yaml = r#"
workflows:
- id: child
version: 1
steps:
- name: do-work
type: shell
config:
run: echo done
- id: parent
version: 1
steps:
- name: run-child
type: workflow
config:
workflow: child
"#;
let config = HashMap::new();
let workflows = load_workflow_from_str(yaml, &config).unwrap();
// Find the parent workflow's factory for the "run-child" step
let parent = workflows.iter().find(|w| w.definition.id == "parent").unwrap();
let factory_key = parent.step_factories.iter()
.find(|(k, _)| k.contains("run-child"))
.map(|(k, _)| k.clone())
.expect("run-child factory should exist");
// Create a step from the factory
let factory = &parent.step_factories.iter()
.find(|(k, _)| *k == factory_key)
.unwrap().1;
let mut step = factory();
// Mock host context that records the start_workflow call
struct MockHost { called: Mutex<bool> }
impl HostContext for MockHost {
fn start_workflow(&self, _def: &str, _ver: u32, _data: serde_json::Value)
-> Pin<Box<dyn Future<Output = wfe_core::Result<String>> + Send + '_>>
{
*self.called.lock().unwrap() = true;
Box::pin(async { Ok("child-instance-id".to_string()) })
}
}
let host = MockHost { called: Mutex::new(false) };
let pointer = ExecutionPointer::new(0);
let wf_step = WfStep::new(0, &factory_key);
let workflow = WorkflowInstance::new("parent", 1, serde_json::json!({}));
let ctx = StepExecutionContext {
item: None,
execution_pointer: &pointer,
persistence_data: None,
step: &wf_step,
workflow: &workflow,
cancellation_token: tokio_util::sync::CancellationToken::new(),
host_context: Some(&host),
};
let result = step.run(&ctx).await.unwrap();
// THE KEY ASSERTION: must NOT proceed immediately.
// It must return wait_for_event so the parent waits for the child.
assert!(
!result.proceed,
"SubWorkflowStep must NOT proceed immediately — it should wait for child completion"
);
assert_eq!(
result.event_name.as_deref(),
Some("wfe.workflow.completed"),
"SubWorkflowStep must wait for wfe.workflow.completed event"
);
assert!(
*host.called.lock().unwrap(),
"SubWorkflowStep must call host_context.start_workflow()"
);
}
// --- Condition compilation tests ---
#[test]
fn compile_simple_condition_into_step_condition() {
let yaml = r#"
workflow:
id: cond-compile
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
field: .inputs.enabled
equals: true
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("deploy"))
.unwrap();
assert!(step.when.is_some(), "Step should have a when condition");
match step.when.as_ref().unwrap() {
wfe_core::models::StepCondition::Comparison(cmp) => {
assert_eq!(cmp.field, ".inputs.enabled");
assert_eq!(cmp.operator, wfe_core::models::ComparisonOp::Equals);
assert_eq!(cmp.value, Some(serde_json::json!(true)));
}
other => panic!("Expected Comparison, got: {other:?}"),
}
}
#[test]
fn compile_nested_condition() {
let yaml = r#"
workflow:
id: nested-cond-compile
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
all:
- field: .inputs.count
gt: 5
- not:
field: .inputs.skip
equals: true
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("deploy"))
.unwrap();
assert!(step.when.is_some());
match step.when.as_ref().unwrap() {
wfe_core::models::StepCondition::All(children) => {
assert_eq!(children.len(), 2);
// First child: comparison
match &children[0] {
wfe_core::models::StepCondition::Comparison(cmp) => {
assert_eq!(cmp.field, ".inputs.count");
assert_eq!(cmp.operator, wfe_core::models::ComparisonOp::Gt);
assert_eq!(cmp.value, Some(serde_json::json!(5)));
}
other => panic!("Expected Comparison, got: {other:?}"),
}
// Second child: not
match &children[1] {
wfe_core::models::StepCondition::Not(inner) => {
match inner.as_ref() {
wfe_core::models::StepCondition::Comparison(cmp) => {
assert_eq!(cmp.field, ".inputs.skip");
assert_eq!(cmp.operator, wfe_core::models::ComparisonOp::Equals);
}
other => panic!("Expected Comparison inside Not, got: {other:?}"),
}
}
other => panic!("Expected Not, got: {other:?}"),
}
}
other => panic!("Expected All, got: {other:?}"),
}
}
#[test]
fn step_without_when_has_none_condition() {
let yaml = r#"
workflow:
id: no-when-compile
version: 1
steps:
- name: step1
type: shell
config:
run: echo hi
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let step = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("step1"))
.unwrap();
assert!(step.when.is_none());
}
#[test]
fn compile_all_comparison_operators() {
use wfe_core::models::ComparisonOp;
let ops = vec![
("equals: 42", ComparisonOp::Equals),
("not_equals: foo", ComparisonOp::NotEquals),
("gt: 10", ComparisonOp::Gt),
("gte: 10", ComparisonOp::Gte),
("lt: 100", ComparisonOp::Lt),
("lte: 100", ComparisonOp::Lte),
("contains: needle", ComparisonOp::Contains),
("is_null: true", ComparisonOp::IsNull),
("is_not_null: true", ComparisonOp::IsNotNull),
];
for (op_yaml, expected_op) in ops {
let yaml = format!(
r#"
workflow:
id: op-test
version: 1
steps:
- name: step1
type: shell
config:
run: echo hi
when:
field: .inputs.x
{op_yaml}
"#
);
let compiled = load_single_workflow_from_str(&yaml, &HashMap::new())
.unwrap_or_else(|e| panic!("Failed to compile with {op_yaml}: {e}"));
let step = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("step1"))
.unwrap();
match step.when.as_ref().unwrap() {
wfe_core::models::StepCondition::Comparison(cmp) => {
assert_eq!(cmp.operator, expected_op, "Operator mismatch for {op_yaml}");
}
other => panic!("Expected Comparison for {op_yaml}, got: {other:?}"),
}
}
}
#[test]
fn compile_condition_on_parallel_container() {
let yaml = r#"
workflow:
id: parallel-cond
version: 1
steps:
- name: parallel-group
when:
field: .inputs.run_parallel
equals: true
parallel:
- name: task-a
type: shell
config:
run: echo a
- name: task-b
type: shell
config:
run: echo b
"#;
let compiled = load_single_workflow_from_str(yaml, &HashMap::new()).unwrap();
let container = compiled
.definition
.steps
.iter()
.find(|s| s.name.as_deref() == Some("parallel-group"))
.unwrap();
assert!(
container.when.is_some(),
"Parallel container should have when condition"
);
}

View File

@@ -41,6 +41,7 @@ fn make_context<'a>(
step,
workflow,
cancellation_token: tokio_util::sync::CancellationToken::new(),
host_context: None,
}
}
@@ -219,7 +220,7 @@ workflow:
type: deno
"#;
let config = HashMap::new();
let result = wfe_yaml::load_workflow_from_str(yaml, &config);
let result = wfe_yaml::load_single_workflow_from_str(yaml, &config);
assert!(result.is_err());
let msg = result.err().unwrap().to_string();
assert!(
@@ -242,7 +243,7 @@ workflow:
FOO: bar
"#;
let config = HashMap::new();
let result = wfe_yaml::load_workflow_from_str(yaml, &config);
let result = wfe_yaml::load_single_workflow_from_str(yaml, &config);
assert!(result.is_err());
let msg = result.err().unwrap().to_string();
assert!(
@@ -264,7 +265,7 @@ workflow:
script: "output('key', 'val');"
"#;
let config = HashMap::new();
let compiled = wfe_yaml::load_workflow_from_str(yaml, &config).unwrap();
let compiled = wfe_yaml::load_single_workflow_from_str(yaml, &config).unwrap();
assert!(!compiled.step_factories.is_empty());
let (key, _factory) = &compiled.step_factories[0];
assert!(key.contains("deno"), "factory key should contain 'deno', got: {key}");

View File

@@ -15,7 +15,7 @@ use wfe::{WorkflowHostBuilder, run_workflow_sync};
use wfe_core::test_support::{
InMemoryLockProvider, InMemoryPersistenceProvider, InMemoryQueueProvider,
};
use wfe_yaml::load_workflow_from_str;
use wfe_yaml::load_single_workflow_from_str;
// ---------------------------------------------------------------------------
// Helpers
@@ -30,7 +30,7 @@ async fn run_yaml_workflow_with_data(
data: serde_json::Value,
) -> wfe::models::WorkflowInstance {
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let persistence = Arc::new(InMemoryPersistenceProvider::new());
let lock = Arc::new(InMemoryLockProvider::new());
@@ -437,7 +437,7 @@ workflow:
script: "output('x', 1);"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
assert!(!compiled.step_factories.is_empty());
let (key, _factory) = &compiled.step_factories[0];
assert!(
@@ -467,7 +467,7 @@ workflow:
dynamic_import: true
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
assert!(!compiled.step_factories.is_empty());
// Verify the step config was serialized correctly.
let step = compiled
@@ -497,7 +497,7 @@ workflow:
timeout: "3s"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let step = compiled
.definition
.steps
@@ -521,7 +521,7 @@ workflow:
file: "./scripts/run.js"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let step = compiled
.definition
.steps
@@ -543,7 +543,7 @@ workflow:
type: deno
"#;
let config = HashMap::new();
let result = load_workflow_from_str(yaml, &config);
let result = load_single_workflow_from_str(yaml, &config);
match result {
Err(e) => {
let msg = e.to_string();
@@ -570,7 +570,7 @@ workflow:
FOO: bar
"#;
let config = HashMap::new();
let result = load_workflow_from_str(yaml, &config);
let result = load_single_workflow_from_str(yaml, &config);
match result {
Err(e) => {
let msg = e.to_string();
@@ -596,7 +596,7 @@ workflow:
script: "1+1;"
"#;
let config = HashMap::new();
assert!(load_workflow_from_str(yaml, &config).is_ok());
assert!(load_single_workflow_from_str(yaml, &config).is_ok());
}
#[test]
@@ -612,7 +612,7 @@ workflow:
file: "./run.js"
"#;
let config = HashMap::new();
assert!(load_workflow_from_str(yaml, &config).is_ok());
assert!(load_single_workflow_from_str(yaml, &config).is_ok());
}
#[test]
@@ -632,7 +632,7 @@ workflow:
script: "output('x', 1);"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let has_shell = compiled.step_factories.iter().any(|(k, _)| k.contains("shell"));
let has_deno = compiled.step_factories.iter().any(|(k, _)| k.contains("deno"));
assert!(has_shell, "should have shell factory");
@@ -655,7 +655,7 @@ workflow:
- "npm:is-number@7"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let step = compiled
.definition
.steps
@@ -685,7 +685,7 @@ workflow:
BAZ: qux
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let step = compiled
.definition
.steps
@@ -744,7 +744,7 @@ workflow:
"#;
let config = HashMap::new();
// This should compile without errors.
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
assert!(compiled.step_factories.len() >= 2);
}
@@ -809,7 +809,7 @@ workflow:
script: "1;"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
assert_eq!(
compiled.definition.description.as_deref(),
Some("A workflow with deno steps")
@@ -834,7 +834,7 @@ workflow:
interval: "2s"
"#;
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let step = compiled
.definition
.steps

View File

@@ -7,11 +7,11 @@ use wfe::{WorkflowHostBuilder, run_workflow_sync};
use wfe_core::test_support::{
InMemoryLockProvider, InMemoryPersistenceProvider, InMemoryQueueProvider,
};
use wfe_yaml::load_workflow_from_str;
use wfe_yaml::load_single_workflow_from_str;
async fn run_yaml_workflow(yaml: &str) -> wfe::models::WorkflowInstance {
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let persistence = Arc::new(InMemoryPersistenceProvider::new());
let lock = Arc::new(InMemoryLockProvider::new());
@@ -91,7 +91,7 @@ workflow:
assert_eq!(greeting.as_str(), Some("hello"));
}
if let Some(count) = data.get("count") {
assert_eq!(count.as_str(), Some("42"));
assert_eq!(count.as_i64(), Some(42)); // auto-converted from string "42"
}
}
}

View File

@@ -1,7 +1,7 @@
use std::collections::HashMap;
use std::io::Write;
use wfe_yaml::{load_workflow, load_workflow_from_str};
use wfe_yaml::{load_workflow, load_single_workflow_from_str};
#[test]
fn load_workflow_from_file() {
@@ -45,7 +45,7 @@ fn load_workflow_from_nonexistent_file_returns_error() {
#[test]
fn load_workflow_from_str_with_invalid_yaml_returns_error() {
let yaml = "this is not valid yaml: [[[";
let result = load_workflow_from_str(yaml, &HashMap::new());
let result = load_single_workflow_from_str(yaml, &HashMap::new());
assert!(result.is_err());
let err = match result { Err(e) => e.to_string(), Ok(_) => panic!("expected error") };
assert!(
@@ -69,7 +69,7 @@ workflow:
let mut config = HashMap::new();
config.insert("message".to_string(), serde_json::json!("hello world"));
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let step = compiled
.definition
.steps
@@ -94,7 +94,7 @@ workflow:
config:
run: echo ((missing))
"#;
let result = load_workflow_from_str(yaml, &HashMap::new());
let result = load_single_workflow_from_str(yaml, &HashMap::new());
assert!(result.is_err());
let err = match result { Err(e) => e.to_string(), Ok(_) => panic!("expected error") };
assert!(

View File

@@ -1,4 +1,4 @@
use wfe_yaml::schema::YamlWorkflow;
use wfe_yaml::schema::{YamlCondition, YamlWorkflow, YamlWorkflowFile};
#[test]
fn parse_minimal_yaml() {
@@ -192,3 +192,361 @@ workflow:
assert_eq!(parsed.workflow.id, "template-wf");
assert_eq!(parsed.workflow.steps.len(), 1);
}
// --- Multi-workflow file tests ---
#[test]
fn parse_single_workflow_file() {
let yaml = r#"
workflow:
id: single
version: 1
steps:
- name: step1
type: shell
config:
run: echo hello
"#;
let parsed: YamlWorkflowFile = serde_yaml::from_str(yaml).unwrap();
assert!(parsed.workflow.is_some());
assert!(parsed.workflows.is_none());
assert_eq!(parsed.workflow.unwrap().id, "single");
}
#[test]
fn parse_multi_workflow_file() {
let yaml = r#"
workflows:
- id: build-wf
version: 1
steps:
- name: build
type: shell
config:
run: cargo build
- id: test-wf
version: 1
steps:
- name: test
type: shell
config:
run: cargo test
"#;
let parsed: YamlWorkflowFile = serde_yaml::from_str(yaml).unwrap();
assert!(parsed.workflow.is_none());
assert!(parsed.workflows.is_some());
let workflows = parsed.workflows.unwrap();
assert_eq!(workflows.len(), 2);
assert_eq!(workflows[0].id, "build-wf");
assert_eq!(workflows[1].id, "test-wf");
}
#[test]
fn parse_workflow_with_input_output_schemas() {
let yaml = r#"
workflow:
id: typed-wf
version: 1
inputs:
repo_url: string
tags: "list<string>"
verbose: bool?
outputs:
artifact_path: string
exit_code: integer
steps:
- name: step1
type: shell
config:
run: echo hello
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
assert_eq!(parsed.workflow.inputs.len(), 3);
assert_eq!(parsed.workflow.inputs.get("repo_url").unwrap(), "string");
assert_eq!(
parsed.workflow.inputs.get("tags").unwrap(),
"list<string>"
);
assert_eq!(parsed.workflow.inputs.get("verbose").unwrap(), "bool?");
assert_eq!(parsed.workflow.outputs.len(), 2);
assert_eq!(
parsed.workflow.outputs.get("artifact_path").unwrap(),
"string"
);
assert_eq!(
parsed.workflow.outputs.get("exit_code").unwrap(),
"integer"
);
}
#[test]
fn parse_step_with_workflow_type() {
let yaml = r#"
workflow:
id: parent-wf
version: 1
steps:
- name: run-child
type: workflow
config:
workflow: child-wf
workflow_version: 2
inputs:
- name: repo_url
path: data.repo
outputs:
- name: result
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let step = &parsed.workflow.steps[0];
assert_eq!(step.step_type.as_deref(), Some("workflow"));
let config = step.config.as_ref().unwrap();
assert_eq!(config.child_workflow.as_deref(), Some("child-wf"));
assert_eq!(config.child_version, Some(2));
assert_eq!(step.inputs.len(), 1);
assert_eq!(step.outputs.len(), 1);
}
#[test]
fn parse_workflow_step_version_defaults() {
let yaml = r#"
workflow:
id: parent-wf
version: 1
steps:
- name: run-child
type: workflow
config:
workflow: child-wf
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let config = parsed.workflow.steps[0].config.as_ref().unwrap();
assert_eq!(config.child_workflow.as_deref(), Some("child-wf"));
// version not specified, should be None in schema (compiler defaults to 1).
assert_eq!(config.child_version, None);
}
#[test]
fn parse_empty_inputs_outputs_default() {
let yaml = r#"
workflow:
id: no-schema-wf
version: 1
steps:
- name: step1
type: shell
config:
run: echo hello
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
assert!(parsed.workflow.inputs.is_empty());
assert!(parsed.workflow.outputs.is_empty());
}
// --- Condition schema tests ---
#[test]
fn parse_step_with_simple_when_condition() {
let yaml = r#"
workflow:
id: cond-wf
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
field: .inputs.enabled
equals: true
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let step = &parsed.workflow.steps[0];
assert!(step.when.is_some());
match step.when.as_ref().unwrap() {
YamlCondition::Comparison(cmp) => {
assert_eq!(cmp.field, ".inputs.enabled");
assert!(cmp.equals.is_some());
}
_ => panic!("Expected Comparison variant"),
}
}
#[test]
fn parse_step_with_nested_combinator_conditions() {
let yaml = r#"
workflow:
id: nested-cond-wf
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
all:
- field: .inputs.count
gt: 5
- any:
- field: .inputs.env
equals: prod
- field: .inputs.env
equals: staging
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let step = &parsed.workflow.steps[0];
assert!(step.when.is_some());
match step.when.as_ref().unwrap() {
YamlCondition::Combinator(c) => {
assert!(c.all.is_some());
let children = c.all.as_ref().unwrap();
assert_eq!(children.len(), 2);
}
_ => panic!("Expected Combinator variant"),
}
}
#[test]
fn parse_step_with_not_condition() {
let yaml = r#"
workflow:
id: not-cond-wf
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
not:
field: .inputs.skip
equals: true
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let step = &parsed.workflow.steps[0];
match step.when.as_ref().unwrap() {
YamlCondition::Combinator(c) => {
assert!(c.not.is_some());
}
_ => panic!("Expected Combinator with not"),
}
}
#[test]
fn parse_step_with_none_condition() {
let yaml = r#"
workflow:
id: none-cond-wf
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
none:
- field: .inputs.skip
equals: true
- field: .inputs.disabled
equals: true
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let step = &parsed.workflow.steps[0];
match step.when.as_ref().unwrap() {
YamlCondition::Combinator(c) => {
assert!(c.none.is_some());
assert_eq!(c.none.as_ref().unwrap().len(), 2);
}
_ => panic!("Expected Combinator with none"),
}
}
#[test]
fn parse_step_with_one_of_condition() {
let yaml = r#"
workflow:
id: one-of-wf
version: 1
steps:
- name: deploy
type: shell
config:
run: deploy.sh
when:
one_of:
- field: .inputs.mode
equals: fast
- field: .inputs.mode
equals: slow
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
let step = &parsed.workflow.steps[0];
match step.when.as_ref().unwrap() {
YamlCondition::Combinator(c) => {
assert!(c.one_of.is_some());
assert_eq!(c.one_of.as_ref().unwrap().len(), 2);
}
_ => panic!("Expected Combinator with one_of"),
}
}
#[test]
fn parse_comparison_with_each_operator() {
// Test that each operator variant deserializes correctly.
let operators = vec![
("equals: 42", "equals"),
("not_equals: foo", "not_equals"),
("gt: 10", "gt"),
("gte: 10", "gte"),
("lt: 100", "lt"),
("lte: 100", "lte"),
("contains: needle", "contains"),
("is_null: true", "is_null"),
("is_not_null: true", "is_not_null"),
];
for (op_yaml, op_name) in operators {
let yaml = format!(
r#"
workflow:
id: op-{op_name}
version: 1
steps:
- name: step1
type: shell
config:
run: echo hi
when:
field: .inputs.x
{op_yaml}
"#
);
let parsed: YamlWorkflow = serde_yaml::from_str(&yaml)
.unwrap_or_else(|e| panic!("Failed to parse operator {op_name}: {e}"));
let step = &parsed.workflow.steps[0];
assert!(
step.when.is_some(),
"Step should have when condition for operator {op_name}"
);
match step.when.as_ref().unwrap() {
YamlCondition::Comparison(_) => {}
_ => panic!("Expected Comparison for operator {op_name}"),
}
}
}
#[test]
fn parse_step_without_when_has_none() {
let yaml = r#"
workflow:
id: no-when-wf
version: 1
steps:
- name: step1
type: shell
config:
run: echo hi
"#;
let parsed: YamlWorkflow = serde_yaml::from_str(yaml).unwrap();
assert!(parsed.workflow.steps[0].when.is_none());
}

View File

@@ -7,14 +7,14 @@ use wfe::{WorkflowHostBuilder, run_workflow_sync};
use wfe_core::test_support::{
InMemoryLockProvider, InMemoryPersistenceProvider, InMemoryQueueProvider,
};
use wfe_yaml::load_workflow_from_str;
use wfe_yaml::load_single_workflow_from_str;
async fn run_yaml_workflow_with_data(
yaml: &str,
data: serde_json::Value,
) -> wfe::models::WorkflowInstance {
let config = HashMap::new();
let compiled = load_workflow_from_str(yaml, &config).unwrap();
let compiled = load_single_workflow_from_str(yaml, &config).unwrap();
let persistence = Arc::new(InMemoryPersistenceProvider::new());
let lock = Arc::new(InMemoryLockProvider::new());
@@ -106,7 +106,7 @@ workflow:
assert_eq!(greeting.as_str(), Some("hello"));
}
if let Some(count) = data.get("count") {
assert_eq!(count.as_str(), Some("42"));
assert_eq!(count.as_i64(), Some(42)); // auto-converted from string "42"
}
if let Some(path) = data.get("path") {
assert_eq!(path.as_str(), Some("/usr/local/bin"));

107
wfe-yaml/tests/types.rs Normal file
View File

@@ -0,0 +1,107 @@
use wfe_yaml::types::{parse_type_string, SchemaType};
#[test]
fn parse_all_primitives() {
assert_eq!(parse_type_string("string").unwrap(), SchemaType::String);
assert_eq!(parse_type_string("number").unwrap(), SchemaType::Number);
assert_eq!(parse_type_string("integer").unwrap(), SchemaType::Integer);
assert_eq!(parse_type_string("bool").unwrap(), SchemaType::Bool);
assert_eq!(parse_type_string("any").unwrap(), SchemaType::Any);
}
#[test]
fn parse_optional_types() {
assert_eq!(
parse_type_string("string?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::String))
);
assert_eq!(
parse_type_string("integer?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::Integer))
);
}
#[test]
fn parse_list_types() {
assert_eq!(
parse_type_string("list<string>").unwrap(),
SchemaType::List(Box::new(SchemaType::String))
);
assert_eq!(
parse_type_string("list<number>").unwrap(),
SchemaType::List(Box::new(SchemaType::Number))
);
}
#[test]
fn parse_map_types() {
assert_eq!(
parse_type_string("map<string>").unwrap(),
SchemaType::Map(Box::new(SchemaType::String))
);
assert_eq!(
parse_type_string("map<any>").unwrap(),
SchemaType::Map(Box::new(SchemaType::Any))
);
}
#[test]
fn parse_nested_generics() {
assert_eq!(
parse_type_string("list<list<string>>").unwrap(),
SchemaType::List(Box::new(SchemaType::List(Box::new(SchemaType::String))))
);
assert_eq!(
parse_type_string("map<list<integer>>").unwrap(),
SchemaType::Map(Box::new(SchemaType::List(Box::new(SchemaType::Integer))))
);
}
#[test]
fn parse_optional_generic() {
assert_eq!(
parse_type_string("list<string>?").unwrap(),
SchemaType::Optional(Box::new(SchemaType::List(Box::new(SchemaType::String))))
);
}
#[test]
fn parse_unknown_type_returns_error() {
let err = parse_type_string("foobar").unwrap_err();
assert!(err.contains("Unknown type"), "Got: {err}");
}
#[test]
fn parse_unknown_generic_container_returns_error() {
let err = parse_type_string("set<string>").unwrap_err();
assert!(err.contains("Unknown generic type"), "Got: {err}");
}
#[test]
fn parse_empty_returns_error() {
let err = parse_type_string("").unwrap_err();
assert!(err.contains("Empty"), "Got: {err}");
}
#[test]
fn parse_malformed_generic_returns_error() {
let err = parse_type_string("list<string").unwrap_err();
assert!(err.contains("Malformed"), "Got: {err}");
}
#[test]
fn display_roundtrip() {
for s in &[
"string",
"number",
"integer",
"bool",
"any",
"list<string>",
"map<number>",
"list<list<string>>",
] {
let parsed = parse_type_string(s).unwrap();
assert_eq!(parsed.to_string(), *s, "Roundtrip failed for {s}");
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -3,6 +3,8 @@ name = "wfe"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
homepage.workspace = true
description = "WFE workflow engine - umbrella crate"
[features]
@@ -39,6 +41,7 @@ opentelemetry-otlp = { workspace = true, optional = true }
[dev-dependencies]
wfe-core = { workspace = true, features = ["test-support"] }
wfe-sqlite = { workspace = true }
wfe-yaml = { workspace = true, features = ["deno"] }
pretty_assertions = { workspace = true }
rstest = { workspace = true }
wiremock = { workspace = true }

View File

@@ -0,0 +1,165 @@
// =============================================================================
// WFE Self-Hosting CI Pipeline Runner
// =============================================================================
//
// Loads the multi-workflow CI pipeline from a YAML file and runs it to
// completion using the WFE engine with in-memory providers.
//
// Usage:
// cargo run --example run_pipeline -p wfe -- workflows.yaml
//
// With config:
// WFE_CONFIG='{"workspace_dir":"/path/to/wfe","registry":"sunbeam"}' \
// cargo run --example run_pipeline -p wfe -- workflows.yaml
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
use serde_json::json;
use wfe::models::WorkflowStatus;
use wfe::test_support::{InMemoryLockProvider, InMemoryQueueProvider, InMemoryPersistenceProvider};
use wfe::WorkflowHostBuilder;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Set up tracing.
tracing_subscriber::fmt()
.with_target(false)
.with_timer(tracing_subscriber::fmt::time::uptime())
.with_env_filter(
std::env::var("RUST_LOG")
.unwrap_or_else(|_| "wfe_core=info,wfe=info,run_pipeline=info".into())
)
.init();
// Read YAML path from args.
let yaml_path = std::env::args()
.nth(1)
.expect("usage: run_pipeline <workflows.yaml>");
// Read config from WFE_CONFIG env var (JSON map), merged over sensible defaults.
let cwd = std::env::current_dir()?
.to_string_lossy()
.to_string();
// Defaults for every ((var)) referenced in the YAML.
let mut config: HashMap<String, serde_json::Value> = HashMap::from([
("workspace_dir".into(), json!(cwd)),
("coverage_threshold".into(), json!(85)),
("registry".into(), json!("sunbeam")),
("git_remote".into(), json!("origin")),
("version".into(), json!("0.0.0")),
]);
// Overlay user-provided config (WFE_CONFIG env var, JSON object).
if let Ok(user_json) = std::env::var("WFE_CONFIG") {
let user: HashMap<String, serde_json::Value> = serde_json::from_str(&user_json)?;
config.extend(user);
}
let config_json = serde_json::to_string(&config)?;
println!("Loading workflows from: {yaml_path}");
println!("Config: {config_json}");
// Load and compile all workflow definitions from the YAML file.
let yaml_content = std::fs::read_to_string(&yaml_path)?;
let workflows = wfe_yaml::load_workflow_from_str(&yaml_content, &config)?;
println!("Compiled {} workflow(s):", workflows.len());
for compiled in &workflows {
println!(
" - {} v{} ({} step factories)",
compiled.definition.id,
compiled.definition.version,
compiled.step_factories.len(),
);
}
// Build the host with in-memory providers.
let persistence = Arc::new(InMemoryPersistenceProvider::default());
let lock = Arc::new(InMemoryLockProvider::default());
let queue = Arc::new(InMemoryQueueProvider::default());
let host = WorkflowHostBuilder::new()
.use_persistence(persistence)
.use_lock_provider(lock)
.use_queue_provider(queue)
.build()?;
// Register all compiled workflows and their step factories.
// We must move the factories out of the compiled workflows since
// register_step_factory requires 'static closures.
for mut compiled in workflows {
let factories = std::mem::take(&mut compiled.step_factories);
for (key, factory) in factories {
host.register_step_factory(&key, move || factory()).await;
}
host.register_workflow_definition(compiled.definition).await;
}
// Start the engine.
host.start().await?;
println!("\nEngine started. Launching 'ci' workflow...\n");
// Determine workspace_dir for initial data (use config value or cwd).
let workspace_dir = config
.get("workspace_dir")
.and_then(|v| v.as_str())
.unwrap_or(&cwd)
.to_string();
let data = json!({
"workspace_dir": workspace_dir,
});
let workflow_id = host.start_workflow("ci", 1, data).await?;
println!("Workflow instance: {workflow_id}");
// Poll for completion with a 1-hour timeout.
let timeout = Duration::from_secs(3600);
let deadline = tokio::time::Instant::now() + timeout;
let poll_interval = Duration::from_millis(500);
let final_instance = loop {
let instance = host.get_workflow(&workflow_id).await?;
match instance.status {
WorkflowStatus::Complete | WorkflowStatus::Terminated => break instance,
_ if tokio::time::Instant::now() > deadline => {
eprintln!("Timeout: workflow did not complete within {timeout:?}");
break instance;
}
_ => tokio::time::sleep(poll_interval).await,
}
};
// Print final status.
println!("\n========================================");
println!("Pipeline status: {:?}", final_instance.status);
println!(
"Execution pointers: {} total, {} complete",
final_instance.execution_pointers.len(),
final_instance
.execution_pointers
.iter()
.filter(|p| p.status == wfe::models::PointerStatus::Complete)
.count()
);
// Print workflow data (contains outputs from all steps).
if let Some(obj) = final_instance.data.as_object() {
println!("\nKey outputs:");
for key in ["version", "all_tests_passed", "coverage", "published", "released"] {
if let Some(val) = obj.get(key) {
println!(" {key}: {val}");
}
}
}
println!("========================================");
host.stop().await;
println!("\nEngine stopped.");
Ok(())
}

View File

@@ -1,3 +1,5 @@
use std::future::Future;
use std::pin::Pin;
use std::sync::Arc;
use tokio::sync::RwLock;
@@ -10,8 +12,8 @@ use wfe_core::models::{
WorkflowStatus,
};
use wfe_core::traits::{
DistributedLockProvider, LifecyclePublisher, PersistenceProvider, QueueProvider, SearchIndex,
StepBody, WorkflowData,
DistributedLockProvider, HostContext, LifecyclePublisher, PersistenceProvider, QueueProvider,
SearchIndex, StepBody, WorkflowData,
};
use wfe_core::traits::registry::WorkflowRegistry;
use wfe_core::{Result, WfeError};
@@ -19,6 +21,51 @@ use wfe_core::builder::WorkflowBuilder;
use crate::registry::InMemoryWorkflowRegistry;
/// A lightweight HostContext implementation that delegates to the WorkflowHost's
/// components. Used by the background consumer task which cannot hold a direct
/// reference to WorkflowHost (it runs in a spawned tokio task).
pub(crate) struct HostContextImpl {
persistence: Arc<dyn PersistenceProvider>,
registry: Arc<RwLock<InMemoryWorkflowRegistry>>,
queue_provider: Arc<dyn QueueProvider>,
}
impl HostContext for HostContextImpl {
fn start_workflow(
&self,
definition_id: &str,
version: u32,
data: serde_json::Value,
) -> Pin<Box<dyn Future<Output = Result<String>> + Send + '_>> {
let def_id = definition_id.to_string();
Box::pin(async move {
// Look up the definition.
let reg = self.registry.read().await;
let definition = reg
.get_definition(&def_id, Some(version))
.ok_or_else(|| WfeError::DefinitionNotFound {
id: def_id.clone(),
version,
})?;
// Create the child workflow instance.
let mut instance = WorkflowInstance::new(&def_id, version, data);
if !definition.steps.is_empty() {
instance.execution_pointers.push(ExecutionPointer::new(0));
}
let id = self.persistence.create_new_workflow(&instance).await?;
// Queue for execution.
self.queue_provider
.queue_work(&id, QueueType::Workflow)
.await?;
Ok(id)
})
}
}
/// The main orchestrator that ties all workflow engine components together.
pub struct WorkflowHost {
pub(crate) persistence: Arc<dyn PersistenceProvider>,
@@ -49,6 +96,7 @@ impl WorkflowHost {
sr.register::<sequence::SequenceStep>();
sr.register::<wait_for::WaitForStep>();
sr.register::<while_step::WhileStep>();
sr.register::<sub_workflow::SubWorkflowStep>();
}
/// Spawn background polling tasks for processing workflows and events.
@@ -66,6 +114,11 @@ impl WorkflowHost {
let step_registry = Arc::clone(&self.step_registry);
let queue = Arc::clone(&self.queue_provider);
let shutdown = self.shutdown.clone();
let host_ctx = Arc::new(HostContextImpl {
persistence: Arc::clone(&self.persistence),
registry: Arc::clone(&self.registry),
queue_provider: Arc::clone(&self.queue_provider),
});
tokio::spawn(async move {
loop {
@@ -94,7 +147,7 @@ impl WorkflowHost {
Some(def) => {
let def_clone = def.clone();
let sr = step_registry.read().await;
if let Err(e) = executor.execute(&workflow_id, &def_clone, &sr).await {
if let Err(e) = executor.execute(&workflow_id, &def_clone, &sr, Some(host_ctx.as_ref())).await {
error!(workflow_id = %workflow_id, error = %e, "Workflow execution failed");
}
}

View File

@@ -0,0 +1,516 @@
use std::sync::Arc;
use std::time::Duration;
use async_trait::async_trait;
use serde_json::json;
use wfe::models::{
ExecutionResult, StepOutcome, WorkflowDefinition, WorkflowStatus, WorkflowStep,
};
use wfe::traits::step::{StepBody, StepExecutionContext};
use wfe::WorkflowHostBuilder;
use wfe_core::primitives::sub_workflow::SubWorkflowStep;
use wfe_core::test_support::{
InMemoryLockProvider, InMemoryPersistenceProvider, InMemoryQueueProvider,
};
// ----- Test step bodies -----
/// A step that sets output data with {"result": "child_done"}.
#[derive(Default)]
struct SetResultStep;
#[async_trait]
impl StepBody for SetResultStep {
async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe_core::Result<ExecutionResult> {
let mut result = ExecutionResult::next();
result.output_data = Some(json!({"result": "child_done"}));
Ok(result)
}
}
/// A step that reads workflow data and proceeds.
#[derive(Default)]
struct ReadDataStep;
#[async_trait]
impl StepBody for ReadDataStep {
async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe_core::Result<ExecutionResult> {
Ok(ExecutionResult::next())
}
}
/// A step that always fails.
#[derive(Default)]
struct FailingStep;
#[async_trait]
impl StepBody for FailingStep {
async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe_core::Result<ExecutionResult> {
Err(wfe_core::WfeError::StepExecution(
"intentional failure".to_string(),
))
}
}
/// A step that sets greeting output based on input.
#[derive(Default)]
struct GreetStep;
#[async_trait]
impl StepBody for GreetStep {
async fn run(&mut self, ctx: &StepExecutionContext<'_>) -> wfe_core::Result<ExecutionResult> {
let name = ctx
.workflow
.data
.get("name")
.and_then(|v| v.as_str())
.unwrap_or("unknown");
let mut result = ExecutionResult::next();
result.output_data = Some(json!({"greeting": format!("hello {name}")}));
Ok(result)
}
}
/// A no-op step.
#[derive(Default)]
struct NoOpStep;
#[async_trait]
impl StepBody for NoOpStep {
async fn run(&mut self, _ctx: &StepExecutionContext<'_>) -> wfe_core::Result<ExecutionResult> {
Ok(ExecutionResult::next())
}
}
// ----- Helpers -----
fn build_host() -> (wfe::WorkflowHost, Arc<InMemoryPersistenceProvider>) {
let persistence = Arc::new(InMemoryPersistenceProvider::new());
let lock = Arc::new(InMemoryLockProvider::new());
let queue = Arc::new(InMemoryQueueProvider::new());
let host = WorkflowHostBuilder::new()
.use_persistence(persistence.clone() as Arc<dyn wfe_core::traits::PersistenceProvider>)
.use_lock_provider(lock as Arc<dyn wfe_core::traits::DistributedLockProvider>)
.use_queue_provider(queue as Arc<dyn wfe_core::traits::QueueProvider>)
.build()
.unwrap();
(host, persistence)
}
fn sub_workflow_step_type() -> String {
std::any::type_name::<SubWorkflowStep>().to_string()
}
/// Build a child workflow definition with a single step that produces output.
fn build_child_definition(child_step_type: &str) -> WorkflowDefinition {
let mut def = WorkflowDefinition::new("child-workflow", 1);
let step0 = WorkflowStep::new(0, child_step_type);
def.steps = vec![step0];
def
}
/// Build a parent workflow with: SubWorkflowStep(0) -> ReadDataStep(1).
fn build_parent_definition() -> WorkflowDefinition {
let mut def = WorkflowDefinition::new("parent-workflow", 1);
let mut step0 = WorkflowStep::new(0, sub_workflow_step_type());
step0.step_config = Some(json!({
"workflow_id": "child-workflow",
"version": 1,
"inputs": {},
"output_keys": ["result"]
}));
step0.outcomes.push(StepOutcome {
next_step: 1,
label: None,
value: None,
});
let step1 = WorkflowStep::new(1, std::any::type_name::<ReadDataStep>());
def.steps = vec![step0, step1];
def
}
async fn wait_for_status(
host: &wfe::WorkflowHost,
id: &str,
status: WorkflowStatus,
timeout: Duration,
) -> wfe::models::WorkflowInstance {
let start = tokio::time::Instant::now();
loop {
if start.elapsed() > timeout {
let instance = host.get_workflow(id).await.unwrap();
panic!(
"Workflow {id} did not reach status {status:?} within {timeout:?}. \
Current status: {:?}, pointers: {:?}",
instance.status, instance.execution_pointers
);
}
let instance = host.get_workflow(id).await.unwrap();
if instance.status == status {
return instance;
}
tokio::time::sleep(Duration::from_millis(25)).await;
}
}
// ----- Tests -----
/// Parent workflow starts a child, waits for the child's completion event,
/// then proceeds to a second step.
#[tokio::test]
async fn parent_workflow_starts_child_and_waits() {
let (host, _persistence) = build_host();
let child_def = build_child_definition(std::any::type_name::<SetResultStep>());
let parent_def = build_parent_definition();
host.register_step::<SetResultStep>().await;
host.register_step::<ReadDataStep>().await;
host.register_workflow_definition(child_def).await;
host.register_workflow_definition(parent_def).await;
host.start().await.unwrap();
let parent_id = host
.start_workflow("parent-workflow", 1, json!({}))
.await
.unwrap();
let parent_instance = wait_for_status(
&host,
&parent_id,
WorkflowStatus::Complete,
Duration::from_secs(10),
)
.await;
assert_eq!(parent_instance.status, WorkflowStatus::Complete);
// The SubWorkflowStep's execution pointer should have received event data
// from the child's completion event.
let sub_wf_pointer = &parent_instance.execution_pointers[0];
assert!(
sub_wf_pointer.event_data.is_some(),
"SubWorkflowStep pointer should have event_data from child completion"
);
let event_data = sub_wf_pointer.event_data.as_ref().unwrap();
assert_eq!(
event_data.get("status").and_then(|v| v.as_str()),
Some("Complete")
);
// The child's output data should be in the event.
let child_data = event_data.get("data").unwrap();
assert_eq!(child_data.get("result"), Some(&json!("child_done")));
host.stop().await;
}
/// Parent workflow starts a child that passes all data through (no output_keys filter).
#[tokio::test]
async fn parent_workflow_passes_all_child_data() {
let (host, _) = build_host();
let child_def = build_child_definition(std::any::type_name::<SetResultStep>());
// Parent with no output_keys filter - passes all child data.
let mut parent_def = WorkflowDefinition::new("parent-all-data", 1);
let mut step0 = WorkflowStep::new(0, sub_workflow_step_type());
step0.step_config = Some(json!({
"workflow_id": "child-workflow",
"version": 1,
"inputs": {}
}));
step0.outcomes.push(StepOutcome {
next_step: 1,
label: None,
value: None,
});
let step1 = WorkflowStep::new(1, std::any::type_name::<NoOpStep>());
parent_def.steps = vec![step0, step1];
host.register_step::<SetResultStep>().await;
host.register_step::<NoOpStep>().await;
host.register_workflow_definition(child_def).await;
host.register_workflow_definition(parent_def).await;
host.start().await.unwrap();
let parent_id = host
.start_workflow("parent-all-data", 1, json!({}))
.await
.unwrap();
let instance = wait_for_status(
&host,
&parent_id,
WorkflowStatus::Complete,
Duration::from_secs(10),
)
.await;
assert_eq!(instance.status, WorkflowStatus::Complete);
host.stop().await;
}
/// Child workflow that uses typed inputs and produces typed output.
#[tokio::test]
async fn nested_workflow_with_typed_inputs_outputs() {
let (host, _) = build_host();
// Child workflow: GreetStep reads "name" from workflow data and produces {"greeting": "hello <name>"}.
let mut child_def = WorkflowDefinition::new("greet-child", 1);
let step0 = WorkflowStep::new(0, std::any::type_name::<GreetStep>());
child_def.steps = vec![step0];
// Parent workflow: SubWorkflowStep starts greet-child with {"name": "world"}.
let mut parent_def = WorkflowDefinition::new("greet-parent", 1);
let mut step0 = WorkflowStep::new(0, sub_workflow_step_type());
step0.step_config = Some(json!({
"workflow_id": "greet-child",
"version": 1,
"inputs": {"name": "world"},
"output_keys": ["greeting"]
}));
step0.outcomes.push(StepOutcome {
next_step: 1,
label: None,
value: None,
});
let step1 = WorkflowStep::new(1, std::any::type_name::<NoOpStep>());
parent_def.steps = vec![step0, step1];
host.register_step::<GreetStep>().await;
host.register_step::<NoOpStep>().await;
host.register_workflow_definition(child_def).await;
host.register_workflow_definition(parent_def).await;
host.start().await.unwrap();
let parent_id = host
.start_workflow("greet-parent", 1, json!({}))
.await
.unwrap();
let instance = wait_for_status(
&host,
&parent_id,
WorkflowStatus::Complete,
Duration::from_secs(10),
)
.await;
assert_eq!(instance.status, WorkflowStatus::Complete);
// Check the SubWorkflowStep pointer got the greeting from the child.
let sub_wf_pointer = &instance.execution_pointers[0];
let event_data = sub_wf_pointer.event_data.as_ref().unwrap();
let child_data = event_data.get("data").unwrap();
assert_eq!(
child_data.get("greeting"),
Some(&json!("hello world")),
"Child should produce greeting from input name"
);
host.stop().await;
}
/// When a child workflow fails, the parent should still complete because the
/// completion event carries the terminated status. The SubWorkflowStep will
/// process the event data even if the child ended in error, and the parent
/// will see the child's status in the event_data.
#[tokio::test]
async fn nested_workflow_child_failure_propagates() {
let (host, _) = build_host();
// Child workflow with a failing step.
let mut child_def = WorkflowDefinition::new("failing-child", 1);
let step0 = WorkflowStep::new(0, std::any::type_name::<FailingStep>());
child_def.steps = vec![step0];
// Parent workflow: SubWorkflowStep starts failing-child.
let mut parent_def = WorkflowDefinition::new("parent-of-failing", 1);
let mut step0 = WorkflowStep::new(0, sub_workflow_step_type());
step0.step_config = Some(json!({
"workflow_id": "failing-child",
"version": 1,
"inputs": {}
}));
// No outcome wired — if the child fails and the sub workflow step
// processes the completion event, the parent can proceed.
step0.outcomes.push(StepOutcome {
next_step: 1,
label: None,
value: None,
});
let step1 = WorkflowStep::new(1, std::any::type_name::<NoOpStep>());
parent_def.steps = vec![step0, step1];
host.register_step::<FailingStep>().await;
host.register_step::<NoOpStep>().await;
host.register_workflow_definition(child_def).await;
host.register_workflow_definition(parent_def).await;
host.start().await.unwrap();
let parent_id = host
.start_workflow("parent-of-failing", 1, json!({}))
.await
.unwrap();
// The child will fail. Depending on executor error handling, the child may
// reach Terminated status. We wait a bit then check states.
tokio::time::sleep(Duration::from_secs(2)).await;
let parent = host.get_workflow(&parent_id).await.unwrap();
// Parent should still be Runnable (waiting for the child's completion event)
// since the failing child may not emit a completion event.
// Check that the parent is stuck waiting (SubWorkflowStep issued wait_for_event).
let has_waiting = parent
.execution_pointers
.iter()
.any(|p| p.status == wfe::models::PointerStatus::WaitingForEvent);
// Either the parent is waiting or already completed (if the child errored
// and published a terminated event).
assert!(
has_waiting || parent.status == WorkflowStatus::Complete,
"Parent should be waiting for child or complete. Status: {:?}",
parent.status
);
host.stop().await;
}
/// Test that SubWorkflowStep is registered as a built-in primitive.
#[tokio::test]
async fn sub_workflow_step_registered_as_primitive() {
let (host, _) = build_host();
// Start registers primitives.
host.start().await.unwrap();
// Verify by checking that the step registry has the SubWorkflowStep type.
// We do this indirectly — if we can run a workflow using SubWorkflowStep
// without explicit registration, it's registered.
let child_def = build_child_definition(std::any::type_name::<NoOpStep>());
let parent_def = build_parent_definition();
// Only register the non-primitive steps, NOT SubWorkflowStep.
host.register_step::<NoOpStep>().await;
host.register_step::<SetResultStep>().await;
host.register_step::<ReadDataStep>().await;
host.register_workflow_definition(child_def).await;
host.register_workflow_definition(parent_def).await;
let parent_id = host
.start_workflow("parent-workflow", 1, json!({}))
.await
.unwrap();
// If SubWorkflowStep is registered as a primitive, the parent should start
// the child and eventually complete.
let instance = wait_for_status(
&host,
&parent_id,
WorkflowStatus::Complete,
Duration::from_secs(10),
)
.await;
assert_eq!(instance.status, WorkflowStatus::Complete);
host.stop().await;
}
/// Test that starting a child with a non-existent definition propagates an error.
#[tokio::test]
async fn nested_workflow_nonexistent_child_definition() {
let (host, _) = build_host();
// Parent references a child that is NOT registered.
let mut parent_def = WorkflowDefinition::new("parent-missing-child", 1);
let mut step0 = WorkflowStep::new(0, sub_workflow_step_type());
step0.step_config = Some(json!({
"workflow_id": "nonexistent-child",
"version": 1,
"inputs": {}
}));
parent_def.steps = vec![step0];
host.register_workflow_definition(parent_def).await;
host.start().await.unwrap();
let parent_id = host
.start_workflow("parent-missing-child", 1, json!({}))
.await
.unwrap();
// The parent should fail because the child definition doesn't exist.
// Give it time to process.
tokio::time::sleep(Duration::from_secs(2)).await;
let parent = host.get_workflow(&parent_id).await.unwrap();
// The error handler should have set the pointer to Failed and the workflow
// to Suspended (default error behavior is Retry then Suspend).
let pointer = &parent.execution_pointers[0];
let is_failed = pointer.status == wfe::models::PointerStatus::Failed
|| pointer.status == wfe::models::PointerStatus::Sleeping;
assert!(
is_failed
|| parent.status == WorkflowStatus::Suspended
|| parent.status == WorkflowStatus::Terminated,
"SubWorkflowStep should error when child definition is missing. \
Pointer status: {:?}, Workflow status: {:?}",
pointer.status,
parent.status
);
host.stop().await;
}
/// Test that starting a workflow via the host works end-to-end,
/// exercising the HostContextImpl path indirectly.
#[tokio::test]
async fn host_context_impl_starts_workflow() {
let (host, _) = build_host();
// Register a child definition.
let child_def = build_child_definition(std::any::type_name::<NoOpStep>());
host.register_step::<NoOpStep>().await;
host.register_workflow_definition(child_def).await;
host.start().await.unwrap();
// Start a child workflow via the host's start_workflow.
let child_id = host
.start_workflow("child-workflow", 1, json!({"test": true}))
.await
.unwrap();
// Verify the child was created and completes.
let child = host.get_workflow(&child_id).await.unwrap();
assert_eq!(child.workflow_definition_id, "child-workflow");
assert_eq!(child.version, 1);
assert_eq!(child.data.get("test"), Some(&json!(true)));
// Wait for the child to complete.
let instance = wait_for_status(
&host,
&child_id,
WorkflowStatus::Complete,
Duration::from_secs(5),
)
.await;
assert_eq!(instance.status, WorkflowStatus::Complete);
host.stop().await;
}

778
workflows.yaml Normal file
View File

@@ -0,0 +1,778 @@
# workflows.yaml — WFE self-hosting CI pipeline
#
# Demonstrates every WFE feature. Idempotent — safe to run repeatedly.
#
# Usage:
# cargo run --example run_pipeline -p wfe -- workflows.yaml
#
# With config:
# WFE_CONFIG='{"workspace_dir":"/path/to/wfe","registry":"sunbeam","git_remote":"origin","coverage_threshold":85}' \
# cargo run --example run_pipeline -p wfe -- workflows.yaml
#
# TODO: Support multi-file merging — individual task files (e.g., lint.yaml,
# test.yaml, publish.yaml) that compose into a single pipeline definition.
# ─── Shared Templates ───────────────────────────────────────────────
# The _templates key is ignored by the workflow parser (extra keys are
# skipped). Anchors are resolved by serde_yaml before parsing.
_templates:
shell_defaults: &shell_defaults
type: shell
config:
shell: bash
timeout: 5m
long_running: &long_running
type: shell
config:
shell: bash
timeout: 30m
# ─── Workflow: preflight ───────────────────────────────────────────────
workflows:
- id: preflight
version: 1
inputs:
workspace_dir: string
outputs:
cargo_ok: bool
nextest_ok: bool
llvm_cov_ok: bool
docker_ok: bool
lima_ok: bool
buildctl_ok: bool
git_ok: bool
steps:
- name: check-tools
type: shell
outputs:
- name: cargo_ok
- name: nextest_ok
- name: llvm_cov_ok
- name: docker_ok
- name: lima_ok
- name: buildctl_ok
- name: git_ok
config:
shell: bash
timeout: 1m
run: |
CARGO_OK=false; NEXTEST_OK=false; LLVM_COV_OK=false
DOCKER_OK=false; LIMA_OK=false; BUILDCTL_OK=false; GIT_OK=false
command -v cargo >/dev/null 2>&1 && CARGO_OK=true
command -v cargo-nextest >/dev/null 2>&1 && NEXTEST_OK=true
command -v cargo-llvm-cov >/dev/null 2>&1 && LLVM_COV_OK=true
command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1 && DOCKER_OK=true
command -v limactl >/dev/null 2>&1 && LIMA_OK=true
command -v buildctl >/dev/null 2>&1 && BUILDCTL_OK=true
command -v git >/dev/null 2>&1 && GIT_OK=true
echo "Tool availability:"
echo " cargo: $CARGO_OK"
echo " nextest: $NEXTEST_OK"
echo " llvm-cov: $LLVM_COV_OK"
echo " docker: $DOCKER_OK"
echo " lima: $LIMA_OK"
echo " buildctl: $BUILDCTL_OK"
echo " git: $GIT_OK"
echo "##wfe[output cargo_ok=$CARGO_OK]"
echo "##wfe[output nextest_ok=$NEXTEST_OK]"
echo "##wfe[output llvm_cov_ok=$LLVM_COV_OK]"
echo "##wfe[output docker_ok=$DOCKER_OK]"
echo "##wfe[output lima_ok=$LIMA_OK]"
echo "##wfe[output buildctl_ok=$BUILDCTL_OK]"
echo "##wfe[output git_ok=$GIT_OK]"
# Fail if essential tools are missing
if [ "$CARGO_OK" = "false" ] || [ "$NEXTEST_OK" = "false" ] || [ "$GIT_OK" = "false" ]; then
echo "ERROR: Essential tools missing (cargo, nextest, or git)"
exit 1
fi
# ─── Workflow: lint ──────────────────────────────────────────────────
- id: lint
version: 1
inputs:
workspace_dir: string
outputs:
fmt_ok: bool
clippy_ok: bool
steps:
- name: fmt-check
<<: *shell_defaults
outputs:
- name: fmt_ok
config:
run: |
cd "$WORKSPACE_DIR"
cargo fmt --all -- --check
echo "##wfe[output fmt_ok=true]"
- name: clippy
<<: *shell_defaults
outputs:
- name: clippy_ok
config:
run: |
cd "$WORKSPACE_DIR"
cargo clippy --workspace -- -D warnings
echo "##wfe[output clippy_ok=true]"
# ─── Workflow: test-unit ─────────────────────────────────────────
- id: test-unit
version: 1
inputs:
workspace_dir: string
outputs:
tests_passed: bool
deno_tests_passed: bool
steps:
- name: core-tests
<<: *long_running
outputs:
- name: tests_passed
config:
run: |
cd "$WORKSPACE_DIR"
cargo nextest run -P ci
echo "##wfe[output tests_passed=true]"
- name: deno-tests
<<: *long_running
outputs:
- name: deno_tests_passed
config:
run: |
cd "$WORKSPACE_DIR"
cargo nextest run -p wfe-yaml --features deno -P ci
echo "##wfe[output deno_tests_passed=true]"
- name: feature-tests
<<: *shell_defaults
config:
run: |
cd "$WORKSPACE_DIR"
cargo nextest run -p wfe-yaml --features buildkit,containerd -P ci
# ─── Workflow: test-integration ──────────────────────────────────
- id: test-integration
version: 1
inputs:
workspace_dir: string
outputs:
postgres_ok: bool
valkey_ok: bool
opensearch_ok: bool
steps:
- name: docker-up
<<: *long_running
outputs:
- name: docker_started
config:
run: |
# Docker runs inside a lima VM. Start it if needed.
if ! command -v limactl >/dev/null 2>&1; then
echo "limactl not available — skipping integration tests"
echo "##wfe[output docker_started=false]"
exit 0
fi
# Start the docker lima VM if not running
if ! limactl list 2>/dev/null | grep -q "docker.*Running"; then
echo "Starting docker lima VM..."
limactl start docker 2>&1 || {
echo "Failed to start docker VM — skipping integration tests"
echo "##wfe[output docker_started=false]"
exit 0
}
fi
# Wait for Docker daemon to be ready
for i in $(seq 1 30); do
if docker info >/dev/null 2>&1; then
break
fi
echo "Waiting for Docker daemon... ($i/30)"
sleep 2
done
if ! docker info >/dev/null 2>&1; then
echo "Docker daemon not ready after 60s — skipping"
echo "##wfe[output docker_started=false]"
exit 0
fi
cd "$WORKSPACE_DIR"
docker compose up -d --wait
echo "##wfe[output docker_started=true]"
on_failure:
name: docker-up-failed
type: shell
config:
run: echo "Failed to start Docker services"
- name: postgres-tests
<<: *shell_defaults
when:
field: .outputs.docker_started
equals: true
outputs:
- name: postgres_ok
config:
run: |
cd "$WORKSPACE_DIR"
cargo nextest run -p wfe-postgres -P ci
echo "##wfe[output postgres_ok=true]"
- name: valkey-tests
<<: *shell_defaults
when:
field: .outputs.docker_started
equals: true
outputs:
- name: valkey_ok
config:
run: |
cd "$WORKSPACE_DIR"
cargo nextest run -p wfe-valkey -P ci
echo "##wfe[output valkey_ok=true]"
- name: opensearch-tests
<<: *shell_defaults
when:
field: .outputs.docker_started
equals: true
outputs:
- name: opensearch_ok
config:
run: |
cd "$WORKSPACE_DIR"
cargo nextest run -p wfe-opensearch -P ci
echo "##wfe[output opensearch_ok=true]"
ensure:
- name: docker-down
<<: *shell_defaults
config:
run: |
if docker info >/dev/null 2>&1; then
cd "$WORKSPACE_DIR"
docker compose down 2>/dev/null || true
fi
# ─── Workflow: test-containers ───────────────────────────────────
- id: test-containers
version: 1
inputs:
workspace_dir: string
outputs:
buildkit_ok: bool
containerd_ok: bool
steps:
- name: lima-up
<<: *long_running
outputs:
- name: lima_started
config:
run: |
if ! command -v limactl >/dev/null 2>&1; then
echo "limactl not available — skipping container tests"
echo "##wfe[output lima_started=false]"
exit 0
fi
# Start the wfe-test VM if not running
if ! limactl list 2>/dev/null | grep -q "wfe-test.*Running"; then
echo "Starting wfe-test lima VM..."
limactl start --name=wfe-test "$WORKSPACE_DIR/test/lima/wfe-test.yaml" 2>&1 || {
echo "Failed to start wfe-test VM — skipping container tests"
echo "##wfe[output lima_started=false]"
exit 0
}
fi
# Wait for sockets to be available
for i in $(seq 1 30); do
if [ -S "$HOME/.lima/wfe-test/sock/buildkitd.sock" ]; then
break
fi
echo "Waiting for buildkitd socket... ($i/30)"
sleep 2
done
echo "##wfe[output lima_started=true]"
- name: buildkit-tests
<<: *shell_defaults
when:
field: .outputs.lima_started
equals: true
outputs:
- name: buildkit_ok
config:
run: |
cd "$WORKSPACE_DIR"
export WFE_BUILDKIT_ADDR="unix://$HOME/.lima/wfe-test/sock/buildkitd.sock"
cargo nextest run -p wfe-buildkit -P ci
echo "##wfe[output buildkit_ok=true]"
- name: containerd-tests
<<: *shell_defaults
when:
field: .outputs.lima_started
equals: true
outputs:
- name: containerd_ok
config:
run: |
cd "$WORKSPACE_DIR"
export WFE_CONTAINERD_ADDR="unix://$HOME/.lima/wfe-test/sock/containerd.sock"
cargo nextest run -p wfe-containerd -P ci
echo "##wfe[output containerd_ok=true]"
ensure:
- name: lima-down
<<: *shell_defaults
config:
run: |
limactl stop wfe-test 2>/dev/null || true
# ─── Workflow: test (orchestrator) ───────────────────────────────
- id: test
version: 1
inputs:
workspace_dir: string
outputs:
all_passed: bool
steps:
- name: run-unit
type: workflow
outputs:
- name: tests_passed
- name: deno_tests_passed
config:
workflow: test-unit
version: 1
- name: run-integration
type: workflow
outputs:
- name: postgres_ok
- name: valkey_ok
- name: opensearch_ok
config:
workflow: test-integration
version: 1
- name: run-containers
type: workflow
outputs:
- name: buildkit_ok
- name: containerd_ok
config:
workflow: test-containers
version: 1
- name: mark-passed
<<: *shell_defaults
outputs:
- name: all_passed
config:
run: |
echo "All test workflows completed"
echo "##wfe[output all_passed=true]"
# ─── Workflow: cover ─────────────────────────────────────────────
- id: cover
version: 1
inputs:
workspace_dir: string
threshold: number?
outputs:
line_coverage: number
meets_threshold: bool
steps:
- name: run-coverage
<<: *shell_defaults
outputs:
- name: coverage_ok
config:
run: |
cd "$WORKSPACE_DIR"
if ! command -v cargo-llvm-cov >/dev/null 2>&1; then
echo "cargo-llvm-cov not installed — skipping coverage"
echo "##wfe[output coverage_ok=false]"
exit 0
fi
cargo llvm-cov nextest -P cover --json 2>&1 | grep '^{' > /tmp/wfe-coverage.json || true
if [ -s /tmp/wfe-coverage.json ]; then
echo "##wfe[output coverage_ok=true]"
else
echo "Coverage JSON not produced — llvm-cov may have failed"
echo "##wfe[output coverage_ok=false]"
fi
echo "##wfe[output coverage_json=/tmp/wfe-coverage.json]"
- name: assert-threshold
type: deno
when:
field: .outputs.coverage_ok
equals: true
outputs:
- name: line_coverage
- name: meets_threshold
config:
script: |
const data = inputs();
const threshold = data.coverage_threshold || 85;
// Read the coverage JSON produced by run-coverage step
const text = await readFile("/tmp/wfe-coverage.json");
const report = JSON.parse(text);
const totals = report.data[0].totals;
const lineCov = (totals.lines.covered / totals.lines.count * 100).toFixed(1);
log(`Line coverage: ${lineCov}% (threshold: ${threshold}%)`);
output("line_coverage", parseFloat(lineCov));
output("meets_threshold", parseFloat(lineCov) >= threshold);
if (parseFloat(lineCov) < threshold) {
throw new Error(`Coverage ${lineCov}% is below threshold ${threshold}%`);
}
permissions:
read: ["/tmp"]
# ─── Workflow: package ───────────────────────────────────────────
- id: package
version: 1
inputs:
workspace_dir: string
outputs:
packages_ok: bool
steps:
- name: package-all
<<: *shell_defaults
outputs:
- name: packages_ok
config:
run: |
echo "Packaging all crates (stub — remove exit 0 for real packaging)"
echo "##wfe[output packages_ok=true]"
exit 0
cd "$WORKSPACE_DIR"
for crate in wfe-core wfe-sqlite wfe-postgres wfe-opensearch wfe-valkey \
wfe-buildkit-protos wfe-containerd-protos wfe-buildkit wfe-containerd \
wfe wfe-yaml; do
echo "Packaging $crate..."
cargo package -p "$crate" --no-verify --allow-dirty 2>&1 || exit 1
done
echo "##wfe[output packages_ok=true]"
# ─── Workflow: tag ───────────────────────────────────────────────
- id: tag
version: 1
inputs:
workspace_dir: string
outputs:
version: string
tag_created: bool
tag_already_existed: bool
steps:
- name: read-version
type: deno
outputs:
- name: version
config:
script: |
// Stub — remove the early return for real tagging
log("Reading version (stub)");
output("version", "1.0.0");
permissions:
read: ["((workspace_dir))"]
- name: check-tag-exists
<<: *shell_defaults
outputs:
- name: tag_already_existed
- name: tag_created
config:
run: |
echo "Checking tag (stub — remove exit 0 for real tagging)"
echo "##wfe[output tag_already_existed=true]"
echo "##wfe[output tag_created=false]"
exit 0
VERSION=$(echo "$VERSION" | tr -d '[:space:]')
TAG="v${VERSION}"
if git tag -l "$TAG" | grep -q "$TAG"; then
echo "Tag $TAG already exists — skipping"
echo "##wfe[output tag_already_existed=true]"
echo "##wfe[output tag_created=false]"
else
echo "Tag $TAG does not exist — will create"
echo "##wfe[output tag_already_existed=false]"
fi
- name: create-tag
<<: *shell_defaults
outputs:
- name: tag_created
config:
run: |
echo "Creating tag (stub — remove exit 0 for real tagging)"
echo "##wfe[output tag_created=false]"
exit 0
if [ "$TAG_ALREADY_EXISTED" = "true" ]; then
echo "Skipping tag creation (already exists)"
echo "##wfe[output tag_created=false]"
exit 0
fi
VERSION=$(echo "$VERSION" | tr -d '[:space:]')
TAG="v${VERSION}"
git tag -a "$TAG" -m "$TAG"
echo "##wfe[output tag_created=true]"
# ─── Workflow: publish ───────────────────────────────────────────
- id: publish
version: 1
inputs:
workspace_dir: string
registry: string?
outputs:
all_published: bool
steps:
- name: publish-protos
<<: *shell_defaults
config:
run: |
echo "Publishing protos (stub — remove exit 0 for real publish)"
exit 0
cd "$WORKSPACE_DIR"
REGISTRY="${REGISTRY:-sunbeam}"
PUBLISHED=""
for crate in wfe-buildkit-protos wfe-containerd-protos; do
echo "Publishing $crate..."
if cargo publish -p "$crate" --registry "$REGISTRY" 2>&1; then
PUBLISHED="$PUBLISHED $crate"
else
echo "Already published or failed: $crate (continuing)"
fi
done
echo "##wfe[output published_protos=$PUBLISHED]"
error_behavior:
type: retry
interval: 10s
max_retries: 2
- name: publish-core
<<: *shell_defaults
config:
run: |
echo "Publishing core (stub — remove exit 0 for real publish)"
exit 0
cd "$WORKSPACE_DIR"
REGISTRY="${REGISTRY:-sunbeam}"
cargo publish -p wfe-core --registry "$REGISTRY" 2>&1 || echo "Already published"
echo "##wfe[output core_published=true]"
error_behavior:
type: retry
interval: 10s
max_retries: 2
- name: publish-providers
<<: *shell_defaults
config:
run: |
echo "Publishing providers (stub — remove exit 0 for real publish)"
exit 0
cd "$WORKSPACE_DIR"
REGISTRY="${REGISTRY:-sunbeam}"
for crate in wfe-sqlite wfe-postgres wfe-opensearch wfe-valkey; do
echo "Publishing $crate..."
cargo publish -p "$crate" --registry "$REGISTRY" 2>&1 || echo "Already published: $crate"
done
echo "##wfe[output providers_published=true]"
error_behavior:
type: retry
interval: 10s
max_retries: 2
- name: publish-executors
<<: *shell_defaults
config:
run: |
echo "Publishing executors (stub — remove exit 0 for real publish)"
exit 0
cd "$WORKSPACE_DIR"
REGISTRY="${REGISTRY:-sunbeam}"
for crate in wfe-buildkit wfe-containerd; do
echo "Publishing $crate..."
cargo publish -p "$crate" --registry "$REGISTRY" 2>&1 || echo "Already published: $crate"
done
echo "##wfe[output executors_published=true]"
- name: publish-framework
<<: *shell_defaults
outputs:
- name: all_published
config:
run: |
echo "Publishing framework (stub — remove exit 0 for real publish)"
echo "##wfe[output all_published=true]"
exit 0
cd "$WORKSPACE_DIR"
REGISTRY="${REGISTRY:-sunbeam}"
for crate in wfe wfe-yaml; do
echo "Publishing $crate..."
cargo publish -p "$crate" --registry "$REGISTRY" 2>&1 || echo "Already published: $crate"
done
echo "##wfe[output all_published=true]"
on_failure:
- name: log-partial-publish
<<: *shell_defaults
outputs:
- name: all_published
config:
run: |
echo "WARNING: Publish partially failed. Check logs above."
echo "##wfe[output all_published=false]"
# ─── Workflow: release ───────────────────────────────────────────
- id: release
version: 1
inputs:
workspace_dir: string
version: string
git_remote: string?
outputs:
pushed: bool
notes: string
steps:
- name: push-tags
<<: *shell_defaults
outputs:
- name: pushed
config:
run: |
echo "Pushing tags (stub — remove exit 0 for real release)"
echo "##wfe[output pushed=true]"
exit 0
REMOTE="${GIT_REMOTE:-origin}"
git push "$REMOTE" --tags
echo "##wfe[output pushed=true]"
- name: generate-notes
type: deno
outputs:
- name: notes
config:
script: |
// Stub — remove the early return for real release
log("Generating release notes (stub)");
output("notes", "stub release notes");
permissions:
run: true
# ─── Workflow: ci (top-level orchestrator) ───────────────────────
- id: ci
version: 1
inputs:
workspace_dir: string
registry: string?
git_remote: string?
coverage_threshold: number?
outputs:
version: string
all_tests_passed: bool
coverage: number
published: bool
released: bool
steps:
- name: run-preflight
type: workflow
outputs:
- name: cargo_ok
- name: nextest_ok
- name: llvm_cov_ok
- name: docker_ok
- name: lima_ok
- name: buildctl_ok
- name: git_ok
config:
workflow: preflight
version: 1
- name: run-lint
type: workflow
outputs:
- name: fmt_ok
- name: clippy_ok
config:
workflow: lint
version: 1
- name: run-tests
type: workflow
outputs:
- name: all_tests_passed
config:
workflow: test
version: 1
- name: run-coverage
type: workflow
outputs:
- name: coverage
config:
workflow: cover
version: 1
- name: run-package
type: workflow
outputs:
- name: packages_ok
config:
workflow: package
version: 1
- name: run-tag
type: workflow
outputs:
- name: version
- name: tag_created
config:
workflow: tag
version: 1
- name: run-publish
type: workflow
outputs:
- name: published
config:
workflow: publish
version: 1
- name: run-release
type: workflow
outputs:
- name: released
config:
workflow: release
version: 1