Documents pipeline architecture, how to run it, WFE features
demonstrated, preflight tool checks, and graceful infrastructure
skipping. Adds nextest cover profile for llvm-cov integration.
Preprocesses <<: *anchor merge keys before serde_yaml 0.9 deserialization.
serde_yaml implements YAML 1.2 which dropped merge keys; the yaml-merge-keys
crate resolves them as a preprocessing step, giving full anchor + merge
support for DRY pipeline definitions.
The YAML compiler was using SubWorkflowPlaceholderStep that returned
next() immediately. Replaced with real SubWorkflowStep from wfe-core
that starts child workflows and waits for completion events.
Added regression test verifying the compiled factory produces a step
that calls host_context.start_workflow() and returns wait_for_event.
Nextest cover profile for cargo llvm-cov integration.
Update existing test imports from load_workflow_from_str to
load_single_workflow_from_str for backward compatibility.
Compiler dispatches type: workflow to SubWorkflowStep. Validation
detects circular workflow references via DFS with coloring. Cross-
workflow reference checking for multi-workflow files. Duplicate
workflow ID detection. 28 edge case tests for validation paths.
YamlWorkflowFile supports both single (workflow:) and multi (workflows:)
formats. WorkflowSpec gains typed inputs/outputs declarations.
Type string parser for inline types ("string?", "list<number>", etc.).
load_workflow_from_str returns Vec<CompiledWorkflow>.
Backward-compatible load_single_workflow_from_str convenience function.
SchemaType enum with inline syntax parsing: "string", "string?",
"list<number>", "map<string>", nested generics. WorkflowSchema
validates inputs/outputs against type declarations at both compile
time and runtime. 39 tests for parse and validate paths.
Replaced third-party buildkit-client git dependency with
wfe-buildkit-protos generated from official moby/buildkit protos.
Direct ControlClient gRPC calls: SolveRequest with frontend attrs,
exporters, cache options. Daemon-local context paths for builds
(session protocol for remote transfer is TODO).
Both proto crates now use tonic 0.14 / prost 0.14 — no transitive
dependency conflicts. 95 combined tests, 85.6% region coverage.
New crate generating Rust gRPC stubs from the official BuildKit
proto files (git submodule from moby/buildkit). Control service,
LLB definitions, session protocols, and source policy.
tonic 0.14 / prost 0.14.
Replaced nerdctl CLI shell-out with direct gRPC communication via
wfe-containerd-protos (tonic 0.14). Connects to containerd daemon
over Unix socket.
Implementation:
- connect() with tonic Unix socket connector
- ensure_image() via ImagesClient (full pull is TODO)
- build_oci_spec() constructing OCI runtime spec with process args,
env, user, cwd, mounts, and linux namespaces
- Container lifecycle: create → snapshot → task create → start →
wait → read FIFOs → cleanup
- containerd-namespace header injection on every request
FIFO-based stdout/stderr capture using named pipes.
40 tests, 88% line coverage (cargo-llvm-cov).
New crate generating Rust gRPC stubs from the official containerd
proto files (vendored as git submodule). Full client-facing API surface
using tonic 0.14 / prost 0.14. No transitive dependency conflicts.
Services: containers, content, diff, events, images, introspection,
leases, mounts, namespaces, sandbox, snapshots, streaming, tasks,
transfer, version.
Replaced buildctl CLI shell-out with direct gRPC communication via
buildkit-client crate. Connects to buildkitd daemon over Unix socket
or TCP with optional TLS.
Implementation:
- connect() with custom tonic UnixStream connector
- execute_build() implementing the solve protocol directly against
ControlClient (session setup, file sync, frontend attributes)
- Extracts digest from containerimage.digest in solve response
Added custom lima template (test/lima/wfe-test.yaml) that provides
both buildkitd and containerd with host-forwarded Unix sockets for
reproducible integration testing.
E2E tests against real buildkitd daemon via WFE_BUILDKIT_ADDR env var.
54 tests total. 89% line coverage (cargo-llvm-cov with E2E).
Standalone workspace crates for BuildKit image building and containerd
container execution. Config types, YAML schema integration, compiler
dispatch, validation rules, and mock-based unit tests.
Current implementation shells out to buildctl/nerdctl — will be
replaced with proper gRPC clients (buildkit-client, containerd protos)
in a follow-up. Config types, YAML integration, and test infrastructure
are stable and reusable.
wfe-buildkit: 60 tests, 97.9% library coverage
wfe-containerd: 61 tests, 97.8% library coverage
447 total workspace tests.
Phase 4 — Permission-gated HTTP fetch op:
- op_fetch with net permission check on every request
- globalThis.fetch() wrapper with .json()/.text() methods
- Supports GET/POST/PUT/DELETE with headers and body
Phase 5 — Module loader:
- WfeModuleLoader resolving npm: → esm.sh, https://, file://, relative paths
- All resolution paths permission-checked
- Bare path resolution (/) for esm.sh sub-module redirects
- Dynamic import rejection unless permissions.dynamic_import: true
- esm.sh auto-added to net allowlist when modules declared
Mandatory npm integration test (is-number via esm.sh).
25 new tests. 133 total deno tests, 326 total workspace tests.
Core plumbing for YAML workflow support:
- Add output_data field to ExecutionResult for step output capture
- Executor merges output_data into workflow.data after step completion
- Add register_step_factory(key, factory) to WorkflowHost for
dynamic step registration by external crates like wfe-yaml
- Add info!-level tracing to workflow executor: logs each execution
round, each step run (with type and name), step completion, and
workflow completion
- WorkflowHost.start() now auto-registers all built-in primitive step
types so users don't need to register them manually
- Add #[derive(Default)] to all primitive steps and PollEndpointConfig
- Add tracing-subscriber to wfe crate for the pizza example
- Pizza example now shows full step-by-step execution logs
OpenSearchIndex implementing SearchIndex trait. Index mapping for
workflow documents. Bool queries with multi_match for terms, term
filters for status/reference, range filters for dates. Pagination support.
ValkeyLockProvider: SET NX EX for acquisition, Lua script for safe release.
ValkeyQueueProvider: LPUSH/RPOP for FIFO queues.
ValkeyLifecyclePublisher: PUBLISH to per-instance and global channels.
Connections obtained once during construction (no per-operation TCP handshakes).
PostgresPersistenceProvider using sqlx with wfc schema. JSONB for
complex fields, TIMESTAMPTZ for dates. Transactions for atomicity.
Proper error propagation on deserialization (no unwrap_or_default).
Status conversion returns Result for unknown values.
SqlitePersistenceProvider using sqlx with in-memory and file-based
SQLite. Schema with 6 tables, 8 indexes, created via ensure_store_exists.
Transactions for multi-table operations. Passes persistence_suite tests.
