Documents pipeline architecture, how to run it, WFE features
demonstrated, preflight tool checks, and graceful infrastructure
skipping. Adds nextest cover profile for llvm-cov integration.
Preprocesses <<: *anchor merge keys before serde_yaml 0.9 deserialization.
serde_yaml implements YAML 1.2 which dropped merge keys; the yaml-merge-keys
crate resolves them as a preprocessing step, giving full anchor + merge
support for DRY pipeline definitions.
The YAML compiler was using SubWorkflowPlaceholderStep that returned
next() immediately. Replaced with real SubWorkflowStep from wfe-core
that starts child workflows and waits for completion events.
Added regression test verifying the compiled factory produces a step
that calls host_context.start_workflow() and returns wait_for_event.
Nextest cover profile for cargo llvm-cov integration.
Update existing test imports from load_workflow_from_str to
load_single_workflow_from_str for backward compatibility.
Compiler dispatches type: workflow to SubWorkflowStep. Validation
detects circular workflow references via DFS with coloring. Cross-
workflow reference checking for multi-workflow files. Duplicate
workflow ID detection. 28 edge case tests for validation paths.
YamlWorkflowFile supports both single (workflow:) and multi (workflows:)
formats. WorkflowSpec gains typed inputs/outputs declarations.
Type string parser for inline types ("string?", "list<number>", etc.).
load_workflow_from_str returns Vec<CompiledWorkflow>.
Backward-compatible load_single_workflow_from_str convenience function.
SchemaType enum with inline syntax parsing: "string", "string?",
"list<number>", "map<string>", nested generics. WorkflowSchema
validates inputs/outputs against type declarations at both compile
time and runtime. 39 tests for parse and validate paths.
Replaced third-party buildkit-client git dependency with
wfe-buildkit-protos generated from official moby/buildkit protos.
Direct ControlClient gRPC calls: SolveRequest with frontend attrs,
exporters, cache options. Daemon-local context paths for builds
(session protocol for remote transfer is TODO).
Both proto crates now use tonic 0.14 / prost 0.14 — no transitive
dependency conflicts. 95 combined tests, 85.6% region coverage.
New crate generating Rust gRPC stubs from the official BuildKit
proto files (git submodule from moby/buildkit). Control service,
LLB definitions, session protocols, and source policy.
tonic 0.14 / prost 0.14.
Replaced nerdctl CLI shell-out with direct gRPC communication via
wfe-containerd-protos (tonic 0.14). Connects to containerd daemon
over Unix socket.
Implementation:
- connect() with tonic Unix socket connector
- ensure_image() via ImagesClient (full pull is TODO)
- build_oci_spec() constructing OCI runtime spec with process args,
env, user, cwd, mounts, and linux namespaces
- Container lifecycle: create → snapshot → task create → start →
wait → read FIFOs → cleanup
- containerd-namespace header injection on every request
FIFO-based stdout/stderr capture using named pipes.
40 tests, 88% line coverage (cargo-llvm-cov).
New crate generating Rust gRPC stubs from the official containerd
proto files (vendored as git submodule). Full client-facing API surface
using tonic 0.14 / prost 0.14. No transitive dependency conflicts.
Services: containers, content, diff, events, images, introspection,
leases, mounts, namespaces, sandbox, snapshots, streaming, tasks,
transfer, version.
Replaced buildctl CLI shell-out with direct gRPC communication via
buildkit-client crate. Connects to buildkitd daemon over Unix socket
or TCP with optional TLS.
Implementation:
- connect() with custom tonic UnixStream connector
- execute_build() implementing the solve protocol directly against
ControlClient (session setup, file sync, frontend attributes)
- Extracts digest from containerimage.digest in solve response
Added custom lima template (test/lima/wfe-test.yaml) that provides
both buildkitd and containerd with host-forwarded Unix sockets for
reproducible integration testing.
E2E tests against real buildkitd daemon via WFE_BUILDKIT_ADDR env var.
54 tests total. 89% line coverage (cargo-llvm-cov with E2E).
Standalone workspace crates for BuildKit image building and containerd
container execution. Config types, YAML schema integration, compiler
dispatch, validation rules, and mock-based unit tests.
Current implementation shells out to buildctl/nerdctl — will be
replaced with proper gRPC clients (buildkit-client, containerd protos)
in a follow-up. Config types, YAML integration, and test infrastructure
are stable and reusable.
wfe-buildkit: 60 tests, 97.9% library coverage
wfe-containerd: 61 tests, 97.8% library coverage
447 total workspace tests.
Phase 4 — Permission-gated HTTP fetch op:
- op_fetch with net permission check on every request
- globalThis.fetch() wrapper with .json()/.text() methods
- Supports GET/POST/PUT/DELETE with headers and body
Phase 5 — Module loader:
- WfeModuleLoader resolving npm: → esm.sh, https://, file://, relative paths
- All resolution paths permission-checked
- Bare path resolution (/) for esm.sh sub-module redirects
- Dynamic import rejection unless permissions.dynamic_import: true
- esm.sh auto-added to net allowlist when modules declared
Mandatory npm integration test (is-number via esm.sh).
25 new tests. 133 total deno tests, 326 total workspace tests.
Core plumbing for YAML workflow support:
- Add output_data field to ExecutionResult for step output capture
- Executor merges output_data into workflow.data after step completion
- Add register_step_factory(key, factory) to WorkflowHost for
dynamic step registration by external crates like wfe-yaml
- Add info!-level tracing to workflow executor: logs each execution
round, each step run (with type and name), step completion, and
workflow completion
- WorkflowHost.start() now auto-registers all built-in primitive step
types so users don't need to register them manually
- Add #[derive(Default)] to all primitive steps and PollEndpointConfig
- Add tracing-subscriber to wfe crate for the pizza example
- Pizza example now shows full step-by-step execution logs
OpenSearchIndex implementing SearchIndex trait. Index mapping for
workflow documents. Bool queries with multi_match for terms, term
filters for status/reference, range filters for dates. Pagination support.
ValkeyLockProvider: SET NX EX for acquisition, Lua script for safe release.
ValkeyQueueProvider: LPUSH/RPOP for FIFO queues.
ValkeyLifecyclePublisher: PUBLISH to per-instance and global channels.
Connections obtained once during construction (no per-operation TCP handshakes).
PostgresPersistenceProvider using sqlx with wfc schema. JSONB for
complex fields, TIMESTAMPTZ for dates. Transactions for atomicity.
Proper error propagation on deserialization (no unwrap_or_default).
Status conversion returns Result for unknown values.
SqlitePersistenceProvider using sqlx with in-memory and file-based
SQLite. Schema with 6 tables, 8 indexes, created via ensure_store_exists.
Transactions for multi-table operations. Passes persistence_suite tests.
WorkflowHost orchestrates the engine: background workflow and event
consumers, start/stop lifecycle, workflow CRUD, event publishing.
WorkflowHostBuilder (owned-self, returns Result).
InMemoryWorkflowRegistry with version support. SyncWorkflowRunner
for testing. WorkflowPurger stub.
Event consumer acquires distributed locks before modifying workflows
to prevent lost updates from concurrent event processing.
InMemoryPersistenceProvider, InMemoryLockProvider, InMemoryQueueProvider,
InMemoryLifecyclePublisher behind test-support feature flag.
Shared test suite macros: persistence_suite!, lock_suite!, queue_suite!
that run the same tests against any provider implementation.
Workspace with 6 crates: wfe-core, wfe-sqlite, wfe-postgres,
wfe-opensearch, wfe-valkey, and wfe (umbrella). Nextest profiles
for default, integration, and CI. Docker compose for PostgreSQL,
Valkey, and OpenSearch integration tests.
