Add wfe-server-protos and wfe-server to workspace members.
Update StepExecutionContext constructions with log_sink: None
in buildkit and containerd test files.
Shell step streaming: when LogSink is present, uses cmd.spawn() with
tokio::select! to interleave stdout/stderr line-by-line. Respects
timeout_ms with child.kill() on timeout. Falls back to buffered mode
when no LogSink.
Security: block sensitive env var overrides (PATH, LD_PRELOAD, etc.)
from workflow data injection. Proper error handling for pipe capture.
4 LogSink regression tests + 2 env var security regression tests.
LogSink trait for real-time step output streaming. Added to
StepExecutionContext as optional field (backward compatible).
Threaded through WorkflowExecutor and WorkflowHostBuilder.
Wired LifecyclePublisher.publish() into executor at 5 points:
StepStarted, StepCompleted, Error, Completed, Terminated.
Also added lifecycle events to host start/suspend/resume/terminate.
Full changelog covering v1.0.0, v1.4.0, and v1.5.0 releases.
Also fix containerd integration test default address to handle
Lima socket forwarding gracefully.
879 tests passing. 88.8% coverage on wfe-rustlang.
Lima wfe-test VM: Alpine with system containerd + BuildKit from apk,
TCP socat proxy for reliable gRPC transport, probes with sudo for
socket permission fixes. 2 core / 4GB / 20GB.
CI pipeline: add wfe-rustlang to feature-tests, package, and publish
steps. Container tests use TCP proxy (http://127.0.0.1:2500) instead
of Unix socket forwarding. Containerd tests set WFE_IO_DIR for shared
filesystem support.
Add rustlang feature flag to wfe-yaml with support for all cargo and
rustup step types (15 total), including cargo-doc-mdx.
Schema additions: output_dir, package, features, all_features,
no_default_features, release, profile, toolchain, extra_args,
components, targets, default_toolchain fields on StepConfig.
Integration tests for compiling all step types from YAML, and
containerd-based end-to-end tests for running Rust toolchain
inside containers from bare Debian images.
Four bugs fixed in the containerd gRPC executor:
- Snapshot parent: resolve image chain ID from content store instead of
using empty parent, which created rootless containers with no binaries
- I/O capture: replace FIFOs with regular files for stdout/stderr since
FIFOs don't work across virtiofs filesystem boundaries (Lima VMs)
- Capabilities: grant Docker-default capability set (SETUID, SETGID,
CHOWN, etc.) when running as root so apt-get and similar tools work
- Shell path: use /bin/sh instead of sh in process args since container
PATH may be empty
Also adds WFE_IO_DIR env var for shared filesystem support with remote
daemons, and documents the remote daemon setup in lib.rs.
External cargo tools (audit, deny, nextest, llvm-cov) auto-install
via cargo install if not found on the system. For llvm-cov, the
llvm-tools-preview rustup component is also installed automatically.
New cargo-doc-mdx step type generates MDX documentation from rustdoc
JSON output. Runs cargo +nightly rustdoc --output-format json, then
transforms the JSON into MDX files with frontmatter, type signatures,
and doc comments grouped by module. Uses the official rustdoc-types
crate for deserialization.
Documents pipeline architecture, how to run it, WFE features
demonstrated, preflight tool checks, and graceful infrastructure
skipping. Adds nextest cover profile for llvm-cov integration.
Preprocesses <<: *anchor merge keys before serde_yaml 0.9 deserialization.
serde_yaml implements YAML 1.2 which dropped merge keys; the yaml-merge-keys
crate resolves them as a preprocessing step, giving full anchor + merge
support for DRY pipeline definitions.
The YAML compiler was using SubWorkflowPlaceholderStep that returned
next() immediately. Replaced with real SubWorkflowStep from wfe-core
that starts child workflows and waits for completion events.
Added regression test verifying the compiled factory produces a step
that calls host_context.start_workflow() and returns wait_for_event.
Nextest cover profile for cargo llvm-cov integration.
Update existing test imports from load_workflow_from_str to
load_single_workflow_from_str for backward compatibility.
Compiler dispatches type: workflow to SubWorkflowStep. Validation
detects circular workflow references via DFS with coloring. Cross-
workflow reference checking for multi-workflow files. Duplicate
workflow ID detection. 28 edge case tests for validation paths.
YamlWorkflowFile supports both single (workflow:) and multi (workflows:)
formats. WorkflowSpec gains typed inputs/outputs declarations.
Type string parser for inline types ("string?", "list<number>", etc.).
load_workflow_from_str returns Vec<CompiledWorkflow>.
Backward-compatible load_single_workflow_from_str convenience function.
SchemaType enum with inline syntax parsing: "string", "string?",
"list<number>", "map<string>", nested generics. WorkflowSchema
validates inputs/outputs against type declarations at both compile
time and runtime. 39 tests for parse and validate paths.
Replaced third-party buildkit-client git dependency with
wfe-buildkit-protos generated from official moby/buildkit protos.
Direct ControlClient gRPC calls: SolveRequest with frontend attrs,
exporters, cache options. Daemon-local context paths for builds
(session protocol for remote transfer is TODO).
Both proto crates now use tonic 0.14 / prost 0.14 — no transitive
dependency conflicts. 95 combined tests, 85.6% region coverage.
New crate generating Rust gRPC stubs from the official BuildKit
proto files (git submodule from moby/buildkit). Control service,
LLB definitions, session protocols, and source policy.
tonic 0.14 / prost 0.14.
Replaced nerdctl CLI shell-out with direct gRPC communication via
wfe-containerd-protos (tonic 0.14). Connects to containerd daemon
over Unix socket.
Implementation:
- connect() with tonic Unix socket connector
- ensure_image() via ImagesClient (full pull is TODO)
- build_oci_spec() constructing OCI runtime spec with process args,
env, user, cwd, mounts, and linux namespaces
- Container lifecycle: create → snapshot → task create → start →
wait → read FIFOs → cleanup
- containerd-namespace header injection on every request
FIFO-based stdout/stderr capture using named pipes.
40 tests, 88% line coverage (cargo-llvm-cov).
New crate generating Rust gRPC stubs from the official containerd
proto files (vendored as git submodule). Full client-facing API surface
using tonic 0.14 / prost 0.14. No transitive dependency conflicts.
Services: containers, content, diff, events, images, introspection,
leases, mounts, namespaces, sandbox, snapshots, streaming, tasks,
transfer, version.
Replaced buildctl CLI shell-out with direct gRPC communication via
buildkit-client crate. Connects to buildkitd daemon over Unix socket
or TCP with optional TLS.
Implementation:
- connect() with custom tonic UnixStream connector
- execute_build() implementing the solve protocol directly against
ControlClient (session setup, file sync, frontend attributes)
- Extracts digest from containerimage.digest in solve response
Added custom lima template (test/lima/wfe-test.yaml) that provides
both buildkitd and containerd with host-forwarded Unix sockets for
reproducible integration testing.
E2E tests against real buildkitd daemon via WFE_BUILDKIT_ADDR env var.
54 tests total. 89% line coverage (cargo-llvm-cov with E2E).
Standalone workspace crates for BuildKit image building and containerd
container execution. Config types, YAML schema integration, compiler
dispatch, validation rules, and mock-based unit tests.
Current implementation shells out to buildctl/nerdctl — will be
replaced with proper gRPC clients (buildkit-client, containerd protos)
in a follow-up. Config types, YAML integration, and test infrastructure
are stable and reusable.
wfe-buildkit: 60 tests, 97.9% library coverage
wfe-containerd: 61 tests, 97.8% library coverage
447 total workspace tests.
