Commit Graph

20 Commits

Author SHA1 Message Date
1a84da40bf feat(wfe-yaml): add HTTP ops, module loader, and npm support via esm.sh
Phase 4 — Permission-gated HTTP fetch op:
- op_fetch with net permission check on every request
- globalThis.fetch() wrapper with .json()/.text() methods
- Supports GET/POST/PUT/DELETE with headers and body

Phase 5 — Module loader:
- WfeModuleLoader resolving npm: → esm.sh, https://, file://, relative paths
- All resolution paths permission-checked
- Bare path resolution (/) for esm.sh sub-module redirects
- Dynamic import rejection unless permissions.dynamic_import: true
- esm.sh auto-added to net allowlist when modules declared

Mandatory npm integration test (is-number via esm.sh).
25 new tests. 133 total deno tests, 326 total workspace tests.
2026-03-25 23:02:51 +00:00
6fec7dbab5 feat(wfe-yaml): add deno_core JS/TS executor with sandboxed permissions
Secure JavaScript/TypeScript execution in workflow steps via deno_core,
behind the `deno` feature flag.

Security features:
- Per-step permission system: net host allowlist, filesystem read/write
  path restrictions, env var allowlist, subprocess spawn control
- V8 heap limits (64MB default) prevent memory exhaustion
- Execution timeout with V8 isolate termination for sync infinite loops
- Path traversal detection blocks ../ escape attempts
- Dynamic import rejection unless explicitly enabled

Workflow I/O ops:
- inputs() — read workflow data as JSON
- output(key, value) — set step outputs
- log(message) — structured tracing

Architecture:
- JsRuntime runs on dedicated thread (V8 is !Send)
- PermissionChecker enforced on every I/O op via OpState
- DenoStep implements StepBody, integrates with existing compiler
- Step type dispatch: "shell" or "deno" in YAML

34 new tests (12 permission unit, 3 config, 2 runtime, 18 integration).
2026-03-25 22:32:07 +00:00
ce68e4beed test(wfe-yaml): coverage pass to 90%+ and fix duration parsing bug
Added 51 tests: compiler hooks/parallel/error behavior (20),
validation error paths (15), shell integration tests (7),
lib.rs file loading (5), interpolation edge cases (4).

Fixed parse_duration_ms: "ms" suffix was unreachable because
strip_suffix('s') matched first. Now checks "ms" before "s".

Coverage: 40% → 90.3%. 326 total workspace tests.
2026-03-25 21:42:26 +00:00
b89563af63 feat(wfe-yaml): add YAML workflow definitions with shell executor
Concourse-CI-inspired YAML format for defining workflows. Compiles
to standard WorkflowDefinition + step factories.

Features:
- Schema parsing with serde_yaml (YamlWorkflow, YamlStep, StepConfig)
- ((var.path)) interpolation from config maps at load time
- YAML anchors (&anchor/*alias) fully supported
- Validation at load time (no runtime surprises)
- Shell executor: runs commands via tokio::process, captures stdout,
  parses ##wfe[output name=value] annotations for structured outputs
- Compiler: sequential wiring, parallel blocks, on_failure/on_success/
  ensure hooks, error behavior mapping
- Public API: load_workflow(), load_workflow_from_str()
- 23 tests (schema, interpolation, compiler, e2e)
2026-03-25 21:32:00 +00:00
8d0f83da3c feat(wfe-core): add output_data to ExecutionResult and register_step_factory to host
Core plumbing for YAML workflow support:
- Add output_data field to ExecutionResult for step output capture
- Executor merges output_data into workflow.data after step completion
- Add register_step_factory(key, factory) to WorkflowHost for
  dynamic step registration by external crates like wfe-yaml
2026-03-25 21:16:09 +00:00
c8582eb514 feat: add OpenTelemetry tracing support behind otel feature flag
- Add tracing::instrument spans to executor (workflow.execute),
  host (workflow.start, event.publish, event.process)
- Add otel feature flag to wfe-core and wfe crates
- Add wfe/src/otel.rs helper for OTLP exporter initialization
- Dependencies: tracing-opentelemetry, opentelemetry, opentelemetry_sdk,
  opentelemetry-otlp (all optional behind otel feature)
- Step execution stays at info level, executor internals at debug
2026-03-25 20:41:34 +00:00
bd51517e9f fix: demote per-round executor log to debug level
The "Executing workflow" message fires every execution round (many
times per workflow) and is noisy at INFO. Step start/completion
remains at INFO.
2026-03-25 20:35:11 +00:00
88fc6bf7ad feat: add executor tracing, auto-register primitives, and Default impls
- Add info!-level tracing to workflow executor: logs each execution
  round, each step run (with type and name), step completion, and
  workflow completion
- WorkflowHost.start() now auto-registers all built-in primitive step
  types so users don't need to register them manually
- Add #[derive(Default)] to all primitive steps and PollEndpointConfig
- Add tracing-subscriber to wfe crate for the pizza example
- Pizza example now shows full step-by-step execution logs
2026-03-25 20:32:47 +00:00
6d57f8ef22 test: add end-to-end tests and pizza workflow example
E2E tests covering: linear workflows, conditional branching, while loops,
foreach iteration, parallel branches, event-driven waiting, delays,
error handling (retry/suspend/terminate), saga compensation, versioning.

Pizza workflow example demonstrating the full API: parallel prep,
quality checks with retry, oven timer events, and delivery routing.

252 tests total, 93%+ coverage, zero clippy warnings.
2026-03-25 20:14:48 +00:00
c74b9b6ad7 feat(wfe-opensearch): add OpenSearch search index provider
OpenSearchIndex implementing SearchIndex trait. Index mapping for
workflow documents. Bool queries with multi_match for terms, term
filters for status/reference, range filters for dates. Pagination support.
2026-03-25 20:14:24 +00:00
f95bef3883 feat(wfe-valkey): add Valkey provider for locks, queues, and lifecycle events
ValkeyLockProvider: SET NX EX for acquisition, Lua script for safe release.
ValkeyQueueProvider: LPUSH/RPOP for FIFO queues.
ValkeyLifecyclePublisher: PUBLISH to per-instance and global channels.

Connections obtained once during construction (no per-operation TCP handshakes).
2026-03-25 20:14:07 +00:00
b2c37701b1 feat(wfe-postgres): add PostgreSQL persistence provider
PostgresPersistenceProvider using sqlx with wfc schema. JSONB for
complex fields, TIMESTAMPTZ for dates. Transactions for atomicity.
Proper error propagation on deserialization (no unwrap_or_default).
Status conversion returns Result for unknown values.
2026-03-25 20:13:44 +00:00
f39766cc3d feat(wfe-sqlite): add SQLite persistence provider
SqlitePersistenceProvider using sqlx with in-memory and file-based
SQLite. Schema with 6 tables, 8 indexes, created via ensure_store_exists.
Transactions for multi-table operations. Passes persistence_suite tests.
2026-03-25 20:11:16 +00:00
8b946e86e3 feat(wfe): add WorkflowHost, registry, sync runner, and purger
WorkflowHost orchestrates the engine: background workflow and event
consumers, start/stop lifecycle, workflow CRUD, event publishing.
WorkflowHostBuilder (owned-self, returns Result).

InMemoryWorkflowRegistry with version support. SyncWorkflowRunner
for testing. WorkflowPurger stub.

Event consumer acquires distributed locks before modifying workflows
to prevent lost updates from concurrent event processing.
2026-03-25 20:11:06 +00:00
a61e68d2a9 feat(wfe-core): add workflow executor with result processing and error handling
WorkflowExecutor: acquire lock, load instance, run steps, process
results, persist, release lock. StepRegistry for resolving step types.

ResultProcessor handles: next, outcome, persist, branch, sleep,
wait_for_event, poll_endpoint. ErrorHandler implements Retry (with
max_retries), Suspend, Terminate, and Compensate behaviors.
2026-03-25 20:10:45 +00:00
456c3c5b2e feat(wfe-core): add fluent workflow builder API
Owned-self builder pattern (no lifetime parameters). WorkflowBuilder
chains start_with/then/end_workflow to produce WorkflowDefinition.

StepBuilder supports: name, id, on_error, compensate_with, then,
then_fn, wait_for, delay, if_do, while_do, for_each, saga, parallel.

ParallelBuilder for branching with join semantics. InlineStep for
closure-based steps. Step config stored on WorkflowStep.step_config.
2026-03-25 20:10:33 +00:00
d0a3f0e185 feat(wfe-core): add step primitives for workflow control flow
12 step primitives implementing StepBody: DecideStep, IfStep, WhileStep,
ForEachStep, SequenceStep, DelayStep, WaitForStep, ScheduleStep,
RecurStep, PollEndpointStep, SagaContainerStep, EndStep.

Each primitive handles its state machine via persistence_data and
branch creation for container steps.
2026-03-25 20:10:03 +00:00
eea8bdb824 feat(wfe-core): add test support with in-memory providers and test suites
InMemoryPersistenceProvider, InMemoryLockProvider, InMemoryQueueProvider,
InMemoryLifecyclePublisher behind test-support feature flag.

Shared test suite macros: persistence_suite!, lock_suite!, queue_suite!
that run the same tests against any provider implementation.
2026-03-25 20:09:17 +00:00
d87d888787 feat(wfe-core): add models, traits, and error types
Core domain models: WorkflowInstance, ExecutionPointer, WorkflowDefinition,
WorkflowStep, Event, EventSubscription, ScheduledCommand, ExecutionError,
LifecycleEvent, PollEndpointConfig. All serde-serializable.

Provider traits: PersistenceProvider (composite of WorkflowRepository,
EventRepository, SubscriptionRepository, ScheduledCommandRepository),
DistributedLockProvider, QueueProvider, SearchIndex, LifecyclePublisher,
WorkflowMiddleware, StepMiddleware, WorkflowRegistry.

StepBody trait with StepExecutionContext for workflow step implementations.
WorkflowData marker trait (blanket impl for Serialize + DeserializeOwned).
2026-03-25 20:07:50 +00:00
098564db51 chore: scaffold workspace with nextest and docker compose
Workspace with 6 crates: wfe-core, wfe-sqlite, wfe-postgres,
wfe-opensearch, wfe-valkey, and wfe (umbrella). Nextest profiles
for default, integration, and CI. Docker compose for PostgreSQL,
Valkey, and OpenSearch integration tests.
2026-03-25 20:05:34 +00:00