marathon/docs/spatial-audio-vendoring-breakdown.md

# Spatial Audio System - Task Breakdown

**Epic:** Spatial Audio System (#4)
**Overall Size:** XXL+ (91+ points across 8 phases)
**Priority:** P1 (High - core immersion feature)

This document breaks down the 8 phases into specific, sized tasks for prioritization and scheduling.

**Note:** We are re-implementing bevy_seedling and bevy_steam_audio, not forking them. We depend on the underlying libraries (Firewheel and Steam Audio) as external crates, but write our own integration code that follows Marathon's patterns and doesn't lag behind Bevy version updates.

---

## Phase 1: Implement Firewheel Integration

**Phase Goal:** Re-implement bevy_seedling's Firewheel integration for Marathon
**Phase Size:** 14 points
**Dependencies:** None (can start immediately)
**Risk:** Medium (lock-free audio graph integration is complex)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 1.1 | Add Firewheel dependency and create audio module structure | S | 2 | Add crate dependency, set up module hierarchy | P1 |
| 1.2 | Implement audio graph initialization and lifecycle | M | 4 | Create graph, manage real-time thread, handle shutdown | P1 |
| 1.3 | Create sample playback nodes and basic routing | M | 4 | Sampler nodes, gain nodes, basic graph connections | P1 |
| 1.4 | Implement cpal audio output integration | S | 2 | Connect Firewheel graph to system audio output | P1 |
| 1.5 | Verify basic playback works (smoke test) | S | 2 | Test on macOS and iOS, verify no glitches | P1 |

**Phase 1 Total:** 14 points

### Lean Analysis
- **Eliminate Waste:** Can we use bevy_seedling directly? NO - lags Bevy updates, doesn't match Marathon patterns
- **Amplify Learning:** What will we learn? How to integrate lock-free audio graphs with ECS
- **Deliver Fast:** Can we implement incrementally? YES - basic playback first, then add features
- **Build Quality In:** Risk of audio glitches? YES - comprehensive playback testing critical

### Phase 1 Recommendations
1. **Do 1.1-1.4 sequentially** - each builds on previous
2. **Do 1.5 thoroughly** - verify no dropouts, glitches, or latency issues
3. **Reference bevy_seedling** - use it as reference implementation, but write our own code

---

## Phase 2: Implement Steam Audio Integration

**Phase Goal:** Re-implement bevy_steam_audio's Steam Audio integration for Marathon
**Phase Size:** 18 points
**Dependencies:** Phase 1 complete
**Risk:** High (C++ bindings, HRTF complexity)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 2.1 | Add steam-audio dependency (audionimbus bindings) | S | 2 | Add crate dependency, verify C++ library linking | P1 |
| 2.2 | Create Firewheel processor node for Steam Audio | L | 8 | Bridge between Firewheel and Steam Audio APIs, handle FFI safely | P1 |
| 2.3 | Implement HRTF initialization with default dataset | M | 4 | Load MIT KEMAR HRTF, verify initialization | P1 |
| 2.4 | Implement distance attenuation and air absorption | S | 2 | Basic spatial processing before HRTF | P1 |
| 2.5 | Test binaural output with positioned source | S | 2 | Create test scene, verify left/right panning and elevation | P1 |

**Phase 2 Total:** 18 points

### Lean Analysis
- **Eliminate Waste:** Can we use simpler panning? NO - HRTF is core to immersion
- **Amplify Learning:** Should we prototype Steam Audio separately? YES - task 2.5 is critical learning
- **Decide Late:** Can we defer HRTF? NO - it's foundational to spatial audio
- **Optimize Whole:** Does this improve both iOS and macOS? YES - cross-platform from start

### Critical Path
```
2.1 (add dependency)
  ↓
2.2 (Firewheel processor node) → 2.3 (HRTF init)
  ↓
2.4 (distance/air absorption)
  ↓
2.5 (binaural test)
```

### Phase 2 Recommendations
1. **Start with 2.1-2.3 sequentially** - Steam Audio setup is delicate
2. **Test heavily at 2.5** - spatial accuracy is mission-critical
3. **Reference bevy_steam_audio** - use as reference for Steam Audio API usage

---

## Phase 3: Bevy Integration

**Phase Goal:** Connect ECS components to audio graph
**Phase Size:** 20 points
**Dependencies:** Phase 2 complete
**Risk:** Medium (lock-free sync between game thread and audio thread)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 3.1 | Create `AudioSource` and `AudioListener` components | S | 2 | Define component API, derive traits | P1 |
| 3.2 | Implement position sync system (Transform → atomics) | L | 8 | Core sync logic, must be lock-free and glitch-free | P1 |
| 3.3 | Implement component lifecycle (Added/Removed) | M | 4 | Handle entity spawn/despawn, cleanup nodes | P1 |
| 3.4 | Create audio asset loading system | M | 4 | Decode audio files, integrate with Bevy assets | P1 |
| 3.5 | Test with moving sources and listener | S | 2 | Verify Doppler-free position updates | P1 |

**Phase 3 Total:** 20 points

### Lean Analysis
- **Eliminate Waste:** This IS the integration work - no waste
- **Amplify Learning:** Will this reveal audio thread issues? YES - explicit test for it (3.5)
- **Build Quality In:** Test concurrency early? YES - that's the whole phase
- **Deliver Fast:** Can we ship without asset loading (3.4)? NO - need real audio files

### Phase 3 Recommendations
1. **Do 3.1 first** - API design gates everything else
2. **3.2 is the critical path** - most complex, needs careful review
3. **Do 3.5 extensively** - test on real hardware, listen for glitches

---

## Phase 4: Bus Mixer

**Phase Goal:** Implement categorical bus-based mixing
**Phase Size:** 14 points
**Dependencies:** Phase 3 complete
**Risk:** Low (straightforward audio routing)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 4.1 | Create `MixerState` resource with bus hierarchy | S | 2 | Define SFX/Ambient/Music/UI/Voice buses | P1 |
| 4.2 | Implement bus Firewheel nodes (gain, EQ, sends) | L | 8 | Multiple node types, routing complexity | P1 |
| 4.3 | Connect all sources to appropriate buses | S | 2 | Route AudioSource components by bus type | P2 |
| 4.4 | Add master bus with limiting | S | 2 | Prevent clipping, add safety limiter | P1 |
| 4.5 | Test bus gain changes propagate correctly | XS | 1 | Verify mixer controls work | P2 |

**Phase 4 Total:** 15 points

### Lean Analysis
- **Eliminate Waste:** Do we need 5 buses initially? YES - categorical thinking is core
- **Amplify Learning:** Can we defer EQ? NO - it's essential for professional mixing
- **Build Quality In:** Is limiting necessary? YES - prevents painful clipping accidents
- **Optimize Whole:** Does bus structure match sound design needs? YES - aligns with RFC requirements

### Phase 4 Recommendations
1. **4.1 and 4.2 together** - design and implementation are coupled
2. **4.4 is critical** - limiter saves ears during development
3. **Fast phase:** Mostly plumbing once Firewheel is solid

---

## Phase 5: Prioritization and Culling

**Phase Goal:** Handle 200+ sources by prioritizing top 64
**Phase Size:** 12 points
**Dependencies:** Phase 4 complete
**Risk:** Medium (performance-critical code path)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 5.1 | Implement priority scoring system | M | 4 | Distance, amplitude, bus type, recency factors | P1 |
| 5.2 | Add distance and amplitude culling | S | 2 | Early exit for inaudible sources | P1 |
| 5.3 | Enforce voice limit (64 simultaneous) | S | 2 | Sort by priority, take top N | P1 |
| 5.4 | Optimize with spatial hashing | M | 4 | Fast neighbor queries for dense scenes | P2 |
| 5.5 | Test with 200+ sources in dense scene | M | 4 | Create test scene, verify <1ms culling time | P1 |

**Phase 5 Total:** 16 points

### Lean Analysis
- **Eliminate Waste:** Can we skip prioritization initially? NO - 200 sources will be muddy
- **Amplify Learning:** What's the real voice limit? Test at 5.5 to find out
- **Decide Late:** Can we defer spatial hashing (5.4)? YES if linear search is fast enough
- **Optimize Whole:** Does this work for both desktop and iOS? YES - same culling logic

### Phase 5 Recommendations
1. **Do 5.1-5.3 first** - core prioritization logic
2. **5.4 is optional optimization** - measure first, optimize if needed
3. **5.5 is GO/NO-GO gate** - if performance fails, revisit 5.4

---

## Phase 6: Debug Visualization

**Phase Goal:** Visual debugging of spatial audio sources
**Phase Size:** 16 points
**Dependencies:** Phase 5 complete (need full system working)
**Risk:** Low (tooling, not core functionality)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 6.1 | Implement gizmo rendering for active sources | M | 4 | Sphere gizmos with falloff ranges | P1 |
| 6.2 | Add color-coding by bus type | S | 2 | Visual differentiation of audio categories | P2 |
| 6.3 | Implement amplitude animation (brightness pulse) | S | 2 | Visual feedback for sound intensity | P2 |
| 6.4 | Add selection raycasting and inspector panel | M | 4 | Click source → show details in egui | P1 |
| 6.5 | Add occlusion ray visualization | S | 2 | Green = clear, red = occluded | P2 |
| 6.6 | Test on complex scene with 50+ sources | S | 2 | Verify visualization remains readable | P2 |

**Phase 6 Total:** 16 points

### Lean Analysis
- **Eliminate Waste:** Is visualization necessary? YES - critical for debugging spatial audio
- **Amplify Learning:** Will this reveal mix problems? YES - that's the purpose
- **Build Quality In:** Should this be P1? YES for 6.1 and 6.4, others are polish
- **Deliver Fast:** Can we ship minimal version? YES - 6.1 and 6.4 are essential, others are nice-to-have

### Phase 6 Recommendations
1. **Do 6.1 and 6.4 first** - core debug functionality
2. **6.2, 6.3, 6.5 are polish** - do when inspired
3. **This is a checkpoint** - use visualization to verify Phases 1-5 work correctly

---

## Phase 7: Mixer Panel

**Phase Goal:** Professional mixing console in egui
**Phase Size:** 18 points
**Dependencies:** Phase 4 complete (needs mixer state)
**Risk:** Low (UI work)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 7.1 | Implement egui mixer panel with channel strips | L | 8 | Layout 5 bus channels + master, faders, meters | P1 |
| 7.2 | Add EQ controls (3-band, collapsible) | M | 4 | Low shelf, mid bell, high shelf UI | P2 |
| 7.3 | Add solo/mute buttons | S | 2 | Isolation for debugging | P1 |
| 7.4 | Implement metering (peak/RMS from audio thread) | M | 4 | Lock-free meter reads, visual bars | P1 |
| 7.5 | Add LUFS integrated loudness meter | S | 2 | Master bus loudness monitoring | P3 |
| 7.6 | Implement preset save/load (JSON) | S | 2 | Serialize mixer state, version control | P2 |

**Phase 7 Total:** 22 points

### Lean Analysis
- **Eliminate Waste:** Do we need LUFS (7.5)? NO - defer to P3
- **Amplify Learning:** Will this improve mix quality? YES - professional tools = professional results
- **Build Quality In:** Is metering (7.4) essential? YES - you can't mix what you can't measure
- **Deliver Fast:** What's minimum viable mixer? 7.1, 7.3, 7.4

### Phase 7 Recommendations
1. **Do 7.1 first** - foundation for all other tasks
2. **7.3 and 7.4 immediately** - essential for mixing
3. **7.2 and 7.6 are P2** - important but not blocking
4. **7.5 is P3** - nice-to-have professional feature

---

## Phase 8: Soundscape Zones

**Phase Goal:** Layered ambient audio zones
**Phase Size:** 14 points
**Dependencies:** Phase 3 complete (needs component system)
**Risk:** Medium (complex activation logic)

### Tasks

| # | Task | Size | Points | Rationale | Priority |
|---|------|------|--------|-----------|----------|
| 8.1 | Implement `SoundscapeZone` component | S | 2 | Define zone shapes, layers, fade distance | P1 |
| 8.2 | Add zone activation system (listener position) | M | 4 | Track listener, activate/deactivate zones | P1 |
| 8.3 | Implement crossfading between overlapping zones | M | 4 | Smooth transitions, prevent popping | P1 |
| 8.4 | Add randomized layer playback | S | 2 | Occasional sounds (birds, creaks) with random timing | P2 |
| 8.5 | Test with 10+ overlapping zones | S | 2 | Verify performance, smooth crossfades | P1 |

**Phase 8 Total:** 14 points

### Lean Analysis
- **Eliminate Waste:** Are zones necessary? YES - essential for hyper-dense soundscapes
- **Amplify Learning:** Will this reveal performance issues? YES - test at 8.5
- **Build Quality In:** Is crossfading (8.3) critical? YES - popping is unacceptable
- **Optimize Whole:** Does this work with prioritization (Phase 5)? YES - zones create sources that get prioritized

### Phase 8 Recommendations
1. **Do 8.1-8.3 sequentially** - core zone system
2. **8.4 is nice-to-have** - adds realism but not essential
3. **8.5 is verification** - test in realistic scenario

---

## Overall Scheduling Recommendations

### Critical Path (Sequential)
```
Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5
                        ↓
                  Phase 6 (can parallelize with Phase 7)
                        ↓
                  Phase 7 → Phase 8
Total: ~91 points
```

### Parallel Opportunities
- **During Phase 1:** Design Phase 3 component API
- **During Phase 5:** Build Phase 6 visualization (uses same source data)
- **After Phase 4:** Phases 6, 7, 8 can partially overlap (different subsystems)

### Risk Mitigation Strategy
1. **Phase 1.6 is a GO/NO-GO gate** - if basic playback fails, stop and debug
2. **Phase 2.4 spatial accuracy test** - verify HRTF works before proceeding
3. **Phase 3.5 concurrency test** - ensure lock-free sync works flawlessly
4. **Phase 5.5 performance test** - verify 200+ sources cull to 64 in <1ms
5. **Incremental commits** - don't batch entire phase into one PR

---

## WSJF Prioritization (P1 Tier)

Scoring against other P1 work:

| Item | Player Value | Time Criticality | Risk Reduction | CoD | Size | WSJF |
|------|--------------|------------------|----------------|-----|------|------|
| **Spatial Audio Epic** | 9 | 6 | 7 | 22 | 91 | **0.24** |
| Phase 1 alone | 3 | 4 | 8 | 15 | 14 | **1.07** |
| Phase 1-3 together | 8 | 5 | 8 | 21 | 52 | **0.40** |
| Phases 1-5 (core) | 9 | 6 | 8 | 23 | 83 | **0.28** |

**Interpretation:**
- **High player value** - spatial audio is core immersion feature
- **High time criticality** - needed for demos and content creation
- **High risk reduction** - vendoring eliminates dependency lag
- **Phases 1-3 are foundation** - nothing works without them
- **Phases 6-8 are tooling** - can defer slightly if needed

---

## Sequencing with Other Work

### Good to do BEFORE this epic:
- ✅ iOS deployment scripts (done)
- ✅ Basic ECS setup (done)
- Bevy rendering vendoring (provides debugging context)

### Good to do AFTER this epic:
- Agent ambient sounds (depends on spatial audio)
- Environmental soundscapes (depends on zones)
- Dialogue system (depends on Voice bus)
- Music system (depends on mixer)

### Can do IN PARALLEL:
- Content creation (3D assets, animations)
- Networking improvements (different subsystem)
- Game design prototyping (can use placeholder audio)

---

## Decision Points

### Before Starting Phase 1:
- [ ] Do we have bandwidth for ~3 months of audio work?
- [ ] Are there higher-priority P1 bugs blocking demos?
- [ ] Have we validated spatial audio is essential to Aspen vision?

### Before Starting Phase 2:
- [ ] Did Phase 1.6 smoke tests pass on both iOS and macOS?
- [ ] Do we understand Firewheel's lock-free guarantees?
- [ ] Is Steam Audio C++ library compatible with iOS?

### Before Starting Phase 3:
- [ ] Does binaural output sound correct? (Phase 2.4)
- [ ] Have we tested with headphones/earbuds on device?
- [ ] Do we understand the game thread → audio thread sync pattern?

### Before Starting Phase 4:
- [ ] Are moving sources glitch-free? (Phase 3.5)
- [ ] Have we tested with 10+ simultaneous sources?

### Before Starting Phase 5:
- [ ] Does bus routing work correctly? (Phase 4.5)
- [ ] Have we identified the voice limit threshold?

### Before Starting Phases 6-8 (Tooling):
- [ ] Is core spatial audio (Phases 1-5) solid?
- [ ] Have we tested on both iOS and macOS extensively?
- [ ] Can we demo spatial audio to stakeholders?

### Before Closing Epic:
- [ ] All platforms tested with 64+ voices?
- [ ] Spatial accuracy test passed (blindfolded pointing <15° error)?
- [ ] Mix quality validated by professional sound engineer?
- [ ] Performance test passed (<2ms audio thread on M1 iPad)?
- [ ] Documentation updated (API docs, audio design guidelines)?

---

## Minimum Viable Implementation

If we need to ship faster, the **minimum viable spatial audio** is:

**Phases 1-5 only** (83 points)
- Firewheel and Steam Audio integration
- ECS integration
- Bus mixer
- Prioritization

This provides:
- 3D positioned audio with HRTF
- Bus-based mixing
- Voice limiting for performance

**Deferred to later:**
- Debug visualization (Phase 6)
- Mixer panel UI (Phase 7)
- Soundscape zones (Phase 8)

**Trade-off:** Harder to debug and mix without tooling, but core spatial audio works.

---

## Summary

**Total Effort:** ~91 points (XXL epic)
**Confidence:** Medium (audio is complex, vendoring reduces some risk)
**Recommendation:** High priority - spatial audio is core to Aspen's immersion
**When to do it:** After basic rendering is stable, before content creation ramps up

This is essential for Aspen's "sense of place" design pillar. Unlike the Bevy renderer epic (P2 technical debt), spatial audio is P1 player-facing immersion.