291 lines
19 KiB
Markdown
291 lines
19 KiB
Markdown
# 03 — HTML5 Rendering Pipeline (Custom WebGL2 + Workers + OffscreenCanvas)
|
||
|
||
This chapter describes the **custom browser-side rendering pipeline** that
|
||
replaces Greenfield's AGPL `@gfld/compositor`. It's the largest single
|
||
component of the MIT approach (~3,000 LOC TypeScript, ~5 dev-weeks).
|
||
|
||
## Critical: clean-room requirement
|
||
|
||
**You CANNOT read Greenfield's `@gfld/compositor` source and reimplement it.**
|
||
Even "inspired-by" code is risky under the *Computer Associates v. Altai*
|
||
Abstraction-Filtration-Comparison test. AGPL §13 network-use clause would
|
||
force you to open-source your entire Odoo fork if a court finds your
|
||
compositor is an AGPL derivative.
|
||
|
||
**Right approach**: True clean-room.
|
||
- Person A (spec author, may read AGPL source) writes a behavioral spec
|
||
(inputs/outputs/edge cases, NO code).
|
||
- Person B (implementer, never opens `packages/compositor/src/`) implements
|
||
from spec + MIT references only:
|
||
- Wayland protocol spec (https://wayland.freedesktop.org/docs/html/)
|
||
- wlroots C source (MIT, https://gitlab.freedesktop.org/wlroots/wlroots)
|
||
- Weston C source (MIT, https://gitlab.freedesktop.org/wayland/weston)
|
||
- Smithay Rust source (MIT, https://github.com/Smithay/smithay)
|
||
- WebGL2/WebCodecs/KCP MDN docs
|
||
|
||
Maintain `SPEC.md` and `CLEAN_ROOM_LOG.md` as legal-defensibility artifacts.
|
||
|
||
## Modernization wins over Greenfield
|
||
|
||
Greenfield's browser compositor uses:
|
||
- WebGL1 (no OffscreenCanvas, no WebGL2)
|
||
- Main-thread WebGL (jank on heavy frames)
|
||
- WebCodecs with hardware decode explicitly disabled (`// FIXME hardware decoding frame offset is wrong`)
|
||
- FFmpeg Wasm fallback (GPL — we can't use it)
|
||
|
||
Our MIT approach modernizes to:
|
||
- **WebGL2** — uniform buffer objects, transform feedback, simplified YUV→RGB shaders
|
||
- **OffscreenCanvas + Web Workers** — render in a worker, never block main thread
|
||
- **WebCodecs with hardware decode enabled** — fix Greenfield's "frame offset is wrong" bug
|
||
- **OpenH264 BSD Wasm fallback** — replaces Greenfield's GPL FFmpeg Wasm
|
||
- **Adaptive bitrate** — consume `EncoderFeedback` data Greenfield produces but doesn't use
|
||
- **Real touch input** — Greenfield has a 46-line stub that never fires
|
||
|
||
## Proposed file structure
|
||
|
||
```
|
||
packages/compositor/ # PROPRIETARY, clean-room TS
|
||
├── SPEC.md # clean-room behavioral spec
|
||
├── CLEAN_ROOM_LOG.md # who read AGPL, who didn't, when
|
||
├── src/
|
||
│ ├── index.ts # public API: createCompositorSession()
|
||
│ ├── core/
|
||
│ │ ├── Session.ts # top-level session
|
||
│ │ ├── Compositor.ts # wl_compositor global
|
||
│ │ └── Globals.ts # Wayland globals registry
|
||
│ ├── surface/ # ~1,200 LOC, 3 weeks
|
||
│ │ ├── Surface.ts # wl_surface, double-buffered state
|
||
│ │ ├── SurfaceState.ts # state struct
|
||
│ │ ├── SurfaceRole.ts # role abstraction
|
||
│ │ ├── SurfaceTree.ts # parent/child tree
|
||
│ │ ├── Subsurface.ts # wl_subsurface
|
||
│ │ ├── Subcompositor.ts # wl_subcompositor global
|
||
│ │ ├── Region.ts # wl_region (pixman-WASM)
|
||
│ │ └── FrameCallback.ts # wl_callback
|
||
│ ├── render/ # ~1,800 LOC, 3 weeks
|
||
│ │ ├── Renderer.ts # 4-phase render pipeline
|
||
│ │ ├── Scene.ts # one per output (canvas)
|
||
│ │ ├── View.ts # positioned surface for compositing
|
||
│ │ ├── Output.ts # wl_output, OffscreenCanvas binding
|
||
│ │ ├── RenderState.ts # per-view-per-scene texture + transform
|
||
│ │ ├── Texture.ts # WebGL2 texture wrapper
|
||
│ │ ├── ShaderProgram.ts # WebGL2 program wrapper
|
||
│ │ ├── Loop.ts # requestAnimationFrame integration
|
||
│ │ └── shaders/
|
||
│ │ ├── YUVA2RGBA.glsl # BT.601 YUVA → RGBA
|
||
│ │ ├── YUV2RGB.glsl # YUV → RGB (no alpha)
|
||
│ │ ├── RGBandA2RGBA.glsl # RGB+Alpha → RGBA
|
||
│ │ └── SceneShader.glsl # final pass: textured quads
|
||
│ ├── buffer/ # ~600 LOC, 1.5 weeks
|
||
│ │ ├── BufferContents.ts # buffer abstraction (mimeType)
|
||
│ │ ├── BufferImplementation.ts # interface
|
||
│ │ ├── Shm.ts # wl_shm, BMP-header hack
|
||
│ │ ├── ImageBitmapBuffer.ts # web-bitmapbuf fast path
|
||
│ │ └── EncodedFrameBuffer.ts # H.264/PNG from proxy
|
||
│ ├── transport/ # ~3,500 LOC, 4 weeks
|
||
│ │ ├── Channel.ts # ARQChannel + SimpleChannel
|
||
│ │ ├── kcp.ts # ported fresh from MIT C ref
|
||
│ │ ├── ReconnectingWebSocket.ts # vendored from pladaria (MIT)
|
||
│ │ ├── BufferStream.ts # serial → frame promise resolution
|
||
│ │ ├── StreamingBuffer.ts # per-surface encoded-frame buffer
|
||
│ │ ├── EncodedFrame.ts # wire format parser
|
||
│ │ ├── EncoderFeedback.ts # 4-byte decode feedback, 1 Hz
|
||
│ │ ├── RemoteInputOutput.ts # FD-over-WebSocket for native apps
|
||
│ │ ├── RemoteAppLauncher.ts # channel setup per app
|
||
│ │ ├── RemoteOutOfBandChannel.ts # OOB opcode dispatch
|
||
│ │ └── decoders/
|
||
│ │ ├── FrameDecoder.ts # interface
|
||
│ │ ├── WebCodecsFrameDecoder.ts # primary: VideoDecoder (HW-accelerated)
|
||
│ │ └── OpenH264FrameDecoder.ts # fallback: OpenH264 BSD Wasm
|
||
│ ├── input/ # ~2,500 LOC, 3 weeks
|
||
│ │ ├── Seat.ts # wl_seat global
|
||
│ │ ├── Keyboard.ts # wl_keyboard, keymap, modifiers
|
||
│ │ ├── Pointer.ts # wl_pointer, motion, buttons, axis
|
||
│ │ ├── Touch.ts # wl_touch (REAL implementation, not stub)
|
||
│ │ ├── Xkb.ts # xkbcommon WASM wrapper
|
||
│ │ ├── InputQueue.ts # event coalescing + RAF dispatch
|
||
│ │ ├── InputOutput.ts # FD abstraction
|
||
│ │ └── events.ts # KeyEvent, ButtonEvent, AxisEvent
|
||
│ ├── roles/ # ~1,800 LOC, 3 weeks (HIGH RISK)
|
||
│ │ ├── XdgWmBase.ts # xdg_wm_base global, ping
|
||
│ │ ├── XdgSurface.ts # xdg_surface, configure/ack
|
||
│ │ ├── XdgToplevel.ts # xdg_toplevel, window state
|
||
│ │ ├── XdgPopup.ts # xdg_popup, transient windows, grabs
|
||
│ │ ├── XdgPositioner.ts # popup positioning math
|
||
│ │ ├── Shell.ts # legacy wl_shell (optional)
|
||
│ │ └── ShellSurface.ts # legacy wl_shell_surface
|
||
│ ├── data/ # ~600 LOC, 1.5 weeks
|
||
│ │ ├── DataDeviceManager.ts # wl_data_device_manager
|
||
│ │ ├── DataSource.ts # wl_data_source
|
||
│ │ ├── DataOffer.ts # wl_data_offer
|
||
│ │ ├── WaylandDataSource.ts # native source wrapper
|
||
│ │ ├── PrimarySelectionDeviceManager.ts # NEW: zwp_primary_selection_v1
|
||
│ │ └── PrimarySelectionSource.ts # NEW: middle-click paste
|
||
│ ├── desktop/ # ~800 LOC, 1.5 weeks
|
||
│ │ ├── Desktop.ts # Floating vs AlwaysFullScreen
|
||
│ │ ├── DesktopSurface.ts # abstraction
|
||
│ │ ├── FloatingDesktopSurface.ts # default mode
|
||
│ │ └── FullscreenDesktopSurface.ts # kiosk mode
|
||
│ ├── math/ # ~500 LOC, 0.5 weeks
|
||
│ │ ├── Mat4.ts # 4×4 matrix
|
||
│ │ ├── Vec4.ts # 4-vector
|
||
│ │ ├── Point.ts # 2D point
|
||
│ │ ├── Size.ts # 2D size
|
||
│ │ ├── Rect.ts # 2D rectangle
|
||
│ │ └── Transformations.ts # affine transforms
|
||
│ ├── userapi/ # ~120 LOC, 0.5 weeks
|
||
│ │ └── UserShellApi.ts # OWL shell integration seam
|
||
│ ├── browser/ # ~750 LOC, 2 weeks
|
||
│ │ ├── input.ts # DOM keyboard events
|
||
│ │ ├── pointer.ts # DOM pointer events
|
||
│ │ ├── dnd.ts # DnD bridge
|
||
│ │ ├── selection.ts # clipboard bridge (native → DOM)
|
||
│ │ ├── text-selection.ts # Firefox fallback (text only)
|
||
│ │ ├── attributes.ts # canvas attributes
|
||
│ │ └── capabilities.ts # browser feature detection
|
||
│ └── (optional, Phase 6)
|
||
│ ├── webapp/ # ~500 LOC, 1 week (WASM web apps)
|
||
│ │ ├── WebAppLauncher.ts
|
||
│ │ ├── WebConnectionHandler.ts
|
||
│ │ ├── WebInputOutput.ts
|
||
│ │ └── WebBuffersFactory.ts
|
||
│ └── xwayland/ # ~5,000 LOC, 8 weeks (X11 apps)
|
||
│ ├── XWindowManager.ts
|
||
│ ├── XWindow.ts
|
||
│ ├── XWindowFrame.ts
|
||
│ └── ... (uses @gfld/xtsb MIT)
|
||
└── test/
|
||
├── unit/ # ~1,000 LOC, Jest unit tests
|
||
├── protocol/ # ~2,000 LOC, Weston test clients ported to TS
|
||
└── e2e/ # ~500 LOC, Playwright E2E with real Linux apps
|
||
```
|
||
|
||
## Modern rendering pipeline (WebGL2 + OffscreenCanvas + Workers)
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ MAIN THREAD (Odoo OWL shell + DOM events) │
|
||
│ │
|
||
│ <canvas> element owned by OWL component │
|
||
│ │ │
|
||
│ │ canvas.transferControlToOffscreen() → OffscreenCanvas │
|
||
│ │ │
|
||
│ │ postMessage({canvas: offscreenCanvas}, [offscreenCanvas]) │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Render Worker (WebGL2 + OffscreenCanvas) │ │
|
||
│ │ │ │
|
||
│ │ Receives: │ │
|
||
│ │ - OffscreenCanvas (transferred) │ │
|
||
│ │ - EncodedVideoChunk / ImageBitmap (transferred from decode) │ │
|
||
│ │ - Surface state updates (postMessage) │ │
|
||
│ │ - Frame callback requests (postMessage) │ │
|
||
│ │ │ │
|
||
│ │ Runs the 4-phase render pipeline: │ │
|
||
│ │ PHASE 1 — pixel upload (per view per scene) │ │
|
||
│ │ PHASE 2 — requestAnimationFrame │ │
|
||
│ │ PHASE 3 — draw (per scene: viewport + clear + drawArrays) │ │
|
||
│ │ PHASE 4 — fire frame callbacks │ │
|
||
│ │ │ │
|
||
│ │ Sends: │ │
|
||
│ │ - Frame callback timestamps (postMessage to main thread) │ │
|
||
│ │ │ │
|
||
│ └────────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ ┌────────────────────────────────────────────────────────────────┐ │
|
||
│ │ Decode Worker (WebCodecs VideoDecoder, hardware-accelerated) │ │
|
||
│ │ │ │
|
||
│ │ Receives: │ │
|
||
│ │ - EncodedVideoChunk (transferred from transport) │ │
|
||
│ │ │ │
|
||
│ │ Decodes: │ │
|
||
│ │ - VideoDecoder.decode(chunk) │ │
|
||
│ │ - VideoFrame output (zero-copy GPU texture source) │ │
|
||
│ │ │ │
|
||
│ │ Sends: │ │
|
||
│ │ - VideoFrame (transferred to render worker) │ │
|
||
│ │ - Decode duration telemetry (for adaptive bitrate) │ │
|
||
│ │ │ │
|
||
│ └────────────────────────────────────────────────────────────────┘ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Why WebGL2 over WebGL1
|
||
|
||
- **Uniform Buffer Objects** — share state between shaders efficiently
|
||
- **Transform feedback** — could be used for vertex skinning (not needed for compositor)
|
||
- **Simplified YUV→RGB shaders** — `texelFetch` for direct texel access
|
||
- **3D textures** — could simplify YUV planar upload (Y, U, V as 3 slices)
|
||
- **Better browser support** — 97%+ global browser support per caniuse
|
||
|
||
## Why OffscreenCanvas + Web Workers
|
||
|
||
- **Never block main thread** — heavy decode + render happens off-thread
|
||
- **Smooth UI** — Odoo shell stays responsive during video playback
|
||
- **Multi-output scaling** — one worker per output for parallel rendering
|
||
- **Recovery** — if render worker crashes, restart without losing Odoo session
|
||
|
||
## WebCodecs vs FFmpeg Wasm
|
||
|
||
| Aspect | WebCodecs (primary) | OpenH264 Wasm (fallback) |
|
||
|---|---|---|
|
||
| License | W3C standard (MIT-equivalent) | BSD-2-Clause |
|
||
| Hardware decode | Yes (Intel/AMD/NVIDIA) | No (CPU only) |
|
||
| Latency | ~5-10ms per frame | ~30-50ms per frame |
|
||
| CPU usage | Low (offloaded to GPU) | High (full decode on CPU) |
|
||
| Browser support | Chrome 94+, Edge 94+, Safari 16+, Firefox 130+ | All browsers with Wasm |
|
||
| Codecs | H.264, VP9, AV1, H.265 (where supported) | H.264 only |
|
||
|
||
**Recommended approach**: Probe `VideoDecoder.isConfigSupported()` at session
|
||
startup. If supported, use WebCodecs. Otherwise, fall back to OpenH264 Wasm
|
||
(loaded lazily, only when needed).
|
||
|
||
## Wire format for encoded frames (REUSED from Greenfield)
|
||
|
||
Same wire format as Greenfield (see `../greenfield-analysis/09-hardware-accel.md`
|
||
for byte layout). The browser side parses `{contentSerial, mimeType, size,
|
||
encodedSize, opaque, alpha?}` and dispatches to the right decoder.
|
||
|
||
## Engineering observations
|
||
|
||
1. **Vestigial damage tracking in Greenfield** — `damageRects` are accumulated
|
||
but not consumed by the renderer. Our MIT approach should actually use
|
||
them for scissoring (per-rect partial repaints).
|
||
2. **Multi-output texture cost** — each view has one texture per scene. Our
|
||
approach should share textures across scenes where possible.
|
||
3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion
|
||
to the browser's BMP decoder. Worth re-implementing.
|
||
4. **Encoder feedback telemetry** — Greenfield produces it but doesn't use
|
||
it. Our approach should consume it for adaptive bitrate.
|
||
5. **Cursor bypass** — cursor is rendered as CSS `cursor: url(dataURL)`,
|
||
not via WebGL. Same approach in our MIT version.
|
||
6. **Frame-callback coalescing** — multiple `wl_surface.frame` requests
|
||
within one render pass fire on the same RAF tick. Same in our version.
|
||
7. **Subsurface sync cascade** — three-layer mechanism (cachedState,
|
||
commitCache, effectiveSync). Hard to get right; clean-room from spec.
|
||
|
||
## Effort estimate
|
||
|
||
| Component | LOC | Weeks (1 dev) | Risk |
|
||
|---|---|---|---|
|
||
| Surface state stack | 1,200 | 3 | Med |
|
||
| Rendering pipeline (WebGL2 + OffscreenCanvas) | 1,800 | 3 | Med |
|
||
| Buffers (SHM + BMP hack) | 600 | 1.5 | Low |
|
||
| Transport (incl. fresh KCP port from MIT C ref) | 3,500 | 4 | **High** |
|
||
| Input (seat/kbd/ptr/touch/xkb) | 2,500 | 3 | Med |
|
||
| Roles (xdg-shell state machine) | 1,800 | 3 | **High** |
|
||
| Data device / clipboard / DnD + primary selection | 600 | 1.5 | Low |
|
||
| Desktop policy | 800 | 1.5 | Low |
|
||
| Math | 500 | 0.5 | Low |
|
||
| User shell API | 120 | 0.5 | Low |
|
||
| Browser glue | 750 | 2 | Med |
|
||
| H.264 decoder (WebCodecs + OpenH264 BSD) | 500 | 1 | Low |
|
||
| Core (Session/Globals) | 200 | 0.5 | Low |
|
||
| **MVP TOTAL** | **~13,400** | **~25 weeks** | |
|
||
| XWayland (optional Phase 6) | 5,000 | 8 | High |
|
||
| WASM web app (optional) | 500 | 1 | Low |
|
||
| Test suite + hardening | — | 3 | — |
|
||
| **GRAND TOTAL** | **~18,900** | **~30-36 weeks** | |
|
||
|
||
With 2 senior devs in parallel: **~15-18 weeks for MVP**.
|