19 KiB
19 KiB
03 — HTML5 Rendering Pipeline (Custom WebGL2 + Workers + OffscreenCanvas)
This chapter describes the custom browser-side rendering pipeline that
replaces Greenfield's AGPL @gfld/compositor. It's the largest single
component of the MIT approach (~3,000 LOC TypeScript, ~5 dev-weeks).
Critical: clean-room requirement
You CANNOT read Greenfield's @gfld/compositor source and reimplement it.
Even "inspired-by" code is risky under the Computer Associates v. Altai
Abstraction-Filtration-Comparison test. AGPL §13 network-use clause would
force you to open-source your entire Odoo fork if a court finds your
compositor is an AGPL derivative.
Right approach: True clean-room.
- Person A (spec author, may read AGPL source) writes a behavioral spec (inputs/outputs/edge cases, NO code).
- Person B (implementer, never opens
packages/compositor/src/) implements from spec + MIT references only:- Wayland protocol spec (https://wayland.freedesktop.org/docs/html/)
- wlroots C source (MIT, https://gitlab.freedesktop.org/wlroots/wlroots)
- Weston C source (MIT, https://gitlab.freedesktop.org/wayland/weston)
- Smithay Rust source (MIT, https://github.com/Smithay/smithay)
- WebGL2/WebCodecs/KCP MDN docs
Maintain SPEC.md and CLEAN_ROOM_LOG.md as legal-defensibility artifacts.
Modernization wins over Greenfield
Greenfield's browser compositor uses:
- WebGL1 (no OffscreenCanvas, no WebGL2)
- Main-thread WebGL (jank on heavy frames)
- WebCodecs with hardware decode explicitly disabled (
// FIXME hardware decoding frame offset is wrong) - FFmpeg Wasm fallback (GPL — we can't use it)
Our MIT approach modernizes to:
- WebGL2 — uniform buffer objects, transform feedback, simplified YUV→RGB shaders
- OffscreenCanvas + Web Workers — render in a worker, never block main thread
- WebCodecs with hardware decode enabled — fix Greenfield's "frame offset is wrong" bug
- OpenH264 BSD Wasm fallback — replaces Greenfield's GPL FFmpeg Wasm
- Adaptive bitrate — consume
EncoderFeedbackdata Greenfield produces but doesn't use - Real touch input — Greenfield has a 46-line stub that never fires
Proposed file structure
packages/compositor/ # PROPRIETARY, clean-room TS
├── SPEC.md # clean-room behavioral spec
├── CLEAN_ROOM_LOG.md # who read AGPL, who didn't, when
├── src/
│ ├── index.ts # public API: createCompositorSession()
│ ├── core/
│ │ ├── Session.ts # top-level session
│ │ ├── Compositor.ts # wl_compositor global
│ │ └── Globals.ts # Wayland globals registry
│ ├── surface/ # ~1,200 LOC, 3 weeks
│ │ ├── Surface.ts # wl_surface, double-buffered state
│ │ ├── SurfaceState.ts # state struct
│ │ ├── SurfaceRole.ts # role abstraction
│ │ ├── SurfaceTree.ts # parent/child tree
│ │ ├── Subsurface.ts # wl_subsurface
│ │ ├── Subcompositor.ts # wl_subcompositor global
│ │ ├── Region.ts # wl_region (pixman-WASM)
│ │ └── FrameCallback.ts # wl_callback
│ ├── render/ # ~1,800 LOC, 3 weeks
│ │ ├── Renderer.ts # 4-phase render pipeline
│ │ ├── Scene.ts # one per output (canvas)
│ │ ├── View.ts # positioned surface for compositing
│ │ ├── Output.ts # wl_output, OffscreenCanvas binding
│ │ ├── RenderState.ts # per-view-per-scene texture + transform
│ │ ├── Texture.ts # WebGL2 texture wrapper
│ │ ├── ShaderProgram.ts # WebGL2 program wrapper
│ │ ├── Loop.ts # requestAnimationFrame integration
│ │ └── shaders/
│ │ ├── YUVA2RGBA.glsl # BT.601 YUVA → RGBA
│ │ ├── YUV2RGB.glsl # YUV → RGB (no alpha)
│ │ ├── RGBandA2RGBA.glsl # RGB+Alpha → RGBA
│ │ └── SceneShader.glsl # final pass: textured quads
│ ├── buffer/ # ~600 LOC, 1.5 weeks
│ │ ├── BufferContents.ts # buffer abstraction (mimeType)
│ │ ├── BufferImplementation.ts # interface
│ │ ├── Shm.ts # wl_shm, BMP-header hack
│ │ ├── ImageBitmapBuffer.ts # web-bitmapbuf fast path
│ │ └── EncodedFrameBuffer.ts # H.264/PNG from proxy
│ ├── transport/ # ~3,500 LOC, 4 weeks
│ │ ├── Channel.ts # ARQChannel + SimpleChannel
│ │ ├── kcp.ts # ported fresh from MIT C ref
│ │ ├── ReconnectingWebSocket.ts # vendored from pladaria (MIT)
│ │ ├── BufferStream.ts # serial → frame promise resolution
│ │ ├── StreamingBuffer.ts # per-surface encoded-frame buffer
│ │ ├── EncodedFrame.ts # wire format parser
│ │ ├── EncoderFeedback.ts # 4-byte decode feedback, 1 Hz
│ │ ├── RemoteInputOutput.ts # FD-over-WebSocket for native apps
│ │ ├── RemoteAppLauncher.ts # channel setup per app
│ │ ├── RemoteOutOfBandChannel.ts # OOB opcode dispatch
│ │ └── decoders/
│ │ ├── FrameDecoder.ts # interface
│ │ ├── WebCodecsFrameDecoder.ts # primary: VideoDecoder (HW-accelerated)
│ │ └── OpenH264FrameDecoder.ts # fallback: OpenH264 BSD Wasm
│ ├── input/ # ~2,500 LOC, 3 weeks
│ │ ├── Seat.ts # wl_seat global
│ │ ├── Keyboard.ts # wl_keyboard, keymap, modifiers
│ │ ├── Pointer.ts # wl_pointer, motion, buttons, axis
│ │ ├── Touch.ts # wl_touch (REAL implementation, not stub)
│ │ ├── Xkb.ts # xkbcommon WASM wrapper
│ │ ├── InputQueue.ts # event coalescing + RAF dispatch
│ │ ├── InputOutput.ts # FD abstraction
│ │ └── events.ts # KeyEvent, ButtonEvent, AxisEvent
│ ├── roles/ # ~1,800 LOC, 3 weeks (HIGH RISK)
│ │ ├── XdgWmBase.ts # xdg_wm_base global, ping
│ │ ├── XdgSurface.ts # xdg_surface, configure/ack
│ │ ├── XdgToplevel.ts # xdg_toplevel, window state
│ │ ├── XdgPopup.ts # xdg_popup, transient windows, grabs
│ │ ├── XdgPositioner.ts # popup positioning math
│ │ ├── Shell.ts # legacy wl_shell (optional)
│ │ └── ShellSurface.ts # legacy wl_shell_surface
│ ├── data/ # ~600 LOC, 1.5 weeks
│ │ ├── DataDeviceManager.ts # wl_data_device_manager
│ │ ├── DataSource.ts # wl_data_source
│ │ ├── DataOffer.ts # wl_data_offer
│ │ ├── WaylandDataSource.ts # native source wrapper
│ │ ├── PrimarySelectionDeviceManager.ts # NEW: zwp_primary_selection_v1
│ │ └── PrimarySelectionSource.ts # NEW: middle-click paste
│ ├── desktop/ # ~800 LOC, 1.5 weeks
│ │ ├── Desktop.ts # Floating vs AlwaysFullScreen
│ │ ├── DesktopSurface.ts # abstraction
│ │ ├── FloatingDesktopSurface.ts # default mode
│ │ └── FullscreenDesktopSurface.ts # kiosk mode
│ ├── math/ # ~500 LOC, 0.5 weeks
│ │ ├── Mat4.ts # 4×4 matrix
│ │ ├── Vec4.ts # 4-vector
│ │ ├── Point.ts # 2D point
│ │ ├── Size.ts # 2D size
│ │ ├── Rect.ts # 2D rectangle
│ │ └── Transformations.ts # affine transforms
│ ├── userapi/ # ~120 LOC, 0.5 weeks
│ │ └── UserShellApi.ts # OWL shell integration seam
│ ├── browser/ # ~750 LOC, 2 weeks
│ │ ├── input.ts # DOM keyboard events
│ │ ├── pointer.ts # DOM pointer events
│ │ ├── dnd.ts # DnD bridge
│ │ ├── selection.ts # clipboard bridge (native → DOM)
│ │ ├── text-selection.ts # Firefox fallback (text only)
│ │ ├── attributes.ts # canvas attributes
│ │ └── capabilities.ts # browser feature detection
│ └── (optional, Phase 6)
│ ├── webapp/ # ~500 LOC, 1 week (WASM web apps)
│ │ ├── WebAppLauncher.ts
│ │ ├── WebConnectionHandler.ts
│ │ ├── WebInputOutput.ts
│ │ └── WebBuffersFactory.ts
│ └── xwayland/ # ~5,000 LOC, 8 weeks (X11 apps)
│ ├── XWindowManager.ts
│ ├── XWindow.ts
│ ├── XWindowFrame.ts
│ └── ... (uses @gfld/xtsb MIT)
└── test/
├── unit/ # ~1,000 LOC, Jest unit tests
├── protocol/ # ~2,000 LOC, Weston test clients ported to TS
└── e2e/ # ~500 LOC, Playwright E2E with real Linux apps
Modern rendering pipeline (WebGL2 + OffscreenCanvas + Workers)
┌─────────────────────────────────────────────────────────────────────────┐
│ MAIN THREAD (Odoo OWL shell + DOM events) │
│ │
│ <canvas> element owned by OWL component │
│ │ │
│ │ canvas.transferControlToOffscreen() → OffscreenCanvas │
│ │ │
│ │ postMessage({canvas: offscreenCanvas}, [offscreenCanvas]) │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Render Worker (WebGL2 + OffscreenCanvas) │ │
│ │ │ │
│ │ Receives: │ │
│ │ - OffscreenCanvas (transferred) │ │
│ │ - EncodedVideoChunk / ImageBitmap (transferred from decode) │ │
│ │ - Surface state updates (postMessage) │ │
│ │ - Frame callback requests (postMessage) │ │
│ │ │ │
│ │ Runs the 4-phase render pipeline: │ │
│ │ PHASE 1 — pixel upload (per view per scene) │ │
│ │ PHASE 2 — requestAnimationFrame │ │
│ │ PHASE 3 — draw (per scene: viewport + clear + drawArrays) │ │
│ │ PHASE 4 — fire frame callbacks │ │
│ │ │ │
│ │ Sends: │ │
│ │ - Frame callback timestamps (postMessage to main thread) │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ Decode Worker (WebCodecs VideoDecoder, hardware-accelerated) │ │
│ │ │ │
│ │ Receives: │ │
│ │ - EncodedVideoChunk (transferred from transport) │ │
│ │ │ │
│ │ Decodes: │ │
│ │ - VideoDecoder.decode(chunk) │ │
│ │ - VideoFrame output (zero-copy GPU texture source) │ │
│ │ │ │
│ │ Sends: │ │
│ │ - VideoFrame (transferred to render worker) │ │
│ │ - Decode duration telemetry (for adaptive bitrate) │ │
│ │ │ │
│ └────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
Why WebGL2 over WebGL1
- Uniform Buffer Objects — share state between shaders efficiently
- Transform feedback — could be used for vertex skinning (not needed for compositor)
- Simplified YUV→RGB shaders —
texelFetchfor direct texel access - 3D textures — could simplify YUV planar upload (Y, U, V as 3 slices)
- Better browser support — 97%+ global browser support per caniuse
Why OffscreenCanvas + Web Workers
- Never block main thread — heavy decode + render happens off-thread
- Smooth UI — Odoo shell stays responsive during video playback
- Multi-output scaling — one worker per output for parallel rendering
- Recovery — if render worker crashes, restart without losing Odoo session
WebCodecs vs FFmpeg Wasm
| Aspect | WebCodecs (primary) | OpenH264 Wasm (fallback) |
|---|---|---|
| License | W3C standard (MIT-equivalent) | BSD-2-Clause |
| Hardware decode | Yes (Intel/AMD/NVIDIA) | No (CPU only) |
| Latency | ~5-10ms per frame | ~30-50ms per frame |
| CPU usage | Low (offloaded to GPU) | High (full decode on CPU) |
| Browser support | Chrome 94+, Edge 94+, Safari 16+, Firefox 130+ | All browsers with Wasm |
| Codecs | H.264, VP9, AV1, H.265 (where supported) | H.264 only |
Recommended approach: Probe VideoDecoder.isConfigSupported() at session
startup. If supported, use WebCodecs. Otherwise, fall back to OpenH264 Wasm
(loaded lazily, only when needed).
Wire format for encoded frames (REUSED from Greenfield)
Same wire format as Greenfield (see ../greenfield-analysis/09-hardware-accel.md
for byte layout). The browser side parses {contentSerial, mimeType, size, encodedSize, opaque, alpha?} and dispatches to the right decoder.
Engineering observations
- Vestigial damage tracking in Greenfield —
damageRectsare accumulated but not consumed by the renderer. Our MIT approach should actually use them for scissoring (per-rect partial repaints). - Multi-output texture cost — each view has one texture per scene. Our approach should share textures across scenes where possible.
- SHM xrgb8888 BMP-header hack — cleverly offloads pixel conversion to the browser's BMP decoder. Worth re-implementing.
- Encoder feedback telemetry — Greenfield produces it but doesn't use it. Our approach should consume it for adaptive bitrate.
- Cursor bypass — cursor is rendered as CSS
cursor: url(dataURL), not via WebGL. Same approach in our MIT version. - Frame-callback coalescing — multiple
wl_surface.framerequests within one render pass fire on the same RAF tick. Same in our version. - Subsurface sync cascade — three-layer mechanism (cachedState, commitCache, effectiveSync). Hard to get right; clean-room from spec.
Effort estimate
| Component | LOC | Weeks (1 dev) | Risk |
|---|---|---|---|
| Surface state stack | 1,200 | 3 | Med |
| Rendering pipeline (WebGL2 + OffscreenCanvas) | 1,800 | 3 | Med |
| Buffers (SHM + BMP hack) | 600 | 1.5 | Low |
| Transport (incl. fresh KCP port from MIT C ref) | 3,500 | 4 | High |
| Input (seat/kbd/ptr/touch/xkb) | 2,500 | 3 | Med |
| Roles (xdg-shell state machine) | 1,800 | 3 | High |
| Data device / clipboard / DnD + primary selection | 600 | 1.5 | Low |
| Desktop policy | 800 | 1.5 | Low |
| Math | 500 | 0.5 | Low |
| User shell API | 120 | 0.5 | Low |
| Browser glue | 750 | 2 | Med |
| H.264 decoder (WebCodecs + OpenH264 BSD) | 500 | 1 | Low |
| Core (Session/Globals) | 200 | 0.5 | Low |
| MVP TOTAL | ~13,400 | ~25 weeks | |
| XWayland (optional Phase 6) | 5,000 | 8 | High |
| WASM web app (optional) | 500 | 1 | Low |
| Test suite + hardening | — | 3 | — |
| GRAND TOTAL | ~18,900 | ~30-36 weeks |
With 2 senior devs in parallel: ~15-18 weeks for MVP.