380 lines
19 KiB
Markdown
380 lines
19 KiB
Markdown
# 03 — HTML5 Rendering Pipeline
|
||
|
||
This chapter covers Greenfield's browser-side rendering pipeline: how
|
||
incoming Wayland buffers (SHM or encoded) end up as pixels on a browser
|
||
canvas/WebGL context, including damage tracking, commit semantics, surface
|
||
state stacking, and the compositing loop. The browser side lives in
|
||
`packages/compositor/src/`.
|
||
|
||
## Three buffer types, one BufferImplementation interface
|
||
|
||
All incoming buffers, regardless of origin, end up as a `BufferContents<T>`
|
||
with a `mimeType` that dispatches to the right upload path. There are three
|
||
concrete buffer types:
|
||
|
||
### 1. SHM buffer (`LazyImageBitmapBuffer`)
|
||
|
||
Comes from `wl_shm_pool` — the standard Wayland shared-memory buffer
|
||
mechanism. Browser-side implementation lives in `Shm.ts`. The raw pixel data
|
||
arrives as a transferable `ArrayBufferView` (transferred zero-copy via
|
||
`MessagePort` from a browser-iframe web app, or fetched via the FD endpoint
|
||
WebSocket from the proxy). Greenfield wraps the bytes as an `ImageData` /
|
||
BMP-Blob and decodes them via `createImageBitmap()` to produce an
|
||
`ImageBitmap` ready for WebGL `texImage2D`.
|
||
|
||
**Notable hack**: for `xrgb8888` SHM buffers, Greenfield prepends a BMP
|
||
header in front of the raw pixel data so the browser's `createImageBitmap`
|
||
decodes it natively, avoiding a per-pixel JS loop. This is a clever trick to
|
||
offload pixel conversion to the browser's BMP decoder.
|
||
|
||
### 2. Web bitmap buffer (`ImageBitmapBuffer` via `web_bitmapbuf_factory`)
|
||
|
||
Comes from Greenfield's custom `web-bitmapbuf-unstable-v1` protocol
|
||
extension. Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to
|
||
the compositor as a `wl_buffer` — bypassing SHM copy, encoding, and dmabuf
|
||
import. The `fd` arg is marshalled as a transferable, so the GPU texture
|
||
never leaves VRAM.
|
||
|
||
### 3. Remote streaming buffer (`StreamingBuffer`)
|
||
|
||
Comes from the native proxy via the FRAME data channel. Receives encoded
|
||
frames (H.264 or PNG) on a separate WebSocket per surface. Includes an
|
||
out-of-band mechanism for `BufferCreation` and `CommitSerial` opcodes so
|
||
the browser can match each encoded frame to the right `wl_surface.commit`
|
||
serial.
|
||
|
||
## Surface state stack
|
||
|
||
`Surface.ts` implements classic Wayland double-buffered state:
|
||
|
||
- `Surface.state` — the committed, currently-visible state.
|
||
- `Surface.pendingState` — the in-flight state, accumulated by client
|
||
requests until `commit` is called.
|
||
- `mergeSurfaceState` — the atomic swap in `commitPending()`.
|
||
|
||
Subsurfaces add a third layer of complexity:
|
||
|
||
- `subsurfaceChildren` arrays (committed + pending, bottom→top)
|
||
- `placeAbove` / `placeBelow` reorder pending
|
||
- `effectiveSync` getter cascades through the ancestor chain (if any
|
||
ancestor is sync, this subsurface is effectively sync)
|
||
- Sync subsurfaces cache to `cachedState` and only flush when an ancestor
|
||
commits via `onParentCommit`
|
||
- `setDesync` immediately calls `onCommit` to flush any pending cache
|
||
|
||
## Commit semantics
|
||
|
||
When a Wayland client calls `wl_surface.commit`:
|
||
|
||
1. If the surface has a remote streaming buffer, Greenfield pulls the
|
||
`serial = commitSerials.shift()` (an out-of-band opcode 7 mechanism that
|
||
lets `StreamingBuffer.getContents(surface, serial)` wait for the right
|
||
encoded frame).
|
||
2. `bufferContents = await bufferImpl.getContents(surface, serial)` —
|
||
resolves when the encoded frame for this serial has arrived.
|
||
3. `role.onCommit(this)` — role-specific logic (XdgToplevel, XdgPopup,
|
||
Subsurface, etc.).
|
||
4. `Surface.commitPending()` runs:
|
||
- Commit `subsurfaceChildren` order + apply pendingPosition → cascade
|
||
`onParentCommit`.
|
||
- Calculate derived pending state (buffer transform matrix, surface size,
|
||
pixman region).
|
||
- Release old buffer if replaced.
|
||
- `mergeSurfaceState(state, pendingState)` — atomic swap.
|
||
- `damaged = true`; clear pending damage + frame callbacks.
|
||
|
||
## Damage tracking
|
||
|
||
`damageRects` (post-transform) and `bufferDamageRects` (buffer-space) are
|
||
accumulated correctly on the surface state via pixman-WASM region math. The
|
||
damage is tracked faithfully — but **vestigial at the GL level**. The
|
||
renderer doesn't consume it for scissoring — it repaints every visible view
|
||
every RAF. The only damage gate is the boolean `surface.damaged`, used to
|
||
decide whether a streaming buffer's frame needs re-upload + release.
|
||
|
||
`RenderState.visibleSceneRegion` (pixman) is used only to decide if a
|
||
`RenderState` exists at all (i.e. whether the view is on this output), not
|
||
for per-rect clipping.
|
||
|
||
## Compositing pipeline
|
||
|
||
The renderer is **WebGL1 only**, no OffscreenCanvas, no WebGL2. The
|
||
context is acquired with `getContext('webgl', {antialias:false, depth:false,
|
||
alpha:true, premultipliedAlpha:false, preserveDrawingBuffer:false,
|
||
desynchronized:true})`. FBOs are used for in-GPU YUV→RGBA and
|
||
RGB+Alpha→RGBA conversion into per-view textures.
|
||
|
||
The compositor runs in **four phases** per render pass:
|
||
|
||
### PHASE 1 — pixel upload (per view per scene)
|
||
|
||
Coalesced via `setTimeout`-based cancellable microtask. For each view, for
|
||
each scene it's visible on:
|
||
|
||
1. `view.applyTransformations()` — compute `parent.transformation *
|
||
translate(position+offset)`.
|
||
2. For each `renderState` in `view.renderStates`:
|
||
`scene[bufferContents.mimeType](bufferContents, renderState)` — dispatch
|
||
to the right upload path:
|
||
- `'image/bitmap'` → `texture.setContent(imageBitmap, size)` — direct
|
||
`texImage2D` / `texSubImage2D` upload.
|
||
- `'video/h264'` → `YUVA2RGBA` or `RGBXandA2RGBA.convertInto(...)` —
|
||
FBO pass that uploads Y/U/V/alpha-Y as 4× LUMINANCE textures, runs the
|
||
YUVA→RGBA shader (BT.601 coefficients), writes into the renderState's
|
||
RGBA texture.
|
||
- `'image/png'` → `texture.setContent(bitmap, size); bitmap.close()`.
|
||
3. Collect `surface.state.frameCallbacks` → `Renderer.frameCallbacks`.
|
||
4. `surface.damaged = false`; `streamingBuffer.release()` (remote only).
|
||
|
||
### PHASE 2 — schedule render frame
|
||
|
||
`Renderer.render()` coalesces via a `setTimeout`-based cancellable
|
||
microtask, then schedules a single `requestAnimationFrame` for PHASE 3.
|
||
|
||
### PHASE 3 — draw (per scene, inside RAF)
|
||
|
||
For each scene (one per canvas/output):
|
||
|
||
1. `sceneShader.use()`; `updateSceneData(canvas)` — viewport, clear, BLEND,
|
||
projection.
|
||
2. For each view in the view stack (flattened from `topLevelViews`
|
||
recursively via `Renderer.updateViewStack`):
|
||
- If `renderState && view.mapped`:
|
||
- `sceneShader.updateViewData(view, renderState)` — bind texture,
|
||
`u_transform`, quad buffer.
|
||
- `sceneShader.draw()` — `gl.drawArrays(TRIANGLE_STRIP, 0, 6)` of a
|
||
textured quad per scene.
|
||
|
||
### PHASE 4 — fire frame callbacks
|
||
|
||
For each callback in `Renderer.frameCallbacks`:
|
||
`callback.done(time)` → `wl_callback.done(time)`; `wl_callback.destroy()`.
|
||
Then `session.flush()` to send any pending wire messages.
|
||
|
||
The `time` value is the `requestAnimationFrame` high-resolution timestamp,
|
||
so apps see real vsync-aligned frame timing.
|
||
|
||
## Multi-output
|
||
|
||
Multi-output = one Scene per canvas. Each view has one `RenderState` (and
|
||
one WebGL texture) per scene it's visible on. So a surface visible on two
|
||
outputs (e.g. mirroring) has two textures — one per canvas — and gets
|
||
uploaded to each on PHASE 1.
|
||
|
||
## Subsurfaces stacking
|
||
|
||
Subsurfaces stack via `subsurfaceChildren` arrays (committed + pending,
|
||
bottom→top). `placeAbove` / `placeBelow` reorder pending;
|
||
`Renderer.updateViewStack` flattens `topLevelViews` recursively.
|
||
|
||
**Important**: no parent-geometry clipping — subsurfaces can render outside
|
||
their parent. The only clip is the scene's canvas region, enforced implicitly
|
||
by NDC mapping.
|
||
|
||
## Cursor and DnD bypass GL entirely
|
||
|
||
Cursor and DnD bypass GL entirely. They draw to a separate singleton 2D
|
||
canvas and use CSS `cursor: url(dataURL)` or HTML5
|
||
`dataTransfer.setDragImage`. Their frame callbacks fire immediately with
|
||
`Date.now()`, not synced to RAF.
|
||
|
||
This is a deliberate optimization: cursor updates must be low-latency, and
|
||
going through the full GL pipeline would add a frame of latency.
|
||
|
||
## Decode pipeline (for H.264 frames)
|
||
|
||
Decoding is delegated to:
|
||
|
||
### Preferred path: WebCodecs `VideoDecoder`
|
||
|
||
At session startup, `Session.create()` probes
|
||
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
|
||
hardwareAcceleration:'prefer-software'})`. If supported, an `opaqueDecoder`
|
||
and `alphaDecoder` are created per surface (lazily, on first H.264 frame).
|
||
Encoded NALs are wrapped as `EncodedVideoChunk {type:key|delta, data:nal}`
|
||
and fed to the decoder. The output `VideoFrame` is uploaded directly to a
|
||
WebGL texture.
|
||
|
||
**Important**: hardware decode is **explicitly disabled** in `Session.ts`
|
||
with a FIXME comment "hardware decoding frame offset is wrong". Only
|
||
software decoding is used today.
|
||
|
||
### Fallback path: FFmpeg Wasm in Web Workers
|
||
|
||
If WebCodecs is unavailable, two persistent Workers (`h264NALDecoderOpaque`,
|
||
`h264NALDecoderAlpha`) are created at session startup. Each worker has one
|
||
`AVCodecContext` per surface, keyed by `surfaceH264DecodeId`. Encoded NALs
|
||
are transferred zero-copy into the worker via
|
||
`worker.postMessage(nal, [nal.buffer])`. Decoded I420 planes are transferred
|
||
back as `[y.buffer, u.buffer, v.buffer]` and uploaded as 4× LUMINANCE
|
||
textures.
|
||
|
||
## PNG path (both decoders)
|
||
|
||
The PNG path is identical in both branches:
|
||
`new Blob([opaque], {type:'image/png'})` →
|
||
`createImageBitmap(blob, sx, sy, w, h)` — the `sx/sy/w/h` crops the
|
||
encoder padding back to logical size. PNG is always used for surfaces ≤
|
||
256×256 pixels because H.264 I-frame overhead dominates at small sizes.
|
||
|
||
## WebGL shaders
|
||
|
||
Three shaders in `render/`:
|
||
|
||
- `YUVA2RGBAShader` — converts YUVA (Y, U, V, alpha-Y as 4 LUMINANCE
|
||
textures) to RGBA in a single FBO pass using BT.601 coefficients. Alpha
|
||
is recovered by running the alpha-Y plane through the same YUV→RGB matrix
|
||
and taking the R channel.
|
||
- `RGBandA2RGBAShader` — for the `DualPlaneRGBA*` WebCodecs outputs
|
||
(opaque + alpha planes composited into RGBA).
|
||
- `SceneShader` — final pass: draws every view's RGBA texture to the canvas
|
||
with `gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`. Note: expects premultiplied
|
||
alpha but the YUVA shader outputs straight alpha — a minor inconsistency
|
||
that works because Wayland surfaces rarely use partial alpha.
|
||
|
||
## ASCII pipeline diagram
|
||
|
||
```
|
||
CLIENT (Wayland app / web worker / compositor-proxy)
|
||
│ wl_surface.attach / damage / frame / commit
|
||
▼
|
||
Surface.commit()
|
||
│ serial = commitSerials.shift() (remote-only)
|
||
│ bufferContents = await bufferImpl.getContents(surface, serial)
|
||
│ role.onCommit(this)
|
||
▼
|
||
Surface.commitPending()
|
||
│ 1. commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit
|
||
│ 2. calculateDerivedPendingState (bufferTransform matrix, surface.size, pixmanRegion)
|
||
│ 3. release old buffer if replaced
|
||
│ 4. mergeSurfaceState(state, pendingState) ◄── atomic swap
|
||
│ 5. damaged = true; clear pending damage/frame callbacks
|
||
▼
|
||
Renderer.render()
|
||
│ coalesce via queueCancellableMicrotask (setTimeout)
|
||
▼
|
||
PHASE 1 — pixel upload (per view per scene):
|
||
view.applyTransformations() // parent.transformation * translate(position+offset)
|
||
for renderState in view.renderStates:
|
||
scene[bufferContents.mimeType](bufferContents, renderState)
|
||
'image/bitmap' → texture.setContent(imageBitmap, size) // texImage2D / texSubImage2D
|
||
'video/h264' → YUVA2RGBA / RGBXandA2RGBA.convertInto(...) // 4 LUMINANCE or 2 RGBA tex
|
||
→ FBO + YUVA2RGBAShader (BT.601) → renderState.texture
|
||
'image/png' → texture.setContent(bitmap, size); bitmap.close()
|
||
collect surface.state.frameCallbacks → Renderer.frameCallbacks
|
||
surface.damaged = false; streamingBuffer.release() (remote only)
|
||
▼
|
||
PHASE 2 — createRenderFrame().then(time => { // = requestAnimationFrame
|
||
PHASE 3 — for scene in scenes: // one per canvas/output
|
||
sceneShader.use(); updateSceneData(canvas) // viewport, clear, BLEND, projection
|
||
for view in viewStack:
|
||
if renderState && view.mapped:
|
||
sceneShader.updateViewData(view, renderState) // bind texture, u_transform, quad buffer
|
||
sceneShader.draw() // gl.drawArrays(TRIANGLE_STRIP, 0, 6)
|
||
PHASE 4 — for callback in frameCallbacks: callback.done(time)
|
||
→ wl_callback.done(time); wl_callback.destroy()
|
||
session.flush()
|
||
})
|
||
```
|
||
|
||
## The remote streaming buffer content flow
|
||
|
||
The remote streaming buffer content arrives in parallel on a separate FRAME
|
||
WebSocket channel:
|
||
|
||
1. `deliverContentToBufferStream` (in `RemoteAppLauncher.ts`) receives
|
||
incoming frame messages.
|
||
2. `BufferStream.onBufferContents` parses the encoded frame envelope.
|
||
3. `createEncodedFrame` (in `EncodedFrame.ts`) parses `{contentSerial,
|
||
mimeType, size, encodedSize, opaque, alpha?}`.
|
||
4. Resolves the `onFrameAvailable(serial)` promise that
|
||
`StreamingBuffer.getContents` is awaiting.
|
||
5. The decoded `DecodedFrame` is then uploaded into the view's
|
||
`RenderState.texture` via the appropriate FBO+shader conversion.
|
||
|
||
## Engineering observations
|
||
|
||
1. **Vestigial damage tracking** — `damageRects` are accumulated correctly
|
||
but the renderer doesn't use them for scissoring. Could be a future
|
||
optimization for partial repaints.
|
||
2. **Multi-output texture cost** — each view has one texture per scene,
|
||
so 4 outputs means 4× texture memory for every surface.
|
||
3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion
|
||
to the browser's BMP decoder. Worth knowing if you need to support
|
||
other SHM formats.
|
||
4. **Commit-serial mechanism** — the out-of-band opcode 7 lets the
|
||
streaming buffer wait for the right encoded frame, ensuring wire-order
|
||
semantics even with async decode.
|
||
5. **Encoder feedback telemetry** — `EncoderFeedback.ts` measures client
|
||
decode duration, but the adaptation loop is wired but not consumed (a
|
||
missed opportunity for adaptive bitrate).
|
||
6. **Context-loss audit gap** — WebGL context loss is not audited in the
|
||
test suite; in production, you should handle `webglcontextlost` and
|
||
`webglcontextrestored` events.
|
||
7. **`queueCancellableMicrotask` misnomer** — the function is actually a
|
||
`setTimeout`-based coalescer, not a microtask. Naming is misleading.
|
||
8. **Frame-callback coalescing** — multiple `wl_surface.frame` requests
|
||
within one render pass fire on the same RAF tick. Correct Wayland
|
||
behavior, but worth knowing.
|
||
9. **Cursor/DnD bypass** — they bypass the GL pipeline entirely for
|
||
latency. The cursor is rendered as a CSS `cursor: url(dataURL)`.
|
||
10. **Subsurface sync cascade** — three-layer mechanism (`cachedState`,
|
||
`commitCache`, `effectiveSync`) is correct but subtle. Easy to break
|
||
in maintenance.
|
||
|
||
## File reference
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `packages/compositor/src/Surface.ts` | `wl_surface` implementation, double-buffered state, commit semantics |
|
||
| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction |
|
||
| `packages/compositor/src/SurfaceRole.ts` | Role abstraction (SurfaceRole interface) |
|
||
| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
|
||
| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global |
|
||
| `packages/compositor/src/View.ts` | Positioned surface for compositing |
|
||
| `packages/compositor/src/Output.ts` | `wl_output`, WebGL canvas binding |
|
||
| `packages/compositor/src/Shm.ts` | `wl_shm` / shm pool, BMP-header hack |
|
||
| `packages/compositor/src/BufferContents.ts` | Buffer abstraction (with mimeType) |
|
||
| `packages/compositor/src/BufferImplementation.ts` | Buffer implementation interface |
|
||
| `packages/compositor/src/ImageBitmapBuffer.ts` | ImageBitmap-backed buffer |
|
||
| `packages/compositor/src/Region.ts` | `wl_region`, pixman-WASM damage regions |
|
||
| `packages/compositor/src/Callback.ts` | Frame callback (wl_callback) |
|
||
| `packages/compositor/src/Loop.ts` | Compositor main loop, RAF integration |
|
||
| `packages/compositor/src/Compositor.ts` | `wl_compositor` global |
|
||
| `packages/compositor/src/Session.ts` | Top-level browser session, decoder setup |
|
||
| `packages/compositor/src/Globals.ts` | Wayland globals registry |
|
||
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
|
||
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
|
||
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
|
||
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
|
||
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
|
||
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
|
||
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
|
||
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
|
||
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
|
||
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
|
||
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
|
||
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
|
||
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
|
||
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||
| `packages/compositor/src/remote/buffer-decoder.ts` | Decoder interface |
|
||
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
|
||
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||
| `packages/compositor/src/remote/EncoderFeedback.ts` | Receives 4-byte feedback messages |
|
||
| `packages/compositor/src/remote/Channel.ts` | ARQChannel + SimpleChannel |
|
||
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
|
||
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Sets up channels per app, frame delivery |
|
||
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer events |
|
||
| `packages/compositor/src/browser/dnd.ts` | DnD bridge |
|
||
| `packages/compositor/src/browser/selection.ts` | Clipboard bridge |
|
||
| `packages/compositor/src/browser/capabilities.ts` | Browser feature detection |
|
||
| `packages/compositor/src/browser/input.ts` | DOM keyboard → wl_keyboard events |
|
||
| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
|
||
| `packages/compositor/src/math/Mat4.ts` | 4×4 matrix math |
|
||
| `packages/compositor/src/math/Rect.ts` | Rectangle math |
|
||
| `packages/compositor/src/polyfill/createImageBitmap.js` | ImageBitmap polyfill |
|