# 03 — HTML5 Rendering Pipeline

This chapter covers Greenfield's browser-side rendering pipeline: how
incoming Wayland buffers (SHM or encoded) end up as pixels on a browser
canvas/WebGL context, including damage tracking, commit semantics, surface
state stacking, and the compositing loop. The browser side lives in
`packages/compositor/src/`.

## Three buffer types, one BufferImplementation interface

All incoming buffers, regardless of origin, end up as a `BufferContents<T>`
with a `mimeType` that dispatches to the right upload path. There are three
concrete buffer types:

### 1. SHM buffer (`LazyImageBitmapBuffer`)

Comes from `wl_shm_pool` — the standard Wayland shared-memory buffer
mechanism. Browser-side implementation lives in `Shm.ts`. The raw pixel data
arrives as a transferable `ArrayBufferView` (transferred zero-copy via
`MessagePort` from a browser-iframe web app, or fetched via the FD endpoint
WebSocket from the proxy). Greenfield wraps the bytes as an `ImageData` /
BMP-Blob and decodes them via `createImageBitmap()` to produce an
`ImageBitmap` ready for WebGL `texImage2D`.

**Notable hack**: for `xrgb8888` SHM buffers, Greenfield prepends a BMP
header in front of the raw pixel data so the browser's `createImageBitmap`
decodes it natively, avoiding a per-pixel JS loop. This is a clever trick to
offload pixel conversion to the browser's BMP decoder.

### 2. Web bitmap buffer (`ImageBitmapBuffer` via `web_bitmapbuf_factory`)

Comes from Greenfield's custom `web-bitmapbuf-unstable-v1` protocol
extension. Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to
the compositor as a `wl_buffer` — bypassing SHM copy, encoding, and dmabuf
import. The `fd` arg is marshalled as a transferable, so the GPU texture
never leaves VRAM.

### 3. Remote streaming buffer (`StreamingBuffer`)

Comes from the native proxy via the FRAME data channel. Receives encoded
frames (H.264 or PNG) on a separate WebSocket per surface. Includes an
out-of-band mechanism for `BufferCreation` and `CommitSerial` opcodes so
the browser can match each encoded frame to the right `wl_surface.commit`
serial.

## Surface state stack

`Surface.ts` implements classic Wayland double-buffered state:

- `Surface.state` — the committed, currently-visible state.
- `Surface.pendingState` — the in-flight state, accumulated by client
  requests until `commit` is called.
- `mergeSurfaceState` — the atomic swap in `commitPending()`.

Subsurfaces add a third layer of complexity:

- `subsurfaceChildren` arrays (committed + pending, bottom→top)
- `placeAbove` / `placeBelow` reorder pending
- `effectiveSync` getter cascades through the ancestor chain (if any
  ancestor is sync, this subsurface is effectively sync)
- Sync subsurfaces cache to `cachedState` and only flush when an ancestor
  commits via `onParentCommit`
- `setDesync` immediately calls `onCommit` to flush any pending cache

## Commit semantics

When a Wayland client calls `wl_surface.commit`:

1. If the surface has a remote streaming buffer, Greenfield pulls the
   `serial = commitSerials.shift()` (an out-of-band opcode 7 mechanism that
   lets `StreamingBuffer.getContents(surface, serial)` wait for the right
   encoded frame).
2. `bufferContents = await bufferImpl.getContents(surface, serial)` —
   resolves when the encoded frame for this serial has arrived.
3. `role.onCommit(this)` — role-specific logic (XdgToplevel, XdgPopup,
   Subsurface, etc.).
4. `Surface.commitPending()` runs:
   - Commit `subsurfaceChildren` order + apply pendingPosition → cascade
     `onParentCommit`.
   - Calculate derived pending state (buffer transform matrix, surface size,
     pixman region).
   - Release old buffer if replaced.
   - `mergeSurfaceState(state, pendingState)` — atomic swap.
   - `damaged = true`; clear pending damage + frame callbacks.

## Damage tracking

`damageRects` (post-transform) and `bufferDamageRects` (buffer-space) are
accumulated correctly on the surface state via pixman-WASM region math. The
damage is tracked faithfully — but **vestigial at the GL level**. The
renderer doesn't consume it for scissoring — it repaints every visible view
every RAF. The only damage gate is the boolean `surface.damaged`, used to
decide whether a streaming buffer's frame needs re-upload + release.

`RenderState.visibleSceneRegion` (pixman) is used only to decide if a
`RenderState` exists at all (i.e. whether the view is on this output), not
for per-rect clipping.

## Compositing pipeline

The renderer is **WebGL1 only**, no OffscreenCanvas, no WebGL2. The
context is acquired with `getContext('webgl', {antialias:false, depth:false,
alpha:true, premultipliedAlpha:false, preserveDrawingBuffer:false,
desynchronized:true})`. FBOs are used for in-GPU YUV→RGBA and
RGB+Alpha→RGBA conversion into per-view textures.

The compositor runs in **four phases** per render pass:

### PHASE 1 — pixel upload (per view per scene)

Coalesced via `setTimeout`-based cancellable microtask. For each view, for
each scene it's visible on:

1. `view.applyTransformations()` — compute `parent.transformation *
   translate(position+offset)`.
2. For each `renderState` in `view.renderStates`:
   `scene[bufferContents.mimeType](bufferContents, renderState)` — dispatch
   to the right upload path:
   - `'image/bitmap'` → `texture.setContent(imageBitmap, size)` — direct
     `texImage2D` / `texSubImage2D` upload.
   - `'video/h264'` → `YUVA2RGBA` or `RGBXandA2RGBA.convertInto(...)` —
     FBO pass that uploads Y/U/V/alpha-Y as 4× LUMINANCE textures, runs the
     YUVA→RGBA shader (BT.601 coefficients), writes into the renderState's
     RGBA texture.
   - `'image/png'` → `texture.setContent(bitmap, size); bitmap.close()`.
3. Collect `surface.state.frameCallbacks` → `Renderer.frameCallbacks`.
4. `surface.damaged = false`; `streamingBuffer.release()` (remote only).

### PHASE 2 — schedule render frame

`Renderer.render()` coalesces via a `setTimeout`-based cancellable
microtask, then schedules a single `requestAnimationFrame` for PHASE 3.

### PHASE 3 — draw (per scene, inside RAF)

For each scene (one per canvas/output):

1. `sceneShader.use()`; `updateSceneData(canvas)` — viewport, clear, BLEND,
   projection.
2. For each view in the view stack (flattened from `topLevelViews`
   recursively via `Renderer.updateViewStack`):
   - If `renderState && view.mapped`:
     - `sceneShader.updateViewData(view, renderState)` — bind texture,
       `u_transform`, quad buffer.
     - `sceneShader.draw()` — `gl.drawArrays(TRIANGLE_STRIP, 0, 6)` of a
       textured quad per scene.

### PHASE 4 — fire frame callbacks

For each callback in `Renderer.frameCallbacks`:
`callback.done(time)` → `wl_callback.done(time)`; `wl_callback.destroy()`.
Then `session.flush()` to send any pending wire messages.

The `time` value is the `requestAnimationFrame` high-resolution timestamp,
so apps see real vsync-aligned frame timing.

## Multi-output

Multi-output = one Scene per canvas. Each view has one `RenderState` (and
one WebGL texture) per scene it's visible on. So a surface visible on two
outputs (e.g. mirroring) has two textures — one per canvas — and gets
uploaded to each on PHASE 1.

## Subsurfaces stacking

Subsurfaces stack via `subsurfaceChildren` arrays (committed + pending,
bottom→top). `placeAbove` / `placeBelow` reorder pending;
`Renderer.updateViewStack` flattens `topLevelViews` recursively.

**Important**: no parent-geometry clipping — subsurfaces can render outside
their parent. The only clip is the scene's canvas region, enforced implicitly
by NDC mapping.

## Cursor and DnD bypass GL entirely

Cursor and DnD bypass GL entirely. They draw to a separate singleton 2D
canvas and use CSS `cursor: url(dataURL)` or HTML5
`dataTransfer.setDragImage`. Their frame callbacks fire immediately with
`Date.now()`, not synced to RAF.

This is a deliberate optimization: cursor updates must be low-latency, and
going through the full GL pipeline would add a frame of latency.

## Decode pipeline (for H.264 frames)

Decoding is delegated to:

### Preferred path: WebCodecs `VideoDecoder`

At session startup, `Session.create()` probes
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
hardwareAcceleration:'prefer-software'})`. If supported, an `opaqueDecoder`
and `alphaDecoder` are created per surface (lazily, on first H.264 frame).
Encoded NALs are wrapped as `EncodedVideoChunk {type:key|delta, data:nal}`
and fed to the decoder. The output `VideoFrame` is uploaded directly to a
WebGL texture.

**Important**: hardware decode is **explicitly disabled** in `Session.ts`
with a FIXME comment "hardware decoding frame offset is wrong". Only
software decoding is used today.

### Fallback path: FFmpeg Wasm in Web Workers

If WebCodecs is unavailable, two persistent Workers (`h264NALDecoderOpaque`,
`h264NALDecoderAlpha`) are created at session startup. Each worker has one
`AVCodecContext` per surface, keyed by `surfaceH264DecodeId`. Encoded NALs
are transferred zero-copy into the worker via
`worker.postMessage(nal, [nal.buffer])`. Decoded I420 planes are transferred
back as `[y.buffer, u.buffer, v.buffer]` and uploaded as 4× LUMINANCE
textures.

## PNG path (both decoders)

The PNG path is identical in both branches:
`new Blob([opaque], {type:'image/png'})` →
`createImageBitmap(blob, sx, sy, w, h)` — the `sx/sy/w/h` crops the
encoder padding back to logical size. PNG is always used for surfaces ≤
256×256 pixels because H.264 I-frame overhead dominates at small sizes.

## WebGL shaders

Three shaders in `render/`:

- `YUVA2RGBAShader` — converts YUVA (Y, U, V, alpha-Y as 4 LUMINANCE
  textures) to RGBA in a single FBO pass using BT.601 coefficients. Alpha
  is recovered by running the alpha-Y plane through the same YUV→RGB matrix
  and taking the R channel.
- `RGBandA2RGBAShader` — for the `DualPlaneRGBA*` WebCodecs outputs
  (opaque + alpha planes composited into RGBA).
- `SceneShader` — final pass: draws every view's RGBA texture to the canvas
  with `gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`. Note: expects premultiplied
  alpha but the YUVA shader outputs straight alpha — a minor inconsistency
  that works because Wayland surfaces rarely use partial alpha.

## ASCII pipeline diagram

```
CLIENT (Wayland app / web worker / compositor-proxy)
  │  wl_surface.attach / damage / frame / commit
  ▼
Surface.commit()
  │  serial = commitSerials.shift()         (remote-only)
  │  bufferContents = await bufferImpl.getContents(surface, serial)
  │  role.onCommit(this)
  ▼
Surface.commitPending()
  │  1. commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit
  │  2. calculateDerivedPendingState (bufferTransform matrix, surface.size, pixmanRegion)
  │  3. release old buffer if replaced
  │  4. mergeSurfaceState(state, pendingState)   ◄── atomic swap
  │  5. damaged = true; clear pending damage/frame callbacks
  ▼
Renderer.render()
  │  coalesce via queueCancellableMicrotask (setTimeout)
  ▼
PHASE 1 — pixel upload (per view per scene):
  view.applyTransformations()                // parent.transformation * translate(position+offset)
  for renderState in view.renderStates:
    scene[bufferContents.mimeType](bufferContents, renderState)
      'image/bitmap'   → texture.setContent(imageBitmap, size)          // texImage2D / texSubImage2D
      'video/h264'      → YUVA2RGBA / RGBXandA2RGBA.convertInto(...)    // 4 LUMINANCE or 2 RGBA tex
                          → FBO + YUVA2RGBAShader (BT.601) → renderState.texture
      'image/png'       → texture.setContent(bitmap, size); bitmap.close()
  collect surface.state.frameCallbacks → Renderer.frameCallbacks
  surface.damaged = false; streamingBuffer.release() (remote only)
  ▼
PHASE 2 — createRenderFrame().then(time => {        // = requestAnimationFrame
PHASE 3 — for scene in scenes:                     // one per canvas/output
    sceneShader.use(); updateSceneData(canvas)     // viewport, clear, BLEND, projection
    for view in viewStack:
      if renderState && view.mapped:
        sceneShader.updateViewData(view, renderState)   // bind texture, u_transform, quad buffer
        sceneShader.draw()                                // gl.drawArrays(TRIANGLE_STRIP, 0, 6)
PHASE 4 — for callback in frameCallbacks: callback.done(time)
          → wl_callback.done(time); wl_callback.destroy()
          session.flush()
})
```

## The remote streaming buffer content flow

The remote streaming buffer content arrives in parallel on a separate FRAME
WebSocket channel:

1. `deliverContentToBufferStream` (in `RemoteAppLauncher.ts`) receives
   incoming frame messages.
2. `BufferStream.onBufferContents` parses the encoded frame envelope.
3. `createEncodedFrame` (in `EncodedFrame.ts`) parses `{contentSerial,
   mimeType, size, encodedSize, opaque, alpha?}`.
4. Resolves the `onFrameAvailable(serial)` promise that
   `StreamingBuffer.getContents` is awaiting.
5. The decoded `DecodedFrame` is then uploaded into the view's
   `RenderState.texture` via the appropriate FBO+shader conversion.

## Engineering observations

1. **Vestigial damage tracking** — `damageRects` are accumulated correctly
   but the renderer doesn't use them for scissoring. Could be a future
   optimization for partial repaints.
2. **Multi-output texture cost** — each view has one texture per scene,
   so 4 outputs means 4× texture memory for every surface.
3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion
   to the browser's BMP decoder. Worth knowing if you need to support
   other SHM formats.
4. **Commit-serial mechanism** — the out-of-band opcode 7 lets the
   streaming buffer wait for the right encoded frame, ensuring wire-order
   semantics even with async decode.
5. **Encoder feedback telemetry** — `EncoderFeedback.ts` measures client
   decode duration, but the adaptation loop is wired but not consumed (a
   missed opportunity for adaptive bitrate).
6. **Context-loss audit gap** — WebGL context loss is not audited in the
   test suite; in production, you should handle `webglcontextlost` and
   `webglcontextrestored` events.
7. **`queueCancellableMicrotask` misnomer** — the function is actually a
   `setTimeout`-based coalescer, not a microtask. Naming is misleading.
8. **Frame-callback coalescing** — multiple `wl_surface.frame` requests
   within one render pass fire on the same RAF tick. Correct Wayland
   behavior, but worth knowing.
9. **Cursor/DnD bypass** — they bypass the GL pipeline entirely for
   latency. The cursor is rendered as a CSS `cursor: url(dataURL)`.
10. **Subsurface sync cascade** — three-layer mechanism (`cachedState`,
    `commitCache`, `effectiveSync`) is correct but subtle. Easy to break
    in maintenance.

## File reference

| File | Role |
|---|---|
| `packages/compositor/src/Surface.ts` | `wl_surface` implementation, double-buffered state, commit semantics |
| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction |
| `packages/compositor/src/SurfaceRole.ts` | Role abstraction (SurfaceRole interface) |
| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global |
| `packages/compositor/src/View.ts` | Positioned surface for compositing |
| `packages/compositor/src/Output.ts` | `wl_output`, WebGL canvas binding |
| `packages/compositor/src/Shm.ts` | `wl_shm` / shm pool, BMP-header hack |
| `packages/compositor/src/BufferContents.ts` | Buffer abstraction (with mimeType) |
| `packages/compositor/src/BufferImplementation.ts` | Buffer implementation interface |
| `packages/compositor/src/ImageBitmapBuffer.ts` | ImageBitmap-backed buffer |
| `packages/compositor/src/Region.ts` | `wl_region`, pixman-WASM damage regions |
| `packages/compositor/src/Callback.ts` | Frame callback (wl_callback) |
| `packages/compositor/src/Loop.ts` | Compositor main loop, RAF integration |
| `packages/compositor/src/Compositor.ts` | `wl_compositor` global |
| `packages/compositor/src/Session.ts` | Top-level browser session, decoder setup |
| `packages/compositor/src/Globals.ts` | Wayland globals registry |
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/buffer-decoder.ts` | Decoder interface |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Receives 4-byte feedback messages |
| `packages/compositor/src/remote/Channel.ts` | ARQChannel + SimpleChannel |
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Sets up channels per app, frame delivery |
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer events |
| `packages/compositor/src/browser/dnd.ts` | DnD bridge |
| `packages/compositor/src/browser/selection.ts` | Clipboard bridge |
| `packages/compositor/src/browser/capabilities.ts` | Browser feature detection |
| `packages/compositor/src/browser/input.ts` | DOM keyboard → wl_keyboard events |
| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
| `packages/compositor/src/math/Mat4.ts` | 4×4 matrix math |
| `packages/compositor/src/math/Rect.ts` | Rectangle math |
| `packages/compositor/src/polyfill/createImageBitmap.js` | ImageBitmap polyfill |