one-os-dev-wayland-html5-compositor-greenfield-analysis/03-html5-rendering.md

03 — HTML5 Rendering Pipeline

This chapter covers Greenfield's browser-side rendering pipeline: how incoming Wayland buffers (SHM or encoded) end up as pixels on a browser canvas/WebGL context, including damage tracking, commit semantics, surface state stacking, and the compositing loop. The browser side lives in packages/compositor/src/.

Three buffer types, one BufferImplementation interface

All incoming buffers, regardless of origin, end up as a BufferContents<T> with a mimeType that dispatches to the right upload path. There are three concrete buffer types:

1. SHM buffer (LazyImageBitmapBuffer)

Comes from wl_shm_pool — the standard Wayland shared-memory buffer mechanism. Browser-side implementation lives in Shm.ts. The raw pixel data arrives as a transferable ArrayBufferView (transferred zero-copy via MessagePort from a browser-iframe web app, or fetched via the FD endpoint WebSocket from the proxy). Greenfield wraps the bytes as an ImageData / BMP-Blob and decodes them via createImageBitmap() to produce an ImageBitmap ready for WebGL texImage2D.

Notable hack: for xrgb8888 SHM buffers, Greenfield prepends a BMP header in front of the raw pixel data so the browser's createImageBitmap decodes it natively, avoiding a per-pixel JS loop. This is a clever trick to offload pixel conversion to the browser's BMP decoder.

2. Web bitmap buffer (ImageBitmapBuffer via web_bitmapbuf_factory)

Comes from Greenfield's custom web-bitmapbuf-unstable-v1 protocol extension. Lets browser JS apps hand a GPU-backed ImageBitmap directly to the compositor as a wl_buffer — bypassing SHM copy, encoding, and dmabuf import. The fd arg is marshalled as a transferable, so the GPU texture never leaves VRAM.

3. Remote streaming buffer (StreamingBuffer)

Comes from the native proxy via the FRAME data channel. Receives encoded frames (H.264 or PNG) on a separate WebSocket per surface. Includes an out-of-band mechanism for BufferCreation and CommitSerial opcodes so the browser can match each encoded frame to the right wl_surface.commit serial.

Surface state stack

Surface.ts implements classic Wayland double-buffered state:

• Surface.state — the committed, currently-visible state.
• Surface.pendingState — the in-flight state, accumulated by client requests until commit is called.
• mergeSurfaceState — the atomic swap in commitPending().

Subsurfaces add a third layer of complexity:

• subsurfaceChildren arrays (committed + pending, bottom→top)
• placeAbove / placeBelow reorder pending
• effectiveSync getter cascades through the ancestor chain (if any ancestor is sync, this subsurface is effectively sync)
• Sync subsurfaces cache to cachedState and only flush when an ancestor commits via onParentCommit
• setDesync immediately calls onCommit to flush any pending cache

Commit semantics

When a Wayland client calls wl_surface.commit:

1. If the surface has a remote streaming buffer, Greenfield pulls the serial = commitSerials.shift() (an out-of-band opcode 7 mechanism that lets StreamingBuffer.getContents(surface, serial) wait for the right encoded frame).
2. bufferContents = await bufferImpl.getContents(surface, serial) — resolves when the encoded frame for this serial has arrived.
3. role.onCommit(this) — role-specific logic (XdgToplevel, XdgPopup, Subsurface, etc.).
4. Surface.commitPending() runs:
   • Commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit.
   • Calculate derived pending state (buffer transform matrix, surface size, pixman region).
   • Release old buffer if replaced.
   • mergeSurfaceState(state, pendingState) — atomic swap.
   • damaged = true; clear pending damage + frame callbacks.

Damage tracking

damageRects (post-transform) and bufferDamageRects (buffer-space) are accumulated correctly on the surface state via pixman-WASM region math. The damage is tracked faithfully — but vestigial at the GL level. The renderer doesn't consume it for scissoring — it repaints every visible view every RAF. The only damage gate is the boolean surface.damaged, used to decide whether a streaming buffer's frame needs re-upload + release.

RenderState.visibleSceneRegion (pixman) is used only to decide if a RenderState exists at all (i.e. whether the view is on this output), not for per-rect clipping.

Compositing pipeline

The renderer is WebGL1 only, no OffscreenCanvas, no WebGL2. The context is acquired with getContext('webgl', {antialias:false, depth:false, alpha:true, premultipliedAlpha:false, preserveDrawingBuffer:false, desynchronized:true}). FBOs are used for in-GPU YUV→RGBA and RGB+Alpha→RGBA conversion into per-view textures.

The compositor runs in four phases per render pass:

PHASE 1 — pixel upload (per view per scene)

Coalesced via setTimeout-based cancellable microtask. For each view, for each scene it's visible on:

1. view.applyTransformations() — compute parent.transformation * translate(position+offset).
2. For each renderState in view.renderStates: scene[bufferContents.mimeType](bufferContents, renderState) — dispatch to the right upload path:
   • 'image/bitmap' → texture.setContent(imageBitmap, size) — direct texImage2D / texSubImage2D upload.
   • 'video/h264' → YUVA2RGBA or RGBXandA2RGBA.convertInto(...) — FBO pass that uploads Y/U/V/alpha-Y as 4× LUMINANCE textures, runs the YUVA→RGBA shader (BT.601 coefficients), writes into the renderState's RGBA texture.
   • 'image/png' → texture.setContent(bitmap, size); bitmap.close().
3. Collect surface.state.frameCallbacks → Renderer.frameCallbacks.
4. surface.damaged = false; streamingBuffer.release() (remote only).

PHASE 2 — schedule render frame

Renderer.render() coalesces via a setTimeout-based cancellable microtask, then schedules a single requestAnimationFrame for PHASE 3.

PHASE 3 — draw (per scene, inside RAF)

For each scene (one per canvas/output):

1. sceneShader.use(); updateSceneData(canvas) — viewport, clear, BLEND, projection.
2. For each view in the view stack (flattened from topLevelViews recursively via Renderer.updateViewStack):
   • If renderState && view.mapped:
     • sceneShader.updateViewData(view, renderState) — bind texture, u_transform, quad buffer.
     • sceneShader.draw() — gl.drawArrays(TRIANGLE_STRIP, 0, 6) of a textured quad per scene.

PHASE 4 — fire frame callbacks

For each callback in Renderer.frameCallbacks: callback.done(time) → wl_callback.done(time); wl_callback.destroy(). Then session.flush() to send any pending wire messages.

The time value is the requestAnimationFrame high-resolution timestamp, so apps see real vsync-aligned frame timing.

Multi-output

Multi-output = one Scene per canvas. Each view has one RenderState (and one WebGL texture) per scene it's visible on. So a surface visible on two outputs (e.g. mirroring) has two textures — one per canvas — and gets uploaded to each on PHASE 1.

Subsurfaces stacking

Subsurfaces stack via subsurfaceChildren arrays (committed + pending, bottom→top). placeAbove / placeBelow reorder pending; Renderer.updateViewStack flattens topLevelViews recursively.

Important: no parent-geometry clipping — subsurfaces can render outside their parent. The only clip is the scene's canvas region, enforced implicitly by NDC mapping.

Cursor and DnD bypass GL entirely

Cursor and DnD bypass GL entirely. They draw to a separate singleton 2D canvas and use CSS cursor: url(dataURL) or HTML5 dataTransfer.setDragImage. Their frame callbacks fire immediately with Date.now(), not synced to RAF.

This is a deliberate optimization: cursor updates must be low-latency, and going through the full GL pipeline would add a frame of latency.

Decode pipeline (for H.264 frames)

Decoding is delegated to:

Preferred path: WebCodecs VideoDecoder

At session startup, Session.create() probes VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true, hardwareAcceleration:'prefer-software'}). If supported, an opaqueDecoder and alphaDecoder are created per surface (lazily, on first H.264 frame). Encoded NALs are wrapped as EncodedVideoChunk {type:key|delta, data:nal} and fed to the decoder. The output VideoFrame is uploaded directly to a WebGL texture.

Important: hardware decode is explicitly disabled in Session.ts with a FIXME comment "hardware decoding frame offset is wrong". Only software decoding is used today.

Fallback path: FFmpeg Wasm in Web Workers

If WebCodecs is unavailable, two persistent Workers (h264NALDecoderOpaque, h264NALDecoderAlpha) are created at session startup. Each worker has one AVCodecContext per surface, keyed by surfaceH264DecodeId. Encoded NALs are transferred zero-copy into the worker via worker.postMessage(nal, [nal.buffer]). Decoded I420 planes are transferred back as [y.buffer, u.buffer, v.buffer] and uploaded as 4× LUMINANCE textures.

PNG path (both decoders)

The PNG path is identical in both branches: new Blob([opaque], {type:'image/png'}) → createImageBitmap(blob, sx, sy, w, h) — the sx/sy/w/h crops the encoder padding back to logical size. PNG is always used for surfaces ≤ 256×256 pixels because H.264 I-frame overhead dominates at small sizes.

WebGL shaders

Three shaders in render/:

• YUVA2RGBAShader — converts YUVA (Y, U, V, alpha-Y as 4 LUMINANCE textures) to RGBA in a single FBO pass using BT.601 coefficients. Alpha is recovered by running the alpha-Y plane through the same YUV→RGB matrix and taking the R channel.
• RGBandA2RGBAShader — for the DualPlaneRGBA* WebCodecs outputs (opaque + alpha planes composited into RGBA).
• SceneShader — final pass: draws every view's RGBA texture to the canvas with gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA). Note: expects premultiplied alpha but the YUVA shader outputs straight alpha — a minor inconsistency that works because Wayland surfaces rarely use partial alpha.

ASCII pipeline diagram

CLIENT (Wayland app / web worker / compositor-proxy)
  │  wl_surface.attach / damage / frame / commit
  ▼
Surface.commit()
  │  serial = commitSerials.shift()         (remote-only)
  │  bufferContents = await bufferImpl.getContents(surface, serial)
  │  role.onCommit(this)
  ▼
Surface.commitPending()
  │  1. commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit
  │  2. calculateDerivedPendingState (bufferTransform matrix, surface.size, pixmanRegion)
  │  3. release old buffer if replaced
  │  4. mergeSurfaceState(state, pendingState)   ◄── atomic swap
  │  5. damaged = true; clear pending damage/frame callbacks
  ▼
Renderer.render()
  │  coalesce via queueCancellableMicrotask (setTimeout)
  ▼
PHASE 1 — pixel upload (per view per scene):
  view.applyTransformations()                // parent.transformation * translate(position+offset)
  for renderState in view.renderStates:
    scene[bufferContents.mimeType](bufferContents, renderState)
      'image/bitmap'   → texture.setContent(imageBitmap, size)          // texImage2D / texSubImage2D
      'video/h264'      → YUVA2RGBA / RGBXandA2RGBA.convertInto(...)    // 4 LUMINANCE or 2 RGBA tex
                          → FBO + YUVA2RGBAShader (BT.601) → renderState.texture
      'image/png'       → texture.setContent(bitmap, size); bitmap.close()
  collect surface.state.frameCallbacks → Renderer.frameCallbacks
  surface.damaged = false; streamingBuffer.release() (remote only)
  ▼
PHASE 2 — createRenderFrame().then(time => {        // = requestAnimationFrame
PHASE 3 — for scene in scenes:                     // one per canvas/output
    sceneShader.use(); updateSceneData(canvas)     // viewport, clear, BLEND, projection
    for view in viewStack:
      if renderState && view.mapped:
        sceneShader.updateViewData(view, renderState)   // bind texture, u_transform, quad buffer
        sceneShader.draw()                                // gl.drawArrays(TRIANGLE_STRIP, 0, 6)
PHASE 4 — for callback in frameCallbacks: callback.done(time)
          → wl_callback.done(time); wl_callback.destroy()
          session.flush()
})

The remote streaming buffer content flow

The remote streaming buffer content arrives in parallel on a separate FRAME WebSocket channel:

1. deliverContentToBufferStream (in RemoteAppLauncher.ts) receives incoming frame messages.
2. BufferStream.onBufferContents parses the encoded frame envelope.
3. createEncodedFrame (in EncodedFrame.ts) parses {contentSerial, mimeType, size, encodedSize, opaque, alpha?}.
4. Resolves the onFrameAvailable(serial) promise that StreamingBuffer.getContents is awaiting.
5. The decoded DecodedFrame is then uploaded into the view's RenderState.texture via the appropriate FBO+shader conversion.

Engineering observations

1. Vestigial damage tracking — damageRects are accumulated correctly but the renderer doesn't use them for scissoring. Could be a future optimization for partial repaints.
2. Multi-output texture cost — each view has one texture per scene, so 4 outputs means 4× texture memory for every surface.
3. SHM xrgb8888 BMP-header hack — cleverly offloads pixel conversion to the browser's BMP decoder. Worth knowing if you need to support other SHM formats.
4. Commit-serial mechanism — the out-of-band opcode 7 lets the streaming buffer wait for the right encoded frame, ensuring wire-order semantics even with async decode.
5. Encoder feedback telemetry — EncoderFeedback.ts measures client decode duration, but the adaptation loop is wired but not consumed (a missed opportunity for adaptive bitrate).
6. Context-loss audit gap — WebGL context loss is not audited in the test suite; in production, you should handle webglcontextlost and webglcontextrestored events.
7. queueCancellableMicrotask misnomer — the function is actually a setTimeout-based coalescer, not a microtask. Naming is misleading.
8. Frame-callback coalescing — multiple wl_surface.frame requests within one render pass fire on the same RAF tick. Correct Wayland behavior, but worth knowing.
9. Cursor/DnD bypass — they bypass the GL pipeline entirely for latency. The cursor is rendered as a CSS cursor: url(dataURL).
10. Subsurface sync cascade — three-layer mechanism (cachedState, commitCache, effectiveSync) is correct but subtle. Easy to break in maintenance.

File reference

File	Role
packages/compositor/src/Surface.ts	wl_surface implementation, double-buffered state, commit semantics
packages/compositor/src/SurfaceChild.ts	Surface tree child abstraction
packages/compositor/src/SurfaceRole.ts	Role abstraction (SurfaceRole interface)
packages/compositor/src/Subsurface.ts	wl_subsurface role, sync/desync mode
packages/compositor/src/Subcompositor.ts	wl_subcompositor global
packages/compositor/src/View.ts	Positioned surface for compositing
packages/compositor/src/Output.ts	wl_output, WebGL canvas binding
packages/compositor/src/Shm.ts	wl_shm / shm pool, BMP-header hack
packages/compositor/src/BufferContents.ts	Buffer abstraction (with mimeType)
packages/compositor/src/BufferImplementation.ts	Buffer implementation interface
packages/compositor/src/ImageBitmapBuffer.ts	ImageBitmap-backed buffer
packages/compositor/src/Region.ts	wl_region, pixman-WASM damage regions
packages/compositor/src/Callback.ts	Frame callback (wl_callback)
packages/compositor/src/Loop.ts	Compositor main loop, RAF integration
packages/compositor/src/Compositor.ts	wl_compositor global
packages/compositor/src/Session.ts	Top-level browser session, decoder setup
packages/compositor/src/Globals.ts	Wayland globals registry
packages/compositor/src/render/Renderer.ts	4-phase render pipeline
packages/compositor/src/render/Scene.ts	One per output, manages view stack + textures
packages/compositor/src/render/SceneShader.ts	Final pass: textured quads to canvas
packages/compositor/src/render/Texture.ts	WebGL texture wrapper
packages/compositor/src/render/RenderState.ts	Per-view-per-scene texture + transform
packages/compositor/src/render/Program.ts	WebGL program wrapper
packages/compositor/src/render/ShaderCompiler.ts	Shader compile/link helper
packages/compositor/src/render/ShaderSources.ts	GLSL source strings
packages/compositor/src/render/YUVA2RGBA.ts	YUVA→RGBA FBO pass
packages/compositor/src/render/YUVA2RGBAShader.ts	BT.601 YUVA→RGBA shader
packages/compositor/src/render/YUV2RGBShader.ts	YUV→RGB shader (no alpha)
packages/compositor/src/render/RGBXandA2RGBA.ts	RGB+Alpha → RGBA FBO pass (WebCodecs output)
packages/compositor/src/render/RGBandA2RGBAShader.ts	Shader for above
packages/compositor/src/remote/StreamingBuffer.ts	Per-surface encoded-frame buffer
packages/compositor/src/remote/BufferStream.ts	Manages serial → frame promise resolution
packages/compositor/src/remote/EncodedFrame.ts	Wire format parser for encoded frames
packages/compositor/src/remote/buffer-decoder.ts	Decoder interface
packages/compositor/src/remote/webcodec-buffer-decoder.ts	WebCodecs VideoDecoder path
packages/compositor/src/remote/wasm-buffer-decoder.ts	FFmpeg Wasm fallback path
packages/compositor/src/remote/H264NALDecoder.worker.ts	Worker entry, runs FFmpeg Wasm decoder
packages/compositor/src/remote/EncoderFeedback.ts	Receives 4-byte feedback messages
packages/compositor/src/remote/Channel.ts	ARQChannel + SimpleChannel
packages/compositor/src/remote/RemoteOutOfBandChannel.ts	OOB opcode dispatch
packages/compositor/src/remote/RemoteAppLauncher.ts	Sets up channels per app, frame delivery
packages/compositor/src/remote/RemoteInputOutput.ts	FD-over-WebSocket for native apps
packages/compositor/src/browser/pointer.ts	DOM pointer → wl_pointer events
packages/compositor/src/browser/dnd.ts	DnD bridge
packages/compositor/src/browser/selection.ts	Clipboard bridge
packages/compositor/src/browser/capabilities.ts	Browser feature detection
packages/compositor/src/browser/input.ts	DOM keyboard → wl_keyboard events
packages/compositor/src/desktop/Desktop.ts	Floating vs AlwaysFullScreen modes
packages/compositor/src/math/Mat4.ts	4×4 matrix math
packages/compositor/src/math/Rect.ts	Rectangle math
packages/compositor/src/polyfill/createImageBitmap.js	ImageBitmap polyfill