# 09 — Hardware Acceleration, Encoding, WASM, GPU Textures

This chapter covers Greenfield's hardware acceleration / encoding / WASM
pipeline: how native buffers get encoded on the proxy side, transported,
decoded in the browser, and turned into WebGL textures or canvas images.

## What Greenfield actually ships

### Two Wasm libraries under `libs/`

**`compositor-wasm`** builds **pixman 0.42.2 + xkbcommon 1.5.0**
(Emscripten 3.1.46, `-msimd128 -O3 -flto`). Used by the browser for
**region math** (damage tracking) and **keyboard layout** — *not* for
pixel format conversion or H.264 decode.

**`compositor-ffmpeg-h264`** is a **stripped-down FFmpeg n5.1.2**
(`--disable-everything --enable-decoder=h264`) compiled to a 1.7 MB
single-file Wasm worker. A 30-line `decoder.c` wrapper exposes
`create_codec_context` / `decode` / `close_frame` / `destroy_codec_context`.
Decodes to **I420 (YUV 4:2:0 planar)** and returns plane pointers; planes
are `postMessage`'d back to the main thread as **transferable
`ArrayBuffer`s**. No threads (single-threaded, SIMD only).

### Native encoder addon (`packages/compositor-proxy/native/encoding/`)

A **GStreamer wrapper** exposed via Node-API. The TS side (`Encoder.ts`)
calls `appEndpointNative.encodeFrame(encoder, bufferId, contentSerial,
creationSerial)`. The C side (`node_encoder.c`) sniffs the wl_resource:

- `zwp_linux_dmabuf_v1` → DMA path
- Legacy `wl_drm` → DMA path
- `wl_shm_buffer` → SHM path

For DMA-BUFs the proxy wraps the dma-buf as an `EGLImage` and then as a
`GstGLMemoryEGL`, so pixels **stay on GPU** through the color-convert
stages.

### Threading

Encoding runs on a dedicated GLib main-loop thread fed by an
`eventfd`-signalled `GAsyncQueue`. Output returns to Node via a
`napi_threadsafe_function` with **zero-copy** external Buffers (finalised
back to C on GC).

## Encoder selection (per-frame, deterministic)

`gst_frame_encoder.c::frame_encoder_descriptions[]` declares **four**
pipelines:

| name | element | when used | split_alpha | coded-size multiple |
|---|---|---|---|---|
| `x264` | `x264enc` (software, default) | >256×256 + ARGB/XRGB/dmabuf | true (parallel alpha H.264 stream) | 128 |
| `nvh264` | `nvh264enc` (Nvidia NVENC) | same, `--encoder=nvh264` | true | 128 |
| `vaapih264` | `vaapih264enc` (Intel/AMD VA-API, "experimental") | same, `--encoder=vaapih264` | true | 128 |
| `png` | `pngenc` (fallback) | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 |

**Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264
encoder. YUV420/NV12/NV21/Y444 SHM formats are *not* supported on the
H.264 path and only by PNG at small sizes — effectively unsupported for
video-class content.

### Canonical H.264 pipeline

```
appsrc → glupload → glcolorconvert → glshader(scale→coded_size) →
glcolorconvert → video/x-raw(memory:GLMemory),format=NV12 → gldownload →
queue → x264enc tune=zerolatency bitrate=12000 vbv-buf-capacity=160
                   bframes=0 rc-lookahead=0 sliced-threads byte-stream=true →
video/x-h264,profile=high,stream-format=byte-stream,alignment=au → appsink
```

The `glshader` element does two jobs:

1. Pad up to the encoder's macroblock multiple (128 for H.264, 16 for PNG)
   using a `u_transformation` matrix.
2. For the **alpha pipeline**, run `vec4(pix.a, pix.a, pix.a, 0.)` so
   alpha is encoded as a grayscale H.264 stream — H.264 has no alpha, so
   Greenfield ships two parallel AU streams.

Bitrates: opaque 12000 / alpha 1200 (x264), 12000 / 1200 (nvh264).

## Wire format

Single contiguous blob, little-endian:

```
struct packed props {
    u32 buffer_id, creation_serial, content_serial,
    u32 frame_encoding_type (0=h264, 1=png),
    u32 width, height, coded_width, coded_height
}
u32 opaque_length
u8  opaque[opaque_length]    // H.264 AU or PNG blob
u32 alpha_length             // 0 if no alpha split
u8  alpha[alpha_length]      // H.264 AU of alpha-only stream
```

Parsed by `remote/EncodedFrame.ts::createEncodedFrame`.

## Transport (`Channel.ts`)

Four logical channels per client multiplexed over WebSockets:

| Channel | Type | Purpose |
|---|---|---|
| PROTOCOL | ARQ (KCP) | Wayland wire protocol + out-of-band ops |
| FRAME | ARQ (KCP) | Encoded surface frames |
| XWM | ARQ (KCP) | XWayland WM messages |
| FEEDBACK | SIMPLE (raw WS) | 4-byte feedback messages |

ARQ = a JS KCP implementation layered over WebSocket. Tuning:
`MTU=64000`, `MAX_BUFFERED_AMOUNT=2949120` (~2.8 MB — the back-pressure
ceiling), `SND_WINDOW=128`, `RCV_WINDOW=1024`,
`setNoDelay(1, 100, 0, 1)`.

## Browser decode (`packages/compositor/src/remote/`)

At session startup `Session.create()` probes
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
hardwareAcceleration:'prefer-software'})`:

### WebCodecs preferred

Software-only; the `prefer-hardware` path is implemented but commented out
with `// FIXME hardware decoding frame offset is wrong`.

### FFmpeg Wasm fallback

If WebCodecs unavailable: two persistent Workers (`h264NALDecoderOpaque`,
`h264NALDecoderAlpha`), one `AVCodecContext` per surface per worker,
keyed by `surfaceH264DecodeId`. Encoded NALs are transferred zero-copy
into the worker; decoded I420 planes are transferred back.

Per-surface decoder context is lazily created on first H.264 frame.

### PNG path (both decoders)

Identical in both branches: `new Blob([opaque], {type:'image/png'})` →
`createImageBitmap(blob, sx, sy, w, h)` (the `sx/sy` crops the encoder
padding back to logical size).

## WebGL upload (`render/`)

WebGL 1 with `desynchronized:true` for low-latency presentation. Per Scene
(canvas/output), three GL helpers:

### `YUVA2RGBA`

FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures**, renders into
the surface's RGBA texture via `YUVA2RGBAShader` (BT.601 coefficients,
alpha recovered by running the alpha-Y plane through the same YUV→RGB
matrix and taking R).

### `RGBXandA2RGBA`

FBO pass for the `DualPlaneRGBA*` WebCodecs outputs: opaque + alpha planes
composited via `RGBandA2RGBAShader`.

### `SceneShader`

Final pass: draws every view's RGBA texture to the canvas with
`gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)` (note: expects premultiplied
alpha but the YUVA shader outputs straight alpha — a minor inconsistency).

PNG and web-bitmapbuf paths skip the FBO and call
`texImage2D(ImageBitmap)` directly.

### Alpha pre-multiplication

Not done explicitly anywhere. The proxy extracts alpha as a Y plane; the
browser shader recombines; the canvas blend mode assumes premultiplied but
receives straight alpha — works acceptably because Wayland surfaces rarely
use partial alpha.

## End-to-end pipeline

```
┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
│  App renders via GTK/Qt/raw EGL into                          │
│   ┌────────────────────────────────────┐                       │
│   │  wl_shm buffer  (ARGB8888/XRGB8888)│  ─── or ─────────┐    │
│   │  zwp_linux_dmabuf_v1 (DMA-BUF)     │                  │    │
│   │  wl_drm create_prime_buffer        │                  │    │
│   └────────────────────────────────────┘                  │    │
└────────────────────────┬──────────────────────────────────┼────┘
                         │ Wayland socket (UNIX domain)     │
                         ▼                                  │
┌────────────────────── COMPOSITOR PROXY (Node + N-API addons)─┐
│  wayland-server-addon.node                                    │
│   └─ wl_display → wl_client → wl_resource (buffer)            │
│   └─ initShm(), initDrm("/dev/dri/renderD128")                │
│   └─ westfield-egl.c: EGLDevice + EGLDisplay + EGLContext     │
│                                                              │
│  proxy-encoding-addon.node (per-surface Encoder)              │
│   ┌─ node_encoder.c → frame_buffer {type: SHM|DMA, w, h, fmt} │
│   │                                                             │
│   │   gst_main_loop.c (own GLib thread, eventfd work queue)    │
│   │    │                                                        │
│   │    ▼                                                        │
│   │   gst_frame_encoder.c                                       │
│   │    ┌─ Encoder selection (per-frame):                       │
│   │    │   pixel_count ≤ 256×256  ─►  PNG pipeline            │
│   │    │   > 256×256 + ARGB/XRGB/dmabuf ─►  configured h264    │
│   │    │                                                       │
│   │    │  H.264 pipelines (parallel):                          │
│   │    │   opaque: appsrc → glupload → glcolorconvert →        │
│   │    │            glshader(scale→coded_size) →              │
│   │    │            glcolorconvert(NV12) → gldownload →        │
│   │    │            queue → x264enc|nvh264enc|vaapih264enc →  │
│   │    │            byte-stream AU → appsink                   │
│   │    │   alpha:   same source, fragment shader              │
│   │    │            `vec4(pix.a,pix.a,pix.a,0.)` →             │
│   │    │            second H.264 stream                        │
│   │    │                                                       │
│   │    │  PNG pipeline (single):                              │
│   │    │   appsrc → glupload → glcolorconvert → glshader →     │
│   │    │   RGBA → gldownload → pngenc → appsink                │
│   │    │                                                       │
│   │    └─ frame_encoding_result_to_encoded_frame():            │
│   │        packed struct {buffer_id, creation_serial,          │
│   │        content_serial, type(0=h264,1=png), w, h,           │
│   │        coded_w, coded_h} + opaque_len + opaque +           │
│   │        alpha_len + alpha  →  Node Buffer (zero-copy ext.)  │
│   │                                                             │
│   └─ SurfaceBufferEncoding.ts                                  │
│       └─ frameDataChannel.send(buffer)                         │
│                                                              │
│  Channel.ts:                                                  │
│   ARQChannel   (KCP-reliable over WebSocket, MTU 64 KB)        │
│   SimpleChannel (raw WebSocket, for FEEDBACK only)             │
│       ▲                                                       │
│       │ FrameFeedback.ts: 16-bit rAF interval + 16-bit avg     │
│       │ decode duration → used only to pace wl_frame.done      │
└───────┼───────────────────────────────────────────────────────┘
        │ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
        ▼
┌────────────────────── BROWSER (compositor) ────────────────────┐
│  WebSocket → ReconnectingWebSocket                              │
│   ↓                                                             │
│  Channel.ts: ARQChannel.input(kcp) → onMessage                  │
│   ↓                                                             │
│  RemoteAppLauncher.setupFrameDataChannel →                      │
│   BufferStream.deliverContentToBufferStream (parse serials)      │
│   ↓                                                             │
│  StreamingBuffer.getContents(surface, serial)                    │
│   ↓                                                             │
│  Session.frameDecoder.decode(surface, encodedFrame)            │
│   │                                                             │
│   ├─ PNG path (both decoders):                                  │
│   │    new Blob([opaque], {type:'image/png'}) →                 │
│   │    createImageBitmap(blob, sx, sy, w, h)  // crop padding  │
│   │    → SinglePlane { bitmap: ImageBitmap }                   │
│   │                                                             │
│   └─ H.264 path:                                                │
│      (preferred) WebCodecs VideoDecoder                        │
│         config: avc1.64001f, prefer-software, optimizeForLatency│
│         opaqueDecoder + alphaDecoder per surface               │
│         EncodedVideoChunk {type:key|delta, data:nal}            │
│         → VideoFrame.output →                                  │
│           if I420: copyTo → DualPlaneYUVAArrayBuffer           │
│           else:    keep VideoFrame → DualPlaneRGBAVideoFrame    │
│      (fallback) FFmpeg Wasm in 2 Web Workers                    │
│         worker.postMessage(nal, [nal.buffer])  // transferable   │
│         libavH264._decode → I420 planes                         │
│         postMessage back with [y.buffer, u.buffer, v.buffer]    │
│         → DualPlaneYUVASplitBuffer                              │
│                                                                 │
│  Renderer.render()  (rAF-driven, queueCancellableMicrotask)    │
│   ↓                                                             │
│  Scene.render(viewStack)                                        │
│   ├─ Scene['image/png']  → texImage2D(ImageBitmap)              │
│   ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D   │
│   └─ Scene['video/h264']:                                       │
│       YUVA2RGBA.convertYUVASplitBufferInto /                    │
│                   convertYUVAArrayBufferInto:                    │
│         Y/U/V/alphaY → 4× LUMINANCE textures                    │
│         bind FBO → renderState.texture (RGBA) as color attach   │
│         YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex     │
│       or RGBXandA2RGBA.convertInto (WebCodecs RGBA):            │
│         VideoFrame/ImageBitmap → texImage2D directly            │
│   ↓                                                             │
│  SceneShader.draw(): drawArrays with surface's RGBA texture,    │
│  blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas    │
│   ↓                                                             │
│  Browser composites canvas into the page                       │
└─────────────────────────────────────────────────────────────────┘


Same-frame web app fast path (web-bitmapbuf):
┌───────────────┐     MessagePort.postMessage([view, ...fds],      ┌──────────────┐
│  Web app      │ ──►  [imageBitmap]) ──────────────────────────►  │ Compositor   │
│  (iframe)     │     web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
└───────────────┘     → wl_buffer → ImageBitmapBuffer            └──────────────┘
                          (no encoding, no WebSocket, no GStreamer)
```

## Key answers

### Why H.264 over PNG?

H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB.
PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split
doubling dominate at small sizes.

### WebCodecs?

Yes — preferred path, software-only (`prefer-hardware` is wired but
disabled with FIXME "frame offset is wrong"). FFmpeg Wasm is the fallback.

### Trade-offs

- H.264 = small wire / decode cost
- PNG = medium wire / browser-native fast decode (`createImageBitmap`)
- raw SHM = **not used** between proxy and browser — every surface commit
  is encoded.

### Per-surface policy?

No — per-frame deterministic. Only configurable knob is
`--encoder=x264|nvh264|vaapih264`. PNG is always automatic fallback.

### Cursors?

Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS
cursor via `setBrowserCursor` — never enters the WebGL compositor.

### Alpha pre-multiplication?

Implicit in shaders. Proxy extracts alpha as a Y plane (no premult);
browser recombines via BT.601 YUVA→RGBA shader outputting straight alpha;
canvas blend is `ONE, ONE_MINUS_SRC_ALPHA` (expects premult) — minor
inconsistency, not visually problematic for typical Wayland content.

### OffscreenCanvas / Web Workers?

Workers yes (two for Wasm decoder, shared across surfaces). OffscreenCanvas
**no** — renderer is main-thread WebGL with `desynchronized:true`.
WebGPU/WebGL2 not used.

## Notable design trade-offs for your compositor

1. Always-encode policy (no raw SHM escape hatch even on LAN).
2. One encoder per surface, EOS-and-recreate on resize/format change.
3. Alpha as a separate H.264 stream — doubles cost for transparent windows.
4. Per-surface decoder context, no decoder pooling.
5. Only aggregate feedback (1 Hz rAF interval + decode duration); no
   per-frame ACKs, no bitrate adaptation; `requestKeyUnit` plumbing exists
   end-to-end but the browser never calls it.
6. KCP over WebSocket (not WebRTC datachannels) — simpler infra, no
   STUN/TURN, but no real congestion control.
7. Emscripten SIMD (not threads) — avoids COOP/COEP requirements.
8. Two parallel GL pipelines (opaque + alpha) for H.264.
9. PNG fallback boundary empirically at 256×256 pixels.
10. WebGL 1, not 2/WebGPU — maximum browser compat at the cost of manual
    LUMINANCE-texture YUV juggling and a hidden FBO intermediate.

## File reference

### Wasm libraries

| Path | Role |
|---|---|
| `libs/compositor-wasm/build_wasm.sh` | Builds pixman 0.42.2 + xkbcommon 1.5.0 to WASM (emsdk 3.1.46) |
| `libs/compositor-wasm/emscripten-build.ini` | Emscripten build config |
| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function |
| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon build (clones 1.5.0, embeds xkeyboard-config DB) |
| `libs/compositor-wasm/pixman/build.sh` | pixman build |
| `libs/compositor-ffmpeg-h264/build_wasm.sh` | Builds FFmpeg n5.1.2 to WASM (single-file, decoder-only) |
| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API |
| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module |
| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point |
| `libs/compositor-ffmpeg-h264/src/index.ts` | Exports |

### Native encoder addon

| Path | Role |
|---|---|
| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon (290 lines) |
| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder (~1400 lines) |
| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
| `packages/compositor-proxy/native/CMakeLists.txt` | Build config |
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | TS-side encoding orchestration |
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback |
| `packages/compositor-proxy/src/encoding/` | High-level encoding files |

### Browser decode

| Path | Role |
|---|---|
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs VideoDecoder path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |

### WebGL rendering

| Path | Role |
|---|---|
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |

### Protocol extensions

| Path | Role |
|---|---|
| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom Wayland extension for browser JS apps |
| `protocol/wayland-drm.xml` | Standard wl_drm for DMA-BUF (proxy-only) |
| `packages/compositor/src/web/WebBuffersFactory.ts` | Browser-side impl of web_bitmapbuf_factory |