Files

424 lines
22 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 09 — Hardware Acceleration, Encoding, WASM, GPU Textures
This chapter covers Greenfield's hardware acceleration / encoding / WASM
pipeline: how native buffers get encoded on the proxy side, transported,
decoded in the browser, and turned into WebGL textures or canvas images.
## What Greenfield actually ships
### Two Wasm libraries under `libs/`
**`compositor-wasm`** builds **pixman 0.42.2 + xkbcommon 1.5.0**
(Emscripten 3.1.46, `-msimd128 -O3 -flto`). Used by the browser for
**region math** (damage tracking) and **keyboard layout** — *not* for
pixel format conversion or H.264 decode.
**`compositor-ffmpeg-h264`** is a **stripped-down FFmpeg n5.1.2**
(`--disable-everything --enable-decoder=h264`) compiled to a 1.7 MB
single-file Wasm worker. A 30-line `decoder.c` wrapper exposes
`create_codec_context` / `decode` / `close_frame` / `destroy_codec_context`.
Decodes to **I420 (YUV 4:2:0 planar)** and returns plane pointers; planes
are `postMessage`'d back to the main thread as **transferable
`ArrayBuffer`s**. No threads (single-threaded, SIMD only).
### Native encoder addon (`packages/compositor-proxy/native/encoding/`)
A **GStreamer wrapper** exposed via Node-API. The TS side (`Encoder.ts`)
calls `appEndpointNative.encodeFrame(encoder, bufferId, contentSerial,
creationSerial)`. The C side (`node_encoder.c`) sniffs the wl_resource:
- `zwp_linux_dmabuf_v1` → DMA path
- Legacy `wl_drm` → DMA path
- `wl_shm_buffer` → SHM path
For DMA-BUFs the proxy wraps the dma-buf as an `EGLImage` and then as a
`GstGLMemoryEGL`, so pixels **stay on GPU** through the color-convert
stages.
### Threading
Encoding runs on a dedicated GLib main-loop thread fed by an
`eventfd`-signalled `GAsyncQueue`. Output returns to Node via a
`napi_threadsafe_function` with **zero-copy** external Buffers (finalised
back to C on GC).
## Encoder selection (per-frame, deterministic)
`gst_frame_encoder.c::frame_encoder_descriptions[]` declares **four**
pipelines:
| name | element | when used | split_alpha | coded-size multiple |
|---|---|---|---|---|
| `x264` | `x264enc` (software, default) | >256×256 + ARGB/XRGB/dmabuf | true (parallel alpha H.264 stream) | 128 |
| `nvh264` | `nvh264enc` (Nvidia NVENC) | same, `--encoder=nvh264` | true | 128 |
| `vaapih264` | `vaapih264enc` (Intel/AMD VA-API, "experimental") | same, `--encoder=vaapih264` | true | 128 |
| `png` | `pngenc` (fallback) | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 |
**Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264
encoder. YUV420/NV12/NV21/Y444 SHM formats are *not* supported on the
H.264 path and only by PNG at small sizes — effectively unsupported for
video-class content.
### Canonical H.264 pipeline
```
appsrc → glupload → glcolorconvert → glshader(scale→coded_size) →
glcolorconvert → video/x-raw(memory:GLMemory),format=NV12 → gldownload →
queue → x264enc tune=zerolatency bitrate=12000 vbv-buf-capacity=160
bframes=0 rc-lookahead=0 sliced-threads byte-stream=true →
video/x-h264,profile=high,stream-format=byte-stream,alignment=au → appsink
```
The `glshader` element does two jobs:
1. Pad up to the encoder's macroblock multiple (128 for H.264, 16 for PNG)
using a `u_transformation` matrix.
2. For the **alpha pipeline**, run `vec4(pix.a, pix.a, pix.a, 0.)` so
alpha is encoded as a grayscale H.264 stream — H.264 has no alpha, so
Greenfield ships two parallel AU streams.
Bitrates: opaque 12000 / alpha 1200 (x264), 12000 / 1200 (nvh264).
## Wire format
Single contiguous blob, little-endian:
```
struct packed props {
u32 buffer_id, creation_serial, content_serial,
u32 frame_encoding_type (0=h264, 1=png),
u32 width, height, coded_width, coded_height
}
u32 opaque_length
u8 opaque[opaque_length] // H.264 AU or PNG blob
u32 alpha_length // 0 if no alpha split
u8 alpha[alpha_length] // H.264 AU of alpha-only stream
```
Parsed by `remote/EncodedFrame.ts::createEncodedFrame`.
## Transport (`Channel.ts`)
Four logical channels per client multiplexed over WebSockets:
| Channel | Type | Purpose |
|---|---|---|
| PROTOCOL | ARQ (KCP) | Wayland wire protocol + out-of-band ops |
| FRAME | ARQ (KCP) | Encoded surface frames |
| XWM | ARQ (KCP) | XWayland WM messages |
| FEEDBACK | SIMPLE (raw WS) | 4-byte feedback messages |
ARQ = a JS KCP implementation layered over WebSocket. Tuning:
`MTU=64000`, `MAX_BUFFERED_AMOUNT=2949120` (~2.8 MB — the back-pressure
ceiling), `SND_WINDOW=128`, `RCV_WINDOW=1024`,
`setNoDelay(1, 100, 0, 1)`.
## Browser decode (`packages/compositor/src/remote/`)
At session startup `Session.create()` probes
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
hardwareAcceleration:'prefer-software'})`:
### WebCodecs preferred
Software-only; the `prefer-hardware` path is implemented but commented out
with `// FIXME hardware decoding frame offset is wrong`.
### FFmpeg Wasm fallback
If WebCodecs unavailable: two persistent Workers (`h264NALDecoderOpaque`,
`h264NALDecoderAlpha`), one `AVCodecContext` per surface per worker,
keyed by `surfaceH264DecodeId`. Encoded NALs are transferred zero-copy
into the worker; decoded I420 planes are transferred back.
Per-surface decoder context is lazily created on first H.264 frame.
### PNG path (both decoders)
Identical in both branches: `new Blob([opaque], {type:'image/png'})` →
`createImageBitmap(blob, sx, sy, w, h)` (the `sx/sy` crops the encoder
padding back to logical size).
## WebGL upload (`render/`)
WebGL 1 with `desynchronized:true` for low-latency presentation. Per Scene
(canvas/output), three GL helpers:
### `YUVA2RGBA`
FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures**, renders into
the surface's RGBA texture via `YUVA2RGBAShader` (BT.601 coefficients,
alpha recovered by running the alpha-Y plane through the same YUV→RGB
matrix and taking R).
### `RGBXandA2RGBA`
FBO pass for the `DualPlaneRGBA*` WebCodecs outputs: opaque + alpha planes
composited via `RGBandA2RGBAShader`.
### `SceneShader`
Final pass: draws every view's RGBA texture to the canvas with
`gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)` (note: expects premultiplied
alpha but the YUVA shader outputs straight alpha — a minor inconsistency).
PNG and web-bitmapbuf paths skip the FBO and call
`texImage2D(ImageBitmap)` directly.
### Alpha pre-multiplication
Not done explicitly anywhere. The proxy extracts alpha as a Y plane; the
browser shader recombines; the canvas blend mode assumes premultiplied but
receives straight alpha — works acceptably because Wayland surfaces rarely
use partial alpha.
## End-to-end pipeline
```
┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
│ App renders via GTK/Qt/raw EGL into │
│ ┌────────────────────────────────────┐ │
│ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │
│ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │
│ │ wl_drm create_prime_buffer │ │ │
│ └────────────────────────────────────┘ │ │
└────────────────────────┬──────────────────────────────────┼────┘
│ Wayland socket (UNIX domain) │
▼ │
┌────────────────────── COMPOSITOR PROXY (Node + N-API addons)─┐
│ wayland-server-addon.node │
│ └─ wl_display → wl_client → wl_resource (buffer) │
│ └─ initShm(), initDrm("/dev/dri/renderD128") │
│ └─ westfield-egl.c: EGLDevice + EGLDisplay + EGLContext │
│ │
│ proxy-encoding-addon.node (per-surface Encoder) │
│ ┌─ node_encoder.c → frame_buffer {type: SHM|DMA, w, h, fmt} │
│ │ │
│ │ gst_main_loop.c (own GLib thread, eventfd work queue) │
│ │ │ │
│ │ ▼ │
│ │ gst_frame_encoder.c │
│ │ ┌─ Encoder selection (per-frame): │
│ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │
│ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │
│ │ │ │
│ │ │ H.264 pipelines (parallel): │
│ │ │ opaque: appsrc → glupload → glcolorconvert → │
│ │ │ glshader(scale→coded_size) → │
│ │ │ glcolorconvert(NV12) → gldownload → │
│ │ │ queue → x264enc|nvh264enc|vaapih264enc → │
│ │ │ byte-stream AU → appsink │
│ │ │ alpha: same source, fragment shader │
│ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │
│ │ │ second H.264 stream │
│ │ │ │
│ │ │ PNG pipeline (single): │
│ │ │ appsrc → glupload → glcolorconvert → glshader → │
│ │ │ RGBA → gldownload → pngenc → appsink │
│ │ │ │
│ │ └─ frame_encoding_result_to_encoded_frame(): │
│ │ packed struct {buffer_id, creation_serial, │
│ │ content_serial, type(0=h264,1=png), w, h, │
│ │ coded_w, coded_h} + opaque_len + opaque + │
│ │ alpha_len + alpha → Node Buffer (zero-copy ext.) │
│ │ │
│ └─ SurfaceBufferEncoding.ts │
│ └─ frameDataChannel.send(buffer) │
│ │
│ Channel.ts: │
│ ARQChannel (KCP-reliable over WebSocket, MTU 64 KB) │
│ SimpleChannel (raw WebSocket, for FEEDBACK only) │
│ ▲ │
│ │ FrameFeedback.ts: 16-bit rAF interval + 16-bit avg │
│ │ decode duration → used only to pace wl_frame.done │
└───────┼───────────────────────────────────────────────────────┘
│ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
▼
┌────────────────────── BROWSER (compositor) ────────────────────┐
│ WebSocket → ReconnectingWebSocket │
│ ↓ │
│ Channel.ts: ARQChannel.input(kcp) → onMessage │
│ ↓ │
│ RemoteAppLauncher.setupFrameDataChannel → │
│ BufferStream.deliverContentToBufferStream (parse serials) │
│ ↓ │
│ StreamingBuffer.getContents(surface, serial) │
│ ↓ │
│ Session.frameDecoder.decode(surface, encodedFrame) │
│ │ │
│ ├─ PNG path (both decoders): │
│ │ new Blob([opaque], {type:'image/png'}) → │
│ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │
│ │ → SinglePlane { bitmap: ImageBitmap } │
│ │ │
│ └─ H.264 path: │
│ (preferred) WebCodecs VideoDecoder │
│ config: avc1.64001f, prefer-software, optimizeForLatency│
│ opaqueDecoder + alphaDecoder per surface │
│ EncodedVideoChunk {type:key|delta, data:nal} │
│ → VideoFrame.output → │
│ if I420: copyTo → DualPlaneYUVAArrayBuffer │
│ else: keep VideoFrame → DualPlaneRGBAVideoFrame │
│ (fallback) FFmpeg Wasm in 2 Web Workers │
│ worker.postMessage(nal, [nal.buffer]) // transferable │
│ libavH264._decode → I420 planes │
│ postMessage back with [y.buffer, u.buffer, v.buffer] │
│ → DualPlaneYUVASplitBuffer │
│ │
│ Renderer.render() (rAF-driven, queueCancellableMicrotask) │
│ ↓ │
│ Scene.render(viewStack) │
│ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │
│ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │
│ └─ Scene['video/h264']: │
│ YUVA2RGBA.convertYUVASplitBufferInto / │
│ convertYUVAArrayBufferInto: │
│ Y/U/V/alphaY → 4× LUMINANCE textures │
│ bind FBO → renderState.texture (RGBA) as color attach │
│ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │
│ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │
│ VideoFrame/ImageBitmap → texImage2D directly │
│ ↓ │
│ SceneShader.draw(): drawArrays with surface's RGBA texture, │
│ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │
│ ↓ │
│ Browser composites canvas into the page │
└─────────────────────────────────────────────────────────────────┘
Same-frame web app fast path (web-bitmapbuf):
┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐
│ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │
│ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
└───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘
(no encoding, no WebSocket, no GStreamer)
```
## Key answers
### Why H.264 over PNG?
H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB.
PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split
doubling dominate at small sizes.
### WebCodecs?
Yes — preferred path, software-only (`prefer-hardware` is wired but
disabled with FIXME "frame offset is wrong"). FFmpeg Wasm is the fallback.
### Trade-offs
- H.264 = small wire / decode cost
- PNG = medium wire / browser-native fast decode (`createImageBitmap`)
- raw SHM = **not used** between proxy and browser — every surface commit
is encoded.
### Per-surface policy?
No — per-frame deterministic. Only configurable knob is
`--encoder=x264|nvh264|vaapih264`. PNG is always automatic fallback.
### Cursors?
Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS
cursor via `setBrowserCursor` — never enters the WebGL compositor.
### Alpha pre-multiplication?
Implicit in shaders. Proxy extracts alpha as a Y plane (no premult);
browser recombines via BT.601 YUVA→RGBA shader outputting straight alpha;
canvas blend is `ONE, ONE_MINUS_SRC_ALPHA` (expects premult) — minor
inconsistency, not visually problematic for typical Wayland content.
### OffscreenCanvas / Web Workers?
Workers yes (two for Wasm decoder, shared across surfaces). OffscreenCanvas
**no** — renderer is main-thread WebGL with `desynchronized:true`.
WebGPU/WebGL2 not used.
## Notable design trade-offs for your compositor
1. Always-encode policy (no raw SHM escape hatch even on LAN).
2. One encoder per surface, EOS-and-recreate on resize/format change.
3. Alpha as a separate H.264 stream — doubles cost for transparent windows.
4. Per-surface decoder context, no decoder pooling.
5. Only aggregate feedback (1 Hz rAF interval + decode duration); no
per-frame ACKs, no bitrate adaptation; `requestKeyUnit` plumbing exists
end-to-end but the browser never calls it.
6. KCP over WebSocket (not WebRTC datachannels) — simpler infra, no
STUN/TURN, but no real congestion control.
7. Emscripten SIMD (not threads) — avoids COOP/COEP requirements.
8. Two parallel GL pipelines (opaque + alpha) for H.264.
9. PNG fallback boundary empirically at 256×256 pixels.
10. WebGL 1, not 2/WebGPU — maximum browser compat at the cost of manual
LUMINANCE-texture YUV juggling and a hidden FBO intermediate.
## File reference
### Wasm libraries
| Path | Role |
|---|---|
| `libs/compositor-wasm/build_wasm.sh` | Builds pixman 0.42.2 + xkbcommon 1.5.0 to WASM (emsdk 3.1.46) |
| `libs/compositor-wasm/emscripten-build.ini` | Emscripten build config |
| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function |
| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon build (clones 1.5.0, embeds xkeyboard-config DB) |
| `libs/compositor-wasm/pixman/build.sh` | pixman build |
| `libs/compositor-ffmpeg-h264/build_wasm.sh` | Builds FFmpeg n5.1.2 to WASM (single-file, decoder-only) |
| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API |
| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module |
| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point |
| `libs/compositor-ffmpeg-h264/src/index.ts` | Exports |
### Native encoder addon
| Path | Role |
|---|---|
| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon (290 lines) |
| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder (~1400 lines) |
| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
| `packages/compositor-proxy/native/CMakeLists.txt` | Build config |
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | TS-side encoding orchestration |
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback |
| `packages/compositor-proxy/src/encoding/` | High-level encoding files |
### Browser decode
| Path | Role |
|---|---|
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs VideoDecoder path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
### WebGL rendering
| Path | Role |
|---|---|
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
### Protocol extensions
| Path | Role |
|---|---|
| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom Wayland extension for browser JS apps |
| `protocol/wayland-drm.xml` | Standard wl_drm for DMA-BUF (proxy-only) |
| `packages/compositor/src/web/WebBuffersFactory.ts` | Browser-side impl of web_bitmapbuf_factory |