Initial commit: MIT-licensed compositor architecture & build plan (14 files, ~80 pages, 10 parallel research agents)
This commit is contained in:
1b-agent committed
commit
582e61e942
15 files changed
+4970
No files matched your search
@@ -0,0 +1,522 @@
|
||||
# 09 — Hardware Acceleration, Encoding, WebCodecs, GPU Textures
|
||||
|
||||
This chapter covers the encoding/decoding/WASM pipeline in the MIT
|
||||
approach. **Critical changes from Greenfield**:
|
||||
- **No x264enc** (GPL-2.0+) — replaced with `vaapih264enc`/`nvh264enc`/`openh264enc`
|
||||
- **No FFmpeg Wasm** (GPL-3.0 in Greenfield's build) — replaced with WebCodecs primary + OpenH264 BSD fallback
|
||||
- **Hardware decode enabled** (Greenfield disables it with a FIXME)
|
||||
- **Adaptive bitrate** (consumes `EncoderFeedback` data Greenfield produces but doesn't use)
|
||||
|
||||
## What's reused (MIT)
|
||||
|
||||
### Browser side
|
||||
|
||||
| Component | License | Role |
|
||||
|---|---|---|
|
||||
| `@gfld/compositor-wasm` | MIT | pixman 0.42.2 + xkbcommon 1.5.0 WASM (region math + keymap) |
|
||||
|
||||
### Vendored C (MIT, used in Python proxy via cffi)
|
||||
|
||||
| File | Lines | Role |
|
||||
|---|---|---|
|
||||
| `westfield-dmabuf.c` | ~200 | DMA-BUF import via EGL/GBM |
|
||||
| `westfield-egl.c` | ~900 | EGL zero-copy setup (EGLDisplay, EGLContext, EGLImage) |
|
||||
| `wlr_drm.c` | ~244 | `wl_drm` global impl |
|
||||
| `wlr_linux_dmabuf_v1.c` | ~1,110 | `zwp_linux_dmabuf_v1` global impl |
|
||||
| `drm_format_set.c` | ~239 | DRM format helpers |
|
||||
|
||||
## What's replaced (proprietary, clean-room)
|
||||
|
||||
### Browser side
|
||||
|
||||
| Component | Replaces | Why |
|
||||
|---|---|---|
|
||||
| Custom `WebCodecsFrameDecoder.ts` | Greenfield's `webcodec-buffer-decoder.ts` (AGPL) | Clean-room; **enables hardware decode** (Greenfield disables with FIXME) |
|
||||
| Custom `OpenH264FrameDecoder.ts` | Greenfield's `wasm-buffer-decoder.ts` + `H264NALDecoder.worker.ts` (AGPL) + `@gfld/compositor-ffmpeg-h264` (GPL) | Clean-room; uses **OpenH264 BSD Wasm** instead of FFmpeg GPL Wasm |
|
||||
| Custom `YUVA2RGBA` / `RGBXandA2RGBA` shaders | Greenfield's (AGPL) | Clean-room; **WebGL2** (Greenfield uses WebGL1) |
|
||||
| Custom `Renderer` / `Scene` / `SceneShader` | Greenfield's (AGPL) | Clean-room; **OffscreenCanvas + Web Workers** (Greenfield is main-thread) |
|
||||
|
||||
### Python side
|
||||
|
||||
| Component | Replaces | Why |
|
||||
|---|---|---|
|
||||
| Custom `gst_encoder.py` (~700 LOC) | Greenfield's `proxy-encoding-addon.node` (AGPL, 290 lines C) + `libproxy-encoding.so` (AGPL, ~1400 lines C) | Pure Python via PyGObject. Same GStreamer, no N-API addon. |
|
||||
| `vaapih264enc` / `nvh264enc` / `openh264enc` | `x264enc` (GPL) | GPL-free H.264 encoding |
|
||||
|
||||
## Encoder selection (per-frame, deterministic)
|
||||
|
||||
Same logic as Greenfield, but **without x264enc**:
|
||||
|
||||
| name | element | when used | split_alpha | coded-size multiple | License |
|
||||
|---|---|---|---|---|---|
|
||||
| `vaapih264` | `vaapih264enc` | >256×256 + ARGB/XRGB/dmabuf, Intel/AMD GPU | true (parallel alpha H.264 stream) | 128 | LGPL + MIT (libva) |
|
||||
| `nvh264` | `nvh264enc` | same, NVIDIA GPU | true | 128 | BSD plugin + proprietary NVENC blob |
|
||||
| `openh264` | `openh264enc` | same, no GPU (CPU fallback) | true | 128 | BSD-2-Clause (libopenh264) |
|
||||
| `png` | `pngenc` | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 | LGPL |
|
||||
|
||||
**Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264
|
||||
encoder (selected by `--encoder=vaapih264|nvh264|openh264`).
|
||||
|
||||
**NOTE**: This means the **default encoder is hardware-accelerated**
|
||||
(vaapih264 on Intel/AMD, nvh264 on NVIDIA). Greenfield defaults to x264
|
||||
(software, GPL).
|
||||
|
||||
## Canonical H.264 pipeline (Python equivalent)
|
||||
|
||||
Same GStreamer pipeline as Greenfield, ported to PyGObject:
|
||||
|
||||
```python
|
||||
# gst_encoder.py (clean-room, ~700 LOC)
|
||||
import gi
|
||||
gi.require_version("Gst", "1.0")
|
||||
gi.require_version("GstGL", "1.0")
|
||||
gi.require_version("GstEGL", "1.0") # if available
|
||||
from gi.repository import Gst, GstGL, GLib
|
||||
|
||||
class H264Encoder:
|
||||
def __init__(self, egl_display_ptr, egl_context_ptr, encoder_type="vaapih264"):
|
||||
# Create GLDisplay + GLContext from existing EGL context (zero-copy)
|
||||
self.gl_display = GstGL.GLDisplayEGL.new_with_egl_display(egl_display_ptr)
|
||||
self.gl_context = GstGL.GLContext.new_wrapped(
|
||||
self.gl_display,
|
||||
egl_context_ptr,
|
||||
GstGL.GLPlatform.EGL,
|
||||
GstGL.GLLAPI.GLES2,
|
||||
)
|
||||
|
||||
# Encoder pipeline
|
||||
encoder_element = {
|
||||
"vaapih264": "vaapih264enc",
|
||||
"nvh264": "nvh264enc",
|
||||
"openh264": "openh264enc",
|
||||
}[encoder_type]
|
||||
|
||||
self.pipeline = Gst.parse_launch(
|
||||
f"appsrc name=src ! "
|
||||
f"glupload ! glcolorconvert ! "
|
||||
f"glshader name=scale ! " # pads to macroblock multiple
|
||||
f"glcolorconvert ! video/x-raw(memory:GLMemory),format=NV12 ! "
|
||||
f"gldownload ! queue ! "
|
||||
f"{encoder_element} tune=zerolatency bitrate=12000 vbv-buf-capacity=160 "
|
||||
f"bframes=0 rc-lookahead=0 sliced-threads byte-stream=true ! "
|
||||
f"video/x-h264,profile=high,stream-format=byte-stream,alignment=au ! "
|
||||
f"appsink name=sink"
|
||||
)
|
||||
|
||||
# Wire up appsink callback (zero-copy external Buffers)
|
||||
sink = self.pipeline.get_by_name("sink")
|
||||
sink.emit("set-property", "emit-signals", True)
|
||||
sink.connect("new-sample", self._on_new_sample)
|
||||
|
||||
# Alpha pipeline (separate, for transparent windows)
|
||||
self.alpha_pipeline = self._build_alpha_pipeline(encoder_element)
|
||||
|
||||
# GLib MainLoop in separate thread (GStreamer requires it)
|
||||
self.glib_loop = GLib.MainLoop()
|
||||
self.glib_context = GLib.MainContext.new()
|
||||
GLib.MainContext.push_thread_default(self.glib_context)
|
||||
import threading
|
||||
self.glib_thread = threading.Thread(target=self.glib_loop.run, daemon=True)
|
||||
self.glib_thread.start()
|
||||
|
||||
def encode_frame(self, buffer_id, content_serial, creation_serial, dma_buf_fd, w, h):
|
||||
# Push buffer into appsrc (zero-copy via GstBuffer wrapping the dmabuf)
|
||||
# ... marshalled from asyncio via GLib.idle_add
|
||||
|
||||
def _on_new_sample(self, appsink):
|
||||
sample = appsink.emit("pull-sample")
|
||||
buffer = sample.get_buffer()
|
||||
# Marshal back to asyncio
|
||||
asyncio.run_coroutine_threadsafe(
|
||||
self._deliver_frame(buffer, self.current_buffer_id),
|
||||
self.asyncio_loop,
|
||||
)
|
||||
```
|
||||
|
||||
### Bitrates (same as Greenfield)
|
||||
|
||||
- Opaque: 12000 (vaapih264 / nvh264 / openh264)
|
||||
- Alpha: 1200 (10% of opaque)
|
||||
|
||||
### Adaptive bitrate (NEW — Greenfield lacks)
|
||||
|
||||
Consume `EncoderFeedback` data (refreshInterval + decode duration):
|
||||
|
||||
```python
|
||||
# adaptive_bitrate.py (clean-room, ~150 LOC)
|
||||
class AdaptiveBitrate:
|
||||
def __init__(self, encoder):
|
||||
self.encoder = encoder
|
||||
|
||||
def on_feedback(self, refresh_interval_ms, avg_decode_ms):
|
||||
# If decode is taking too long, reduce bitrate
|
||||
if avg_decode_ms > refresh_interval_ms * 0.8:
|
||||
new_bitrate = max(self.current_bitrate * 0.7, MIN_BITRATE)
|
||||
self.encoder.set_property("bitrate", new_bitrate)
|
||||
# If decode is fast and we have headroom, increase bitrate
|
||||
elif avg_decode_ms < refresh_interval_ms * 0.3:
|
||||
new_bitrate = min(self.current_bitrate * 1.3, MAX_BITRATE)
|
||||
self.encoder.set_property("bitrate", new_bitrate)
|
||||
```
|
||||
|
||||
## Wire format (REUSED from Greenfield)
|
||||
|
||||
Same binary format as Greenfield (see `../greenfield-analysis/09-hardware-accel.md`):
|
||||
|
||||
```
|
||||
struct packed props {
|
||||
u32 buffer_id, creation_serial, content_serial,
|
||||
u32 frame_encoding_type (0=h264, 1=png),
|
||||
u32 width, height, coded_width, coded_height
|
||||
}
|
||||
u32 opaque_length
|
||||
u8 opaque[opaque_length] // H.264 AU or PNG blob
|
||||
u32 alpha_length // 0 if no alpha split
|
||||
u8 alpha[alpha_length] // H.264 AU of alpha-only stream
|
||||
```
|
||||
|
||||
## Browser decode (clean-room TS)
|
||||
|
||||
### Preferred path: WebCodecs `VideoDecoder` (with HW decode enabled)
|
||||
|
||||
```typescript
|
||||
// WebCodecsFrameDecoder.ts (clean-room, ~250 LOC)
|
||||
export class WebCodecsFrameDecoder implements FrameDecoder {
|
||||
private opaqueDecoders: Map<SurfaceId, VideoDecoder> = new Map();
|
||||
private alphaDecoders: Map<SurfaceId, VideoDecoder> = new Map();
|
||||
|
||||
async isSupported(): Promise<boolean> {
|
||||
const config = {
|
||||
codec: 'avc1.64001f',
|
||||
optimizeForLatency: true,
|
||||
// NEW: enable hardware acceleration (Greenfield disables this)
|
||||
hardwareAcceleration: 'prefer-hardware',
|
||||
};
|
||||
const result = await VideoDecoder.isConfigSupported(config);
|
||||
return result.supported;
|
||||
}
|
||||
|
||||
async decode(surface: Surface, encodedFrame: EncodedFrame): Promise<DecodedFrame> {
|
||||
let opaqueDecoder = this.opaqueDecoders.get(surface.id);
|
||||
if (!opaqueDecoder) {
|
||||
opaqueDecoder = new VideoDecoder({
|
||||
output: (frame) => this.onDecoded(surface, frame),
|
||||
error: (e) => this.onError(surface, e),
|
||||
});
|
||||
opaqueDecoder.configure({
|
||||
codec: 'avc1.64001f',
|
||||
optimizeForLatency: true,
|
||||
hardwareAcceleration: 'prefer-hardware', // NEW
|
||||
});
|
||||
this.opaqueDecoders.set(surface.id, opaqueDecoder);
|
||||
}
|
||||
|
||||
const chunk = new EncodedVideoChunk({
|
||||
type: encodedFrame.isKeyframe ? 'key' : 'delta',
|
||||
data: encodedFrame.opaque,
|
||||
timestamp: encodedFrame.timestamp,
|
||||
});
|
||||
opaqueDecoder.decode(chunk);
|
||||
|
||||
// Alpha path (if present)
|
||||
if (encodedFrame.alpha) {
|
||||
// Same pattern for alphaDecoder
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Key difference from Greenfield**: We use `hardwareAcceleration:
|
||||
'prefer-hardware'` (Greenfield uses `'prefer-software'` with a FIXME
|
||||
about "frame offset is wrong"). This needs testing, but should give
|
||||
significant CPU savings.
|
||||
|
||||
### Fallback path: OpenH264 BSD Wasm (NOT FFmpeg — FFmpeg is GPL)
|
||||
|
||||
```typescript
|
||||
// OpenH264FrameDecoder.ts (clean-room, ~250 LOC)
|
||||
// Uses OpenH264 BSD-2-Clause Wasm build (Cisco's binary release or self-built)
|
||||
// Available from: https://github.com/cisco/openh264 (BSD-2-Clause)
|
||||
```
|
||||
|
||||
OpenH264 is BSD-2-Clause — fully proprietary-compatible. Less featureful
|
||||
than FFmpeg (no VP9/AV1), but sufficient for H.264 fallback.
|
||||
|
||||
## WebGL2 upload (clean-room TS, modernized)
|
||||
|
||||
WebGL2 with `desynchronized:true` + OffscreenCanvas in Web Worker.
|
||||
|
||||
Per Scene (canvas/output), three GL helpers (same as Greenfield, but
|
||||
WebGL2 instead of WebGL1):
|
||||
|
||||
### `YUVA2RGBA`
|
||||
|
||||
FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures** (WebGL1
|
||||
compatible) OR **1× R8UI texture array** (WebGL2, more efficient),
|
||||
renders into the surface's RGBA texture via `YUVA2RGBAShader` (BT.601
|
||||
coefficients).
|
||||
|
||||
### `RGBXandA2RGBA`
|
||||
|
||||
FBO pass for the `DualPlaneRGBA*` WebCodecs outputs (opaque + alpha
|
||||
planes composited via `RGBandA2RGBAShader`).
|
||||
|
||||
### `SceneShader`
|
||||
|
||||
Final pass: draws every view's RGBA texture to the canvas with
|
||||
`gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`.
|
||||
|
||||
**NEW**: WebGL2 `texStorage2D` for immutable textures (faster upload).
|
||||
**NEW**: `VideoFrame` directly as a WebGL texture source (zero-copy from
|
||||
WebCodecs to GPU, no `copyTo` needed).
|
||||
|
||||
## End-to-end pipeline
|
||||
|
||||
```
|
||||
┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
|
||||
│ App renders via GTK/Qt/raw EGL into │
|
||||
│ ┌────────────────────────────────────┐ │
|
||||
│ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │
|
||||
│ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │
|
||||
│ │ wl_drm create_prime_buffer │ │ │
|
||||
│ └────────────────────────────────────┘ │ │
|
||||
└────────────────────────┬──────────────────────────────────┼────┘
|
||||
│ Wayland socket (UNIX domain) │
|
||||
▼ │
|
||||
┌────────────────────── PYTHON PROXY (aiohttp + pywayland + GStreamer)─┐
|
||||
│ pywayland (Apache-2.0) │
|
||||
│ └─ Display → Client → Resource (buffer) │
|
||||
│ └─ display.init_shm(), init_drm("/dev/dri/renderD128") │
|
||||
│ └─ Vendored westfield-egl.c (MIT, via cffi): │
|
||||
│ EGLDevice + EGLDisplay + EGLContext │
|
||||
│ │
|
||||
│ gst_encoder.py (~700 LOC Python, clean-room) │
|
||||
│ ┌─ frame_buffer {type: SHM|DMA, w, h, fmt} │
|
||||
│ │ │
|
||||
│ │ GLib MainLoop in separate thread (GLib.idle_add) │
|
||||
│ │ │ │
|
||||
│ │ ▼ │
|
||||
│ │ gst_parse_launch: │
|
||||
│ │ ┌─ Encoder selection (per-frame): │
|
||||
│ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │
|
||||
│ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │
|
||||
│ │ │ │
|
||||
│ │ │ H.264 pipelines (parallel, NO x264 — GPL): │
|
||||
│ │ │ opaque: appsrc → glupload → glcolorconvert → │
|
||||
│ │ │ glshader(scale→coded_size) → │
|
||||
│ │ │ glcolorconvert(NV12) → gldownload → │
|
||||
│ │ │ queue → vaapih264enc|nvh264enc|openh264enc │
|
||||
│ │ │ → byte-stream AU → appsink │
|
||||
│ │ │ alpha: same source, fragment shader │
|
||||
│ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │
|
||||
│ │ │ second H.264 stream │
|
||||
│ │ │ │
|
||||
│ │ │ PNG pipeline (single): │
|
||||
│ │ │ appsrc → glupload → glcolorconvert → glshader → │
|
||||
│ │ │ RGBA → gldownload → pngenc → appsink │
|
||||
│ │ │ │
|
||||
│ │ └─ asyncio.run_coroutine_threadsafe for cross-thread │
|
||||
│ │ │
|
||||
│ └─ surface_buffer_encoding.py │
|
||||
│ └─ frame_data_channel.send(buffer) │
|
||||
│ │
|
||||
│ channel.py: │
|
||||
│ ARQChannel (kcp PyPI package, MIT, over WebSocket) │
|
||||
│ SimpleChannel (raw WebSocket, for FEEDBACK only) │
|
||||
│ ▲ │
|
||||
│ │ frame_feedback.py: 16-bit rAF interval + 16-bit avg │
|
||||
│ │ decode duration → used to pace wl_frame.done AND │
|
||||
│ │ drive adaptive_bitrate.py (NEW vs Greenfield) │
|
||||
└───────┼───────────────────────────────────────────────────────┘
|
||||
│ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
|
||||
▼
|
||||
┌────────────────────── BROWSER (Custom Compositor TS, clean-room) ──────┐
|
||||
│ WebSocket → ReconnectingWebSocket (vendored MIT) │
|
||||
│ ↓ │
|
||||
│ channel.ts: ARQChannel.input(kcp) → onMessage │
|
||||
│ ↓ │
|
||||
│ RemoteAppLauncher.setupFrameDataChannel → │
|
||||
│ BufferStream.deliverContentToBufferStream (parse serials) │
|
||||
│ ↓ │
|
||||
│ StreamingBuffer.getContents(surface, serial) │
|
||||
│ ↓ │
|
||||
│ Session.frameDecoder.decode(surface, encodedFrame) │
|
||||
│ │ │
|
||||
│ ├─ PNG path (both decoders): │
|
||||
│ │ new Blob([opaque], {type:'image/png'}) → │
|
||||
│ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │
|
||||
│ │ → SinglePlane { bitmap: ImageBitmap } │
|
||||
│ │ │
|
||||
│ └─ H.264 path: │
|
||||
│ (preferred) WebCodecs VideoDecoder │
|
||||
│ config: avc1.64001f, prefer-hardware (NEW!), │
|
||||
│ optimizeForLatency │
|
||||
│ opaqueDecoder + alphaDecoder per surface │
|
||||
│ EncodedVideoChunk {type:key|delta, data:nal} │
|
||||
│ → VideoFrame.output → │
|
||||
│ if I420: copyTo → DualPlaneYUVAArrayBuffer │
|
||||
│ else: keep VideoFrame → DualPlaneRGBAVideoFrame │
|
||||
│ (fallback) OpenH264 BSD Wasm in 2 Web Workers │
|
||||
│ (NOT FFmpeg — GPL contamination) │
|
||||
│ worker.postMessage(nal, [nal.buffer]) // transferable │
|
||||
│ OpenH264._decode → I420 planes │
|
||||
│ postMessage back with [y.buffer, u.buffer, v.buffer] │
|
||||
│ → DualPlaneYUVASplitBuffer │
|
||||
│ │
|
||||
│ Renderer.render() (rAF-driven, in Web Worker via OffscreenCanvas)│
|
||||
│ ↓ │
|
||||
│ Scene.render(viewStack) │
|
||||
│ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │
|
||||
│ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │
|
||||
│ └─ Scene['video/h264']: │
|
||||
│ YUVA2RGBA.convertYUVASplitBufferInto / │
|
||||
│ convertYUVAArrayBufferInto: │
|
||||
│ Y/U/V/alphaY → 4× LUMINANCE textures (WebGL1 compat) │
|
||||
│ OR 1× R8UI texture array (WebGL2, more efficient) │
|
||||
│ bind FBO → renderState.texture (RGBA) as color attach │
|
||||
│ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │
|
||||
│ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │
|
||||
│ VideoFrame/ImageBitmap → texImage2D directly │
|
||||
│ ↓ │
|
||||
│ SceneShader.draw(): drawArrays with surface's RGBA texture, │
|
||||
│ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │
|
||||
│ ↓ │
|
||||
│ Browser composites canvas into the page │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
|
||||
|
||||
Same-frame web app fast path (web-bitmapbuf, REUSED):
|
||||
┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐
|
||||
│ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │
|
||||
│ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
|
||||
└───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘
|
||||
(no encoding, no WebSocket, no GStreamer)
|
||||
```
|
||||
|
||||
## Key answers (quick reference)
|
||||
|
||||
### Why H.264 over PNG?
|
||||
|
||||
H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB.
|
||||
PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split
|
||||
doubling dominate at small sizes.
|
||||
|
||||
### WebCodecs?
|
||||
|
||||
Yes — preferred path. **Hardware decode enabled** (Greenfield disables
|
||||
with FIXME). OpenH264 BSD Wasm is the fallback (NOT FFmpeg — GPL).
|
||||
|
||||
### Trade-offs?
|
||||
|
||||
- H.264 = small wire / decode cost
|
||||
- PNG = medium wire / browser-native fast decode (`createImageBitmap`)
|
||||
- raw SHM = **not used** between proxy and browser — every surface commit
|
||||
is encoded.
|
||||
|
||||
### Per-surface policy?
|
||||
|
||||
No — per-frame deterministic. Only configurable knob is
|
||||
`--encoder=vaapih264|nvh264|openh264`. PNG is always automatic fallback.
|
||||
|
||||
### Cursors?
|
||||
|
||||
Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS
|
||||
cursor via `setBrowserCursor` — never enters the WebGL compositor.
|
||||
|
||||
### Alpha pre-multiplication?
|
||||
|
||||
Implicit in shaders (same as Greenfield). Works acceptably for typical
|
||||
Wayland content.
|
||||
|
||||
### OffscreenCanvas / Web Workers?
|
||||
|
||||
**Yes** — Render Worker (WebGL2 + OffscreenCanvas) + Decode Worker
|
||||
(WebCodecs VideoDecoder). Greenfield uses only 2 workers for the WASM
|
||||
fallback path; our MIT approach uses Workers for ALL rendering.
|
||||
|
||||
### Adaptive bitrate (NEW)?
|
||||
|
||||
**Yes** — `adaptive_bitrate.py` consumes `EncoderFeedback` data and
|
||||
adjusts `vaapih264enc`/`nvh264enc`/`openh264enc` bitrate at runtime.
|
||||
Greenfield produces the feedback data but doesn't use it.
|
||||
|
||||
## Notable design trade-offs for the MIT approach
|
||||
|
||||
1. **Hardware-accelerated encoding by default** (vaapih264 on Intel/AMD,
|
||||
nvh264 on NVIDIA). Greenfield defaults to x264 (software, GPL).
|
||||
2. **Hardware-accelerated decoding by default** (WebCodecs
|
||||
`prefer-hardware`). Greenfield uses `prefer-software` with a FIXME.
|
||||
3. **No GPL anywhere in the encoding/decoding pipeline** (no x264, no
|
||||
FFmpeg Wasm).
|
||||
4. **Adaptive bitrate** consumes the feedback data Greenfield ignores.
|
||||
5. **OpenH264 BSD Wasm** as fallback (Cisco's binary release or
|
||||
self-built). Sufficient for H.264 fallback when WebCodecs is
|
||||
unavailable.
|
||||
6. **OffscreenCanvas + Web Workers** for all rendering (Greenfield is
|
||||
main-thread).
|
||||
7. **WebGL2** instead of WebGL1 (uniform buffer objects, simplified
|
||||
shaders).
|
||||
8. **VideoFrame directly as WebGL texture source** (zero-copy from
|
||||
WebCodecs to GPU, no `copyTo` needed).
|
||||
9. **Same alpha-split H.264 approach** as Greenfield (doubles cost for
|
||||
transparent windows) — could be replaced with AV1 with native alpha
|
||||
support in the future.
|
||||
10. **Future: AV1/VP9 support** via WebCodecs (when browser support is
|
||||
widespread) — sidesteps H.264 MPEG-LA patent royalties entirely.
|
||||
|
||||
## File reference
|
||||
|
||||
### Browser side (custom, clean-room TS)
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `packages/compositor/src/transport/decoders/WebCodecsFrameDecoder.ts` | WebCodecs VideoDecoder (HW-accelerated) |
|
||||
| `packages/compositor/src/transport/decoders/OpenH264FrameDecoder.ts` | OpenH264 BSD Wasm fallback |
|
||||
| `packages/compositor/src/transport/decoders/FrameDecoder.ts` | Decoder interface |
|
||||
| `packages/compositor/src/transport/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||||
| `packages/compositor/src/transport/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
|
||||
| `packages/compositor/src/transport/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||||
| `packages/compositor/src/transport/BufferStream.ts` | Manages serial → frame promise resolution |
|
||||
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline (WebGL2 + OffscreenCanvas) |
|
||||
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
|
||||
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
|
||||
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
|
||||
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
|
||||
|
||||
### Browser side (REUSED MIT)
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `libs/compositor-wasm/src/index.ts` | pixman + xkbcommon WASM (region math + keymap) |
|
||||
|
||||
### Python side (custom, clean-room)
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `proxy/encoding/gst_encoder.py` | GStreamer H.264/PNG encoder (~700 LOC) |
|
||||
| `proxy/encoding/adaptive_bitrate.py` | Adaptive bitrate controller (NEW) |
|
||||
| `proxy/surface_buffer_encoding.py` | Encoding pipeline orchestration |
|
||||
| `proxy/proxy_buffer.py` | Proxy-side buffer representation |
|
||||
| `proxy/frame_feedback.py` | Receives 4-byte decode feedback |
|
||||
| `proxy/encoding/glib_thread.py` | GLib MainLoop in separate thread + asyncio marshalling |
|
||||
|
||||
### Vendored Greenfield C files (MIT, used via cffi)
|
||||
|
||||
| Path | Role |
|
||||
|---|---|
|
||||
| `native/westfield-egl.c` | EGL zero-copy setup |
|
||||
| `native/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM |
|
||||
| `native/wlr_drm.c` | `wl_drm` global impl |
|
||||
| `native/wlr_linux_dmabuf_v1.c` | `zwp_linux_dmabuf_v1` global impl |
|
||||
| `native/drm_format_set.c` | DRM format helpers |
|
||||
|
||||
### Reused (LGPL/BSD/MIT)
|
||||
|
||||
| Component | Role |
|
||||
|---|---|
|
||||
| PyGObject (`gi.repository.Gst`) | GStreamer Python bindings (LGPL-2.1+) |
|
||||
| GStreamer core | Media pipeline framework (LGPL-2.0+) |
|
||||
| gst-plugins-base, gst-plugins-good | PNG, appsrc, appsink, videoconvert (LGPL) |
|
||||
| gst-vaapi | `vaapih264enc` for Intel/AMD (LGPL + MIT libva) |
|
||||
| gst-plugins-bad (BSD subset) | `nvh264enc` for NVIDIA, `openh264enc` for CPU fallback |
|
||||
| libopenh264 | OpenH264 BSD Wasm fallback decoder (BSD-2-Clause) |
|
||||
Reference in new issue
Block a user