1b-agent/one-os-dev-wayland-html5-compositor-mit-compositor-analysis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
one-os-dev-wayland-html5-co…/09-hardware-accel.md
T

25 KiB
Raw Permalink Blame History

09 — Hardware Acceleration, Encoding, WebCodecs, GPU Textures

This chapter covers the encoding/decoding/WASM pipeline in the MIT approach. Critical changes from Greenfield:

  • No x264enc (GPL-2.0+) — replaced with vaapih264enc/nvh264enc/openh264enc
  • No FFmpeg Wasm (GPL-3.0 in Greenfield's build) — replaced with WebCodecs primary + OpenH264 BSD fallback
  • Hardware decode enabled (Greenfield disables it with a FIXME)
  • Adaptive bitrate (consumes EncoderFeedback data Greenfield produces but doesn't use)

What's reused (MIT)

Browser side

Component License Role
@gfld/compositor-wasm MIT pixman 0.42.2 + xkbcommon 1.5.0 WASM (region math + keymap)

Vendored C (MIT, used in Python proxy via cffi)

File Lines Role
westfield-dmabuf.c ~200 DMA-BUF import via EGL/GBM
westfield-egl.c ~900 EGL zero-copy setup (EGLDisplay, EGLContext, EGLImage)
wlr_drm.c ~244 wl_drm global impl
wlr_linux_dmabuf_v1.c ~1,110 zwp_linux_dmabuf_v1 global impl
drm_format_set.c ~239 DRM format helpers

What's replaced (proprietary, clean-room)

Browser side

Component Replaces Why
Custom WebCodecsFrameDecoder.ts Greenfield's webcodec-buffer-decoder.ts (AGPL) Clean-room; enables hardware decode (Greenfield disables with FIXME)
Custom OpenH264FrameDecoder.ts Greenfield's wasm-buffer-decoder.ts + H264NALDecoder.worker.ts (AGPL) + @gfld/compositor-ffmpeg-h264 (GPL) Clean-room; uses OpenH264 BSD Wasm instead of FFmpeg GPL Wasm
Custom YUVA2RGBA / RGBXandA2RGBA shaders Greenfield's (AGPL) Clean-room; WebGL2 (Greenfield uses WebGL1)
Custom Renderer / Scene / SceneShader Greenfield's (AGPL) Clean-room; OffscreenCanvas + Web Workers (Greenfield is main-thread)

Python side

Component Replaces Why
Custom gst_encoder.py (~700 LOC) Greenfield's proxy-encoding-addon.node (AGPL, 290 lines C) + libproxy-encoding.so (AGPL, ~1400 lines C) Pure Python via PyGObject. Same GStreamer, no N-API addon.
vaapih264enc / nvh264enc / openh264enc x264enc (GPL) GPL-free H.264 encoding

Encoder selection (per-frame, deterministic)

Same logic as Greenfield, but without x264enc:

name element when used split_alpha coded-size multiple License
vaapih264 vaapih264enc >256×256 + ARGB/XRGB/dmabuf, Intel/AMD GPU true (parallel alpha H.264 stream) 128 LGPL + MIT (libva)
nvh264 nvh264enc same, NVIDIA GPU true 128 BSD plugin + proprietary NVENC blob
openh264 openh264enc same, no GPU (CPU fallback) true 128 BSD-2-Clause (libopenh264)
png pngenc ≤256×256 + ARGB/XRGB/dmabuf false (PNG has native alpha) 16 LGPL

Rule: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264 encoder (selected by --encoder=vaapih264|nvh264|openh264).

NOTE: This means the default encoder is hardware-accelerated (vaapih264 on Intel/AMD, nvh264 on NVIDIA). Greenfield defaults to x264 (software, GPL).

Canonical H.264 pipeline (Python equivalent)

Same GStreamer pipeline as Greenfield, ported to PyGObject:

# gst_encoder.py (clean-room, ~700 LOC)
import gi
gi.require_version("Gst", "1.0")
gi.require_version("GstGL", "1.0")
gi.require_version("GstEGL", "1.0")  # if available
from gi.repository import Gst, GstGL, GLib

class H264Encoder:
    def __init__(self, egl_display_ptr, egl_context_ptr, encoder_type="vaapih264"):
        # Create GLDisplay + GLContext from existing EGL context (zero-copy)
        self.gl_display = GstGL.GLDisplayEGL.new_with_egl_display(egl_display_ptr)
        self.gl_context = GstGL.GLContext.new_wrapped(
            self.gl_display,
            egl_context_ptr,
            GstGL.GLPlatform.EGL,
            GstGL.GLLAPI.GLES2,
        )
        
        # Encoder pipeline
        encoder_element = {
            "vaapih264": "vaapih264enc",
            "nvh264":    "nvh264enc",
            "openh264":  "openh264enc",
        }[encoder_type]
        
        self.pipeline = Gst.parse_launch(
            f"appsrc name=src ! "
            f"glupload ! glcolorconvert ! "
            f"glshader name=scale ! "  # pads to macroblock multiple
            f"glcolorconvert ! video/x-raw(memory:GLMemory),format=NV12 ! "
            f"gldownload ! queue ! "
            f"{encoder_element} tune=zerolatency bitrate=12000 vbv-buf-capacity=160 "
            f"bframes=0 rc-lookahead=0 sliced-threads byte-stream=true ! "
            f"video/x-h264,profile=high,stream-format=byte-stream,alignment=au ! "
            f"appsink name=sink"
        )
        
        # Wire up appsink callback (zero-copy external Buffers)
        sink = self.pipeline.get_by_name("sink")
        sink.emit("set-property", "emit-signals", True)
        sink.connect("new-sample", self._on_new_sample)
        
        # Alpha pipeline (separate, for transparent windows)
        self.alpha_pipeline = self._build_alpha_pipeline(encoder_element)
        
        # GLib MainLoop in separate thread (GStreamer requires it)
        self.glib_loop = GLib.MainLoop()
        self.glib_context = GLib.MainContext.new()
        GLib.MainContext.push_thread_default(self.glib_context)
        import threading
        self.glib_thread = threading.Thread(target=self.glib_loop.run, daemon=True)
        self.glib_thread.start()
    
    def encode_frame(self, buffer_id, content_serial, creation_serial, dma_buf_fd, w, h):
        # Push buffer into appsrc (zero-copy via GstBuffer wrapping the dmabuf)
        # ... marshalled from asyncio via GLib.idle_add
        
    def _on_new_sample(self, appsink):
        sample = appsink.emit("pull-sample")
        buffer = sample.get_buffer()
        # Marshal back to asyncio
        asyncio.run_coroutine_threadsafe(
            self._deliver_frame(buffer, self.current_buffer_id),
            self.asyncio_loop,
        )

Bitrates (same as Greenfield)

  • Opaque: 12000 (vaapih264 / nvh264 / openh264)
  • Alpha: 1200 (10% of opaque)

Adaptive bitrate (NEW — Greenfield lacks)

Consume EncoderFeedback data (refreshInterval + decode duration):

# adaptive_bitrate.py (clean-room, ~150 LOC)
class AdaptiveBitrate:
    def __init__(self, encoder):
        self.encoder = encoder
    
    def on_feedback(self, refresh_interval_ms, avg_decode_ms):
        # If decode is taking too long, reduce bitrate
        if avg_decode_ms > refresh_interval_ms * 0.8:
            new_bitrate = max(self.current_bitrate * 0.7, MIN_BITRATE)
            self.encoder.set_property("bitrate", new_bitrate)
        # If decode is fast and we have headroom, increase bitrate
        elif avg_decode_ms < refresh_interval_ms * 0.3:
            new_bitrate = min(self.current_bitrate * 1.3, MAX_BITRATE)
            self.encoder.set_property("bitrate", new_bitrate)

Wire format (REUSED from Greenfield)

Same binary format as Greenfield (see ../greenfield-analysis/09-hardware-accel.md):

struct packed props {
    u32 buffer_id, creation_serial, content_serial,
    u32 frame_encoding_type (0=h264, 1=png),
    u32 width, height, coded_width, coded_height
}
u32 opaque_length
u8  opaque[opaque_length]    // H.264 AU or PNG blob
u32 alpha_length             // 0 if no alpha split
u8  alpha[alpha_length]      // H.264 AU of alpha-only stream

Browser decode (clean-room TS)

Preferred path: WebCodecs VideoDecoder (with HW decode enabled)

// WebCodecsFrameDecoder.ts (clean-room, ~250 LOC)
export class WebCodecsFrameDecoder implements FrameDecoder {
    private opaqueDecoders: Map<SurfaceId, VideoDecoder> = new Map();
    private alphaDecoders: Map<SurfaceId, VideoDecoder> = new Map();
    
    async isSupported(): Promise<boolean> {
        const config = {
            codec: 'avc1.64001f',
            optimizeForLatency: true,
            // NEW: enable hardware acceleration (Greenfield disables this)
            hardwareAcceleration: 'prefer-hardware',
        };
        const result = await VideoDecoder.isConfigSupported(config);
        return result.supported;
    }
    
    async decode(surface: Surface, encodedFrame: EncodedFrame): Promise<DecodedFrame> {
        let opaqueDecoder = this.opaqueDecoders.get(surface.id);
        if (!opaqueDecoder) {
            opaqueDecoder = new VideoDecoder({
                output: (frame) => this.onDecoded(surface, frame),
                error: (e) => this.onError(surface, e),
            });
            opaqueDecoder.configure({
                codec: 'avc1.64001f',
                optimizeForLatency: true,
                hardwareAcceleration: 'prefer-hardware',  // NEW
            });
            this.opaqueDecoders.set(surface.id, opaqueDecoder);
        }
        
        const chunk = new EncodedVideoChunk({
            type: encodedFrame.isKeyframe ? 'key' : 'delta',
            data: encodedFrame.opaque,
            timestamp: encodedFrame.timestamp,
        });
        opaqueDecoder.decode(chunk);
        
        // Alpha path (if present)
        if (encodedFrame.alpha) {
            // Same pattern for alphaDecoder
        }
    }
}

Key difference from Greenfield: We use hardwareAcceleration: 'prefer-hardware' (Greenfield uses 'prefer-software' with a FIXME about "frame offset is wrong"). This needs testing, but should give significant CPU savings.

Fallback path: OpenH264 BSD Wasm (NOT FFmpeg — FFmpeg is GPL)

// OpenH264FrameDecoder.ts (clean-room, ~250 LOC)
// Uses OpenH264 BSD-2-Clause Wasm build (Cisco's binary release or self-built)
// Available from: https://github.com/cisco/openh264 (BSD-2-Clause)

OpenH264 is BSD-2-Clause — fully proprietary-compatible. Less featureful than FFmpeg (no VP9/AV1), but sufficient for H.264 fallback.

WebGL2 upload (clean-room TS, modernized)

WebGL2 with desynchronized:true + OffscreenCanvas in Web Worker.

Per Scene (canvas/output), three GL helpers (same as Greenfield, but WebGL2 instead of WebGL1):

YUVA2RGBA

FBO pass: uploads Y/U/V/alphaY as 4× LUMINANCE textures (WebGL1 compatible) OR 1× R8UI texture array (WebGL2, more efficient), renders into the surface's RGBA texture via YUVA2RGBAShader (BT.601 coefficients).

RGBXandA2RGBA

FBO pass for the DualPlaneRGBA* WebCodecs outputs (opaque + alpha planes composited via RGBandA2RGBAShader).

SceneShader

Final pass: draws every view's RGBA texture to the canvas with gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA).

NEW: WebGL2 texStorage2D for immutable textures (faster upload). NEW: VideoFrame directly as a WebGL texture source (zero-copy from WebCodecs to GPU, no copyTo needed).

End-to-end pipeline

┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
│  App renders via GTK/Qt/raw EGL into                          │
│   ┌────────────────────────────────────┐                       │
│   │  wl_shm buffer  (ARGB8888/XRGB8888)│  ─── or ─────────┐    │
│   │  zwp_linux_dmabuf_v1 (DMA-BUF)     │                  │    │
│   │  wl_drm create_prime_buffer        │                  │    │
│   └────────────────────────────────────┘                  │    │
└────────────────────────┬──────────────────────────────────┼────┘
                         │ Wayland socket (UNIX domain)     │
                         ▼                                  │
┌────────────────────── PYTHON PROXY (aiohttp + pywayland + GStreamer)─┐
│  pywayland (Apache-2.0)                                       │
│   └─ Display → Client → Resource (buffer)                    │
│   └─ display.init_shm(), init_drm("/dev/dri/renderD128")      │
│   └─ Vendored westfield-egl.c (MIT, via cffi):                │
│      EGLDevice + EGLDisplay + EGLContext                     │
│                                                              │
│  gst_encoder.py (~700 LOC Python, clean-room)                  │
│   ┌─ frame_buffer {type: SHM|DMA, w, h, fmt}                  │
│   │                                                             │
│   │   GLib MainLoop in separate thread (GLib.idle_add)          │
│   │    │                                                        │
│   │    ▼                                                        │
│   │   gst_parse_launch:                                         │
│   │    ┌─ Encoder selection (per-frame):                       │
│   │    │   pixel_count ≤ 256×256  ─►  PNG pipeline            │
│   │    │   > 256×256 + ARGB/XRGB/dmabuf ─►  configured h264    │
│   │    │                                                       │
│   │    │  H.264 pipelines (parallel, NO x264 — GPL):           │
│   │    │   opaque: appsrc → glupload → glcolorconvert →        │
│   │    │            glshader(scale→coded_size) →              │
│   │    │            glcolorconvert(NV12) → gldownload →        │
│   │    │            queue → vaapih264enc|nvh264enc|openh264enc │
│   │    │            → byte-stream AU → appsink                │
│   │    │   alpha:   same source, fragment shader              │
│   │    │            `vec4(pix.a,pix.a,pix.a,0.)` →             │
│   │    │            second H.264 stream                        │
│   │    │                                                       │
│   │    │  PNG pipeline (single):                              │
│   │    │   appsrc → glupload → glcolorconvert → glshader →     │
│   │    │   RGBA → gldownload → pngenc → appsink                │
│   │    │                                                       │
│   │    └─ asyncio.run_coroutine_threadsafe for cross-thread    │
│   │                                                          │
│   └─ surface_buffer_encoding.py                              │
│       └─ frame_data_channel.send(buffer)                      │
│                                                              │
│  channel.py:                                                  │
│   ARQChannel   (kcp PyPI package, MIT, over WebSocket)         │
│   SimpleChannel (raw WebSocket, for FEEDBACK only)             │
│       ▲                                                       │
│       │ frame_feedback.py: 16-bit rAF interval + 16-bit avg   │
│       │ decode duration → used to pace wl_frame.done AND       │
│       │ drive adaptive_bitrate.py (NEW vs Greenfield)          │
└───────┼───────────────────────────────────────────────────────┘
        │ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
        ▼
┌────────────────────── BROWSER (Custom Compositor TS, clean-room) ──────┐
│  WebSocket → ReconnectingWebSocket (vendored MIT)                       │
│   ↓                                                             │
│  channel.ts: ARQChannel.input(kcp) → onMessage                  │
│   ↓                                                             │
│  RemoteAppLauncher.setupFrameDataChannel →                      │
│   BufferStream.deliverContentToBufferStream (parse serials)      │
│   ↓                                                             │
│  StreamingBuffer.getContents(surface, serial)                    │
│   ↓                                                             │
│  Session.frameDecoder.decode(surface, encodedFrame)            │
│   │                                                             │
│   ├─ PNG path (both decoders):                                  │
│   │    new Blob([opaque], {type:'image/png'}) →                 │
│   │    createImageBitmap(blob, sx, sy, w, h)  // crop padding  │
│   │    → SinglePlane { bitmap: ImageBitmap }                   │
│   │                                                             │
│   └─ H.264 path:                                                │
│      (preferred) WebCodecs VideoDecoder                        │
│         config: avc1.64001f, prefer-hardware (NEW!),           │
│                 optimizeForLatency                              │
│         opaqueDecoder + alphaDecoder per surface               │
│         EncodedVideoChunk {type:key|delta, data:nal}            │
│         → VideoFrame.output →                                  │
│           if I420: copyTo → DualPlaneYUVAArrayBuffer           │
│           else:    keep VideoFrame → DualPlaneRGBAVideoFrame    │
│      (fallback) OpenH264 BSD Wasm in 2 Web Workers             │
│         (NOT FFmpeg — GPL contamination)                       │
│         worker.postMessage(nal, [nal.buffer])  // transferable   │
│         OpenH264._decode → I420 planes                          │
│         postMessage back with [y.buffer, u.buffer, v.buffer]    │
│         → DualPlaneYUVASplitBuffer                              │
│                                                                 │
│  Renderer.render()  (rAF-driven, in Web Worker via OffscreenCanvas)│
│   ↓                                                             │
│  Scene.render(viewStack)                                        │
│   ├─ Scene['image/png']  → texImage2D(ImageBitmap)              │
│   ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D   │
│   └─ Scene['video/h264']:                                       │
│       YUVA2RGBA.convertYUVASplitBufferInto /                    │
│                   convertYUVAArrayBufferInto:                    │
│         Y/U/V/alphaY → 4× LUMINANCE textures (WebGL1 compat)  │
│         OR 1× R8UI texture array (WebGL2, more efficient)       │
│         bind FBO → renderState.texture (RGBA) as color attach   │
│         YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex     │
│       or RGBXandA2RGBA.convertInto (WebCodecs RGBA):            │
│         VideoFrame/ImageBitmap → texImage2D directly            │
│   ↓                                                             │
│  SceneShader.draw(): drawArrays with surface's RGBA texture,    │
│  blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas    │
│   ↓                                                             │
│  Browser composites canvas into the page                       │
└─────────────────────────────────────────────────────────────────┘


Same-frame web app fast path (web-bitmapbuf, REUSED):
┌───────────────┐     MessagePort.postMessage([view, ...fds],      ┌──────────────┐
│  Web app      │ ──►  [imageBitmap]) ──────────────────────────►  │ Compositor   │
│  (iframe)     │     web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
└───────────────┘     → wl_buffer → ImageBitmapBuffer            └──────────────┘
                          (no encoding, no WebSocket, no GStreamer)

Key answers (quick reference)

Why H.264 over PNG?

H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB. PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split doubling dominate at small sizes.

WebCodecs?

Yes — preferred path. Hardware decode enabled (Greenfield disables with FIXME). OpenH264 BSD Wasm is the fallback (NOT FFmpeg — GPL).

Trade-offs?

  • H.264 = small wire / decode cost
  • PNG = medium wire / browser-native fast decode (createImageBitmap)
  • raw SHM = not used between proxy and browser — every surface commit is encoded.

Per-surface policy?

No — per-frame deterministic. Only configurable knob is --encoder=vaapih264|nvh264|openh264. PNG is always automatic fallback.

Cursors?

Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS cursor via setBrowserCursor — never enters the WebGL compositor.

Alpha pre-multiplication?

Implicit in shaders (same as Greenfield). Works acceptably for typical Wayland content.

OffscreenCanvas / Web Workers?

Yes — Render Worker (WebGL2 + OffscreenCanvas) + Decode Worker (WebCodecs VideoDecoder). Greenfield uses only 2 workers for the WASM fallback path; our MIT approach uses Workers for ALL rendering.

Adaptive bitrate (NEW)?

Yes — adaptive_bitrate.py consumes EncoderFeedback data and adjusts vaapih264enc/nvh264enc/openh264enc bitrate at runtime. Greenfield produces the feedback data but doesn't use it.

Notable design trade-offs for the MIT approach

  1. Hardware-accelerated encoding by default (vaapih264 on Intel/AMD, nvh264 on NVIDIA). Greenfield defaults to x264 (software, GPL).
  2. Hardware-accelerated decoding by default (WebCodecs prefer-hardware). Greenfield uses prefer-software with a FIXME.
  3. No GPL anywhere in the encoding/decoding pipeline (no x264, no FFmpeg Wasm).
  4. Adaptive bitrate consumes the feedback data Greenfield ignores.
  5. OpenH264 BSD Wasm as fallback (Cisco's binary release or self-built). Sufficient for H.264 fallback when WebCodecs is unavailable.
  6. OffscreenCanvas + Web Workers for all rendering (Greenfield is main-thread).
  7. WebGL2 instead of WebGL1 (uniform buffer objects, simplified shaders).
  8. VideoFrame directly as WebGL texture source (zero-copy from WebCodecs to GPU, no copyTo needed).
  9. Same alpha-split H.264 approach as Greenfield (doubles cost for transparent windows) — could be replaced with AV1 with native alpha support in the future.
  10. Future: AV1/VP9 support via WebCodecs (when browser support is widespread) — sidesteps H.264 MPEG-LA patent royalties entirely.

File reference

Browser side (custom, clean-room TS)

Path Role
packages/compositor/src/transport/decoders/WebCodecsFrameDecoder.ts WebCodecs VideoDecoder (HW-accelerated)
packages/compositor/src/transport/decoders/OpenH264FrameDecoder.ts OpenH264 BSD Wasm fallback
packages/compositor/src/transport/decoders/FrameDecoder.ts Decoder interface
packages/compositor/src/transport/EncodedFrame.ts Wire format parser for encoded frames
packages/compositor/src/transport/EncoderFeedback.ts Sends 4-byte decode feedback at 1 Hz
packages/compositor/src/transport/StreamingBuffer.ts Per-surface encoded-frame buffer
packages/compositor/src/transport/BufferStream.ts Manages serial → frame promise resolution
packages/compositor/src/render/Renderer.ts 4-phase render pipeline (WebGL2 + OffscreenCanvas)
packages/compositor/src/render/Scene.ts One per output, manages view stack + textures
packages/compositor/src/render/YUVA2RGBA.ts YUVA→RGBA FBO pass
packages/compositor/src/render/YUVA2RGBAShader.ts BT.601 YUVA→RGBA shader
packages/compositor/src/render/RGBXandA2RGBA.ts RGB+Alpha → RGBA FBO pass (WebCodecs output)

Browser side (REUSED MIT)

Path Role
libs/compositor-wasm/src/index.ts pixman + xkbcommon WASM (region math + keymap)

Python side (custom, clean-room)

Path Role
proxy/encoding/gst_encoder.py GStreamer H.264/PNG encoder (~700 LOC)
proxy/encoding/adaptive_bitrate.py Adaptive bitrate controller (NEW)
proxy/surface_buffer_encoding.py Encoding pipeline orchestration
proxy/proxy_buffer.py Proxy-side buffer representation
proxy/frame_feedback.py Receives 4-byte decode feedback
proxy/encoding/glib_thread.py GLib MainLoop in separate thread + asyncio marshalling

Vendored Greenfield C files (MIT, used via cffi)

Path Role
native/westfield-egl.c EGL zero-copy setup
native/westfield-dmabuf.c DMA-BUF import via EGL/GBM
native/wlr_drm.c wl_drm global impl
native/wlr_linux_dmabuf_v1.c zwp_linux_dmabuf_v1 global impl
native/drm_format_set.c DRM format helpers

Reused (LGPL/BSD/MIT)

Component Role
PyGObject (gi.repository.Gst) GStreamer Python bindings (LGPL-2.1+)
GStreamer core Media pipeline framework (LGPL-2.0+)
gst-plugins-base, gst-plugins-good PNG, appsrc, appsink, videoconvert (LGPL)
gst-vaapi vaapih264enc for Intel/AMD (LGPL + MIT libva)
gst-plugins-bad (BSD subset) nvh264enc for NVIDIA, openh264enc for CPU fallback
libopenh264 OpenH264 BSD Wasm fallback decoder (BSD-2-Clause)
Reference in new issue View Git Blame Copy Permalink