From c4b45cf2180d382e2d47e6665ea28c19bb891f75 Mon Sep 17 00:00:00 2001 From: 1b-agent Date: Thu, 8 Oct 2026 08:48:25 +0000 Subject: [PATCH] Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents) --- .gitignore | 24 ++ 00-overview.md | 386 +++++++++++++++++++ 01-architecture.md | 443 ++++++++++++++++++++++ 02-wayland-protocol.md | 325 ++++++++++++++++ 03-html5-rendering.md | 379 +++++++++++++++++++ 04-transport.md | 397 ++++++++++++++++++++ 05-input-handling.md | 366 ++++++++++++++++++ 06-data-clipboard.md | 314 ++++++++++++++++ 07-native-app-bridge.md | 404 ++++++++++++++++++++ 08-app-lifecycle.md | 304 +++++++++++++++ 09-hardware-accel.md | 423 +++++++++++++++++++++ 10-porting-to-odoo-js-python.md | 644 ++++++++++++++++++++++++++++++++ 11-pitfalls.md | 532 ++++++++++++++++++++++++++ 12-appendix.md | 400 ++++++++++++++++++++ README.md | 103 +++++ 15 files changed, 5444 insertions(+) create mode 100644 .gitignore create mode 100644 00-overview.md create mode 100644 01-architecture.md create mode 100644 02-wayland-protocol.md create mode 100644 03-html5-rendering.md create mode 100644 04-transport.md create mode 100644 05-input-handling.md create mode 100644 06-data-clipboard.md create mode 100644 07-native-app-bridge.md create mode 100644 08-app-lifecycle.md create mode 100644 09-hardware-accel.md create mode 100644 10-porting-to-odoo-js-python.md create mode 100644 11-pitfalls.md create mode 100644 12-appendix.md create mode 100644 README.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..4d5081d --- /dev/null +++ b/.gitignore @@ -0,0 +1,24 @@ +# Python +__pycache__/ +*.pyc +*.pyo +.venv/ +venv/ + +# OS +.DS_Store +Thumbs.db + +# Editor +.vscode/ +.idea/ +*.swp + +# Logs +*.log + +# Build +node_modules/ +dist/ +build/ +*.egg-info/ diff --git a/00-overview.md b/00-overview.md new file mode 100644 index 0000000..8403860 --- /dev/null +++ b/00-overview.md @@ -0,0 +1,386 @@ +# 00 — What Greenfield Is and How It's Structured + +## The one-paragraph summary + +Greenfield is a Wayland compositor that runs partly in the browser (the actual +compositing and rendering, written in TypeScript against WebGL / WebCodecs / +WASM) and partly on a Linux host (a Node.js process that hosts real Linux +Wayland apps via `libwayland-server` and bridges their wire traffic to the +browser over WebSocket). The two halves together let you display **real Linux +applications** (GTK3/4, Qt, xterm, X11 apps via XWayland) inside a browser tab, +as if they were native browser apps. An optional Emscripten-based SDK lets you +compile separate Wayland apps to WASM and run them entirely in the browser with +no Linux host required. + +## Repo layout + +``` +greenfield/ +├── README.md # "The in-browser wayland compositor" +├── package.json # yarn 4.5.0 workspaces: libs/*, packages/*, examples/webapps/*, examples/compositor/* +├── .yarnrc.yml # nodeLinker: pnpm, global cache off +├── yarn.lock # 9368 lines +├── publish.sh # yarn workspaces foreach npm publish +├── protocol/ # Wayland XML protocol definitions (4 files) +│ ├── wayland.xml # Core Wayland protocol (124KB) +│ ├── xdg-shell.xml # Desktop shell protocol (48KB) +│ ├── wayland-drm.xml # DRM buffer formats (8.8KB) +│ └── web-bitmapbuf-unstable-v1.xml # *** Greenfield custom extension (2KB) *** +├── libs/ # 9 internal @gfld/* libraries +│ ├── common/ # @gfld/common - shared client/server runtime (wire format) +│ ├── client-protocol/ # @gfld/client-protocol - browser-side generated Wayland bindings +│ ├── compositor-protocol/ # @gfld/compositor-protocol - Node.js-side generated Wayland bindings +│ ├── client-generator/ # CLI: XML → TS browser protocol source +│ ├── compositor-generator/ # CLI: XML → TS Node protocol source +│ ├── compositor-proxy-generator/ # CLI: XML → TS proxy protocol source +│ ├── compositor-wasm/ # pixman + xkbcommon compiled to WASM +│ ├── compositor-ffmpeg-h264/ # FFmpeg n5.1.2 compiled to WASM (H.264 software fallback decoder) +│ └── xtsb/ # X11/XCB TypeScript bindings (vendored) +├── packages/ # 4 user-facing @gfld/* packages +│ ├── compositor/ # @gfld/compositor - browser-side Wayland compositor library (AGPL) +│ ├── compositor-shell/ # @gfld/compositor-shell - Preact+Tailwind SPA shell (AGPL) *** OUT OF SCOPE *** +│ ├── compositor-proxy/ # @gfld/compositor-proxy - native Wayland compositor proxy (AGPL, Linux-only) +│ └── compositor-proxy-cli/ # @gfld/compositor-proxy-cli - CLI binary (AGPL, Linux-only) +├── sdk/ # Greenfield WebAssembly SDK (for browser-resident WASM apps) +│ ├── README.md +│ ├── build.sh # clones emsdk 3.1.60, builds 30+ sysrootlibs +│ ├── sdk_env.sh # exports PKG_CONFIG_LIBDIR + PATH=toolkit/bin +│ ├── toolkit/ # gfcc/gf++/gfar/gfnm wrappers around emcc + wayland-scanner + meson cross inis +│ └── sysrootlibs/ # 34 build scripts for cross-compiling graphics stack (wayland, pixman, cairo, pango, gtk4, etc.) +├── examples/ +│ ├── compositor/experimental-fullscreen/ # vite PWA kiosk running a single remote app +│ ├── docker-compose/ # Dockerfile + applications.json + docker-compose.yml for gtk3-demo/gtk4-demo/xterm/glxgears +│ ├── sdk/simple-shm/ # Weston simple-shm.c built with gfcc + wayland-scanner +│ ├── sdk/weston/ # Weston clients (terminal, dnd, editor, keyboard, flower, ...) meson-built +│ ├── sdk/gtk4/ # GTK4 demos meson-built to WASM +│ └── webapps/ # Pure-JS browser demos: simple-shm, webgl, weston-clients (symlink), gtk4 +├── docs/ # Jekyll site (CNAME: udev.be/greenfield) +│ └── pages/{components, design, future_plans, getting_started, sdk} +└── docker/ # Production Dockerfile for compositor-proxy-cli +``` + +## The dual-side architecture + +Greenfield is split into **two cooperating halves**, connected over WebSocket +(or, in the future, WebTransport). Understanding this split is fundamental to +everything else in this bundle. + +``` +┌─────────────────────────────────────────────────────────────────────────────┐ +│ BROWSER SIDE (TypeScript + WASM) │ +│ │ +│ ┌─────────────────────────┐ ┌──────────────────────────────────────────┐ │ +│ │ @gfld/compositor-shell │ │ @gfld/compositor │ │ +│ │ (Preact + Tailwind SPA)│───▶│ - Session, Compositor, Surface, Seat │ │ +│ │ *** OUT OF SCOPE *** │ │ - Keyboard, Pointer, Touch, Xkb (WASM) │ │ +│ └─────────────────────────┘ │ - Output (WebGL canvas) │ │ +│ │ - render/ (WebGL shaders: YUV→RGB, etc.) │ │ +│ ┌─────────────────────────┐ │ - desktop/ (Floating vs AlwaysFullScreen) │ │ +│ │ JS/WASM webapps │───▶│ - remote/ (Channel, kcp.ts, decoders) │ │ +│ │ (use @gfld/client- │ │ - browser/ (dnd, selection, input) │ │ +│ │ protocol + common) │ │ │ │ +│ └─────────────────────────┘ └──────────────┬──────────────┬─────────────┘ │ +│ │ │ │ +│ depends on ▼ ▼ │ +│ ┌─────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ +│ │ @gfld/compositor│ │ @gfld/common │ │ @gfld/xtsb │ │ +│ │ -protocol │ │ (Connection, │ │ (X11/XCB bindings,│ │ +│ │ (Node-side │ │ wire format)│ │ used by both │ │ +│ │ generated │ │ │ │ compositor+proxy)│ │ +│ │ bindings) │ │ │ └──────────────────┘ │ +│ └─────────────────┘ └──────────────┘ │ +└─────────────────────────────────────────────────────────────────────────────┘ + │ + │ WebSocket (TCP, perMessageDeflate=false) + │ + KCP-over-WebSocket (reliable UDP, ready for WebTransport) + │ + 9 endpoint paths per session + ▼ +┌─────────────────────────────────────────────────────────────────────────────┐ +│ NATIVE SIDE (Linux, Node 18+, C) │ +│ │ +│ ┌──────────────────────────┐ ┌──────────────────────────────────────────┐ │ +│ │ @gfld/compositor-proxy-cli│───▶│ @gfld/compositor-proxy │ │ +│ │ - main.ts: http + ws │ │ - Session, SessionController │ │ +│ │ WebSocketServer │ │ - NativeWaylandClientSession │ │ +│ │ - forks SessionProcess │ │ - NativeWaylandCompositorSession │ │ +│ │ per browser session │ │ - NativeAppContext (launch) │ │ +│ │ - --bind-port=8081 │ │ - SurfaceBufferEncoding │ │ +│ │ - --applications= │ │ - Channel, FrameFeedback │ │ +│ │ - --encoder=x264|nvh264 │ │ - XWaylandSession │ │ +│ │ - --render-device=/dev/ │ │ - kcp.ts (1100-line TypeScript impl) │ │ +│ │ dri/renderD128 │ │ - wayland-server.ts (TS wrapper) │ │ +│ └──────────────────────────┘ │ - io/, encoding/, protocol/ │ │ +│ └──────────────┬──────────────┬─────────────┘ │ +│ │ │ │ +│ depends on ▼ ▼ │ +│ ┌────────────────────────┐ ┌────────────────────────────────┐ │ +│ │ @gfld/compositor-proxy-│ │ native/ (C, built via CMake + │ │ +│ │ generator │ │ Ninja) │ │ +│ │ (generates *_interface │ │ - wayland/src/addon.c │ │ +│ │ and *_interceptor.js) │ │ - encoding/src/gst_frame_ │ │ +│ └────────────────────────┘ │ encoder.c │ │ +│ │ - poll/src/poll.c (libuv) │ │ +│ │ - wayland-server/ (patched) │ │ +│ │ - westfield-*.c (EGL, dmabuf, │ │ +│ │ XWayland, etc.) │ │ +│ └────────────────────────────────┘ │ +│ │ +│ Native Wayland apps (GTK3/4, Qt, xterm, X11 via XWayland) attach to the │ +│ proxy as if it were a real wl_display. The proxy encodes each surface's │ +│ buffer to H.264 via GStreamer (x264 software or nvh264 hardware) using │ +│ zero-copy dmabuf, sends frames over WebSocket/KCP to the browser. │ +└─────────────────────────────────────────────────────────────────────────────┘ +``` + +## Tech-stack table + +| Category | Component | Tech / Version | Role | +|---|---|---|---| +| **Languages** | Browser TS | TypeScript ^5.5.2 | All `@gfld/*` TS packages | +| | Native C | C99 | `packages/compositor-proxy/native/` | +| | Shell | bash | sdk/build.sh, toolkit wrappers | +| | Python | python3 (xcbgen) | libs/xtsb protocol generation | +| **Build tools** | Workspace | Yarn 4.5.0 (Berry, pnpm nodeLinker) | Monorepo workspace mgmt | +| | Browser bundler | Vite ^5.3.1 | compositor-shell, examples | +| | TS compiler | tsc ^5.5.2 | libs/* and packages/* build | +| | Native build | CMake 3.13+ + Ninja | packages/compositor-proxy `build:native` | +| | Native pkg | @vercel/pkg (Brotli) + @vercel/ncc | compositor-proxy-cli single-binary | +| | Lint/format | ESLint ^8.57 + Prettier ^3.3.2 | All TS packages | +| | Test | Jest ^29.7.0 + ts-jest ^29.1.5 | libs/xtsb, packages/compositor-proxy | +| **WASM toolchain** | Emscripten (libs build) | emsdk 3.1.46 | libs/compositor-wasm, libs/compositor-ffmpeg-h264 | +| | Emscripten (SDK build) | emsdk 3.1.60 | sdk/build.sh (sysrootlibs) | +| | Meson cross | meson-gf-cross.ini (host_machine: wasm32/emscripten) | sdk/toolkit | +| **Wayland / GL stack** | wayland.xml / xdg-shell.xml / wayland-drm.xml | upstream freedesktop.org | protocol/ | +| | wayland-scanner | vendored at sdk/toolkit/bin/wayland-scanner | C header/codegen for native clients | +| | libwayland (server) | vendored + patched at packages/compositor-proxy/native/wayland/src/wayland-server/ | native compositor server side | +| | wl_drm + linux-dmabuf-unstable-v1 | implemented in native/wayland/src/wlr_*.c | zero-copy dmabuf to GStreamer | +| **Custom protocol** | web-bitmapbuf-unstable-v1.xml | Greenfield extension (Erik De Rijcke, 2023) | browser JS/WASM clients wrap HTML ImageBitmap as wl_buffer via fd | +| **Native GL/media runtime** | GStreamer | 1.20 (built from gitlab.freedesktop.org in docker) | H.264 encoding pipeline, plugins: base/good/bad/ugly/gl + nvcodec | +| | GStreamer encoders | x264 (software), nvh264 (NVIDIA hardware), vaapih264 (Intel, experimental) | --encoder flag on compositor-proxy-cli | +| | Mesa / OpenGL / EGL | libgbm1, libdrm2, libopengl0, libglvnd0, libegl1-mesa | GPU buffer mgmt + EGL rendering | +| | NVIDIA | libnvidia-egl-wayland1, libnvidia-egl-gbm1 | optional HW accel | +| **Browser transport** | WebSocket | ws ^8.17.1 (Node), native WebSocket (browser) | primary transport (TCP) | +| | KCP | TypeScript impl (1100+ lines per side) | planned reliable UDP — future switch to WebTransport | +| | WebTransport | mentioned in docs/design as future | not yet implemented | +| | XWayland | xwayland package + xauth + xxd | X11 backward compat on native side | +| **Browser rendering** | WebGL | WebGL1 native, `desynchronized:true` | Output.ts canvas, render/*.ts shaders | +| | WebCodecs API | @types/dom-webcodecs ^0.1.11 | H.264 GPU decode in browser (preferred path) | +| | WebAssembly (FFmpeg) | libs/compositor-ffmpeg-h264 (FFmpeg n5.1.2) | software fallback H.264 decode | +| | WebAssembly (pixman+xkbcommon) | libs/compositor-wasm (emsdk 3.1.46) | pixel ops + XKB keymap math | +| | SharedArrayBuffer + Web Workers | browser-native (COOP/COEP required) | async compositor, H.264 NAL decoder workers | +| | ImageBitmap | browser-native | web-bitmapbuf-unstable-v1 backing store | +| **Shell UI (out of scope)** | Preact | ^10.22.0 | compositor-shell SPA | +| | Tailwind CSS | ^3.4.4 | compositor-shell styling | +| | @preact/signals | ^1.2.3 | reactivity | + +## Package inventory + +### Browser side + +**`packages/compositor`** (`@gfld/compositor`, AGPL-3.0) — The browser-side +Wayland compositor library. Pure TypeScript + WebGL + WebCodecs + WASM, no +3rd-party runtime deps. Exports `createCompositorSession()` and +`createAppLauncher(session, 'web' | 'remote')`. Internally: `Session.ts` +orchestrates, `Compositor.ts`/`Surface.ts`/`Seat.ts`/`Pointer.ts`/ +`Keyboard.ts`/`Output.ts` implement Wayland semantics, `render/` holds WebGL +shaders (YUV/YUVA/RGB→RGBA conversion), `desktop/` arranges surfaces +(Floating vs AlwaysFullScreen), `web/` speaks to same-origin-iframe JS/WASM +clients, `remote/` speaks to the native proxy (with `kcp.ts`, +`websocket-events.ts`, `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`, +`H264NALDecoder.worker.ts`, `xwayland/`), `browser/` handles dnd/selection/ +input. Output is a WebGL texture inside an HTML5 canvas. Async — a slow client +never blocks another. + +**`libs/client-protocol`** (`@gfld/client-protocol`, MIT) — Browser-side +generated Wayland protocol bindings (wayland.xml + xdg-shell.xml + +web-bitmapbuf-unstable-v1.xml). Used by JS/WASM webapps via `@gfld/common` +for the wire format. + +**`libs/compositor-protocol`** (`@gfld/compositor-protocol`, MIT) — Node-side +generated Wayland protocol bindings (same 3 XMLs). Used by +`@gfld/compositor` for the in-browser server-side marshalling of Wayland +messages. + +**`libs/common`** (`@gfld/common`, MIT) — Shared client/server Wayland +wire-format runtime. Two source files: `Connection.ts` (message framing + +argument codec) + `index.ts`. Tiny core that both `client-protocol` and +`compositor-protocol` build on. + +**`libs/compositor-wasm`** (`@gfld/compositor-wasm`, MIT) — pixman 0.42.2 + +xkbcommon 1.5.0 compiled to WASM via emsdk 3.1.46. Exposes +`src/index.ts` (`init` function). Used by `@gfld/compositor` for pixel-region +math and XKB keyboard layout state. + +**`libs/compositor-ffmpeg-h264`** (`@gfld/compositor-ffmpeg-h264`, Apache-2.0) +— FFmpeg n5.1.2 compiled to a single WASM module. TypeScript wrapper in +`src/{H264Decoder.ts, H264Worker.ts, index.ts}` runs in a Web Worker. Used as +the software fallback H.264 decoder when the browser lacks WebCodecs. + +### Native side + +**`packages/compositor-proxy`** (`@gfld/compositor-proxy`, AGPL-3.0, +Linux-only, Node ≥18) — Native Wayland compositor that bridges Linux apps to +the browser. Has BOTH a TypeScript layer (`src/Session.ts`, +`NativeWaylandClientSession.ts`, `NativeWaylandCompositorSession.ts`, +`NativeAppContext.ts`, `SurfaceBufferEncoding.ts`, `XWaylandSession.ts`, +`kcp.ts`, `Channel.ts`, `FrameFeedback.ts`, `wayland-server.ts`) AND a C +layer (`native/`) built with `cmake -G Ninja`. The C layer produces: + +- `wayland-server.so` — patched libwayland-server (vendored at + `native/wayland/src/wayland-server/`, with `changes.diff` and + `generate_wayland_server_patch.sh`) +- `westfield.so` — Greenfield's westfield-* helpers (buffer, dmabuf, egl, + xwayland, fdutils) + `wlr_drm.c` + `wlr_linux_dmabuf_v1.c` + + `drm-protocol.c` + `linux-dmabuf-unstable-v1-protocol.c` +- `proxy-encoding.so` — GStreamer frame encoder (`gst_frame_encoder.c` + + `gst_main_loop.c`) +- `proxy-encoding-addon.node` + `proxy-poll-addon.node` + + `wayland-server-addon.node` — Node-API addons (vendored node-api headers in + `native/node-api/`) loaded by `src/addons/` +- `poll.c` — epoll wrapper backed by libuv + +CMake deps: glib-2.0, gstreamer-1.0/app/video/gl/allocators ≥1.18, +graphene-1.0, egl ≥1.5, opengl ≥3.3, libffi, gbm, libdrm. Runtime also needs +libudev, mesa, NVIDIA EGL wayland/gbm drivers, xwayland. Implements `wl_drm` ++ `linux-dmabuf-v1` for zero-copy GPU buffer transfer to GStreamer encoder +pipeline (x264 software or nvh264 hardware). Connects to browser via +WebSocket; has TypeScript KCP implementation ready for the future WebTransport +switch. Supports direct peer-to-peer copy-paste between two proxy instances +to avoid browser round-trip. + +**`packages/compositor-proxy-cli`** (`@gfld/compositor-proxy-cli`, AGPL-3.0, +Linux-only, Node ≥18) — CLI binary that wraps `@gfld/compositor-proxy`. +`src/main.ts` runs an HTTP server + `ws` WebSocketServer on `--bind-port` +(default 8081), and on first connection per `compositorSessionId` query +param, `fork()`s a `SessionProcess` (src/SessionProcess.ts) — one child per +active browser session. CLI args: `--applications=` (URL→executable +map), `--encoder=x264|nvh264`, `--render-device=/dev/dri/renderD128`, +`--base-url`, `--allow-origin`, `--bind-ip`, `--bind-port`, `--basic-auth`. +The `--applications` JSON maps paths like `/gtk4-demo`, `/kwrite`, `/xterm` +to executables + args + env. Packaged with `@vercel/pkg` (Brotli, target +`node18-linux`) into a single static binary. + +**`libs/compositor-proxy-generator`** (`@gfld/compositor-proxy-generator`, +MIT) — CLI that converts `protocol/{wayland,xdg-shell,wayland-drm}.xml` into +TypeScript source under `packages/compositor-proxy/src/protocol/`. Used at +build time only (yarn `generate` script). + +### Shell side (OUT OF SCOPE for user) + +**`packages/compositor-shell`** (`@gfld/compositor-shell`, AGPL-3.0) — Preact ++ Tailwind single-page-app shell built with Vite. Depends only on +`@gfld/compositor` + `preact` at runtime. Provides application launcher, +keyboard config, window management UI. **Explicitly out of scope for the +user** per task brief. The protocol-level integration point the user needs +to know about is `UserShellApi` (see `08-app-lifecycle.md`). + +### Generators (3 sibling CLIs) + +All three generators share the same dependency set (`camelcase`, +`uppercamelcase`, `meow`, `xml2js`) and similar layout +(`ProtocolParser.mjs` + `ProtocolArguments.mjs` + `index.mjs`). They consume +the same `protocol/*.xml` files but emit slightly different TypeScript +flavors for each side: + +- **`libs/client-generator`** — emits browser-side client bindings into + `libs/client-protocol/src/protocol/`. +- **`libs/compositor-generator`** — emits Node.js-side server bindings into + `libs/compositor-protocol/src/protocol/`. +- **`libs/compositor-proxy-generator`** — emits proxy-side bindings into + `packages/compositor-proxy/src/protocol/`. Different file layout: + `src/EndpointProtocolArguments.mjs` + `EndpointProtocolParser.mjs` + + `index.mjs`. + +### XWayland support + +**`libs/xtsb`** (`@gfld/xtsb`, MIT, repo: github.com/udevbe/xtsb) — Vendored +TypeScript X11/XCB bindings. Builds from `xcbgen/ts_client.py` against 5 +XCB protocol XMLs in `libs/xtsb/protocol/`: `xproto.xml`, `shape.xml`, +`xfixes.xml`, `render.xml`, `composite.xml`. Dual main/module entries: +`dist/main/index.node.js` (Node) + `dist/module/index.web.js` (browser). +Used by BOTH `@gfld/compositor` (browser side, +`packages/compositor/src/remote/xwayland/`) AND `@gfld/compositor-proxy` +(native side, `src/XWaylandSession.ts`) — same TS API on both ends of the +wire. On the native C side, +`packages/compositor-proxy/native/wayland/src/westfield-xwayland.c` +actually launches XWayland and bridges it. + +### Wire between browser and native + +- **Transport today**: WebSocket over TCP. Browser connects to + `ws(s)://:/?compositorSessionId=`. + `compositor-proxy-cli` uses `ws` ^8.17.1 with `perMessageDeflate: false` + (compression off for latency). The `compositorSessionId` is generated by + the browser and used by the CLI to route the upgrade to the correct + per-session child process. +- **Transport future**: KCP-over-WebTransport (reliable UDP). The codebase + already has a full TS KCP implementation in both + `packages/compositor-proxy/src/kcp.ts` (~1100 lines) and + `packages/compositor/src/remote/kcp.ts`. `docs/pages/design/index.md` + explicitly calls out that WebSockets-over-TCP is ill-suited for real-time, + that WebRTC DataChannels' SCTP congestion control is too slow, and that + they're waiting for WebTransport + will pair it with KCP + + forward-error-correction. +- **Out-of-band data**: copy-paste between two proxy instances goes + peer-to-peer (proxy → proxy) without browser round-trip, configured via + `--base-url` so each proxy can advertise itself to peers. + +### SDK (`sdk/`) — for browser-resident WASM apps, NOT native apps + +The Greenfield SDK lets you compile existing native Wayland C/C++ apps to +WebAssembly and run them directly in the browser compositor (no proxy +needed). Based on Emscripten 3.1.60 with three Greenfield-specific +extensions: + +1. `sdk/toolkit/bin/gfcc` and `gf++` — wrapper scripts around `emcc`/`em++` + that inject pthread pool size, asyncify, and the unix-socket fs shim. +2. `sdk/toolkit/bin/wayland-scanner` — symlink to the natively-built + wayland-scanner inside `build-sysroot/bin/wayland-scanner`, used to + generate C client headers/source from xdg-shell.xml during app builds. +3. `sdk/sysrootlibs/` — 34 build scripts that cross-compile a full graphics + stack (wayland, wayland-protocols, pixman, zlib, png, xml2, xkbcommon, + icu, harfbuzz, freetype, fontconfig, pcre2, glib, jpeg, cairo, fribidi, + tiff4, pango, gdk-pixbuf, egl-stub, glesv2-stub, epoxy-emscripten, + graphene, sass, sassc, uapi-stub, upoll, cygepoll, expat, libffi; + gtk4 currently broken) into `sdk/sysroot/` for static linking. + +`sdk/build.sh` runs everything end-to-end and writes `meson-gf-toolchain.ini` +(with absolute `_SDK_DIR`). `sdk/sdk_env.sh` exports `PKG_CONFIG_LIBDIR` +(pointing at `sysroot/lib/pkgconfig` + `build-sysroot/lib/pkgconfig`) and +prepends `toolkit/bin` to `PATH`. Meson projects use +`--cross-file toolkit/meson-gf-cross.ini --cross-file toolkit/meson-gf-toolchain.ini` +(host_machine: cpu=mvp, cpu_family=wasm32, system=emscripten). + +**Important**: the `sdk/` is **not for native apps**. It's exclusively for +**WASM apps** that run inside the browser via Emscripten. Native Linux apps +on the proxy use the host's own system libraries — the Dockerfile installs +them via `apt-get install libffi8 libgbm1 libgstreamer1.0-plugins-base … +xwayland xauth`. + +## Versioning note + +Every workspace package is `1.0.0-rc1`. This signals: + +1. **Feature freeze for 1.0** — the architecture, protocol bindings, and + public APIs are expected to be stable; no breaking changes anticipated + before 1.0.0 final. +2. **Not yet production-blessed** — known issues remain (e.g. gtk4 WASM is + "broken for now" per `sdk/build.sh` comment; WebTransport/KCP-over-UDP + is designed but not yet active; Linux Kernel Library WASM + JuiceFS web + FS are explicitly future work). +3. **Yarn Berry semantics** — `workspace:^` resolution means all `@gfld/*` + deps are pinned to rc1 across the tree; if the user forks and bumps, + they bump the whole tree consistently. + +For the user's purpose (building their own Wayland-to-HTML compositor for an +Odoo 20 + JS/Python web OS), the rc1 status means they should treat +Greenfield as a stable architectural reference but plan to pin to a +specific commit if they fork it, since 1.0.0 final may land breaking-ish +changes before the rc1 → 1.0.0 transition completes. + +## License of the source + +The four `packages/*` (compositor, compositor-proxy, compositor-proxy-cli, +compositor-shell) are **AGPL-3.0-or-later**. The lower-level `libs/*` are +MIT or Apache-2.0. Anyone forking or building on the runtime packages must +respect AGPL — see `11-pitfalls.md` §License for implications. diff --git a/01-architecture.md b/01-architecture.md new file mode 100644 index 0000000..a37875d --- /dev/null +++ b/01-architecture.md @@ -0,0 +1,443 @@ +# 01 — Layered Architecture Map + +This chapter walks Greenfield's architecture layer by layer, from the lowest +abstraction (Wayland XML) up to the highest (the user-facing shell). Each layer +is described in terms of: **responsibility**, **key abstractions**, +**dependencies**, and **files of interest**. The chapter ends with an ASCII +diagram showing how the layers stack and where data flows between them. + +## The eight layers + +``` +┌──────────────────────────────────────────────────────────────────────────┐ +│ Layer 8: USER SHELL (Odoo 20 OWL addon, supplied by user) │ +│ Hooks: UserShellApi (events + actions) │ +│ ─ NOT covered in this chapter (see 08-app-lifecycle.md) │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 7: DESKTOP / LAYOUT POLICY │ +│ Floating vs AlwaysFullScreen, view stacking, focus activation refcount │ +│ packages/compositor/src/desktop/Desktop.ts │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 6: SURFACE LIFECYCLE / ROLES │ +│ XdgWmBase, XdgSurface, XdgToplevel, XdgPopup, XdgPositioner, Shell, │ +│ ShellSurface, Subcompositor, Subsurface, SurfaceRole │ +│ packages/compositor/src/{Xdg*, Shell*, Sub*, SurfaceRole, SurfaceChild} │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 5: RENDERING PIPELINE │ +│ Surface (state stack), View, Output (WebGL canvas), Renderer, Scene, │ +│ Shaders (YUV→RGBA, YUVA→RGBA, RGB+A→RGBA), Loop (RAF), Callback │ +│ packages/compositor/src/{Surface, View, Output, Loop, Callback, Region} │ +│ packages/compositor/src/render/ │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 4: TRANSPORT (browser side) │ +│ Channel (ARQ + Simple), ReconnectingWebSocket, RemoteAppLauncher, │ +│ StreamingBuffer, BufferStream, EncodedFrame, EncoderFeedback, decoders │ +│ (WebCodecs + WASM fallback) │ +│ packages/compositor/src/remote/ │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 4: TRANSPORT (native side) │ +│ Channel (ARQ + Simple), SessionController, kcp.ts, NativeAppContext, │ +│ FrameFeedback, SurfaceBufferEncoding, wayland-server.ts │ +│ packages/compositor-proxy/src/{Channel, kcp, Session*, Native*, ...} │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 3: WAYLAND RUNTIME (browser side) │ +│ Connection (wire framing), Client, WlObject registry, marshallers │ +│ libs/common/src/Connection.ts │ +│ libs/compositor-protocol/src/{westfield-runtime-server.ts, generated/*} │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 3: WAYLAND RUNTIME (native side) │ +│ wayland-server.ts + wayland-server-addon.node (N-API → libwayland-server)│ +│ MessageInterceptor (per-resource request/event routing) │ +│ packages/compositor-proxy/src/{wayland-server.ts, protocol/*_interceptor}│ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 2: PROTOCOL GENERATORS │ +│ client-generator, compositor-generator, compositor-proxy-generator │ +│ libs/{client,compositor,compositor-proxy}-generator/ │ +├──────────────────────────────────────────────────────────────────────────┤ +│ Layer 1: WAYLAND PROTOCOL XML │ +│ protocol/{wayland, xdg-shell, wayland-drm, web-bitmapbuf-unstable-v1}.xml│ +└──────────────────────────────────────────────────────────────────────────┘ +``` + +## Layer 1 — Wayland Protocol XML + +**Responsibility**: declare the protocol surface — what interfaces, requests, +events, enums, and bitfield types exist on the wire. + +**Files of interest**: + +- `protocol/wayland.xml` — standard upstream Wayland core. 22 interfaces, + `wl_compositor` v4, `wl_seat` v6, `wl_surface` v4 (roughly Wayland + 1.20-ish). Kristian Høgsberg copyright. Used by all 3 generators. +- `protocol/xdg-shell.xml` — stable `xdg_shell` v1, 5 interfaces (xdg_wm_base, + xdg_surface, xdg_toplevel, xdg_popup, xdg_positioner). Used by all 3 + generators. +- `protocol/wayland-drm.xml` — legacy `wl_drm` v2, used by native apps (Mesa, + XWayland) to advertise dmabuf buffers. **Only the proxy generator + processes this file** — so the proxy can intercept `create_buffer` / + `create_planar_buffer` / `create_prime_buffer` and turn dmabuf fds into + ProxyFDs for the browser's GStreamer encoding pipeline. Browser-side libs + do not generate `wl_drm` stubs. +- `protocol/web-bitmapbuf-unstable-v1.xml` — **Greenfield's custom + extension**. Single interface `web_bitmapbuf_factory` v1, single request + `create_buffer(id: new_id, bitmap: fd)`. Lets browser JS apps + hand a GPU-backed `ImageBitmap` directly to the compositor as a `wl_buffer` + — bypassing SHM copy, encoding, and dmabuf import. The `fd` arg is + marshalled via the same `fileDescriptor(...)` path as + `wl_shm.create_pool`'s fd, but holds an `ImageBitmap` (becomes a + transferable on the flush path). **Not processed by the proxy generator** + — native apps never use it. + +**Dependencies**: none. This is the source of truth. + +## Layer 2 — Protocol Generators + +**Responsibility**: parse the XML with `xml2js`, walk the interface tree, +emit typed TypeScript bindings for each side. + +**Key abstractions**: `ProtocolParser.mjs` (shared XML walker) + +`ProtocolArguments.mjs` (per-side argument-type dispatch table). + +**Files of interest**: + +- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs` — + emits TS `*Proxy` classes for browser apps that want to be Wayland clients + of the browser compositor (via `MessagePort`). +- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs` + — emits TS `*Resource` + `*Requests` classes for the browser-resident + compositor itself (receives requests, dispatches to `*Requests` impls, + emits events). Skips `wl_display`, `wl_registry` (hand-written in runtime). +- `libs/compositor-proxy-generator/{EndpointProtocolParser, + EndpointProtocolArguments, index}.mjs` — emits CommonJS + `*_interface.js` (wl_interface struct definitions for the N-API addon) + + `*_interceptor.js` (per-opcode interceptors that decide routing) for + the Node.js native proxy. + +**Why three generators?** Because each side has a fundamentally different job: + +1. **client-generator** → browser apps that want to be Wayland clients of the + browser compositor (via `MessagePort`). +2. **compositor-generator** → the browser-resident compositor itself (receives + requests, dispatches to `*Requests` impls, emits events). +3. **compositor-proxy-generator** → the Node.js native proxy bridging real + libwayland-client apps to the browser. Emits `wl_interface` struct + definitions for the N-API addon + per-opcode interceptors that decide + routing. + +**Dependencies**: `xml2js`, `camelcase`, `uppercamelcase`, `meow`. + +## Layer 3 — Wayland Runtime + +**Responsibility**: marshal and unmarshal the actual wire bytes. Implement +object lifecycle (creation, dispatch, destruction), new-id tracking, listener +dispatch, file descriptor handling. + +**Browser side** (`libs/common/src/Connection.ts`) — the shared runtime for +both `@gfld/client-protocol` and `@gfld/compositor-protocol`: + +- Wire format is **standard upstream Wayland**, byte-for-byte. 8-byte + header: `u32 object_id | u16 opcode | u16 size-incl-header`, then per-arg + bytes. `u/i/f/n/o` are 4 bytes; `s` and `a` are length-prefixed u32 + + payload padded to 4-byte alignment; `h` (FD) takes zero bytes — FDs travel + out-of-band in `wireMsg.fds[]`. +- Reading: `wlObject[opcode](message)` — the generated classes use + computed-property-key methods `[0](message) { ... }`, `[1](message) { ... }` + so opcode is the array index. +- Client side: `Proxy.marshallConstructor` calls `display.generateNextId()` + (1-indexed, recycles ids from `wl_display.delete_id`), constructs the new + Proxy (self-registers in constructor), patches the `new_id` arg's `value`, + then flushes. +- Server side: `Client.marshallConstructor` allocates from + `SERVER_OBJECT_ID_BASE = 0xff000000` (or recycles locally). When a + client-allocated id is destroyed, the server sends `wl_display.delete_id` + so the client can recycle. +- Connection registry: `wlObjects: { [id]: WlObject }`; `registerWlObject` + throws on collision. +- `Connection.message` dispatches by literal `wlObject[opcode](message)`. + Each handler returns void or a Promise; `Connection` awaits the promise + before reading the next message — this preserves wire-order while still + allowing async implementations. Per-client Connections mean a slow client + doesn't block others. + +**Native side** (`packages/compositor-proxy/src/wayland-server.ts`) — a TS +wrapper around `wayland-server-addon.node` (the C N-API addon that wraps a +**patched** `libwayland-server`): + +- `MessageInterceptor` — per-resource request/event dispatch. Each + `wl_*_interceptor.js` file declares `R(...args)` and + `E(...args)` functions that decide whether a message is routed + `{native: true}` (handle in the proxy), `{browser: true}` (forward to + browser compositor), or both. +- Special cases: `wl_display.sync` → both native and browser (must complete + before `done` event), `wl_display.get_registry` → both, `wl_registry.bind` + → native if id is in `nativeGlobalNames[]` (e.g. `wl_drm`, `wl_shm`), + else browser. Fast-sync optimization skips browser round trip when no + prior browser-bound request occurred. + +**File descriptor handling — the Greenfield innovation**: + +`type FD = unknown` — it's intentionally opaque. Browser-side FDs are +**transferable JS objects**: + +- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives + in browser, zero-copy transferred via `MessagePort.postMessage(..., + transferList)`). +- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg + (GPU-backed texture, zero-copy). +- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe). + +Native-side (proxy) FDs are real POSIX ints. +`NativeWaylandClientSession.onWireMessageEnd` converts each to a JSON +`ProxyFD = {handle, type, host}` (host = proxy's HTTP base URL), +length-prefixed + 4-byte aligned, prepended to the wire message before +sending over WebSocket/KCP. Browser's `webFS` resolves them by fetching +data over HTTP. + +**Dependencies**: Layer 2's generated bindings + `@gfld/common`. + +## Layer 4 — Transport + +**Responsibility**: carry wire bytes between the browser compositor and the +native proxy. Multiplex logical channels. Handle reconnection, backpressure, +and latency. + +**Browser side** (`packages/compositor/src/remote/`): + +- `Channel.ts` — defines `ARQChannel` (KCP-wrapped, reliable, low-latency) + and `SimpleChannel` (raw WebSocket). +- `ReconnectingWebSocket` — vendored from `pladaria/reconnecting-websocket` + with `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`, `minUptime=5s`, + `connTimeout=4s`, `retries=∞`. +- `RemoteAppLauncher.ts` — sets up the 4 logical channels per app client + (PROTOCOL, FRAME, XWM, FEEDBACK) and routes incoming messages. +- `RemoteInputOutput.ts` — implements the `InputOutput` interface for native + apps, with `mkfifo`, `mkstempMmap`, `readFd`, `writeFdAsStream`, `closeFd` + over separate WebSocket endpoints. +- `StreamingBuffer.ts`, `BufferStream.ts`, `EncodedFrame.ts`, + `EncoderFeedback.ts` — decode pipeline for incoming H.264 / PNG frames. +- `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`, + `H264NALDecoder.worker.ts` — the actual decoders. + +**Native side** (`packages/compositor-proxy/src/`): + +- `Channel.ts` — mirror image of the browser side. `ARQChannel` and + `SimpleChannel`. +- `kcp.ts` — ~1100-line TypeScript port of the canonical C `ikcp.c`. +- `SessionController.ts` — WS server with 9 endpoint paths: + `/signal`, `/channel`, `/mkfifo`, `/mkstemp-mmap`, `/read-fd`, + `/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`. +- `Session.ts`, `SessionProcess.ts`, `SessionController.ts` — process model: + one main process, one forked child per browser session. +- `NativeAppContext.ts` — per-app state, signaling WS, lifecycle (10-min + SIGHUP grace on disconnect, SIGTERM→SIGKILL escalation). +- `FrameFeedback.ts` — receives 4-byte feedback messages from the browser, + uses them to pace `wl_frame.done` callbacks. +- `SurfaceBufferEncoding.ts`, `ProxyBuffer.ts` — encoding pipeline. +- `io/` subdirectory — WebSocket server glue. +- `encoding/` subdirectory — high-level encoding orchestration. + +**Wire format**: see `04-transport.md` for the full byte-level breakdown of +PROTOCOL, FRAME, XWM, and FEEDBACK channels. + +**Dependencies**: Layer 3 + `ws` (Node) + browser-native WebSocket + `ReconnectingWebSocket`. + +## Layer 5 — Rendering Pipeline + +**Responsibility**: take incoming buffer frames (SHM or encoded), turn them +into pixels on a browser canvas/WebGL context. Manage surface state, damage +tracking, frame callbacks, and the compositing loop. + +**Key abstractions**: + +- `Surface.ts` — the `wl_surface` implementation. Double-buffered state + (`state` vs `pendingState`), atomic merge in `commitPending()`. Subsurface + sync mode cascades through the ancestor chain. +- `SurfaceChild.ts`, `SurfaceRole.ts`, `Subsurface.ts`, `Subcompositor.ts` — + surface tree / role system. +- `View.ts` — represents a surface's position on an output, used for + compositing. +- `Output.ts` — the `wl_output` implementation; how output is rendered to a + WebGL canvas. +- `Shm.ts` — `wl_shm` / shm pool handling. How shared memory buffers arrive + in the browser. +- `BufferContents.ts`, `BufferImplementation.ts`, `ImageBitmapBuffer.ts` — + buffer abstraction hierarchy. +- `Region.ts` — `wl_region`, damage regions (pixman-WASM backed). +- `Callback.ts` — frame callback mechanism. +- `Loop.ts` — the compositor main loop / `requestAnimationFrame` integration. +- `Compositor.ts` — the `wl_compositor` global; how it creates surfaces and + regions. +- `Session.ts` — top-level browser-side session. +- `Globals.ts` — how Wayland globals are registered. +- `render/` subdirectory — the WebGL renderer: `Renderer.ts`, `Scene.ts`, + `SceneShader.ts`, `Texture.ts`, `RenderState.ts`, `Program.ts`, + `ShaderCompiler.ts`, `ShaderSources.ts`, `YUVA2RGBA.ts`, `YUVA2RGBAShader.ts`, + `YUV2RGBShader.ts`, `RGBXandAToRGBA.ts`, `RGBandA2RGBAShader.ts`. +- `browser/` subdirectory — browser integration glue (canvas, RAF, + ResizeObserver, dnd, selection, input). +- `math/` subdirectory — matrix/region math (`Mat4.ts`, `Rect.ts`). + +**Pipeline**: see `03-html5-rendering.md` for the full +commit → damage → render → present pipeline. + +**Dependencies**: Layer 4 + WebGL + WebCodecs + `@gfld/compositor-wasm` (pixman + xkbcommon). + +## Layer 6 — Surface Lifecycle / Roles + +**Responsibility**: implement the role-specific surface state machines +(xdg-toplevel, xdg-popup, xdg-positioner, wl_shell_surface, subsurface). +Manage configure/ack-configure round-trips, popup grabs, window state. + +**Files of interest**: + +- `XdgWmBase.ts` — the `xdg_wm_base` global. How clients advertise they're + XDG clients. Includes ping-based unresponsiveness detection. +- `XdgSurface.ts` — `xdg_surface`. The role wrapper around a `wl_surface`. +- `XdgToplevel.ts` — `xdg_toplevel`. Top-level windows. State (maximized, + fullscreen, minimized, activated, resizing). Configure/ack-configure + round-trip. +- `XdgPopup.ts` — `xdg_popup`. Transient windows, menus, tooltips. Grab + semantics. +- `XdgPositioner.ts` — `xdg_positioner`. How popups are positioned relative + to parent. +- `Shell.ts`, `ShellSurface.ts` — the legacy `wl_shell` (deprecated but still + implemented for old clients). +- `Subcompositor.ts`, `Subsurface.ts`, `SurfaceChild.ts`, `SurfaceRole.ts` + — subsurface compositing, the surface tree. + +**State machines**: see `08-app-lifecycle.md` for the full +XdgToplevel lifecycle, configure/ack round-trip semantics, popup +positioner math, and subsurface sync/desync mode. + +**Dependencies**: Layer 5 + Layer 3 (Wayland runtime). + +## Layer 7 — Desktop / Layout Policy + +**Responsibility**: arrange surfaces on the output. Decide stacking order, +focus policy, and the high-level window management mode (Floating vs +AlwaysFullScreen). + +**Files of interest**: + +- `desktop/Desktop.ts` — implements both `FloatingDesktopSurface` and + `AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`. + Floating lets the user move/resize windows freely; AlwaysFullScreen + enforces one window per output (kiosk mode). + +**Dependencies**: Layer 6 + Layer 5 (for View rendering). + +## Layer 8 — User Shell + +**Responsibility**: the actual user-facing UI. The taskbar, window +decorations, app launcher, keyboard layout picker, etc. + +**Out of scope for the user** — they will build their own shell as an Odoo 20 +OWL addon. The integration point is `UserShellApi.ts` (see +`08-app-lifecycle.md`). + +**Greenfield's reference shell** (`packages/compositor-shell`) is a Preact + +Tailwind SPA, 562 lines of TypeScript total. Mentioned here only for +orientation; not analyzed in this bundle. + +## How data flows across the layers + +``` + NATIVE APP (e.g. gtk4-demo, X11 via XWayland) + │ + │ libwayland-client → UNIX socket + SCM_RIGHTS FDs + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ NATIVE PROXY (Node.js + C N-API addons) │ +│ │ +│ Layer 3 (native): wayland-server.ts + wayland-server-addon.node │ +│ ↳ patched libwayland-server parses wire bytes │ +│ ↳ MessageInterceptor routes each request to {native, browser} │ +│ ↳ FDs wrapped as ProxyFD JSON │ +│ │ +│ Layer 4 (native): Channel.ts + kcp.ts + SessionController │ +│ ↳ 4 logical channels: PROTOCOL, FRAME, XWM, FEEDBACK │ +│ ↳ KCP-over-WebSocket for ARQ channels │ +│ ↳ SimpleChannel for FEEDBACK │ +│ │ +│ Layer 5 (encoding): SurfaceBufferEncoding + proxy-encoding-addon │ +│ ↳ GStreamer H.264 (x264 / nvh264 / vaapih264) on dedicated thread │ +│ ↳ PNG fallback for ≤256×256 surfaces │ +│ ↳ Zero-copy dmabuf via EGLImage → GstGLMemoryEGL │ +│ │ +│ Layer 1 (XML) → Layer 2 (proxy-generator) feeds Layer 3 interceptors │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + │ WebSocket (TCP) + KCP-over-WebSocket + │ 9 endpoint paths per session + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ BROWSER COMPOSITOR (TypeScript + WebGL + WebCodecs + WASM) │ +│ │ +│ Layer 4 (browser): Channel.ts + RemoteAppLauncher + decoders │ +│ ↳ ReconnectingWebSocket with exponential backoff │ +│ ↳ ARQChannel + SimpleChannel mirror the native side │ +│ ↳ WebCodecs VideoDecoder preferred, FFmpeg WASM fallback │ +│ ↳ H264NALDecoder.worker.ts (2 workers: opaque + alpha) │ +│ │ +│ Layer 3 (browser): @gfld/common Connection + compositor-protocol │ +│ ↳ Standard Wayland wire format, byte-for-byte │ +│ ↳ WlObject registry, opcode dispatch via wlObject[opcode](msg) │ +│ ↳ Per-client Connections (slow client never blocks another) │ +│ │ +│ Layer 5: Surface.ts + View.ts + Output.ts + render/ │ +│ ↳ Double-buffered state (state vs pendingState) │ +│ ↳ Damage tracking via pixman-WASM regions │ +│ ↳ WebGL1 with FBOs for YUV→RGBA, RGB+Alpha→RGBA conversion │ +│ ↳ desynchronized:true canvas for low-latency presentation │ +│ ↳ Frame callbacks fired on requestAnimationFrame vsync │ +│ │ +│ Layer 6: XdgWmBase + XdgSurface + XdgToplevel + XdgPopup + ... │ +│ ↳ Configure/ack-configure round-trips │ +│ ↳ Popup positioner math (anchor + gravity + constraints) │ +│ ↳ Subsurface sync/desync mode cascading │ +│ │ +│ Layer 7: desktop/Desktop.ts │ +│ ↳ Floating vs AlwaysFullScreen mode │ +│ ↳ View stacking, focus refcount │ +│ │ +│ Layer 8: UserShellApi (events + actions) │ +│ ↳ Events: clientCreated, surfaceCreated, surfaceTitleUpdated, │ +│ surfaceActivationUpdated, ... │ +│ ↳ Actions: initScene, refreshScene, destroyScene, │ +│ closeClient, activateSurface, setUserConfiguration │ +│ ↳ THE INTEGRATION SEAM — Odoo 20 OWL shell hooks here │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + │ UserShellApi.events.* fired by compositor + │ UserShellApi.actions.* called by shell + ▼ +┌─────────────────────────────────────────────────────────────────────────┐ +│ USER SHELL (Odoo 20 OWL addon — supplied by user) │ +│ │ +│ - Application launcher (lists apps from --applications JSON) │ +│ - Window management UI (taskbar, decorations, minimize/maximize) │ +│ - Keyboard layout picker (RMLVO) │ +│ - Per-output canvas (HTMLCanvasElement, one per Output) │ +│ - Trigger app launches via fetch('http://proxy:8081/') │ +│ - Open signalURL WebSocket to receive CONNECT_CHANNEL messages │ +│ │ +│ Not analyzed in this bundle (out of scope). │ +└─────────────────────────────────────────────────────────────────────────┘ +``` + +## Reading guide for the rest of the bundle + +Now that you've seen the layer map, the remaining chapters drill into each +layer in detail: + +- `02-wayland-protocol.md` — Layers 1, 2, 3 (XML → generators → runtime) +- `03-html5-rendering.md` — Layer 5 (rendering pipeline) +- `04-transport.md` — Layer 4 (transport, both sides) +- `05-input-handling.md` — Layer 5 input subsystem (cross-cuts Layers 3, 4, 5) +- `06-data-clipboard.md` — Layer 5 data device (cross-cuts Layers 3, 4, 5) +- `07-native-app-bridge.md` — Layer 4 native + Layer 3 native + Layer 1 XML native (proxy-only `wl_drm`) +- `08-app-lifecycle.md` — Layer 6 + Layer 8 (UserShellApi seam) +- `09-hardware-accel.md` — Layer 5 encoding + Layer 4 frame channel + Layer 3 WASM +- `10-porting-to-odoo-js-python.md` — how to rebuild all 8 layers in your stack +- `11-pitfalls.md` — what Greenfield does wrong, layer by layer +- `12-appendix.md` — glossary, file index, external references diff --git a/02-wayland-protocol.md b/02-wayland-protocol.md new file mode 100644 index 0000000..6e8be66 --- /dev/null +++ b/02-wayland-protocol.md @@ -0,0 +1,325 @@ +# 02 — Wayland Protocol Layer + +This chapter covers Greenfield's protocol layer in depth: how Wayland XML +protocol definitions get parsed, how typed TypeScript stubs get generated for +each of the three sides (browser client, browser compositor, native proxy), +how wire messages are marshalled/unmarshalled at runtime, how objects +lifecycle is managed, and how the FD-over-WebSocket problem is solved. + +## The three generators + +Greenfield ships **three sibling generator CLIs** that consume the same +Wayland XML files but emit slightly different TypeScript flavors: + +| | `client-generator` | `compositor-generator` | `compositor-proxy-generator` | +|---|---|---|---| +| Output | TS `*Proxy` classes (browser) | TS `*Resource` + `*Requests` (Node) | CommonJS `*_interface.js` + `*_interceptor.js` (Node + N-API) | +| Listener pattern | `listener?: Events` (optional, duck-typed, async) | `implementation: Requests` (required) | `requestHandlers` / `eventHandlers` maps | +| Skips | none | `wl_display`, `wl_registry` (hand-written in runtime) | none — proxy must intercept display/registry/bind | +| Arg-type table | `ProtocolArguments.mjs` returns `{signature, jsType, marshallGen}` | Same shape, uses `Westfield.${resourceName}` for objects | `EndpointProtocolArguments.mjs` returns plain wayland sig letters (`u`, `?s`, `n`, `h`, ...) | +| XMLs consumed | `wayland.xml` + `xdg-shell.xml` + `web-bitmapbuf-unstable-v1.xml` | same 3 | `wayland.xml` + `xdg-shell.xml` + `wayland-drm.xml` | + +All three share `xml2js` + `camelcase` + `uppercamelcase` and a near-identical +`ProtocolParser.mjs`. The XML → JSON → walk `interface[]`/`request[]`/ +`event[]`/`enum[]` → per-arg type-dispatch pipeline is identical. + +### Why three generators? + +Because each side has a fundamentally different job: + +1. **`client-generator`** → browser apps that want to be Wayland clients of + the browser compositor (via `MessagePort`). These are the same-origin + iframe JS/WASM apps that ship in `examples/webapps/`. +2. **`compositor-generator`** → the browser-resident compositor itself + (`@gfld/compositor`). It receives requests, dispatches them to + `*Requests` implementation classes, and emits events back. +3. **`compositor-proxy-generator`** → the Node.js native proxy bridging real + `libwayland-client` apps to the browser. Emits `wl_interface` struct + definitions for the N-API addon + per-opcode interceptors that decide + routing. + +## Wire format (`libs/common/src/Connection.ts`) + +**Standard upstream Wayland wire format, byte-for-byte.** No custom binary +encoding at the protocol layer. + +``` +┌─────────────┬──────────┬──────────┬─────────────────────────────────────────────┐ +│ object id │ opcode │ size │ args (variable per arg type, see §4) │ +│ uint32 │ uint16 │ uint16 │ u,i,f,n,o = 4 bytes; s,a = 4+len+pad; │ +│ (4 bytes) │ (2 B) │ (2 B) │ h = 0 bytes, FD goes in side-band fds[] │ +└─────────────┴──────────┴──────────┴─────────────────────────────────────────────┘ + ▲ + └── size includes this 8-byte header. + Total message size = size field. +``` + +### Per-arg marshalling table + +| Arg signature letter | Type | Size on wire | Notes | +|---|---|---|---| +| `u` | uint32 | 4 bytes | unsigned integer | +| `i` | int32 | 4 bytes | signed integer | +| `f` | fixed | 4 bytes | wl_fixed_t (24.8 signed) | +| `s` | string | 4 + len + pad | length-prefixed u32, padded to 4-byte alignment | +| `a` | array | 4 + len + pad | length-prefixed u32, padded to 4-byte alignment | +| `o` | object | 4 bytes | object id (0 means null) | +| `n` | new_id | 4 bytes | newly-allocated object id | +| `h` | file descriptor | 0 bytes | FD goes in side-band `wireMsg.fds[]` | + +### Reading + +`Connection.message` dispatches by literal `wlObject[opcode](message)`. The +generated classes use computed-property-key methods: +`[0](message) { ... }`, `[1](message) { ... }` so opcode is the array index. +Each handler returns void or a Promise; `Connection` awaits the promise +before reading the next message — this preserves wire-order while still +allowing async implementations. Per-client Connections mean a slow client +doesn't block others. + +## Object lifecycle & new-id tracking + +### Client side + +`Proxy.marshallConstructor` calls `display.generateNextId()` (1-indexed, +recycles ids from `wl_display.delete_id`), constructs the new Proxy +(self-registers in constructor), patches the `new_id` arg's `value` (it was +`0` placeholder from `newObject()`), then flushes. + +### Server side + +`Client.marshallConstructor` allocates from `SERVER_OBJECT_ID_BASE = +0xff000000` (or recycles locally). When a client-allocated id is destroyed, +the server sends `wl_display.delete_id` so the client can recycle. On +`Client.close()`, objects are destroyed in **descending id order** (children +first). + +### Connection registry + +`wlObjects: { [id]: WlObject }`; `registerWlObject` throws on collision. + +## Listener dispatch + +`Connection.message` dispatches by literal `wlObject[opcode](message)`. Each +handler returns void or a Promise; `Connection` awaits the promise before +reading the next message — this preserves wire-order while still allowing +async implementations. Per-client Connections mean a slow client doesn't +block others. + +## File descriptor handling — the Greenfield innovation + +`type FD = unknown` — it's intentionally opaque. The actual FD +representation depends on which link it's traversing. + +### Browser ↔ Browser (in-iframe, via `MessagePort`) + +FDs are transferable JS objects: + +- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives + in browser, zero-copy transferred via `MessagePort.postMessage(..., + transferList)`). +- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg + (GPU-backed texture, zero-copy). +- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe). + +### Browser ↔ Proxy (over WebSocket/KCP) + +FDs are virtualized as JSON `ProxyFD` descriptors: + +``` +ProxyFD = { handle: number, type: 'pipe-read' | 'pipe-write' | 'shm' | 'unknown', host: string } +``` + +Where: +- `handle` is the proxy's internal numeric fd id +- `type` is the kind of fd (so the browser knows what to do with it) +- `host` is the proxy's HTTP base URL (for proxy-to-proxy FD relay) + +`NativeWaylandClientSession.onWireMessageEnd` converts each POSIX fd to a +ProxyFD JSON, length-prefixed + 4-byte aligned, prepended to the wire +message before sending over WebSocket/KCP. The browser's `webFS` resolves +them by fetching data over a separate ad-hoc WebSocket endpoint +(`/read-fd`, `/write-fd-as-stream`, etc.). + +### Native App ↔ Proxy (libwayland socket) + +FDs are real POSIX ints, sent via `SCM_RIGHTS` by libwayland-server. + +### Most wl_buffer FDs never need to be read + +Because the proxy intercepts `wl_surface.commit`, reads the buffer locally +via DRM/SHM, encodes to H.264, and ships only the encoded bytes — the +browser never needs to read the original FD. + +## How the proxy translates between native and browser + +**It doesn't translate wire bytes** — both sides speak identical Wayland +wire format. What it translates is: + +- **FDs**: POSIX int ↔ `ProxyFD` JSON. +- **Object-id space**: native libwayland's id range vs browser's + `0xff000000`+ server range. +- **Routing per message** via `MessageInterceptor.interceptRequest` / + `interceptEvent`. Special cases: + - `wl_display.sync` → both native and browser (must complete before `done` + event) + - `wl_display.get_registry` → both + - `wl_registry.bind` → native if id is in `nativeGlobalNames[]` (e.g. + `wl_drm`, `wl_shm`), else browser + - Fast-sync optimization skips browser round trip when no prior + browser-bound request occurred + +## The four protocol XMLs + +### `protocol/wayland.xml` + +Standard upstream Wayland core. 22 interfaces, `wl_compositor` v4, `wl_seat` +v6, `wl_surface` v4 (roughly Wayland 1.20-ish). Kristian Høgsberg copyright. +Used by all 3 generators. + +### `protocol/xdg-shell.xml` + +Stable `xdg_shell` v1, 5 interfaces: `xdg_wm_base`, `xdg_surface`, +`xdg_toplevel`, `xdg_popup`, `xdg_positioner`. Used by all 3 generators. + +### `protocol/wayland-drm.xml` + +Legacy `wl_drm` v2, used by native apps (Mesa, XWayland) to advertise dmabuf +buffers. **Only the proxy generator processes this file** — so the proxy can +intercept `create_buffer` / `create_planar_buffer` / `create_prime_buffer` +and turn dmabuf fds into ProxyFDs for the browser's GStreamer encoding +pipeline. Browser-side libs do not generate `wl_drm` stubs. + +### `protocol/web-bitmapbuf-unstable-v1.xml` — Greenfield's custom extension + +This is **Greenfield's own Wayland protocol extension** for in-browser +JS/WASM clients. It defines a single singleton interface: + +- **`web_bitmapbuf_factory` (v1)** with one request + `create_buffer(new_id wl_buffer id, fd bitmap)` — "Create a wl_buffer + object by wrapping an HTML bitmap, so it can be used with a surface." + +The `fd` argument is an HTML bitmap file descriptor (a Greenfield-specific +construct: an `ImageBitmap`-like object exposed as an FD-like handle). This +lets JavaScript or WebAssembly Wayland clients back a surface with a native +browser bitmap (`ImageBitmap`, `OffscreenCanvas` transferable) and zero-copy +it into the compositor's WebGL texture pipeline — bypassing the SHM +pixel-copy path. + +The browser-side factory lives in +`packages/compositor/src/web/WebBuffersFactory.ts`. **Not processed by the +proxy generator** — native apps never use it. + +This is a trade-off worth noting: the fast path is Greenfield-specific. To +get zero-copy `ImageBitmap` upload to a GPU texture, apps must be +Greenfield-aware (i.e. built against `@gfld/client-protocol`). Standard +Wayland apps fall back to the H.264-encoded path even when running in the +browser compositor. + +## ASCII pipeline diagram + +``` + ┌──────────────────────────────────────────────────────────────────────────────┐ + │ PROTOCOL XML SOURCES │ + │ wayland.xml xdg-shell.xml wayland-drm.xml web-bitmapbuf-unstable-v1 │ + └──────────┬──────────────┬──────────────┬─────────────────┬─────────────────────┘ + │ │ │ │ + ▼ ▼ ▼ ▼ + ┌────────────────┐ ┌────────────────┐ ┌───────────────────┐ ┌────────────────────┐ + │ client- │ │ compositor- │ │ compositor-proxy- │ │ (no proxy gen for │ + │ generator │ │ generator │ │ generator │ │ web-bitmapbuf) │ + │ ProtocolParser │ │ ProtocolParser │ │ EndpointProtocol │ │ │ + │ .mjs │ │ .mjs │ │ Parser.mjs │ │ │ + └───────┬────────┘ └───────┬────────┘ └─────────┬─────────┘ └─────────┬──────────┘ + │ xml2js │ xml2js │ xml2js │ + ▼ ▼ ▼ ▼ + ┌────────────────┐ ┌────────────────┐ ┌─────────────────────┐ ┌────────────────────┐ + │ wayland.ts │ │ wayland.ts │ │ *_interface.js │ │ wayland.ts │ + │ xdg_shell.ts │ │ xdg_shell.ts │ │ *_interceptor.js │ │ xdg_shell.ts │ + │ web_bitmapbuf.ts│ │ web_bitmapbuf.ts│ │ (wayland,xdg,drm) │ │ web_bitmapbuf.ts │ + │ *Proxy classes │ │ *Resource + │ │ wl_interface structs│ │ *Proxy classes │ + │ *Events ifaces │ │ *Requests │ │ + R*/E* handlers │ │ *Events interfaces │ + │ *ProtocolName │ │ *ProtocolName │ │ + requestHandlers │ │ *ProtocolName │ + │ *Enum │ │ *Enum │ │ /eventHandlers │ │ *Enum │ + └───────┬─────────┘ └───────┬─────────┘ └──────────┬──────────┘ └─────────┬──────────┘ + │ imports │ imports │ require()'d │ imports + ▼ ▼ ▼ ▼ + ┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐ + │ @gfld/common │ │ packages/compositor-proxy/ │ + │ Connection.marshallMsg() ── writes │ │ src/wayland-server.ts │ + │ u32 id | u16 opcode | u16 size | args │ │ ↳ re-exports wayland-server-addon │ + │ Connection.message() ── parses header, │ │ ↳ MessageInterceptor (R*/E* dispatch) │ + │ dispatches wlObject[opcode](message) │ │ ↳ unmarshallArgs(message, 'usun') etc │ + │ Per-arg marshallers: uint/int/fixed/ │ │ NativeWaylandClientSession.ts │ + │ string/array/object/objectOptional/ │ │ ↳ onWireMessageRequest → interceptRequest │ + │ newObject/fileDescriptor │ │ ↳ onWireMessageEnd → pack ProxyFD JSON │ + │ Per-arg unmarshallers: u/i/f/s/a/o/n/h │ │ ↳ onWireMessageEvents → sendEvents (C) │ + └──────────────┬───────────────────────────┘ └─────────────────────┬────────────────────────┘ + │ │ + ▼ ▼ + ┌────────────────────────────────────────┐ ┌────────────────────────────────────────────────────┐ + │ Browser (client iframe) │ │ Node (compositor-proxy) │ + │ │ │ │ + │ DisplayImpl.messagePort.onmessage ←───┼─────────┼─ protocolChannel (WS/KCP) sends/receives │ + │ connection.message({buffer, fds}) │ │ [u32 oob-op][u32 lastSerial][u32 fdCount] ... │ + │ wlObject[opcode](message) │ │ [u32 len][ProxyFD JSON, padded][wayland msg bytes]│ + │ listener?.(...unmarshalled) │ │ │ + │ │ │ ← from native: wl_client wire callback (C addon) │ + │ proxy.marshall(id, opcode, [args]) │ │ → to native: sendEvents(wlClient, buf, fds) (C) │ + │ connection.marshallMsg writes Uint32 │ │ │ + │ onFlush → postMessage(buf, transfer) │ │ Native libwayland-server │ + │ ↳ buffer + ArrayBufferView/ImageBitmap│ │ accepts socket from native wayland apps │ + │ /MessagePort as transferables │ │ parses wire bytes, dispatches to interceptors │ + └────────────────────────────────────────┘ └────────────────────────────────────────────────────┘ + + WIRE FORMAT (identical on all links — standard Wayland): + ┌─────────────┬──────────┬──────────┬─────────────────────────────────────────────┐ + │ object id │ opcode │ size │ args (variable per arg type, see §4) │ + │ uint32 │ uint16 │ uint16 │ u,i,f,n,o = 4 bytes; s,a = 4+len+pad; │ + │ (4 bytes) │ (2 B) │ (2 B) │ h = 0 bytes, FD goes in side-band fds[] │ + └─────────────┴──────────┴──────────┴─────────────────────────────────────────────┘ + ▲ + └── size includes this 8-byte header. Total message size = size field. + + FD transport varies per link: + Browser↔Browser (MessagePort): FD is a transferable JS object (ArrayBufferView, ImageBitmap, MessagePort). + Browser↔Proxy (WebSocket/KCP): FD is a JSON ProxyFD {handle, type, host}, length-prefixed + 4-byte aligned, prepended before the message header. + Native App↔Proxy (libwayland socket): FD is a real POSIX fd, sent via SCM_RIGHTS by libwayland-server. +``` + +## Build-time workflow + +If you edit a protocol XML in your fork, you regenerate stubs by running +the matching `generate` script: + +- `yarn workspace @gfld/client-protocol generate` — regenerates browser + client stubs. +- `yarn workspace @gfld/compositor-protocol generate` — regenerates browser + compositor stubs. +- `yarn workspace @gfld/compositor-proxy generate` — regenerates proxy + interceptor modules, then `build:native` (because `*_interface.js` + declares `wl_interface` structs the N-API addon consumes). + +To add a new Greenfield-only extension: drop XML in `/protocol/`, add a +`generate:` script to `client-protocol` / `compositor-protocol` +`package.json`, regenerate. Only add to `compositor-proxy` if native apps +will speak it. + +## Notable file references + +- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs` +- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs` +- `libs/compositor-proxy-generator/src/{EndpointProtocolParser, EndpointProtocolArguments, index}.mjs` +- `libs/common/src/Connection.ts` — the runtime wire codec (shared) +- `libs/compositor-protocol/src/westfield-runtime-server.ts` — server-side runtime +- `libs/client-protocol/src/westfield-runtime-client.ts` — client-side runtime +- `packages/compositor-proxy/src/wayland-server.ts` — proxy's TS wrapper around the N-API addon +- `packages/compositor-proxy/src/NativeWaylandClientSession.ts` — per-client message interceptor dispatch +- `packages/compositor-proxy/src/protocol/wl_*_interceptor.js` — generated per-opcode routing decisions +- `packages/compositor/src/web/WebBuffersFactory.ts` — browser-side implementation of the custom `web_bitmapbuf_factory` global + +## One stale file spotted + +`libs/compositor-protocol/index.js` (CommonJS, references non-existent +`./src/Resource` / `./src/Client` / `./src/Display` / `./src/Global`). The +actual entry point is `src/index.ts`. Safe to delete in a fork. diff --git a/03-html5-rendering.md b/03-html5-rendering.md new file mode 100644 index 0000000..4e5a67e --- /dev/null +++ b/03-html5-rendering.md @@ -0,0 +1,379 @@ +# 03 — HTML5 Rendering Pipeline + +This chapter covers Greenfield's browser-side rendering pipeline: how +incoming Wayland buffers (SHM or encoded) end up as pixels on a browser +canvas/WebGL context, including damage tracking, commit semantics, surface +state stacking, and the compositing loop. The browser side lives in +`packages/compositor/src/`. + +## Three buffer types, one BufferImplementation interface + +All incoming buffers, regardless of origin, end up as a `BufferContents` +with a `mimeType` that dispatches to the right upload path. There are three +concrete buffer types: + +### 1. SHM buffer (`LazyImageBitmapBuffer`) + +Comes from `wl_shm_pool` — the standard Wayland shared-memory buffer +mechanism. Browser-side implementation lives in `Shm.ts`. The raw pixel data +arrives as a transferable `ArrayBufferView` (transferred zero-copy via +`MessagePort` from a browser-iframe web app, or fetched via the FD endpoint +WebSocket from the proxy). Greenfield wraps the bytes as an `ImageData` / +BMP-Blob and decodes them via `createImageBitmap()` to produce an +`ImageBitmap` ready for WebGL `texImage2D`. + +**Notable hack**: for `xrgb8888` SHM buffers, Greenfield prepends a BMP +header in front of the raw pixel data so the browser's `createImageBitmap` +decodes it natively, avoiding a per-pixel JS loop. This is a clever trick to +offload pixel conversion to the browser's BMP decoder. + +### 2. Web bitmap buffer (`ImageBitmapBuffer` via `web_bitmapbuf_factory`) + +Comes from Greenfield's custom `web-bitmapbuf-unstable-v1` protocol +extension. Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to +the compositor as a `wl_buffer` — bypassing SHM copy, encoding, and dmabuf +import. The `fd` arg is marshalled as a transferable, so the GPU texture +never leaves VRAM. + +### 3. Remote streaming buffer (`StreamingBuffer`) + +Comes from the native proxy via the FRAME data channel. Receives encoded +frames (H.264 or PNG) on a separate WebSocket per surface. Includes an +out-of-band mechanism for `BufferCreation` and `CommitSerial` opcodes so +the browser can match each encoded frame to the right `wl_surface.commit` +serial. + +## Surface state stack + +`Surface.ts` implements classic Wayland double-buffered state: + +- `Surface.state` — the committed, currently-visible state. +- `Surface.pendingState` — the in-flight state, accumulated by client + requests until `commit` is called. +- `mergeSurfaceState` — the atomic swap in `commitPending()`. + +Subsurfaces add a third layer of complexity: + +- `subsurfaceChildren` arrays (committed + pending, bottom→top) +- `placeAbove` / `placeBelow` reorder pending +- `effectiveSync` getter cascades through the ancestor chain (if any + ancestor is sync, this subsurface is effectively sync) +- Sync subsurfaces cache to `cachedState` and only flush when an ancestor + commits via `onParentCommit` +- `setDesync` immediately calls `onCommit` to flush any pending cache + +## Commit semantics + +When a Wayland client calls `wl_surface.commit`: + +1. If the surface has a remote streaming buffer, Greenfield pulls the + `serial = commitSerials.shift()` (an out-of-band opcode 7 mechanism that + lets `StreamingBuffer.getContents(surface, serial)` wait for the right + encoded frame). +2. `bufferContents = await bufferImpl.getContents(surface, serial)` — + resolves when the encoded frame for this serial has arrived. +3. `role.onCommit(this)` — role-specific logic (XdgToplevel, XdgPopup, + Subsurface, etc.). +4. `Surface.commitPending()` runs: + - Commit `subsurfaceChildren` order + apply pendingPosition → cascade + `onParentCommit`. + - Calculate derived pending state (buffer transform matrix, surface size, + pixman region). + - Release old buffer if replaced. + - `mergeSurfaceState(state, pendingState)` — atomic swap. + - `damaged = true`; clear pending damage + frame callbacks. + +## Damage tracking + +`damageRects` (post-transform) and `bufferDamageRects` (buffer-space) are +accumulated correctly on the surface state via pixman-WASM region math. The +damage is tracked faithfully — but **vestigial at the GL level**. The +renderer doesn't consume it for scissoring — it repaints every visible view +every RAF. The only damage gate is the boolean `surface.damaged`, used to +decide whether a streaming buffer's frame needs re-upload + release. + +`RenderState.visibleSceneRegion` (pixman) is used only to decide if a +`RenderState` exists at all (i.e. whether the view is on this output), not +for per-rect clipping. + +## Compositing pipeline + +The renderer is **WebGL1 only**, no OffscreenCanvas, no WebGL2. The +context is acquired with `getContext('webgl', {antialias:false, depth:false, +alpha:true, premultipliedAlpha:false, preserveDrawingBuffer:false, +desynchronized:true})`. FBOs are used for in-GPU YUV→RGBA and +RGB+Alpha→RGBA conversion into per-view textures. + +The compositor runs in **four phases** per render pass: + +### PHASE 1 — pixel upload (per view per scene) + +Coalesced via `setTimeout`-based cancellable microtask. For each view, for +each scene it's visible on: + +1. `view.applyTransformations()` — compute `parent.transformation * + translate(position+offset)`. +2. For each `renderState` in `view.renderStates`: + `scene[bufferContents.mimeType](bufferContents, renderState)` — dispatch + to the right upload path: + - `'image/bitmap'` → `texture.setContent(imageBitmap, size)` — direct + `texImage2D` / `texSubImage2D` upload. + - `'video/h264'` → `YUVA2RGBA` or `RGBXandA2RGBA.convertInto(...)` — + FBO pass that uploads Y/U/V/alpha-Y as 4× LUMINANCE textures, runs the + YUVA→RGBA shader (BT.601 coefficients), writes into the renderState's + RGBA texture. + - `'image/png'` → `texture.setContent(bitmap, size); bitmap.close()`. +3. Collect `surface.state.frameCallbacks` → `Renderer.frameCallbacks`. +4. `surface.damaged = false`; `streamingBuffer.release()` (remote only). + +### PHASE 2 — schedule render frame + +`Renderer.render()` coalesces via a `setTimeout`-based cancellable +microtask, then schedules a single `requestAnimationFrame` for PHASE 3. + +### PHASE 3 — draw (per scene, inside RAF) + +For each scene (one per canvas/output): + +1. `sceneShader.use()`; `updateSceneData(canvas)` — viewport, clear, BLEND, + projection. +2. For each view in the view stack (flattened from `topLevelViews` + recursively via `Renderer.updateViewStack`): + - If `renderState && view.mapped`: + - `sceneShader.updateViewData(view, renderState)` — bind texture, + `u_transform`, quad buffer. + - `sceneShader.draw()` — `gl.drawArrays(TRIANGLE_STRIP, 0, 6)` of a + textured quad per scene. + +### PHASE 4 — fire frame callbacks + +For each callback in `Renderer.frameCallbacks`: +`callback.done(time)` → `wl_callback.done(time)`; `wl_callback.destroy()`. +Then `session.flush()` to send any pending wire messages. + +The `time` value is the `requestAnimationFrame` high-resolution timestamp, +so apps see real vsync-aligned frame timing. + +## Multi-output + +Multi-output = one Scene per canvas. Each view has one `RenderState` (and +one WebGL texture) per scene it's visible on. So a surface visible on two +outputs (e.g. mirroring) has two textures — one per canvas — and gets +uploaded to each on PHASE 1. + +## Subsurfaces stacking + +Subsurfaces stack via `subsurfaceChildren` arrays (committed + pending, +bottom→top). `placeAbove` / `placeBelow` reorder pending; +`Renderer.updateViewStack` flattens `topLevelViews` recursively. + +**Important**: no parent-geometry clipping — subsurfaces can render outside +their parent. The only clip is the scene's canvas region, enforced implicitly +by NDC mapping. + +## Cursor and DnD bypass GL entirely + +Cursor and DnD bypass GL entirely. They draw to a separate singleton 2D +canvas and use CSS `cursor: url(dataURL)` or HTML5 +`dataTransfer.setDragImage`. Their frame callbacks fire immediately with +`Date.now()`, not synced to RAF. + +This is a deliberate optimization: cursor updates must be low-latency, and +going through the full GL pipeline would add a frame of latency. + +## Decode pipeline (for H.264 frames) + +Decoding is delegated to: + +### Preferred path: WebCodecs `VideoDecoder` + +At session startup, `Session.create()` probes +`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true, +hardwareAcceleration:'prefer-software'})`. If supported, an `opaqueDecoder` +and `alphaDecoder` are created per surface (lazily, on first H.264 frame). +Encoded NALs are wrapped as `EncodedVideoChunk {type:key|delta, data:nal}` +and fed to the decoder. The output `VideoFrame` is uploaded directly to a +WebGL texture. + +**Important**: hardware decode is **explicitly disabled** in `Session.ts` +with a FIXME comment "hardware decoding frame offset is wrong". Only +software decoding is used today. + +### Fallback path: FFmpeg Wasm in Web Workers + +If WebCodecs is unavailable, two persistent Workers (`h264NALDecoderOpaque`, +`h264NALDecoderAlpha`) are created at session startup. Each worker has one +`AVCodecContext` per surface, keyed by `surfaceH264DecodeId`. Encoded NALs +are transferred zero-copy into the worker via +`worker.postMessage(nal, [nal.buffer])`. Decoded I420 planes are transferred +back as `[y.buffer, u.buffer, v.buffer]` and uploaded as 4× LUMINANCE +textures. + +## PNG path (both decoders) + +The PNG path is identical in both branches: +`new Blob([opaque], {type:'image/png'})` → +`createImageBitmap(blob, sx, sy, w, h)` — the `sx/sy/w/h` crops the +encoder padding back to logical size. PNG is always used for surfaces ≤ +256×256 pixels because H.264 I-frame overhead dominates at small sizes. + +## WebGL shaders + +Three shaders in `render/`: + +- `YUVA2RGBAShader` — converts YUVA (Y, U, V, alpha-Y as 4 LUMINANCE + textures) to RGBA in a single FBO pass using BT.601 coefficients. Alpha + is recovered by running the alpha-Y plane through the same YUV→RGB matrix + and taking the R channel. +- `RGBandA2RGBAShader` — for the `DualPlaneRGBA*` WebCodecs outputs + (opaque + alpha planes composited into RGBA). +- `SceneShader` — final pass: draws every view's RGBA texture to the canvas + with `gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`. Note: expects premultiplied + alpha but the YUVA shader outputs straight alpha — a minor inconsistency + that works because Wayland surfaces rarely use partial alpha. + +## ASCII pipeline diagram + +``` +CLIENT (Wayland app / web worker / compositor-proxy) + │ wl_surface.attach / damage / frame / commit + ▼ +Surface.commit() + │ serial = commitSerials.shift() (remote-only) + │ bufferContents = await bufferImpl.getContents(surface, serial) + │ role.onCommit(this) + ▼ +Surface.commitPending() + │ 1. commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit + │ 2. calculateDerivedPendingState (bufferTransform matrix, surface.size, pixmanRegion) + │ 3. release old buffer if replaced + │ 4. mergeSurfaceState(state, pendingState) ◄── atomic swap + │ 5. damaged = true; clear pending damage/frame callbacks + ▼ +Renderer.render() + │ coalesce via queueCancellableMicrotask (setTimeout) + ▼ +PHASE 1 — pixel upload (per view per scene): + view.applyTransformations() // parent.transformation * translate(position+offset) + for renderState in view.renderStates: + scene[bufferContents.mimeType](bufferContents, renderState) + 'image/bitmap' → texture.setContent(imageBitmap, size) // texImage2D / texSubImage2D + 'video/h264' → YUVA2RGBA / RGBXandA2RGBA.convertInto(...) // 4 LUMINANCE or 2 RGBA tex + → FBO + YUVA2RGBAShader (BT.601) → renderState.texture + 'image/png' → texture.setContent(bitmap, size); bitmap.close() + collect surface.state.frameCallbacks → Renderer.frameCallbacks + surface.damaged = false; streamingBuffer.release() (remote only) + ▼ +PHASE 2 — createRenderFrame().then(time => { // = requestAnimationFrame +PHASE 3 — for scene in scenes: // one per canvas/output + sceneShader.use(); updateSceneData(canvas) // viewport, clear, BLEND, projection + for view in viewStack: + if renderState && view.mapped: + sceneShader.updateViewData(view, renderState) // bind texture, u_transform, quad buffer + sceneShader.draw() // gl.drawArrays(TRIANGLE_STRIP, 0, 6) +PHASE 4 — for callback in frameCallbacks: callback.done(time) + → wl_callback.done(time); wl_callback.destroy() + session.flush() +}) +``` + +## The remote streaming buffer content flow + +The remote streaming buffer content arrives in parallel on a separate FRAME +WebSocket channel: + +1. `deliverContentToBufferStream` (in `RemoteAppLauncher.ts`) receives + incoming frame messages. +2. `BufferStream.onBufferContents` parses the encoded frame envelope. +3. `createEncodedFrame` (in `EncodedFrame.ts`) parses `{contentSerial, + mimeType, size, encodedSize, opaque, alpha?}`. +4. Resolves the `onFrameAvailable(serial)` promise that + `StreamingBuffer.getContents` is awaiting. +5. The decoded `DecodedFrame` is then uploaded into the view's + `RenderState.texture` via the appropriate FBO+shader conversion. + +## Engineering observations + +1. **Vestigial damage tracking** — `damageRects` are accumulated correctly + but the renderer doesn't use them for scissoring. Could be a future + optimization for partial repaints. +2. **Multi-output texture cost** — each view has one texture per scene, + so 4 outputs means 4× texture memory for every surface. +3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion + to the browser's BMP decoder. Worth knowing if you need to support + other SHM formats. +4. **Commit-serial mechanism** — the out-of-band opcode 7 lets the + streaming buffer wait for the right encoded frame, ensuring wire-order + semantics even with async decode. +5. **Encoder feedback telemetry** — `EncoderFeedback.ts` measures client + decode duration, but the adaptation loop is wired but not consumed (a + missed opportunity for adaptive bitrate). +6. **Context-loss audit gap** — WebGL context loss is not audited in the + test suite; in production, you should handle `webglcontextlost` and + `webglcontextrestored` events. +7. **`queueCancellableMicrotask` misnomer** — the function is actually a + `setTimeout`-based coalescer, not a microtask. Naming is misleading. +8. **Frame-callback coalescing** — multiple `wl_surface.frame` requests + within one render pass fire on the same RAF tick. Correct Wayland + behavior, but worth knowing. +9. **Cursor/DnD bypass** — they bypass the GL pipeline entirely for + latency. The cursor is rendered as a CSS `cursor: url(dataURL)`. +10. **Subsurface sync cascade** — three-layer mechanism (`cachedState`, + `commitCache`, `effectiveSync`) is correct but subtle. Easy to break + in maintenance. + +## File reference + +| File | Role | +|---|---| +| `packages/compositor/src/Surface.ts` | `wl_surface` implementation, double-buffered state, commit semantics | +| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction | +| `packages/compositor/src/SurfaceRole.ts` | Role abstraction (SurfaceRole interface) | +| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode | +| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global | +| `packages/compositor/src/View.ts` | Positioned surface for compositing | +| `packages/compositor/src/Output.ts` | `wl_output`, WebGL canvas binding | +| `packages/compositor/src/Shm.ts` | `wl_shm` / shm pool, BMP-header hack | +| `packages/compositor/src/BufferContents.ts` | Buffer abstraction (with mimeType) | +| `packages/compositor/src/BufferImplementation.ts` | Buffer implementation interface | +| `packages/compositor/src/ImageBitmapBuffer.ts` | ImageBitmap-backed buffer | +| `packages/compositor/src/Region.ts` | `wl_region`, pixman-WASM damage regions | +| `packages/compositor/src/Callback.ts` | Frame callback (wl_callback) | +| `packages/compositor/src/Loop.ts` | Compositor main loop, RAF integration | +| `packages/compositor/src/Compositor.ts` | `wl_compositor` global | +| `packages/compositor/src/Session.ts` | Top-level browser session, decoder setup | +| `packages/compositor/src/Globals.ts` | Wayland globals registry | +| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline | +| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures | +| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas | +| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper | +| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform | +| `packages/compositor/src/render/Program.ts` | WebGL program wrapper | +| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper | +| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings | +| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass | +| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader | +| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) | +| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) | +| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above | +| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer | +| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution | +| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames | +| `packages/compositor/src/remote/buffer-decoder.ts` | Decoder interface | +| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path | +| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path | +| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder | +| `packages/compositor/src/remote/EncoderFeedback.ts` | Receives 4-byte feedback messages | +| `packages/compositor/src/remote/Channel.ts` | ARQChannel + SimpleChannel | +| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch | +| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Sets up channels per app, frame delivery | +| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps | +| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer events | +| `packages/compositor/src/browser/dnd.ts` | DnD bridge | +| `packages/compositor/src/browser/selection.ts` | Clipboard bridge | +| `packages/compositor/src/browser/capabilities.ts` | Browser feature detection | +| `packages/compositor/src/browser/input.ts` | DOM keyboard → wl_keyboard events | +| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes | +| `packages/compositor/src/math/Mat4.ts` | 4×4 matrix math | +| `packages/compositor/src/math/Rect.ts` | Rectangle math | +| `packages/compositor/src/polyfill/createImageBitmap.js` | ImageBitmap polyfill | diff --git a/04-transport.md b/04-transport.md new file mode 100644 index 0000000..7bb098f --- /dev/null +++ b/04-transport.md @@ -0,0 +1,397 @@ +# 04 — Transport Layer (WebSocket / KCP / Channel Layer) + +This chapter covers Greenfield's transport: how Wayland wire traffic flows +between the Linux-side proxy and the browser-side compositor over WebSocket, +the binary framing, message ordering, latency/backpressure, KCP usage, +multiplexing, and connection lifecycle. + +## Architecture at a glance + +Greenfield uses a **multi-WebSocket, KCP-wrapped, asymmetric flow-control** +design: + +- **Two-tier process model**: A `main.ts` HTTP hub forks one `SessionProcess` + per browser session (per `compositorSessionId`). Raw `net.Socket`s are + handed to children via Node IPC `sendHandle`. Each session child owns its + own `wl_display` (via `wayland-server-addon.node`). +- **9 WebSocket endpoints** per session: `/signal` (JSON control), + `/channel` (binary data channels), `/mkfifo`, `/mkstemp-mmap`, `/read-fd`, + `/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`. + All with `perMessageDeflate: false`, TCP `noDelay: true`. + +## Channel types + +There are **4 logical channel types** per app client, each implemented as a +separate WebSocket: + +| Type | Class | KCP? | Purpose | +|------|-------|------|---------| +| PROTOCOL | ARQChannel | Yes | Wayland wire protocol, per `wl_client` | +| FRAME | ARQChannel | Yes | H.264/PNG encoded surface content, per `wl_surface` | +| XWM | ARQChannel | Yes | XWayland window manager protocol | +| FEEDBACK | SimpleChannel | No | 4-byte telemetry (refreshInterval + decode avg), 1 Hz | + +**There is no multiplexing** of multiple Wayland clients onto one WS. Each +logical channel is its own WS, eliminating head-of-line blocking between +video, input, and control at the cost of many connections per app (10+ for +complex apps). + +## KCP-over-WebSocket + +`kcp.ts` is a ~1100-line TypeScript port of the canonical C `ikcp.c`. +Greenfield runs it **on top of WebSocket** (not real UDP) for: + +- 30 ms min RTO vs TCP's 200 ms. +- 100 ms update tick (`setInterval(kcp.update, 100)`). +- Per-channel congestion domains independent of TCP. +- `setNoDelay(1, 100, 0, 1)` — nodelay enabled, congestion control disabled + (`nc=1`). +- 24-byte segment header: `conv|cmd|frg|wnd|ts|sn|una|len`. +- MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC + design). + +### Asymmetric windows (deliberate) + +- Proxy side: `snd_wnd=128, rcv_wnd=1024` +- Browser side: `snd_wnd=1024, rcv_wnd=128` + +So proxy→browser (heavy video) has a smaller in-flight cap to bound latency; +browser→proxy (bursty input) has a large window to avoid stalls. + +### Why KCP-over-WebSocket? + +Because real-time video over plain TCP suffers from head-of-line blocking +and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in +user-space, while still riding on WebSocket (the only bidirectional binary +transport universally available in browsers today). + +The long-term plan, documented in `docs/pages/design/index.md`, is to +switch to **WebTransport** (real UDP) when browser support is widespread, +and pair it with KCP for reliability. The KCP code is already in place; +only the WebSocket transport layer needs to be swapped out. + +## Binary message format + +All multi-byte fields are **little-endian**. + +### PROTOCOL channel + +``` +u32 outOfBandOpcode (0 = in-band events, else 1=ResourceDestroyed, + 2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial) +u32 eventSerial +u32 fdsCount + for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned) +concatenated Wayland wire messages (object_id + opcode/size + args) +``` + +### FRAME channel (per H.264 frame) + +``` +u32 bufferId +u32 bufferCreationSerial +u32 bufferContentSerial +encodedFrame: + u32 contentSerial, u32 encodingType (0=h264/1=png), + u32 width, u32 height, u32 encodedWidth, u32 encodedHeight, + u32 opaqueLength, u8[] opaqueData, + u32 alphaLength, u8[] alphaData ← alpha encoded as a SEPARATE H.264 stream +``` + +### FEEDBACK channel + +Just 4 bytes: `u16 refreshIntervalMs + u16 avgDecodeDurationMs`. Sent at +1 Hz from browser to proxy. Used by the proxy to pace `wl_frame.done` +callbacks so the native app sees a virtual monitor refresh rate matching +the browser's actual RAF cadence. + +## File descriptor handling + +Can't send FDs over WebSocket — Greenfield virtualizes them: + +- **ProxyFD** = `{handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown', + host:string}`. +- Serialized as JSON inline in the message stream with a u32 length prefix. +- Browser opens **separate ad-hoc WebSocket** to `/read-fd`, + `/write-fd-as-stream`, etc., only when it actually needs the bytes. +- Most `wl_buffer` FDs never need to be read because the proxy intercepts + `wl_surface.commit`, reads the buffer locally via DRM/SHM, encodes to + H.264, and ships only the encoded bytes. +- **Foreign ProxyFD** (`proxyFD.host !== local baseURL`) triggers + proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the + foreign proxy's `/write-fd-as-stream`, and pipes the data through. This + is how copy-paste between two proxy instances avoids a browser round-trip. + +## Reconnection & lifecycle + +### Browser side + +Vendored `ReconnectingWebSocket` (pladaria) with: + +- `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s` +- `minUptime=5s`, `connTimeout=4s` +- `retries=∞` + +Channel state preservation: only close codes 4000–4999 (application-level) +tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps +ticking and the WS reconnects underneath. + +### Proxy side + +No auto-reconnect (it's the server). The Channel object lives on, waiting +for a new WS upgrade with matching `id` + `key`. Outbound messages queue +in `outboundMessages` and flush on `onOpen`. + +### App kill timers + +When the browser disconnects from the `/signal` WebSocket: + +- **10-minute SIGHUP grace period** — gives the user time to reconnect + (e.g. if they refresh the browser tab). +- After 10 minutes of no reconnect, escalating to **SIGKILL after another + 10s**. +- Close code 1001 → immediate SIGHUP (user closed tab intentionally). +- Close code 4001 → immediate SIGTERM (user clicked "force quit" in the + shell). + +### Session close + +When all `wl_clients` disconnect, 1-second grace then `session.close()` → +`process.exit()` in the session child. Main process deletes the entry +from its `sessionProcesses` map. + +## Process model — three layers + +### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`) + +A single Node.js process listens on `:8081`. Per browser session, it +`fork()`s one `SessionProcess` child — one child per `compositorSessionId`, +**not** per app. Multiple apps in the same browser tab share one child. +WebSocket upgrades are handed off to the matching child via +`childProcess.send(wsUpgrade, socket)`. + +### Layer 2: Session Process (`SessionProcess.ts`) + +The forked child. Receives `start` (creates `Session` + +`SessionController`), `launchApp` (spawns the actual app binary), and +`wsUpgrade` (hands WS to the controller) IPC messages. On `launchApp` it +calls `launchApplication()` from `@gfld/compositor-proxy`, which `spawn()`s +the executable, captures `pid`, replies to main with `{pid, key}`. On +`session.close()` the process exits. + +### Layer 3: Proxy Core (`compositor-proxy/src/`) + +- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via + `createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket + at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this + becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/ + NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM, + registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the + wl_display FD into Node's libuv loop via `startPoll()`. +- **`NativeWaylandClientSession`** — per-client wire-message interceptor. + Uses `MessageInterceptor` + auto-generated per-resource interceptor + modules (`src/protocol/wl_*_interceptor.js`). Each request returns + `{native, browser, neverReplies?}` routing decision. Handles + `wl_display.sync` fast-path (skips browser round-trip when no reply + expected). Marshalls FDs as `ProxyFD` JSON. +- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random + signaling auth), signaling WS, client sessions, data channels. Lifecycle: + 10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation + (10 s) on force-quit, auto-destroy on app exit. +- **`SessionController`** — WS demultiplexer by pathname: `/signal` + (per-app control), `/channel` (multiplexed data channels), `/mkfifo`, + `/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`. +- **`XWaylandSession`** — lazy Xwayland fork (only on first X client + connect). Xwayland itself becomes a Wayland client of the proxy. The + `wm_fd` is bridged via xtsb to the browser, which implements the actual + XWM logic. +- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency) + for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK. + +## ASCII diagram of the full transport stack + +``` + LINUX SIDE | BROWSER SIDE + | +┌─────────────────────────┐ | ┌──────────────────────────────┐ +│ gtk4-demo (process) │ | │ Browser tab │ +│ libwayland-client.so │ | │ (Greenfield compositor) │ +└───────────┬─────────────┘ | └──────────────┬───────────────┘ + │ WAYLAND_DISPLAY=wayland-0 │ + │ Unix socket + SCM_RIGHTS FD passing │ DOM events → input + ▼ │ WebSocket API +┌──────────────────────────────────────────┐ │ +│ SessionProcess (fork'd Node child) │ │ +│ ┌────────────────────────────────────┐ │ │ +│ │ wayland-server-addon.node │ │ │ +│ │ (libwayland-server N-API) │ │ │ +│ │ - wl_display + wl_client │ │ │ +│ │ - epoll via proxy-poll-addon │ │ │ +│ └─────────────┬──────────────────────┘ │ │ +│ │ wire msg + FDs │ │ +│ ▼ │ │ +│ ┌─────────────────────────────────────┐ │ │ +│ │ NativeWaylandClientSession │ │ │ +│ │ - interceptRequest / interceptEvent│ │ │ +│ │ - onWireMessageEnd: serialize │ │ │ +│ │ FDs as JSON ProxyFDs inline │ │ │ +│ │ - fastSync optimization │ │ │ +│ └─────────────┬───────────────────────┘ │ │ +│ │ outbound buffer │ │ +│ ▼ │ │ +│ ┌─────────────────────────────────────┐ │ │ +│ │ ARQChannel (KCP-wrapped) │ │ │ +│ │ MTU=64000, snd_wnd=128, rcv_wnd=1024│ │ │ +│ │ KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data │ +│ │ kcp.update() every 100ms │ │ │ +│ │ - PROTOCOL (per wl_client) │ │ │ +│ │ - FRAME (per wl_surface) │ │ │ +│ │ - XWM (per XWayland) │ │ │ +│ └─────────────┬───────────────────────┘ │ │ +│ ┌─────────────┴───────────────────────┐ │ │ +│ │ SimpleChannel (FEEDBACK, no KCP) │ │ │ +│ │ 4 bytes: refreshIntervalMs + │ │ │ +│ │ avgDecodeDurationMs │ │ │ +│ └─────────────┬───────────────────────┘ │ │ +│ ┌─────────────▼───────────────────────┐ │ │ +│ │ NativeAppContext │ │ │ +│ │ - signalingWebSocket (JSON msgs) │ │ │ +│ │ - kill timers: SIGHUP@10min, │ │ │ +│ │ SIGKILL@+10s │ │ │ +│ └─────────────┬───────────────────────┘ │ │ +│ ┌─────────────▼───────────────────────┐ │ │ +│ │ SessionController │ │ │ +│ │ ws.WebSocketServer (noServer) │ │ │ +│ │ perMessageDeflate: false │ │ │ +│ │ Endpoints: /signal /channel /mkfifo│ │ │ +│ │ /mkstemp-mmap /read-fd /write-fd-as-stream│ │ +│ │ /read-fd-as-stream /close-fd /request-keyframe│ │ +│ └─────────────┬───────────────────────┘ │ │ +└────────────────┼─────────────────────────────┘ │ + │ │ + │ ┌──────────────────────────┐ │ + │ │ main.ts (HTTP+WS hub) │ │ + │ │ - socket.pause() + IPC │ │ + │ │ send to session proc │ │ + │ │ - noDelay:true (TCP) │ │ + │ └────────────┬────────────┘ │ + │ │ │ + │ ◄──── 1 WebSocket per channel ────► │ + │ ◄── 1 signaling WS per app ──► │ + │ ◄── ad-hoc WebFD streams ─► │ + │ │ │ + │ ▼ │ + │ ┌────────────────────────────────────────┐ │ + │ │ ReconnectingWebSocket │ │ + │ │ min 1-5s, factor 1.3, max 10s │ │ + │ │ minUptime 5s, connTimeout 4s │ │ + │ └────────────┬───────────────────────────┘ │ + │ ▼ │ + │ ┌────────────────────────────────────────────┐ │ + │ │ ARQChannel / SimpleChannel (mirror image) │ │ + │ │ snd_wnd=1024, rcv_wnd=128 │ │ + │ │ kcp.input → peekSize/recv → onMessage │ │ + │ └────────────┬───────────────────────────────┘ │ + │ ▼ │ + │ ┌────────────────────────────────────────────┐ │ + │ │ RemoteAppLauncher.onChannel │ │ + │ │ PROTOCOL → session.display.createClient │ │ + │ │ FRAME → deliverContentToBufferStream │ │ + │ │ XWM → XWindowManagerConnection │ │ + │ │ FEEDBACK → ClientEncodersFeedback │ │ + │ └────────────┬───────────────────────────────┘ │ + │ ▼ │ + │ ┌────────────────────────────────────────────┐ │ + │ │ @gfld/compositor-protocol Client │ │ + │ │ - parses Wayland wire format │ │ + │ │ - dispatches to wl_*_resource impls │ │ + │ │ - H264NALDecoder.worker.ts decodes frames │ │ + │ └────────────┬───────────────────────────────┘ │ + │ ▼ │ + │ ┌────────────────────────────────────────────┐ │ + │ │ Surface + Scene rendering │ │ + │ │ - ImageBitmap / VideoFrame / Canvas │ │ + │ │ - DOM input → wl_keyboard/pointer reqs │ │ + │ └────────────┬───────────────────────────────┘ │ + │ │ input events flow back via the │ + │ │ PROTOCOL channel as ordinary │ + │ │ Wayland requests (same framing) │ + └──────────────────┘ │ + Browser DOM ◄─┘ +``` + +## Key takeaways for building your own transport + +1. **Separate control plane from data plane** — Greenfield uses `/signal` + (JSON) vs `/channel` (binary). The control plane handles app launch, + termination, kill timers; the data plane handles Wayland wire + encoded + frames. +2. **One WS per logical stream** — simpler, no HoL blocking, but many TCP + connections. +3. **KCP-over-WS** trades double-reliability overhead for fast (30 ms) + retransmits; reasonable for real-time video. +4. **Virtualize FDs as JSON tokens** with lazy separate-WS endpoints for + actual byte transfer. +5. **`bufferedAmount` check ≠ backpressure** — Greenfield's check is a drop + gate, not real flow control (acknowledged TODO). +6. **Distinguish transient vs permanent close** — codes 4000-4999 tear + down; lower codes auto-reconnect. +7. **Piggyback out-of-band opcodes** on the data channel via a leading u32 + to avoid extra control WS. +8. **Asymmetric KCP windows** match each direction's bandwidth profile. +9. **Decode feedback drives frame pacing** — the proxy synthesizes + `wl_frame.done` callbacks at the browser's reported cadence. +10. **One Node child per browser session** for fault isolation and + per-session `wl_display` ownership. + +## File reference + +### Native side + +| File | Role | +|---|---| +| `packages/compositor-proxy/src/Channel.ts` | `ARQChannel` + `SimpleChannel` | +| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation | +| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session | +| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths | +| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers | +| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor | +| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm | +| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done | +| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation | +| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration | +| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge | +| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node | +| `packages/compositor-proxy/src/io/` | WebSocket server glue | +| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration | +| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules | +| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) | + +### Browser side + +| File | Role | +|---|---| +| `packages/compositor/src/remote/Channel.ts` | `ARQChannel` + `SimpleChannel` (mirror) | +| `packages/compositor/src/remote/kcp.ts` | KCP mirror | +| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery | +| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch | +| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps | +| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz | +| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer | +| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution | +| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames | +| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path | +| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path | +| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder | +| `packages/compositor/src/remote/api.ts` | Public exports | + +### CLI + +| File | Role | +|---|---| +| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session | +| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing | +| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions | +| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler | +| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps | +| `packages/compositor-proxy-cli/wait-until-ready.sh` | Startup readiness check | diff --git a/05-input-handling.md b/05-input-handling.md new file mode 100644 index 0000000..dadae3a --- /dev/null +++ b/05-input-handling.md @@ -0,0 +1,366 @@ +# 05 — Input Handling (Keyboard, Pointer, Touch, Focus) + +This chapter covers Greenfield's input handling: keyboard (with xkbcommon in +browser), pointer, touch, focus management, and how input events travel from +the browser to native Linux apps via the proxy. + +## Headline + +Greenfield is a **browser-native Wayland compositor**. All seat logic — +keymap compilation, modifier state, pointer grabs, button/axis dispatch — +runs **in the browser**, against a **WASM build of libxkbcommon 1.5.0 + +pixman**. The `compositor-proxy` is *not* a compositor — it is a thin +libwayland-server bridge that relays Wayland wire messages between the +browser and native clients and provides file-descriptor shims. + +There is no second xkbcommon on the proxy: the keymap is generated in the +browser from RMLVO and shipped to the proxy as a real native shm fd via a +custom `/mkstemp-mmap` WebSocket endpoint. `wl_touch` is advertised but +**never populated with events** — touch input is collapsed into Pointer +Events. + +## The Seat abstraction + +`Seat.ts` (in `packages/compositor/src/`) is the `wl_seat` global. It +aggregates keyboard, pointer, and touch, manages focus, and runs the input +queue. Each `Seat` has: + +- `Keyboard` (always present) +- `Pointer` (always present) +- `Touch` (advertised only if `'ontouchstart' in document.documentElement`, + but inert — see §Touch below) +- `InputQueue` — coalesces events and dispatches them on the next RAF +- `InputOutput` — abstracts FD operations (browser pipes for in-browser + apps, WebSocket endpoints for native apps via proxy) + +Each seat also tracks: + +- The currently focused `Surface` for keyboard input +- The currently focused `Surface` for pointer input (transient) +- The active pointer grab (default, popup, drag, or resize) +- The active keyboard grab (default or popup) + +## Keyboard — xkbcommon in the browser + +### WASM xkbcommon build + +`libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, +applies a patch (removes tests, shims `strdup`), builds with emscripten +3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, +exports the keymap/state C API as a single-file ES module. + +The same build approach is used for pixman (for region math). + +### Keymap compilation flow + +The browser, not the proxy, is the source of truth for the keymap. The +flow is: + +1. Browser receives a keymap config (either default or user-supplied via + `userShell.actions.setUserConfiguration({ keyboardLayoutName })`). +2. `Seat.notifyUpdateKeymap(nrmlvo)` is called. +3. `xkb_context_new` + `xkb_keymap_new_from_names` (WASM calls) compile the + RMLVO into a keymap. +4. `xkb_keymap_get_as_string` serializes the keymap as a long string. +5. `Keyboard.sendKeymap(resource)` is called for each `wl_keyboard` + resource: + - `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a + `/mkstemp-mmap` WebSocket, sends the blob, and receives a `ProxyFD { + handle: , type: 'shm', host: baseURL }`. + - `keyboardResource.keymap(xkbV1, ProxyFD, size)` is sent to the + native client. +6. The proxy deserializes the ProxyFD, looks up the native fd via + `proxyFDtoNativeFD` (identity when `host === baseURL`), ships it via + `sendEvents(wlClient, msg, fds)` from the N-API addon — + `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain + socket. +7. The native client's libwayland receives the keymap fd, mmaps it, and + calls native `xkb_keymap_new_from_string` to compile its own copy. + +So the keymap flows **browser → proxy → native**, not the reverse. This +is the opposite of what you might expect (since the proxy is on the Linux +host where xkbcommon natively lives), but it makes sense: the browser is +the actual compositor and is the one that needs the keymap to translate +DOM key events to Wayland keycodes. + +### DOM → Wayland key event translation + +Browser DOM `KeyboardEvent` has `event.code` (e.g. `"KeyA"`) and +`event.key` (the resolved glyph). Greenfield uses **only the physical +`.code`**, mirroring evdev: + +1. `event.code` is looked up in `EvDevKeyCode` reverse-name-map → evdev + keycode (e.g. `30` for `KeyA`). +2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset). +3. `capsLock` / `numLock` are read from `event.getModifierState(...)`. + +The DOM `.key` glyph is **ignored** — only physical `.code` is used, so +the same physical key always produces the same evdev keycode regardless +of layout. This is correct Wayland behavior. + +### Modifier tracking + +Modifier state is tracked entirely via the WASM libxkbcommon state +machine: + +1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or + `xkb.keyUp(x11)`. +2. Internally, `_xkb_state_update_key` is invoked. +3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` / + `modsLocked` / `group` (each a `_xkb_state_serialize_mods` call). +4. High-level `KeyboardModifier` bits are computed by checking xkb + mod-indices resolved at keymap-compile time. +5. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)` + sends the `wl_keyboard.modifiers` event to the native client. + +### Notable bug + +`Seat.ts` lines 511-513 contain a copy/paste typo: +`modsLatched = modsLocked` (should be `= modsLatched`). The cached +`keyboard.modifiers` struct is unreliable but doesn't affect protocol +output because `notifyModifiers` reads from `xkb` directly each time. + +## Pointer + +### DOM → Wayland pointer events + +Browser DOM `MouseEvent` is converted to Greenfield's internal +`ButtonEvent` / `AxisEvent`: + +- `pointermove` → `createButtonEventFromMouseEvent` → `ButtonEvent` + (motion-only) +- `pointerdown` / `pointerup` → `ButtonEvent` (with button + state) +- `wheel` → `createAxisEventFromWheelEvent` → `AxisEvent` + +### Pointer enter / leave between surfaces + +`DefaultPointerGrab.focus()` runs on every motion event: + +1. `Renderer.pickView` walks the view stack front-to-back. +2. Returns the first view whose `Surface.isWithinInputRegion(point)` + (pixman-WASM region) is true. +3. On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to + the old surface, then `wl_pointer.enter` + `frame` (+ modifiers if + keyboard is grabbed) to the new surface. + +### Pointer capture / window leave + +- `canvas.setPointerCapture` is called on `pointerdown`. +- `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus, + releases depressed keys, sets `needFocusInit=true` for re-sync), + resets `buttonCount=0`. +- DnD suppresses this handler mid-drag. + +### Pointer acceleration / cursor images + +**No acceleration** — verbatim `offsetX/Y`. + +Cursor images are **CSS `cursor: url(data:...) x y, auto`**: + +1. Native client calls `wl_pointer.set_cursor`. +2. The cursor surface commits a buffer. +3. `CursorRole` draws the bitmap to a hidden canvas. +4. `toDataURL()` produces a data URL. +5. Set on `document.body.style.cursor`. + +OS composites the cursor. No rotation/scale transforms. + +### Scroll wheel / touchpad + +`WheelEvent.deltaMode`: + +- `DOM_DELTA_LINE × 12` — discrete wheel clicks +- `DOM_DELTA_PAGE × surface size` — page-sized scrolls +- `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad + +Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version +≥5 also gets `axis_source(wheel)` + `frame()` (groups events). + +No `axis_discrete` / `value120` / `axis_stop` events are sent. + +### Linux button codes + +DOM `event.button` (0=left, 1=middle, 2=right, 3=back, 4=forward) is +mapped to Linux input protocol button codes via the `linuxInput` table +in `browser/input.ts`. + +## Touch — declared but inert + +`Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. +`wl_touch` capability is advertised if `'ontouchstart' in +document.documentElement`, but no events are ever sent. Touch input flows +through Pointer Events as mouse input. + +**For the user**: if you need real touch support (e.g. for tablets), this +is a feature you'll have to implement from scratch. The protocol stubs +exist (via generated `wl_touch` bindings), but no event source populates +them. + +## Focus management + +### DOM focus + +The `` owns keyboard focus permanently (`tabIndex=1`). +`Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between +native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface` +activation handlers. + +### DOM → Wayland focus + +- Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores + saved focus, re-syncs modifiers). +- Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves + focus, releases depressed keys, sets `needFocusInit=true`). + +### Surface-level focus + +- Click on a surface → `XdgToplevel` activation request → + `desktopSurface.activate()` → `gainFocus()` (refcounted via + `focusCount`, fires `surfaceActivationUpdated` event to the user shell + on 0↔1 transitions). +- Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`, + validating the serial against the keyboard/pointer `grabSerial`. + +## Input queue + +`InputQueue.ts` coalesces events and dispatches them on the next RAF +(`InputQueue.drainQueues`). This batches bursts of input events (e.g. fast +mouse motion) into a single dispatch, avoiding redundant work. + +The queue is FIFO per event type (motion, axis, key, button). Each +dispatch calls the appropriate `seat.notify*` method. + +## ASCII event-flow diagram + +``` +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ BROWSER (DOM) │ +│ │ +│ ─┐ │ +│ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │ +│ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │ +│ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │ +│ │ • evdevKeyCode = EvDevKeyCode[event.code] │ +│ │ • x11KeyCode = evdevKeyCode + 8 │ +│ │ • capsLock/numLock = event.getModifierState(...) │ +│ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │ +│ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │ +└──────────────────────────────┬──────────────────────────────────────────────────────┘ + │ session.inputQueue.queue{Motion,Axis,Key,Button} + ▼ +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │ +│ │ +│ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame │ +│ │ │ +│ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │ +│ │ └─ Keyboard.sendKey(event) │ +│ │ └─ keyboardResource.key(serial,time,evdev,st) │ +│ │ │ +│ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │ +│ │ └─ Pointer.sendButton │ +│ │ └─ pointerResource.button(serial,time, │ +│ │ linuxInput[btn], state) │ +│ │ │ +│ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │ +│ │ └─ pointerResource.motion(time, sx_f, sy_f) │ +│ │ │ +│ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │ +│ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │ +│ │ └─ pointerResource.axis(time, axis, Fixed) │ +│ │ + axisSource(wheel) + frame() if v>=5 │ +│ │ │ +│ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │ +│ │ │ +│ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │ +│ └─ Keyboard.sendModifiers(modsDepressed,Latched, │ +│ Locked,group) │ +│ │ +│ Keymap path (once per WlKeyboardResource): │ +│ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │ +│ → xkb_context_new + xkb_keymap_new_from_names (libxkbcommon WASM) │ +│ → xkb_keymap_get_as_string ──► keymapString │ +│ Keyboard.sendKeymap(resource): │ +│ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │ +│ keyboardResource.keymap(xkbV1, ProxyFD, size) │ +└──────────────────────────────┬──────────────────────────────────────────────────────┘ + │ client.connection.onFlush(wireMessages) + │ ─ one big Uint32Array, FDs serialized as ProxyFD JSON + ▼ +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ WIRE (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts) │ +│ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ ws on proxy :SessionController │ +│ Message layout (Uint32Array): │ +│ [0] outOfBandOpcode (0 = in-band) │ +│ [1] eventSerial │ +│ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │ +│ then concatenated Wayland wire messages │ +└──────────────────────────────┬──────────────────────────────────────────────────────┘ + │ NativeWaylandClientSession.onMessage + ▼ +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ PROXY (Node.js, compositor-proxy) │ +│ │ +│ SessionController wss paths: /mkfifo /mkstemp-mmap /read-fd /close-fd │ +│ /read-fd-as-stream /write-fd-as-stream │ +│ /signal /channel /request-keyframe │ +│ │ +│ mkstempMmap(ws): │ +│ receives Blob → Buffer.concat → createMemoryMappedFile(buf) ← native addon │ +│ returns ProxyFD { handle: , type: 'shm', host: baseURL } │ +│ │ +│ onWireMessageEvents: │ +│ for each ProxyFD in message: │ +│ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) // just returns .handle │ +│ └─ if proxyFD.host === baseURL → identity │ +│ └─ if foreign host → spawn a bridging pipe + WebSocket │ +│ messageInterceptor.interceptEvent(...) // empty stubs for kbd/ptr/touch │ +│ sendEvents(wlClient, messageBuffer, fdsBuffer) ← libwayland-server addon │ +│ flush(wlClient) │ +└──────────────────────────────┬──────────────────────────────────────────────────────┘ + │ libwayland-server sends wire msg + SCM_RIGHTS FDs + ▼ +┌─────────────────────────────────────────────────────────────────────────────────────┐ +│ NATIVE (Wayland client e.g. weston-terminal, Xwayland app) │ +│ wl_display.fd readable → libwayland-client demarshals │ +│ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │ +│ wl_keyboard.key / enter / leave / modifiers → client app │ +│ wl_pointer.enter / motion / button / axis / frame → client app │ +└──────────────────────────────────────────────────────────────────────────────────────┘ +``` + +## Key answers (quick reference) + +| Question | Answer | +|---|---| +| DOM→Wayland key event translation | `event.code` (e.g. `"KeyA"`) is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30`). `x11KeyCode = evdevKeyCode + 8`. DOM `.key` glyph is **ignored** — only physical `.code` is used, mirroring evdev. | +| Modifier tracking | Entirely via WASM libxkbcommon state machine. `Seat.updateModifierState` calls `xkb.keyDown/Up(x11)` → `_xkb_state_update_key`, then `notifyModifiers` reads `xkb.modsDepressed/Latched/Locked/group` (each a `_xkb_state_serialize_mods` call). High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. | +| Focus: DOM ↔ native surfaces | The `` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave/enter` between native surfaces. Activation is driven by `XdgToplevel`/`ShellSurface` activation handlers. | +| Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` calls `notifyKeyboardFocusOut` (saves kbd focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. DnD suppresses this handler mid-drag. | +| Touch event mapping | **Not implemented.** `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. | +| Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView` walks the view stack front-to-back, returns first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. On change, `Pointer.setFocus` emits `leave+frame` to old surface, `enter+frame` (+modifiers) to new. | +| Pointer acceleration / cursor images | **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: client calls `wl_pointer.set_cursor` → `CursorRole` → draw bitmap to hidden canvas → `toDataURL()` → set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. | +| Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: `DOM_DELTA_LINE × 12`, `DOM_DELTA_PAGE × surface size`, `DOM_DELTA_PIXEL ÷ 3`. Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. v≥5 also gets `axis_source(wheel)` + `frame()`. No `axis_discrete/value120/stop`. | +| WASM xkbcommon? | Yes. `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch, builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. Sibling build does the same for `pixman`. | +| Keymap FD workaround for WebSocket | Direction is **browser → proxy**, not the reverse. Browser calls `inputOutput.mkstempMmap(Blob)` → opens `/mkstemp-mmap` WebSocket → proxy creates real native shm fd via `createMemoryMappedFile` addon → returns `ProxyFD {handle: , type: 'shm', host: baseURL}`. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`; the ProxyFD is **serialized as JSON** in the wire message's `fds[]` array (WebSockets can't pass FDs natively). Proxy deserializes, maps `handle` → native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. | + +## File reference + +| File | Role | +|---|---| +| `packages/compositor/src/Seat.ts` | `wl_seat` global, multi-seat support, focus management | +| `packages/compositor/src/Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events | +| `packages/compositor/src/Pointer.ts` | `wl_pointer` impl, motion, buttons, axis | +| `packages/compositor/src/Touch.ts` | `wl_touch` (46-line stub, inert) | +| `packages/compositor/src/Xkb.ts` | xkbcommon WASM wrapper | +| `packages/compositor/src/KeyEvent.ts` | Key event wrapper | +| `packages/compositor/src/ButtonEvent.ts` | Pointer button event wrapper | +| `packages/compositor/src/AxisEvent.ts` | Pointer axis (scroll) event wrapper | +| `packages/compositor/src/InputQueue.ts` | Event coalescing + RAF dispatch | +| `packages/compositor/src/InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) | +| `packages/compositor/src/browser/input.ts` | DOM event listeners → Greenfield events | +| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer | +| `libs/compositor-wasm/` | pixman + xkbcommon WASM build | +| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node | +| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor | +| `packages/compositor-proxy/src/io/` | WebSocket server glue (mkstemp-mmap endpoint etc.) | diff --git a/06-data-clipboard.md b/06-data-clipboard.md new file mode 100644 index 0000000..c4ce4ea --- /dev/null +++ b/06-data-clipboard.md @@ -0,0 +1,314 @@ +# 06 — Data Device, Selection, DnD (`wl_data_device`) + +This chapter covers Greenfield's data device / clipboard / drag-and-drop +implementation: how `wl_data_device_manager`, `wl_data_source`, +`wl_data_offer`, `wl_data_device` are implemented, how clipboard contents +are transferred between browser and native apps, and how DnD is coordinated. + +## Faithful Wayland model + +`DataDeviceManager` is a v3 global singleton. `wl_data_device` is per +seat-per-client and the implementation lives directly on `Seat` (which +implements `WlDataDeviceRequests`). `WaylandDataSource` / `DataOffer` are +the standard source/offer pair, ported almost verbatim from weston, +including the `chooseAction()` bit-trick for DnD action negotiation. + +## The FD-over-WebSocket problem + +Wayland's `wl_data_offer.receive(mimeType, fd)` / +`wl_data_source.send(mimeType, fd)` carry a real UNIX fd. Greenfield solves +this with a **`ProxyFD` JSON descriptor**: + +``` +ProxyFD = { + handle: number, + type: 'pipe-read' | 'pipe-write' | 'shm' | 'unknown', + host: string // proxy's HTTP base URL +} +``` + +Three concrete `InputOutput` / `InputOutputFD` implementations: + +### `webInputOutput` — for in-browser web apps + +Pipes are `MessageChannel` ports, transferred as `ArrayBuffer`s. All +"fd" operations are in-browser zero-copy transfers. + +### `RemoteInputOutput` — for native apps + +All fd operations go over a secondary WebSocket to endpoints: +`/mkfifo`, `/mkstemp-mmap`, `/read-fd-as-stream`, `/write-fd-as-stream`, +`/read-fd`, `/close-fd`. + +### `ProxyInputOutput` (proxy-side) — owns the real OS pipes and memfd files + +`proxyFDtoNativeFD(proxyFD)` returns the raw int fd because every ProxyFD +has `host === proxy.baseURL`. + +The proxy interceptors for `wl_data_source` / `wl_data_offer` / +`wl_data_device` / `wl_data_device_manager` are auto-generated no-ops that +only handle resource construction and lifetime — all actual semantics live +in the browser compositor. + +### Crucial consequence + +For native→native clipboard transfers (e.g. user copies in one Linux app, +pastes in another), the browser compositor never sees a single byte of the +data — it only relays the fd reference. The actual content stays inside +the proxy host's OS kernel pipe. + +## MIME-type negotiation + +At the Wayland level, mime types are just relayed strings. Mapping to the +browser's hard-coded allow-list `['text/plain', 'text/html', 'image/png']` +happens only at the DOM boundary: + +### Native → browser + +`selection.ts:handleWaylandDataSourceUpdate`: each source mime type is +substring-matched against the allow-list (`text/plain;charset=utf-8` +contains `text/plain`, so it maps), survivors are pulled as separate blobs +via `mkfifo + send + readBlob`, and finally +`navigator.clipboard.write([new ClipboardItem(...)])`. + +### Browser → native + +`BrowserDataSource.ts`: `ClipboardItem.types` become the source mime types, +plus a synthetic `text/plain;charset=utf-8` alias is appended (Linux apps +expect the charset form). On `send`, the alias is normalised back to +`text/plain` for the lookup. + +### Caveat: substring matching is fragile + +The substring match `dataSourceMimeType.indexOf(browserMimeType) !== -1` +can collide (`image/png` matching `image/png;base64` is fine, but +`text/plain;charset=iso-8859-1` silently becomes `text/plain`). Consider +an explicit prefix or exact-match policy in your implementation. + +## Clipboard mirroring is pull-based + +The browser has no `clipboardchange` event (Chromium issue 933608 is +cited in the code). Greenfield polls `navigator.clipboard.read()`: + +- on init +- on every `window.focus` event + +De-duplicates offers by mime-type set + total blob size, then calls +`seat.setSelectionInternal(createBrowserDataSource(offers), serial)`. + +The feedback loop is broken because `handleWaylandDataSourceUpdate` returns +early when `dataSource instanceof BrowserDataSource`. + +### Firefox fallback + +`text-selection.ts` uses only `readText` / `writeText` and a +`BrowserTextDataSource` that serves `text/plain;charset=utf-8` — no images +or HTML. + +## Drag-and-drop + +### Wayland-side (Seat.startDrag) + +1. Validates the implicit pointer grab. +2. Attaches `DragIconRole` to the optional icon surface. +3. Installs a `PointerDrag` grab on both pointer and keyboard. + +`PointerDrag` tracks the focused data device, sends +`wl_data_device.enter` / `leave` / `motion` / `drop`, creates fresh +`wl_data_offer`s via `sendOffer`, and runs the v3 action negotiation. + +### Browser-side (`BrowserDnD` class) + +Creates a hidden full-canvas `dragHelper` div with `draggable=true` when +the Wayland drag-icon surface commits, so the browser shows the Wayland icon +via `event.dataTransfer.setDragImage(cursorCanvas, ...)`. While the user +drags, `dragover` events on the dragHelper are converted back to +`ButtonEvent`s and pushed through `session.inputQueue.queueMotion`, keeping +`PointerDrag` fed. + +### Not implemented: dragging data FROM a Wayland app INTO a real DOM target + +The FIXME in `dnd.ts` admits: dragging data FROM a Wayland app INTO a real +DOM drop target (file drag-out, etc.) is **not implemented**. + +## XWayland bridge + +`XWindowManager` listens on the X11 `CLIPBOARD` atom via `XFixes` and +builds `XDataSource` (clipboard) or `XDnDDataSource` (XDnD). When the +receiver pulls data: + +1. `XDataSource.send` calls `XConvertSelection` to have the X server push + the bytes into a property on `selectionWindow`. +2. `writeProperty` copies them into the receiver's `InputOutputFD`. +3. INCR transfers are handled (`getIncrChunk`, `sendIncrChunk`). + +## Primary selection is NOT implemented + +A grep for `primary_selection`, `gtk_primary`, `zwp_primary_selection` +across the entire `packages/` tree returns zero hits. XWayland only +watches the `CLIPBOARD` atom, not `PRIMARY`. **Middle-click paste does +not work in Greenfield today.** + +**For the user**: if your users expect Linux-style middle-click paste, +you'll need to implement `zwp_primary_selection_device_manager_v1` (or +`gtk_primary_selection`) on the compositor side and forward `PRIMARY` +notifications in `XWindowManager`. + +## ASCII diagram — clipboard flow in both directions + +``` + CLIPBOARD / SELECTION FLOW + +================================================================= +A) NATIVE APP COPIES ──► BROWSER DOM CLIPBOARD (Ctrl+C in Linux app) +================================================================= + +┌──────────────┐ wl_data_device.set_selection(src,serial) ┌────────────────────────┐ +│ Native app │ ─────────────────────────────────────────────► │ compositor │ +│ (Wayland or │ │ (browser, in DOM) │ +│ XWayland) │ ◄────── wl_data_offer.offer("text/plain") ─── │ │ +└──────────────┘ (only if this app has keyboard focus)└───────────┬────────────┘ + │ selectionListeners fire + ▼ + ┌──────────────────────────┐ + │ handleWaylandDataSource │ + │ Update() (selection.ts)│ + └──────────┬──────────────┘ + mkfifo() ──────────────────────────► │ proxy /mkfifo + ProxyFD read+write (real OS pipe) ◄───── returns 2 ProxyFDs + ▼ + dataSource.send("text/plain", writeFD) + │ + ▼ + wl_data_source.send("text/plain", writeFDProxyFD) + ─────────────────────────────────────────► │ Native source + │ receives real write fd + │ (proxyFDtoNativeFD + │ is a passthrough) + ▼ + Native app writes bytes + ──────────────► OS pipe ───┐ + │ + readFD.readBlob() ◄──────────── read from /read-fd-as-stream ◄─┘ + ▼ (proxy streams over WS) + Blob {type:"text/plain"} + ▼ + navigator.clipboard.write([ new ClipboardItem({"text/plain": blob}) ]) + ▼ + ┌────────────────────────────┐ + │ DOM clipboard populated │ + │ (Ctrl+V anywhere now works) │ + └────────────────────────────┘ + + +================================================================= +B) USER COPIES IN BROWSER ──► NATIVE APP GETS SELECTION (Ctrl+C in DOM) +================================================================= + +┌─────────────────────────────┐ +│ User copies in DOM │ +│ (browser tab, textarea...) │ +└──────────────┬──────────────┘ + │ on window 'focus' event (or initial load) + ▼ + navigator.clipboard.read() ──► ClipboardItems[] + ▼ + handleBrowserDataSourceUpdate() — de-dup vs previous offers + ▼ + createBrowserDataSource(offers) + .mimeTypes = offer.types + ['text/plain;charset=utf-8'] alias + ▼ + seat.setSelectionInternal(browserDataSource, serial) + │ + │ 1) selectionListeners fire — handleWaylandDataSourceUpdate + │ returns early (instanceof BrowserDataSource), breaking loop + │ + │ 2) sendSelection(focusedClient) + ▼ +┌────────────────────────┐ wl_data_offer.offer("text/plain"), ┌──────────────┐ +│ compositor │ ─ wl_data_device.selection(offer) ──► │ Native app │ +│ (browser, in DOM) │ │ (focused) │ +└───────────┬────────────┘ └──────┬───────┘ + │ │ + │ When the native app calls wl_data_offer.receive("text/plain", fd): + │ │ + │ ◄──────── wl_data_offer.receive(mimeType, writeFDProxyFD) ◄── native pipe + │ (fd was created by the native app's libwayland; proxy + │ wrapped it as ProxyFD{host:proxy.baseURL, type:'unknown'} + │ then compositor's DataOffer.receive calls wrapFD → RemoteInputOutputFD) + ▼ + BrowserDataSource.send(mimeType, ioFD) + │ + ▼ + matchingOffer.getType(mimeType) ─► Promise (from ClipboardItem) + │ + ▼ + ioFD.write(blob) ─► POST /write-fd-as-stream?fd= ─► proxy + │ + ▼ + createWriteStream({fd, autoClose:true}) + ws stream ──► native write fd + │ + ▼ + Bytes flow through OS pipe into native app + │ + ▼ + ioFD.close() ─► /close-fd ─► EOF ─► native app's read returns 0 + + +================================================================= + PROXY FD TRANSLATION (works the same way in B and A) +================================================================= + + Native fd (in proxy process) + │ + │ wrap into ProxyFD JSON: {handle, type, host: proxy.baseURL} + │ JSON-encode, length-prefix, send over WS protocol channel + ▼ + ┌─────────────────────────────────────────────────┐ + │ Browser compositor │ + │ client.userData.inputOutput.wrapFD(fd, type) │ + │ → RemoteInputOutputFD (RemoteInputOutput) │ + │ → WebPipeInputOutputFD (webInputOutput, │ + │ for in-browser apps)│ + └─────────────────────────────────────────────────┘ +``` + +## File reference + +| File | Role | +|---|---| +| `packages/compositor/src/DataDeviceManager.ts` | `wl_data_device_manager` global | +| `packages/compositor/src/DataSource.ts` | `wl_data_source` impl, the provider side | +| `packages/compositor/src/DataOffer.ts` | `wl_data_offer` impl, the receiver side | +| `packages/compositor/src/WaylandDataSource.ts` | Wrapper around native `wl_data_source` events | +| `packages/compositor/src/Seat.ts` | `wl_data_device` impl (Seat implements `WlDataDeviceRequests`) | +| `packages/compositor/src/browser/selection.ts` | Browser clipboard bridge (native → DOM) | +| `packages/compositor/src/browser/text-selection.ts` | Firefox fallback (text-only) | +| `packages/compositor/src/browser/dnd.ts` | Browser DnD bridge (Wayland → DOM) | +| `packages/compositor/src/InputOutput.ts` | FD abstraction interface | +| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps | +| `packages/compositor/src/remote/XWindowManager.ts` | X11 CLIPBOARD atom watcher | +| `packages/compositor/src/remote/XDataSource.ts` | X11 clipboard source (XWayland apps) | +| `packages/compositor/src/remote/XDnDDataSource.ts` | X11 DnD source | + +## Next actions / suggested improvements + +1. **Primary selection** — implement `zwp_primary_selection_device_manager` + v1 (or `gtk_primary_selection`) on the compositor side and forward + `PRIMARY` notifications in `XWindowManager`. This unlocks middle-click + paste from native apps. +2. **`clipboardchange` event** — when browsers ship it, replace the + `window.focus`-based polling in `selection.ts` with a proper event-driven + subscription. +3. **Native→DOM DnD** — implement the dataTransfer side of `BrowserDnD` + so a Wayland drag can deliver bytes to a real DOM drop target (the + `dnd.ts` FIXME). +4. **MIME allow-list discovery** — replace the hard-coded + `['text/plain', 'text/html', 'image/png']` with a runtime check once + browsers expose the supported types. +5. **MIME-type strict matching** — the substring match + `dataSourceMimeType.indexOf(browserMimeType) !== -1` can collide + (`image/png` matching `image/png;base64` is fine, but + `text/plain;charset=iso-8859-1` silently becomes `text/plain`). + Consider an explicit prefix or exact-match policy. diff --git a/07-native-app-bridge.md b/07-native-app-bridge.md new file mode 100644 index 0000000..bc924b1 --- /dev/null +++ b/07-native-app-bridge.md @@ -0,0 +1,404 @@ +# 07 — Native App Bridge (How Real Linux Apps Get Connected) + +This chapter covers Greenfield's native app bridge: how real Linux +applications get launched, connected to the proxy, and routed to the +browser. This is the bridge that makes a Linux Wayland app appear inside +the browser. **This is the most important chapter for the user** — they +want to run real Linux apps in their Odoo 20 web app. + +## Two app models — don't conflate them + +Greenfield has **two app models**: + +### Model 1: WASM web apps (browser-resident) + +Built with the Greenfield SDK (Emscripten), run inside the browser via +`MessageChannel`. The `sdk/` directory serves only this model. Apps are +compiled to WASM, run in the same browser tab as the compositor, and +speak Wayland over a `MessagePort` instead of a UNIX socket. + +### Model 2: Native Linux apps (the focus of this chapter) + +Real ELF binaries (`gtk4-demo`, `kwrite`, `xterm`) running as child +processes of the Node.js proxy. Apps use the host's own system libraries +— **no sysroot, no chroot, no container**. + +## Process model — three layers + +### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`) + +A single Node.js process listens on `:8081`. Per browser session, it +`fork()`s one `SessionProcess` child — **one child per `compositorSessionId`**, +not per app. Multiple apps in the same browser tab share one child. +WebSocket upgrades are handed off to the matching child via +`childProcess.send(wsUpgrade, socket)`. + +CLI args: + +- `--bind-port=8081` (default) +- `--applications=` (URL → executable map) +- `--encoder=x264|nvh264|vaapih264` +- `--render-device=/dev/dri/renderD128` +- `--base-url`, `--allow-origin`, `--bind-ip`, `--basic-auth` + +### Layer 2: Session Process (`SessionProcess.ts`) + +The forked child. Receives IPC messages: + +- `start` — creates `Session` + `SessionController` +- `launchApp` — spawns the actual app binary +- `wsUpgrade` — hands WS to the controller + +On `launchApp` it calls `launchApplication()` from `@gfld/compositor-proxy`, +which `spawn()`s the executable, captures `pid`, replies to main with +`{pid, key}`. On `session.close()` the process exits. + +### Layer 3: Proxy Core (`compositor-proxy/src/`) + +- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via + `createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket + at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this + becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/ + NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM, + registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the + wl_display FD into Node's libuv loop via `startPoll()`. +- **`NativeWaylandClientSession`** — per-client wire-message interceptor. + Uses `MessageInterceptor` + auto-generated per-resource interceptor + modules (`src/protocol/wl_*_interceptor.js`). Each request returns + `{native, browser, neverReplies?}` routing decision. Handles + `wl_display.sync` fast-path (skips browser round-trip when no reply + expected). Marshalls FDs as `ProxyFD` JSON. +- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random + signaling auth), signaling WS, client sessions, data channels. Lifecycle: + 10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation + (10 s) on force-quit, auto-destroy on app exit. +- **`SessionController`** — WS demultiplexer by pathname: `/signal` + (per-app control), `/channel` (multiplexed data channels), `/mkfifo`, + `/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`. +- **`XWaylandSession`** — lazy Xwayland fork (only on first X client + connect). Xwayland itself becomes a Wayland client of the proxy. The + `wm_fd` is bridged via xtsb to the browser, which implements the actual + XWM logic. +- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency) + for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK. + +## App launch flow (definitive answer) + +1. Browser sends `GET http://proxy:8081/` with + `X-Compositor-Session-Id: ` header. +2. Main process forks a `SessionProcess` if none exists for that session + ID, sends `start` IPC. +3. Main sends `launchApp` IPC with `{name, executable, args, env, serial}`. +4. `SessionProcess` calls `launchApplication()`: + - `appEnv = {...process.env, ...env, WAYLAND_DISPLAY: + session.nativeWaylandCompositorSession.waylandDisplay}` + - `child_process.spawn(executable, args, {env: appEnv})` — **plain + `spawn`, no chroot/container** +5. On `spawn` event, captures `pid`, creates `NativeAppContext`, registers + `exit` listener. +6. Replies to main with `{pid, key}` → main returns HTTP 201 with + `{baseURL, signalURL, key, pid, name}`. +7. Browser opens `signalURL` as WebSocket → control channel established. +8. App connects to `$XDG_RUNTIME_DIR/wayland-N` → libwayland fires + `on_client_created` → proxy looks up PID via `getCredentials()`, walks + `/proc//status` PPid chain to match a `NativeAppContext`. +9. For each new `wl_client`, creates a `PROTOCOL` ARQ channel and sends + `CONNECT_CHANNEL` over the signaling WS; browser opens a new WS at + `/channel?id=…`. +10. Wayland wire messages now flow: app → libwayland → + `setWireMessageCallback` → JS interceptor → routed to `{native, + browser}`. + +## Native addon architecture + +**Important**: The "native" code is **C, not C++** (one GLSL shader string + +GObject-style). Uses Node-API (N-API). + +### `wayland-server-addon.node` (1231 lines C) + +Wraps a **patched** `libwayland-server` (`libwayland-server.so.0` with +Greenfield additions in `native/wayland/changes.diff`): + +- `wl_client_set_wire_message_cb` +- `wl_client_set_registry_created_cb` +- `wl_client_set_sync_done_cb` +- `wl_client_add_resource_created_listener` +- `wl_display_set_global_created_cb` +- `wl_resource_destroy_silently` +- `wl_get_server_object_ids_batch` +- `wl_connection_copy_fds_in` + +Plus the `westfield` shared lib (EGL/GBM init, `wlr_linux_dmabuf_v1`, +`wlr_drm`, `westfield-xwayland`). **31 N-API functions** exposed. This is +what makes the proxy a real Wayland compositor. + +### `proxy-poll-addon.node` (127 lines C) + +Tiny wrapper around `uv_poll_t` (libuv, not raw epoll — though libuv uses +epoll under the hood). `startPoll(fd, cb)` integrates the wl_display +event-loop FD into Node's libuv loop so wayland events fire as Node +callbacks without blocking. + +### `proxy-encoding-addon.node` (290 lines C) + `libproxy-encoding.so` (~1400 lines C) + +GStreamer H.264/PNG encoder. Runs on a **dedicated GMainLoop thread** with +an `eventfd`-based message queue (`SyncSource` GSource). Two pipelines per +surface (opaque + alpha) for translucent windows. Supports `x264` +(software), `nvh264` (Nvidia NVENC), `vaapih264` (Intel, experimental). +Zero-copy DMA-BUF path via EGLImage → GStreamer GL. Uses +`napi_threadsafe_function` to marshal encoded samples back to Node. + +## XWayland bridge + +### Native side (`westfield-xwayland.c`, 466 lines C) + +1. Creates `/tmp/.X11-unix/X` listener socket +2. `setenv("DISPLAY", ":", 1)` +3. Waits for X client connect +4. On connect: `socketpair(AF_UNIX)` × 3 (wayland / wm / display) +5. `wl_client_create(wl_display, sv[0])` — **Xwayland itself becomes a + Wayland client of the proxy!** +6. `fork()` + `execlp("Xwayland", "Xwayland", ":", "-ac", "-rootless", + "-listen", "", "-displayfd", "", "-wm", "", + NULL)` + +### xtsb (X11 TypeScript Bindings) + +`libs/xtsb/`: pure-TS X11 client, no libxcb. Dual-environment (Node +`net.Socket` + browser WebSocket). Auto-generated from XCB XML protocol +files via `xcbgen/ts_client.py`. Handles MIT-MAGIC-COOKIE-1 auth from +`~/.Xauthority`. The browser-side xtsb implements the actual XWM logic +(translates X `CreateNotify` events → `xdg_toplevel` Wayland surfaces). + +`XWaylandSession.ts` uses `nodeFDConnectionSetup(wmFd)` to wrap the wm_fd +as a `net.Socket`, performs X11 auth, and pipes the X11 wire protocol +through the XWM data channel to the browser. + +## SDK misconception corrected + +The `sdk/` directory is **NOT for native apps**. It's exclusively for +**WASM apps** that run inside the browser via Emscripten. The +`sysrootlibs/` cross-compiles wayland/gtk/cairo/etc. to wasm32. Native +Linux apps on the proxy use the host's own system libraries — the +Dockerfile installs them via `apt-get install libffi8 libgbm1 +libgstreamer1.0-plugins-base … xwayland xauth`. The proxy's `pkg` config +only bundles three libs as assets (`libproxy-encoding.so`, +`libwayland-server.so.0`, `libwestfield.so`) — these are the proxy's OWN +addon dependencies, not the apps'. + +## ASCII diagram — native app launch & connection + +``` +Browser (compositor-shell) Proxy host (Linux) App host (often same as proxy) +========================== ==================== ============================== + +[1] User clicks app in shell $ compositor-proxy-cli \ + │ └─── main.ts listens on :8081 + │ sessionProcesses: Record + │ + │ HTTP GET /gtk4-demo + │ Header: X-Compositor-Session-Id: + ├──────────────────────────────► │ + │ │ if no child for : fork(SessionProcess.ts) + │ │ └── IPC: {type:'start', payload:{config, sessionId}} + │ │ └── createSession() → NativeWaylandCompositorSession + │ │ ├── createDisplay() + │ │ │ └── wl_display_create() (libwayland-server) + │ │ ├── addSocketAuto() + │ │ │ └── creates $XDG_RUNTIME_DIR/wayland-N + │ │ │ returns "wayland-N" ◄── this becomes WAYLAND_DISPLAY + │ │ ├── initShm() + YUV420/NV12/YUV444/NV21 + │ │ ├── initDrm("/dev/dri/renderD128") + │ │ │ └── westfield_egl_new() → EGLDisplay via GBM + │ │ │ wlr_linux_dmabuf_v1 global (zero-copy GPU buffers) + │ │ │ wlr_drm global (legacy dmabuf) + │ │ └── startPoll(wlDisplayFd, dispatchRequests) + │ │ └── uv_poll_t on EPOLLIN + │ │ (proxy-poll-addon.node) + │ │ + │ │ IPC: {type:'launchApp', payload:{name, executable, args, env, serial}} + │ │ └── launchApplication() (NativeAppContext.ts) + │ │ ├── appEnv = {...process.env, ...env, + │ │ │ WAYLAND_DISPLAY: "wayland-N"} + │ │ ├── child_process.spawn("gtk4-demo", [], {env: appEnv}) + │ │ │ ◄── posix_spawn/execvp, NO chroot, NO container + │ │ │ ──────────────────────────────► [2] gtk4-demo process starts + │ │ │ │ + │ │ │ │ libwayland-client.so + │ │ │ │ connects to + │ │ │ │ $XDG_RUNTIME_DIR/wayland-N + │ │ │ │ + │ │ │ ▼ + │ │ │ ┌──────────────────────────┐ + │ │ │ │ wl_display event loop │ + │ │ │ │ (in proxy, libwayland- │ + │ │ │ │ server.so.0) │ + │ │ │ │ │ + │ │ │ ◄── uv_poll fires, │ on_client_created → │ + │ │ │ dispatchRequests() │ getCredentials(pid) │ + │ │ │ wl_event_loop_dispatch() │ findMatchingNativeApp │ + │ │ │ │ Context(pid) by PPID │ + │ │ │ │ walk │ + │ │ │ │ │ + │ │ │ │ setWireMessageCallback │ + │ │ │ │ setBufferCreated... │ + │ │ │ │ setSyncDoneCallback │ + │ │ │ │ setClientDestroyed... │ + │ │ │ │ │ + │ │ │ │ NativeWaylandClientSession + │ │ │ │ .messageInterceptor │ + │ │ │ │ dispatches R │ + │ │ │ │ → {native, browser} │ + │ │ │ └──────────────────────────┘ + │ │ │ + │ │ ├── child.on('spawn') → capture pid + │ │ │ createNativeAppContext(pid, name, external:false) + │ │ │ nativeAppContext.onDisconnect() // start 10-min SIGHUP timer + │ │ │ + │ │ └── child.on('exit') → nativeAppContext.onExit({exitCode|signal}) + │ │ + │ │ IPC reply: {type:'launchAppSuccess', payload:{pid, key}} + │ + │ HTTP 201 Created + │ Body: { baseURL, signalURL:"ws://…/signal?key=…", key, pid, name, internal:false } + ◄────────────────────────────── │ + │ │ + │ Open WS: ws://…/signal?key=<16-hex> + ├──────────────────────────────► │ + │ │ SessionController demux /signal → nativeAppContext.onConnect(ws) + │ │ clears 10-min SIGHUP timer + │ │ + │ │ For each new wl_client (one per wayland connection): + │ │ createProtocolChannel() → ARQChannel (KCP over WS) + │ │ sends CONNECT_CHANNEL {url:"ws://…/channel?id=…&key=…", desc} + │ ◄──── CONNECT_CHANNEL JSON ───┤ + │ │ + │ Open WS: ws://…/channel?id=…&key=… + ├──────────────────────────────► │ binds WS to the ARQChannel + │ │ + │ Wayland wire protocol flows over the ARQ channel (KCP reliable, 100ms interval): + │ + │ App → wl_surface.commit ──── proxy intercepts R1 (attach) + R6 (commit) ────► browser compositor + │ App ← wl_surface.frame ──── proxy intercepts E1 (frame callback) ◄────── FrameFeedback paces callback + │ App → wl_display.sync ────── proxy fast-path sync (no roundtrip if no reply expected) + │ App ← wl_keyboard.enter ─── browser → proxy → sendEvents(wlClient, …) → flush(wlClient) + │ + │ Buffer path (per wl_surface): + │ - App submits wl_buffer (dmabuf or shm) + │ - setBufferCreatedCallback → ProxyBuffer with creationSerial + │ - On commit: wl_surface_interceptor.R6 → Encoder.encodeFrame() + │ └── proxy-encoding-addon.encodeFrame(bufferId, contentSerial, creationSerial) + │ └── GStreamer pipeline (x264/nvh264/vaapih264) on dedicated GMainLoop thread + │ └── encoded H.264 sample → napi_threadsafe_function → Node main thread + │ └── sent over the surface's FRAME data channel (separate WS, also CONNECT_CHANNEL'd) + │ + │ X11 path (xterm etc.): + │ - App connects to /tmp/.X11-unix/X1 (DISPLAY=:1, set by westfield_xwayland.c) + │ - First X connect → spawn_xserver() forks: execlp("Xwayland", …, "-rootless", "-wm", wm_fd) + │ - Xwayland itself becomes a Wayland client of the proxy (wl_client_create) + │ - When Xwayland ready (displayFd readable): + │ └── xtsb.nodeFDConnectionSetup(wmFd) → wraps as net.Socket + │ └── pipes through XWM data channel (4th channel type) to browser + │ └── browser-side xtsb speaks X11 → translates X windows → xdg_toplevel surfaces + │ + │ Lifecycle: + │ - User closes browser tab → /signal WS closes with code 1001 → kill('SIGHUP') + │ - User clicks "force quit" in shell → /signal WS sends KILL_APP {signal:'SIGTERM'} → kill('SIGTERM') + │ └── if not exited in 10s → SIGKILL + │ - App crashes → child 'exit' event → nativeAppContext.onExit() → APP_TERMINATED msg → browser UI updates + │ - All clients gone for >1s → session.close() → process.exit() in SessionProcess.ts → main deletes entry + │ + │ Native addon role summary: + │ - wayland-server-addon.node: full Wayland compositor (libwayland-server fork + westfield EGL/GBM/dmabuf/XWayland) + │ - proxy-poll-addon.node: libuv uv_poll_t hook so wayland FD events fire as Node callbacks + │ - proxy-encoding-addon.node: GStreamer H.264/PNG encoder on a dedicated GMainLoop thread, threadsafe_function back to Node +``` + +## File reference + +### CLI (`packages/compositor-proxy-cli/src/`) + +| File | Role | +|---|---| +| `main.ts` | HTTP + WS hub, forks per session | +| `main-controller.ts` | CLI arg parsing | +| `main-args.ts` | CLI arg definitions | +| `SessionProcess.ts` | Per-session child, IPC handler | +| `example-applications.json` | Pre-configured apps | +| `wait-until-ready.sh` | Startup readiness check | + +### Proxy TS layer (`packages/compositor-proxy/src/`) + +| File | Role | +|---|---| +| `index.ts` | Exports `createSession()`, `launchApplication()` | +| `Session.ts` | Top-level proxy session | +| `SessionController.ts` | WS server with 9 endpoint paths | +| `NativeAppContext.ts` | Per-app state, signaling WS, kill timers | +| `NativeWaylandClientSession.ts` | Per-client message interceptor | +| `NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm | +| `Channel.ts` | ARQChannel + SimpleChannel | +| `kcp.ts` | ~1100-line KCP implementation | +| `FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done | +| `ProxyBuffer.ts` | Proxy-side buffer representation | +| `SurfaceBufferEncoding.ts` | Encoding pipeline orchestration | +| `XWaylandSession.ts` | XWayland spawn + bridge | +| `wayland-server.ts` | TS wrapper around wayland-server-addon.node | +| `io/` | WebSocket server glue | +| `encoding/` | High-level encoding orchestration | +| `protocol/` | Generated interceptor modules | +| `addons/` | .node file shims (Linux-x64) | +| `config.ts` | Encoding-related configuration | + +### Proxy C layer (`packages/compositor-proxy/native/`) + +| Path | Role | +|---|---| +| `CMakeLists.txt` | Build config | +| `wayland/src/addon.c` | Node-API addon: wayland-server-addon.node | +| `wayland/src/westfield-buffer.c` | Buffer helpers | +| `wayland/src/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM | +| `wayland/src/westfield-egl.c` | EGL zero-copy setup | +| `wayland/src/westfield-xwayland.c` | XWayland spawn + bridge | +| `wayland/src/wlr_drm.c` | wl_drm global impl | +| `wayland/src/wlr_linux_dmabuf_v1.c` | zwp_linux_dmabuf_v1 global impl | +| `wayland/src/drm-protocol.c` | DRM protocol helpers | +| `wayland/src/linux-dmabuf-unstable-v1-protocol.c` | linux-dmabuf-v1 protocol stubs | +| `wayland/src/wayland-server/` | Vendored + patched libwayland-server (with changes.diff) | +| `encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder | +| `encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding | +| `encoding/src/node_encoder.c` | Node-API addon: proxy-encoding-addon.node | +| `poll/src/poll.c` | libuv uv_poll_t wrapper | +| `node-api/` | Vendored Node-API headers | + +### XWayland support (`libs/xtsb/`) + +| File | Role | +|---|---| +| `README.md` | Project intro | +| `package.json` | @gfld/xtsb package config | +| `protocol/xproto.xml` | X11 core protocol XML (5800+ lines) | +| `protocol/shape.xml` | Shape extension XML | +| `protocol/xfixes.xml` | XFIXES extension XML | +| `protocol/render.xml` | Render extension XML | +| `protocol/composite.xml` | Composite extension XML | +| `src/` | Generated TS bindings | + +## Next actions for user (Odoo 20 integration) + +1. Build the proxy on a Linux host (or use `udevbe/compositor-proxy-cli` + Docker image). +2. Write an `applications.json` mapping desired apps to HTTP paths. +3. Start proxy with `--allow-origin=https://your-odoo.example.com`. +4. Embed the Greenfield `compositor-shell` Preact SPA in Odoo (iframe or + QWeb template) — OR replace it with your Odoo 20 OWL shell (see + `10-porting-to-odoo-js-python.md`). +5. Trigger app launches from Odoo via + `fetch('http://:8081/', { headers: { 'X-Compositor-Session-Id': sessionId }})` + — Odoo frontend JS uses the returned `signalURL` to open the per-app + control WS. +6. For multi-tenant isolation: one `SessionProcess` per browser session is + automatic, but file-system isolation between tenants requires separate + proxy instances or containers per tenant. +7. For GPU acceleration: pass `--device /dev/dri:/dev/dri` to Docker; use + `--encoder=nvh264|vaapih264|x264` based on GPU. diff --git a/08-app-lifecycle.md b/08-app-lifecycle.md new file mode 100644 index 0000000..e5fd0aa --- /dev/null +++ b/08-app-lifecycle.md @@ -0,0 +1,304 @@ +# 08 — App Lifecycle (XDG Shell, Toplevels, Popups, Subsurfaces) + +This chapter covers Greenfield's application lifecycle layer: how xdg-shell +(toplevels, popups, positioners), wl_shell (legacy), and subsurface +compositing are implemented, and how window state transitions are handled. +EXCLUDE the actual UI shell (`packages/compositor-shell`) — focus on the +protocol-level lifecycle. + +## Three role families, one shared abstraction + +All surface roles implement `SurfaceRole` (`{ view, optional +desktopSurface, onCommit(surface) }`). `XdgToplevel`, `XdgPopup`, and +`ShellSurface` additionally implement `DesktopSurfaceRole`, which bridges +them to a `DesktopSurface` (`FloatingDesktopSurface` or +`AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`). The +`DesktopSurface` is where **every** `userShell.events.*` emission +originates — making it the true integration boundary. + +## XdgToplevel lifecycle — the headline diagram + +``` + ┌───────────────────────────────────────────────┐ + │ wl_surface created (no role, no buffer yet) │ + └─────────────────────┬─────────────────────────┘ + │ client: xdg_wm_base.get_xdg_surface + │ (errors if surface already has buffer) + ▼ + ┌───────────────────────────────────────────────┐ + │ XdgSurface created │ + │ • surface.hasKeyboard/Pointer/Touch = true │ + │ • configured=false, configureList=[] │ + └─────────────────────┬─────────────────────────┘ + │ client: xdg_surface.get_toplevel + │ (errors if surface has another role) + ▼ + ┌───────────────────────────────────────────────┐ + │ XdgToplevel created │ + │ • surface.role = xdgToplevel │ + │ • View created │ + │ • DesktopSurface created + init() │ + │ • ackConfigure override installed │ + └─────────────────────┬─────────────────────────┘ + │ client: xdg_toplevel.set_title / + │ set_app_id / set_min/max_size / + │ set_parent (stored, propagated) + ▼ + ┌───────────────────────────────────────────────┐ + │ First wl_surface.commit (buffer === undefined) │ + │ XdgToplevel.onCommit() → ensureAdded() │ + │ • desktopSurface.add() │ + │ → userShell.events.surfaceCreated(...) │ + │ • scheduleConfigure(first) │ + │ → microtask → sendConfigure() │ + │ → resource.configure(w,h,states) │ + │ • added=true │ + └─────────────────────┬─────────────────────────┘ + │ client receives configure + │ client: xdg_surface.ack_configure(serial) + ▼ + ┌───────────────────────────────────────────────┐ + │ ackConfigure(serial, configure) │ + │ • handleAckConfigure trims older serials │ + │ • configured=true │ + │ • next.state = configure.state │ + │ • next.size = configure.size │ + └─────────────────────┬─────────────────────────┘ + │ client: wl_surface.commit (with buffer) + ▼ + ┌───────────────────────────────────────────────┐ + │ ★ MAPPED (steady state) ★ │ + │ onCommit(): │ + │ • surface.commitPending() (state latch) │ + │ • xdgSurface.commit() (geometry latch) │ + │ • current.state ← next.state │ + │ • current.min/max ← next.min/max │ + │ • desktopSurface.commit() → render() │ + └─────────────────────┬─────────────────────────┘ + │ + ┌────────────────────────────────────────┼────────────────────────────┐ + │ │ │ + user/shell requests client requests destruction + │ │ │ + ▼ ▼ ▼ + ┌──────────────────────────┐ ┌─────────────────────────────┐ ┌──────────────────────┐ + │ configureMaximized(b) │ │ setMaximized / unsetMaximized│ │ wl_surface.destroy │ + │ configureFullscreen(b) │ │ setFullscreen/unsetFullscreen│ │ or xdg_toplevel. │ + │ configureActivated(b) │ │ setMinimized │ │ destroy │ + │ configureResizing(b) │ │ setParent / setTitle / │ ├──────────────────────┤ + │ configureSize(w,h) │ │ setAppId │ │ destroy listener: │ + └────────────┬─────────────┘ │ move / resize / showWindowMenu│ │ • surface.unmap() │ + │ └──────────────┬──────────────┘ │ • desktopSurface. │ + │ each mutates │ │ removed() │ + │ pending.state / pending.size │ │ → surfaceDestroyed │ + ▼ │ │ • view.destroy() │ + ┌──────────────────────────┐ │ └──────────────────────┘ + │ scheduleConfigure( │ │ + │ stateCompareEquals(), │ ┌─────────────────────────────┐ + │ sendConfigure) │ │ setMinimized → │ + │ • coalesce via µtask │ │ desktopSurface.minimize() │ + │ • if pending==current → │ │ • view.mapped=false │ + │ skip │ │ • seat drops keyboard focus │ + │ • else: push serial, │ │ if this was the focused │ + │ emit configure event │ │ surface │ + │ │ └─────────────────────────────┘ + └────────────┬─────────────┘ + │ + │ cycle repeats: ack_configure → commit → render + └────────────────────────────────────► back to ★ MAPPED ★ +``` + +## Configure / ack-configure round-trip + +This is the **most important mechanism** to understand in xdg-shell. It's +an asynchronous protocol-level sync between the compositor and the +client. + +### Why it's needed + +The compositor wants to tell the client "please resize to (800, 600) +because the user maximized you". But the client may not be ready to +resize, or may want a different size (e.g. the app enforces a 16:9 aspect +ratio). The configure/ack round-trip lets the client acknowledge the +configure with a serial, then on the next commit, the compositor knows +which configure the client has applied. + +### How Greenfield implements it + +1. Compositor mutates `pending.{state,size}`. +2. Coalesces via `queueCancellableMicrotask` with a `stateCompareEquals` + short-circuit (if pending == current, skip). +3. Emits `resource.configure(w,h,states)` with a fresh serial. +4. Client acks: `xdg_surface.ack_configure(serial)`. +5. `handleAckConfigure` trims older serials, sets `configured=true`, + copies `configure.{state,size}` into `next`. +6. Next commit latches `next → current`. + +The `configured` flag gates buffer attachment: an `unconfiguredBuffer` +error is raised if the client tries to attach a buffer before acking. + +## XdgPositioner math + +`anchor` selects a point on the anchor rect (in parent window-geometry +space); `gravity` selects which corner/edge of the popup is placed at +that point; `offset` is applied last. + +### Constraint adjustment cascade + +Runs as X-then-Y cascade: + +1. Try **flip** — swap anchor+gravity via `inverseX` / `inverseY` tables, + revert if still violating. +2. Try **slide** — shift offset by violation delta. +3. Try **resize** — shrink `sizeRect`. + +Slides/flips are skipped if popup size on that axis exceeds viewport. + +## Popup grabs + +`XdgPopup.grab` rejects if: + +- Surface is already mapped, OR +- Parent isn't either: + - a toplevel (root of new chain), OR + - the topmost popup of an existing grab (`notTheTopmostPopup` error) + +`seat.popupGrabStart` validates the serial against the keyboard/pointer +`grabSerial`, then starts a `PopupGrab` on both devices. `popupGrabEnd` +walks the chain in reverse calling `requestClose()` on each (sends +`popup_done` / `xdg_popup.popup_done`). + +## Subsurfaces sync/desync + +Three-layer mechanism: + +1. **`cachedState`** — full pending-state shadow on the subsurface role. +2. **`commitCache`** — flush on parent commit. +3. **`effectiveSync`** getter — cascades: if any ancestor is sync, this + subsurface is effectively sync. + +### Decision matrix + +| Sync mode | Dirty | Action | +|---|---|---| +| Sync | yes | Cache to `cachedState`, flush when parent commits | +| Sync | no | No-op (already cached) | +| Desync | yes | Flush immediately to parent's pending state | +| Desync | no | Acts like a normal surface (commit directly to its own pendingState) | + +`setDesync` immediately calls `onCommit` to flush any pending cache. + +## wl_shell (legacy path) + +Legacy path. Reduced feature set: + +- `configureActivated` / `configureResizing` are explicit no-ops. +- State machine via `SurfaceStates` enum (MAXIMIZED / FULLSCREEN / POPUP / + TRANSIENT / TOP_LEVEL) and `changeState()`. +- Per-surface ping (separate from `XdgWmBase`'s per-client ping). + +## The UserShellApi seam — critical for the Odoo 20 OWL shell + +Located in `UserShellApi.ts`, exposed as `session.userShell: UserShellApi = +{ events, actions }`. The default `createUserShellApi(session)` returns +`events: {}` (empty) and a populated `actions` object. **The shell +assigns the event callbacks; the compositor always invokes them with +`?.()` so unhandled events are silent no-ops.** This is the single +integration seam — the protocol layer above, and the shell UI below. + +### Events (compositor → shell) — all optional + +| Event | Payload | Fired from | When | +|---|---|---|---| +| `clientCreated` | `CompositorClient` | `WebConnectionHandler`, `RemoteAppLauncher` | A new app client connects. | +| `clientDestroyed` | `CompositorClient` | same | A client disconnects. | +| `clientUnresponsiveUpdated` | `(client, unresponsive: boolean)` | `XdgWmBase`, `ShellSurface` | Ping (2s timeout, every 5s) — `true` = hung, `false` = recovered. | +| `surfaceCreated` | `CompositorSurface` | `FloatingDesktopSurface.add()`, `AlwaysFullScreenDesktopSurface` | Toplevel first added to desktop. | +| `surfaceDestroyed` | `CompositorSurface` | `.removed()` | Window destroyed. | +| `surfaceTitleUpdated` | `(surface, title)` | `.setTitle()` | `xdg_toplevel.set_title` / `wl_shell_surface.set_title`. | +| `surfaceAppIdUpdated` | `(surface, appId)` | `.setAppId()` | `xdg_toplevel.set_app_id` / `wl_shell_surface.set_class`. | +| `surfaceActivationUpdated` | `(surface, active: boolean)` | `gainFocus()` / `loseFocus()` | Refcounted via `focusCount` — only fires on 0↔1 transitions. | +| `notify` | `(variant, message)` | (declared, no direct emissions) | Free-form notification channel. | +| `sceneRefreshed` | `sceneId: string` | `Scene:75` | Render finished. | + +### Actions (shell → compositor) — provided by default + +| Action | Effect | +|---|---| +| `initScene(canvasCreator)` | Bootstraps a WebGL canvas + input/output. One per output. | +| `refreshScene()` | `session.renderer.render()` — manual redraw. | +| `destroyScene(sceneId)` | Destroys a specific scene. | +| `setUserConfiguration(config)` | Updates `scrollFactor`, `keyboardLayoutName`. | +| `closeClient({ id })` | Closes a client connection. | +| `activateSurface(compositorSurface)` | **Click-to-focus entry point from the shell's taskbar.** Reverse-lookups the surface via `session.display.clients[clientId].connection.wlObjects[surfaceId]` and calls `desktopSurface.activate()`. | + +### Opaque handle types + +``` +interface CompositorSurface { id: number; client: CompositorClient } +interface CompositorClient { id: string } +``` + +`toCompositorSurface(desktopSurface)` adapts `{ surface.resource.id, +surface.resource.client.id }`; `lookupSurface(session, compositorSurface)` +is the reverse walk. + +## Notable gaps for the Odoo OWL shell + +1. **No maximize/fullscreen/resize/minimize events** in + `UserShellApiEvents`. The shell can't directly observe window-state + transitions through this API — only focus (`surfaceActivationUpdated`) + and metadata (title/appId). For a taskbar this is fine; for + window-decorations rendering it is not. Options: (a) extend + `UserShellApiEvents` with `surfaceStateChanged`; (b) the OWL shell + imports `DesktopSurface` directly and introspects; (c) add a + `querySurfaceState` action. +2. **No geometry/position event** — the shell can't render a frame that + hugs surface bounds from the shell side without taking over rendering. +3. **`notify` declared but unused** in compositor core — dead interface + slot. +4. **`surfaceCreated` fires once on first commit; no separate map/unmap + signal** — minimization is internal (`view.mapped=false`) and not + propagated. + +## Risks flagged + +- `XdgWmBase.doPing` uses serial 0 (`// FIXME use a proper serial`) — works + only because there's a single outstanding ping per client. +- `XdgToplevel.onCommit` has commented-out buffer-size-vs-configured-state + validation (lines 140–158) — the compositor trusts clients to honor + configured sizes. +- `XdgSurface.scheduleConfigure`'s `pendingSame` argument inverts meaning + depending on whether a microtask is already queued — subtle and easy to + break in maintenance. + +## File reference + +| File | Role | +|---|---| +| `packages/compositor/src/Shell.ts` | `wl_shell` global (legacy) | +| `packages/compositor/src/ShellSurface.ts` | `wl_shell_surface` role | +| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global | +| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode | +| `packages/compositor/src/SurfaceRole.ts` | SurfaceRole interface | +| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction | +| `packages/compositor/src/UserShellApi.ts` | The integration seam — events + actions | +| `packages/compositor/src/XdgWmBase.ts` | `xdg_wm_base` global, ping-based unresponsiveness detection | +| `packages/compositor/src/XdgSurface.ts` | `xdg_surface` role, configure/ack round-trip | +| `packages/compositor/src/XdgToplevel.ts` | `xdg_toplevel` role, window state | +| `packages/compositor/src/XdgPopup.ts` | `xdg_popup` role, transient windows, grabs | +| `packages/compositor/src/XdgPositioner.ts` | `xdg_positioner`, popup positioning math | +| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes | + +## Next actions (for Odoo 20 OWL shell) + +1. Design `createOdooUserShellApi(session)` returning the default + `actions` and OWL-signal-bound callbacks for the 8 events. +2. Decide how to surface maximize/fullscreen/resize/minimize state (extend + events vs. introspect `DesktopSurface`). +3. Wire `clientCreated` / `clientDestroyed` to an Odoo `AppContext`-style + registry for the app launcher UI. +4. Feed `surfaceActivationUpdated` into an OWL reactive store as the + single source of truth for active-window focus. +5. `initScene(canvasCreator)` — OWL shell owns the `HTMLCanvasElement` + lifecycle, one per output. diff --git a/09-hardware-accel.md b/09-hardware-accel.md new file mode 100644 index 0000000..7059033 --- /dev/null +++ b/09-hardware-accel.md @@ -0,0 +1,423 @@ +# 09 — Hardware Acceleration, Encoding, WASM, GPU Textures + +This chapter covers Greenfield's hardware acceleration / encoding / WASM +pipeline: how native buffers get encoded on the proxy side, transported, +decoded in the browser, and turned into WebGL textures or canvas images. + +## What Greenfield actually ships + +### Two Wasm libraries under `libs/` + +**`compositor-wasm`** builds **pixman 0.42.2 + xkbcommon 1.5.0** +(Emscripten 3.1.46, `-msimd128 -O3 -flto`). Used by the browser for +**region math** (damage tracking) and **keyboard layout** — *not* for +pixel format conversion or H.264 decode. + +**`compositor-ffmpeg-h264`** is a **stripped-down FFmpeg n5.1.2** +(`--disable-everything --enable-decoder=h264`) compiled to a 1.7 MB +single-file Wasm worker. A 30-line `decoder.c` wrapper exposes +`create_codec_context` / `decode` / `close_frame` / `destroy_codec_context`. +Decodes to **I420 (YUV 4:2:0 planar)** and returns plane pointers; planes +are `postMessage`'d back to the main thread as **transferable +`ArrayBuffer`s**. No threads (single-threaded, SIMD only). + +### Native encoder addon (`packages/compositor-proxy/native/encoding/`) + +A **GStreamer wrapper** exposed via Node-API. The TS side (`Encoder.ts`) +calls `appEndpointNative.encodeFrame(encoder, bufferId, contentSerial, +creationSerial)`. The C side (`node_encoder.c`) sniffs the wl_resource: + +- `zwp_linux_dmabuf_v1` → DMA path +- Legacy `wl_drm` → DMA path +- `wl_shm_buffer` → SHM path + +For DMA-BUFs the proxy wraps the dma-buf as an `EGLImage` and then as a +`GstGLMemoryEGL`, so pixels **stay on GPU** through the color-convert +stages. + +### Threading + +Encoding runs on a dedicated GLib main-loop thread fed by an +`eventfd`-signalled `GAsyncQueue`. Output returns to Node via a +`napi_threadsafe_function` with **zero-copy** external Buffers (finalised +back to C on GC). + +## Encoder selection (per-frame, deterministic) + +`gst_frame_encoder.c::frame_encoder_descriptions[]` declares **four** +pipelines: + +| name | element | when used | split_alpha | coded-size multiple | +|---|---|---|---|---| +| `x264` | `x264enc` (software, default) | >256×256 + ARGB/XRGB/dmabuf | true (parallel alpha H.264 stream) | 128 | +| `nvh264` | `nvh264enc` (Nvidia NVENC) | same, `--encoder=nvh264` | true | 128 | +| `vaapih264` | `vaapih264enc` (Intel/AMD VA-API, "experimental") | same, `--encoder=vaapih264` | true | 128 | +| `png` | `pngenc` (fallback) | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 | + +**Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264 +encoder. YUV420/NV12/NV21/Y444 SHM formats are *not* supported on the +H.264 path and only by PNG at small sizes — effectively unsupported for +video-class content. + +### Canonical H.264 pipeline + +``` +appsrc → glupload → glcolorconvert → glshader(scale→coded_size) → +glcolorconvert → video/x-raw(memory:GLMemory),format=NV12 → gldownload → +queue → x264enc tune=zerolatency bitrate=12000 vbv-buf-capacity=160 + bframes=0 rc-lookahead=0 sliced-threads byte-stream=true → +video/x-h264,profile=high,stream-format=byte-stream,alignment=au → appsink +``` + +The `glshader` element does two jobs: + +1. Pad up to the encoder's macroblock multiple (128 for H.264, 16 for PNG) + using a `u_transformation` matrix. +2. For the **alpha pipeline**, run `vec4(pix.a, pix.a, pix.a, 0.)` so + alpha is encoded as a grayscale H.264 stream — H.264 has no alpha, so + Greenfield ships two parallel AU streams. + +Bitrates: opaque 12000 / alpha 1200 (x264), 12000 / 1200 (nvh264). + +## Wire format + +Single contiguous blob, little-endian: + +``` +struct packed props { + u32 buffer_id, creation_serial, content_serial, + u32 frame_encoding_type (0=h264, 1=png), + u32 width, height, coded_width, coded_height +} +u32 opaque_length +u8 opaque[opaque_length] // H.264 AU or PNG blob +u32 alpha_length // 0 if no alpha split +u8 alpha[alpha_length] // H.264 AU of alpha-only stream +``` + +Parsed by `remote/EncodedFrame.ts::createEncodedFrame`. + +## Transport (`Channel.ts`) + +Four logical channels per client multiplexed over WebSockets: + +| Channel | Type | Purpose | +|---|---|---| +| PROTOCOL | ARQ (KCP) | Wayland wire protocol + out-of-band ops | +| FRAME | ARQ (KCP) | Encoded surface frames | +| XWM | ARQ (KCP) | XWayland WM messages | +| FEEDBACK | SIMPLE (raw WS) | 4-byte feedback messages | + +ARQ = a JS KCP implementation layered over WebSocket. Tuning: +`MTU=64000`, `MAX_BUFFERED_AMOUNT=2949120` (~2.8 MB — the back-pressure +ceiling), `SND_WINDOW=128`, `RCV_WINDOW=1024`, +`setNoDelay(1, 100, 0, 1)`. + +## Browser decode (`packages/compositor/src/remote/`) + +At session startup `Session.create()` probes +`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true, +hardwareAcceleration:'prefer-software'})`: + +### WebCodecs preferred + +Software-only; the `prefer-hardware` path is implemented but commented out +with `// FIXME hardware decoding frame offset is wrong`. + +### FFmpeg Wasm fallback + +If WebCodecs unavailable: two persistent Workers (`h264NALDecoderOpaque`, +`h264NALDecoderAlpha`), one `AVCodecContext` per surface per worker, +keyed by `surfaceH264DecodeId`. Encoded NALs are transferred zero-copy +into the worker; decoded I420 planes are transferred back. + +Per-surface decoder context is lazily created on first H.264 frame. + +### PNG path (both decoders) + +Identical in both branches: `new Blob([opaque], {type:'image/png'})` → +`createImageBitmap(blob, sx, sy, w, h)` (the `sx/sy` crops the encoder +padding back to logical size). + +## WebGL upload (`render/`) + +WebGL 1 with `desynchronized:true` for low-latency presentation. Per Scene +(canvas/output), three GL helpers: + +### `YUVA2RGBA` + +FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures**, renders into +the surface's RGBA texture via `YUVA2RGBAShader` (BT.601 coefficients, +alpha recovered by running the alpha-Y plane through the same YUV→RGB +matrix and taking R). + +### `RGBXandA2RGBA` + +FBO pass for the `DualPlaneRGBA*` WebCodecs outputs: opaque + alpha planes +composited via `RGBandA2RGBAShader`. + +### `SceneShader` + +Final pass: draws every view's RGBA texture to the canvas with +`gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)` (note: expects premultiplied +alpha but the YUVA shader outputs straight alpha — a minor inconsistency). + +PNG and web-bitmapbuf paths skip the FBO and call +`texImage2D(ImageBitmap)` directly. + +### Alpha pre-multiplication + +Not done explicitly anywhere. The proxy extracts alpha as a Y plane; the +browser shader recombines; the canvas blend mode assumes premultiplied but +receives straight alpha — works acceptably because Wayland surfaces rarely +use partial alpha. + +## End-to-end pipeline + +``` +┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐ +│ App renders via GTK/Qt/raw EGL into │ +│ ┌────────────────────────────────────┐ │ +│ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │ +│ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │ +│ │ wl_drm create_prime_buffer │ │ │ +│ └────────────────────────────────────┘ │ │ +└────────────────────────┬──────────────────────────────────┼────┘ + │ Wayland socket (UNIX domain) │ + ▼ │ +┌────────────────────── COMPOSITOR PROXY (Node + N-API addons)─┐ +│ wayland-server-addon.node │ +│ └─ wl_display → wl_client → wl_resource (buffer) │ +│ └─ initShm(), initDrm("/dev/dri/renderD128") │ +│ └─ westfield-egl.c: EGLDevice + EGLDisplay + EGLContext │ +│ │ +│ proxy-encoding-addon.node (per-surface Encoder) │ +│ ┌─ node_encoder.c → frame_buffer {type: SHM|DMA, w, h, fmt} │ +│ │ │ +│ │ gst_main_loop.c (own GLib thread, eventfd work queue) │ +│ │ │ │ +│ │ ▼ │ +│ │ gst_frame_encoder.c │ +│ │ ┌─ Encoder selection (per-frame): │ +│ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │ +│ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │ +│ │ │ │ +│ │ │ H.264 pipelines (parallel): │ +│ │ │ opaque: appsrc → glupload → glcolorconvert → │ +│ │ │ glshader(scale→coded_size) → │ +│ │ │ glcolorconvert(NV12) → gldownload → │ +│ │ │ queue → x264enc|nvh264enc|vaapih264enc → │ +│ │ │ byte-stream AU → appsink │ +│ │ │ alpha: same source, fragment shader │ +│ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │ +│ │ │ second H.264 stream │ +│ │ │ │ +│ │ │ PNG pipeline (single): │ +│ │ │ appsrc → glupload → glcolorconvert → glshader → │ +│ │ │ RGBA → gldownload → pngenc → appsink │ +│ │ │ │ +│ │ └─ frame_encoding_result_to_encoded_frame(): │ +│ │ packed struct {buffer_id, creation_serial, │ +│ │ content_serial, type(0=h264,1=png), w, h, │ +│ │ coded_w, coded_h} + opaque_len + opaque + │ +│ │ alpha_len + alpha → Node Buffer (zero-copy ext.) │ +│ │ │ +│ └─ SurfaceBufferEncoding.ts │ +│ └─ frameDataChannel.send(buffer) │ +│ │ +│ Channel.ts: │ +│ ARQChannel (KCP-reliable over WebSocket, MTU 64 KB) │ +│ SimpleChannel (raw WebSocket, for FEEDBACK only) │ +│ ▲ │ +│ │ FrameFeedback.ts: 16-bit rAF interval + 16-bit avg │ +│ │ decode duration → used only to pace wl_frame.done │ +└───────┼───────────────────────────────────────────────────────┘ + │ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK + ▼ +┌────────────────────── BROWSER (compositor) ────────────────────┐ +│ WebSocket → ReconnectingWebSocket │ +│ ↓ │ +│ Channel.ts: ARQChannel.input(kcp) → onMessage │ +│ ↓ │ +│ RemoteAppLauncher.setupFrameDataChannel → │ +│ BufferStream.deliverContentToBufferStream (parse serials) │ +│ ↓ │ +│ StreamingBuffer.getContents(surface, serial) │ +│ ↓ │ +│ Session.frameDecoder.decode(surface, encodedFrame) │ +│ │ │ +│ ├─ PNG path (both decoders): │ +│ │ new Blob([opaque], {type:'image/png'}) → │ +│ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │ +│ │ → SinglePlane { bitmap: ImageBitmap } │ +│ │ │ +│ └─ H.264 path: │ +│ (preferred) WebCodecs VideoDecoder │ +│ config: avc1.64001f, prefer-software, optimizeForLatency│ +│ opaqueDecoder + alphaDecoder per surface │ +│ EncodedVideoChunk {type:key|delta, data:nal} │ +│ → VideoFrame.output → │ +│ if I420: copyTo → DualPlaneYUVAArrayBuffer │ +│ else: keep VideoFrame → DualPlaneRGBAVideoFrame │ +│ (fallback) FFmpeg Wasm in 2 Web Workers │ +│ worker.postMessage(nal, [nal.buffer]) // transferable │ +│ libavH264._decode → I420 planes │ +│ postMessage back with [y.buffer, u.buffer, v.buffer] │ +│ → DualPlaneYUVASplitBuffer │ +│ │ +│ Renderer.render() (rAF-driven, queueCancellableMicrotask) │ +│ ↓ │ +│ Scene.render(viewStack) │ +│ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │ +│ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │ +│ └─ Scene['video/h264']: │ +│ YUVA2RGBA.convertYUVASplitBufferInto / │ +│ convertYUVAArrayBufferInto: │ +│ Y/U/V/alphaY → 4× LUMINANCE textures │ +│ bind FBO → renderState.texture (RGBA) as color attach │ +│ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │ +│ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │ +│ VideoFrame/ImageBitmap → texImage2D directly │ +│ ↓ │ +│ SceneShader.draw(): drawArrays with surface's RGBA texture, │ +│ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │ +│ ↓ │ +│ Browser composites canvas into the page │ +└─────────────────────────────────────────────────────────────────┘ + + +Same-frame web app fast path (web-bitmapbuf): +┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐ +│ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │ +│ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │ +└───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘ + (no encoding, no WebSocket, no GStreamer) +``` + +## Key answers + +### Why H.264 over PNG? + +H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB. +PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split +doubling dominate at small sizes. + +### WebCodecs? + +Yes — preferred path, software-only (`prefer-hardware` is wired but +disabled with FIXME "frame offset is wrong"). FFmpeg Wasm is the fallback. + +### Trade-offs + +- H.264 = small wire / decode cost +- PNG = medium wire / browser-native fast decode (`createImageBitmap`) +- raw SHM = **not used** between proxy and browser — every surface commit + is encoded. + +### Per-surface policy? + +No — per-frame deterministic. Only configurable knob is +`--encoder=x264|nvh264|vaapih264`. PNG is always automatic fallback. + +### Cursors? + +Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS +cursor via `setBrowserCursor` — never enters the WebGL compositor. + +### Alpha pre-multiplication? + +Implicit in shaders. Proxy extracts alpha as a Y plane (no premult); +browser recombines via BT.601 YUVA→RGBA shader outputting straight alpha; +canvas blend is `ONE, ONE_MINUS_SRC_ALPHA` (expects premult) — minor +inconsistency, not visually problematic for typical Wayland content. + +### OffscreenCanvas / Web Workers? + +Workers yes (two for Wasm decoder, shared across surfaces). OffscreenCanvas +**no** — renderer is main-thread WebGL with `desynchronized:true`. +WebGPU/WebGL2 not used. + +## Notable design trade-offs for your compositor + +1. Always-encode policy (no raw SHM escape hatch even on LAN). +2. One encoder per surface, EOS-and-recreate on resize/format change. +3. Alpha as a separate H.264 stream — doubles cost for transparent windows. +4. Per-surface decoder context, no decoder pooling. +5. Only aggregate feedback (1 Hz rAF interval + decode duration); no + per-frame ACKs, no bitrate adaptation; `requestKeyUnit` plumbing exists + end-to-end but the browser never calls it. +6. KCP over WebSocket (not WebRTC datachannels) — simpler infra, no + STUN/TURN, but no real congestion control. +7. Emscripten SIMD (not threads) — avoids COOP/COEP requirements. +8. Two parallel GL pipelines (opaque + alpha) for H.264. +9. PNG fallback boundary empirically at 256×256 pixels. +10. WebGL 1, not 2/WebGPU — maximum browser compat at the cost of manual + LUMINANCE-texture YUV juggling and a hidden FBO intermediate. + +## File reference + +### Wasm libraries + +| Path | Role | +|---|---| +| `libs/compositor-wasm/build_wasm.sh` | Builds pixman 0.42.2 + xkbcommon 1.5.0 to WASM (emsdk 3.1.46) | +| `libs/compositor-wasm/emscripten-build.ini` | Emscripten build config | +| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function | +| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon build (clones 1.5.0, embeds xkeyboard-config DB) | +| `libs/compositor-wasm/pixman/build.sh` | pixman build | +| `libs/compositor-ffmpeg-h264/build_wasm.sh` | Builds FFmpeg n5.1.2 to WASM (single-file, decoder-only) | +| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API | +| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module | +| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point | +| `libs/compositor-ffmpeg-h264/src/index.ts` | Exports | + +### Native encoder addon + +| Path | Role | +|---|---| +| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon (290 lines) | +| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder (~1400 lines) | +| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding | +| `packages/compositor-proxy/native/CMakeLists.txt` | Build config | +| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | TS-side encoding orchestration | +| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation | +| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback | +| `packages/compositor-proxy/src/encoding/` | High-level encoding files | + +### Browser decode + +| Path | Role | +|---|---| +| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames | +| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs VideoDecoder path | +| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path | +| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder | +| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz | +| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer | +| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution | + +### WebGL rendering + +| Path | Role | +|---|---| +| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline | +| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures | +| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas | +| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper | +| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform | +| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass | +| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader | +| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) | +| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) | +| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above | +| `packages/compositor/src/render/Program.ts` | WebGL program wrapper | +| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper | +| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings | + +### Protocol extensions + +| Path | Role | +|---|---| +| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom Wayland extension for browser JS apps | +| `protocol/wayland-drm.xml` | Standard wl_drm for DMA-BUF (proxy-only) | +| `packages/compositor/src/web/WebBuffersFactory.ts` | Browser-side impl of web_bitmapbuf_factory | diff --git a/10-porting-to-odoo-js-python.md b/10-porting-to-odoo-js-python.md new file mode 100644 index 0000000..6ca10e3 --- /dev/null +++ b/10-porting-to-odoo-js-python.md @@ -0,0 +1,644 @@ +# 10 — Porting Blueprint (JS Browser + Python Host + Odoo 20 OWL Shell) + +This is the standalone porting chapter. It maps every Greenfield layer +analyzed in chapters 02-09 to concrete guidance for building a +Wayland-to-HTML compositor that integrates with an Odoo 20 addon shell and +a JS + Python backend. + +## Target architecture + +The recommended architecture: + +``` +┌──────────────────────────────────────────────────────────────────────────────┐ +│ BROWSER (Odoo 20 web client) │ +│ │ +│ ┌──────────────────────────────────────────────────────────────────────────┐ │ +│ │ Odoo 20 OWL Shell addon (your code) │ │ +│ │ - Application launcher (lists apps from /api/compositor/apps) │ │ +│ │ - Window management UI (taskbar, decorations) │ │ +│ │ - Keyboard layout picker (RMLVO) │ │ +│ │ - Per-output element │ │ +│ │ - Triggers app launches via Odoo /api/compositor/launch │ │ +│ │ - Implements UserShellApi.events.* callbacks │ │ +│ └──────────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ userShell.events + userShell.actions │ +│ ▼ │ +│ ┌──────────────────────────────────────────────────────────────────────────┐ │ +│ │ @gfld/compositor (browser-side compositor library) │ │ +│ │ - Use as-is from npm: @gfld/compositor │ │ +│ │ - Replaces compositor-shell with createOdooUserShellApi(session) │ │ +│ │ - Renders surfaces to via WebGL │ │ +│ │ - Manages keyboard/pointer/touch │ │ +│ └──────────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────────┘ + │ + │ WebSocket (TCP, KCP-over-WS) + │ 9 endpoint paths per session + ▼ +┌──────────────────────────────────────────────────────────────────────────────┐ +│ PYTHON HOST (Linux server) │ +│ │ +│ ┌──────────────────────────────────────────────────────────────────────────┐ │ +│ │ Odoo 20 backend addon (your code) │ │ +│ │ - /api/compositor/apps → list of available apps │ │ +│ │ - /api/compositor/launch → triggers proxy to spawn app │ │ +│ │ - Session management (one Odoo session ↔ one proxy session) │ │ +│ │ - Per-tenant isolation, ACL, audit log │ │ +│ └──────────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ spawn / IPC │ +│ ▼ │ +│ ┌──────────────────────────────────────────────────────────────────────────┐ │ +│ │ Python compositor-proxy (your code — replaces @gfld/compositor-proxy) │ │ +│ │ - aiohttp/FastAPI WebSocket server (replaces ws + ws-server) │ │ +│ │ - pywayland (cffi bindings to libwayland-server) │ │ +│ │ - python-xkbcommon for keymap compilation │ │ +│ │ - GStreamer Python bindings (gi.repository.Gst) for H.264 encoding │ │ +│ │ - asyncio event loop integrating libwayland FD polling │ │ +│ │ - Per-app child_process spawning (subprocess.Popen with env) │ │ +│ │ - Sandboxing via bubblewrap + seccomp (NEW — Greenfield has none) │ │ +│ └──────────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ WAYLAND_DISPLAY=wayland-N │ +│ ▼ │ +│ ┌──────────────────────────────────────────────────────────────────────────┐ │ +│ │ Native Linux apps (gtk4-demo, xterm, kwrite, ...) │ │ +│ │ - Use host system libraries (no sysroot) │ │ +│ │ - Sandboxed via bubblewrap + seccomp │ │ +│ └──────────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────────┘ +``` + +## Per-layer porting guidance + +For each Greenfield layer analyzed in chapters 02-09: + +### Layer 1: Wayland Protocol XML + +**What to copy as-is**: All four protocol XML files (`wayland.xml`, +`xdg-shell.xml`, `wayland-drm.xml`, `web-bitmapbuf-unstable-v1.xml`). +They are upstream standard + Greenfield's custom extension. No need to +rewrite. + +**What to replace**: Nothing. The XML is the source of truth. + +**Where Python fits**: Python doesn't need to parse the XML directly — +use `pywayland` which already ships with parsed bindings for the standard +protocols. For Greenfield's custom `web-bitmapbuf-unstable-v1`, you can +either skip it (native apps don't use it) or generate Python bindings via +`pywayland-protocol-scanner`. + +**Where the Odoo 20 OWL shell hooks in**: Not applicable. This layer is +below the shell. + +### Layer 2: Protocol Generators + +**What to copy as-is**: The TypeScript generators for the browser side +(`client-generator`, `compositor-generator`). They produce stubs for the +browser compositor and browser-resident JS apps. + +**What to replace**: The `compositor-proxy-generator`. Greenfield's proxy +generates CommonJS `*_interface.js` + `*_interceptor.js` files for the +Node N-API addon. For Python, you don't need a generator — use +`pywayland`'s built-in protocol bindings. + +**Where Python fits**: `pywayland` provides `wl_interface` structs and +request/event dispatch in pure Python via cffi. You write Python classes +that implement `WlCompositorRequests`, `WlSeatRequests`, etc. + +**Where the Odoo 20 OWL shell hooks in**: Not applicable. + +### Layer 3: Wayland Runtime + +**What to copy as-is**: `@gfld/common` (`Connection.ts`) and +`@gfld/compositor-protocol` for the browser side. These implement the +standard Wayland wire format and object lifecycle, and they're MIT-licensed. + +**What to replace**: The proxy's `wayland-server.ts` (which wraps the +`wayland-server-addon.node` N-API addon). Replace with `pywayland`'s +`Display` and `Client` classes. + +**Where Python fits**: `pywayland` provides: +- `pywayland.server.Display` — wraps `wl_display` +- `pywayland.server.Client` — wraps `wl_client` +- `pywayland.server.Resource` — wraps `wl_resource` +- `pywayland.server.Global` — wraps `wl_global` +- Per-resource event/request dispatch via Python method overrides + +You write Python classes that subclass `WlCompositor`, `WlSeat`, etc. — +similar to how Greenfield's TypeScript classes implement `*Requests` +interfaces. + +**Where the Odoo 20 OWL shell hooks in**: Not applicable. The runtime +layer is below the shell. + +### Layer 4: Transport + +**What to copy as-is**: The browser side of `Channel.ts` (`ARQChannel` ++ `SimpleChannel` + `ReconnectingWebSocket`) and `kcp.ts`. The TypeScript +KCP implementation is solid and battle-tested. + +**What to replace**: The native side of `Channel.ts`. Replace Node's `ws` +library with `aiohttp` WebSocket or `FastAPI` WebSocket. + +**Where Python fits**: + +- Use `aiohttp.web.WebSocketResponse` or `starlette.websockets.WebSocket` + for the WebSocket server. +- Port the KCP implementation from TypeScript to Python (or wrap a C + implementation via cffi — `pykcp` exists). +- Use `asyncio` event loop integration: `pywayland` exposes the + libwayland event-loop FD, which you can integrate via + `asyncio.loop.add_reader()`. +- For multiplexing 9 endpoints, use `aiohttp.web.Application` with 9 + routes, or `FastAPI` with 9 WebSocket endpoints. + +**Where the Odoo 20 OWL shell hooks in**: The shell connects to the +proxy via the same WebSocket URLs. No change needed. + +### Layer 5: Rendering Pipeline (browser side) + +**What to copy as-is**: The entire `packages/compositor/src/` rendering +pipeline. This is the heart of Greenfield and it's where most of the +engineering effort went. Reuse it wholesale: +- `Surface.ts`, `View.ts`, `Output.ts`, `Loop.ts`, `Callback.ts`, + `Region.ts`, `Shm.ts`, `BufferContents.ts`, `BufferImplementation.ts`, + `ImageBitmapBuffer.ts` +- `render/` directory (Renderer, Scene, SceneShader, Texture, RenderState, + shaders) +- `remote/` directory (StreamingBuffer, BufferStream, EncodedFrame, + EncoderFeedback, decoders) + +**What to replace**: Nothing. This layer is pure browser TypeScript and +works as-is. + +**Where Python fits**: Not applicable. The browser does all rendering. + +**Where the Odoo 20 OWL shell hooks in**: The shell owns the +`` element and passes it to +`userShell.actions.initScene(canvasCreator)`. The compositor creates a +`Scene` per canvas. + +### Layer 6: Surface Lifecycle / Roles + +**What to copy as-is**: All of `XdgWmBase.ts`, `XdgSurface.ts`, +`XdgToplevel.ts`, `XdgPopup.ts`, `XdgPositioner.ts`, `Shell.ts`, +`ShellSurface.ts`, `Subcompositor.ts`, `Subsurface.ts`, `SurfaceRole.ts`, +`SurfaceChild.ts`. These implement Wayland's role state machines correctly. + +**What to replace**: Nothing. + +**Where Python fits**: Not applicable (this is browser-side). + +**Where the Odoo 20 OWL shell hooks in**: The shell implements +`UserShellApi.events` callbacks (see `08-app-lifecycle.md`). Specifically: +- `surfaceCreated` → add a window to the OWL taskbar +- `surfaceDestroyed` → remove from taskbar +- `surfaceTitleUpdated` → update window title +- `surfaceActivationUpdated` → mark window as active +- `clientCreated` / `clientDestroyed` → update app registry +- `clientUnresponsiveUpdated` → show "not responding" indicator + +### Layer 7: Desktop / Layout Policy + +**What to copy as-is**: `desktop/Desktop.ts` provides Floating and +AlwaysFullScreen modes. Floating is what most users want (windows can be +moved/resized freely). + +**What to replace**: If your Odoo shell needs custom layout policies +(e.g. tiling, tabbed), you'll need to extend `DesktopSurface` with new +modes. But you can start with Floating. + +**Where Python fits**: Not applicable. + +**Where the Odoo 20 OWL shell hooks in**: The shell calls +`userShell.actions.activateSurface(compositorSurface)` when the user +clicks a taskbar entry. + +### Layer 8: User Shell (the integration seam) + +**What to copy as-is**: `UserShellApi.ts`. The interface is clean and +shell-agnostic. + +**What to replace**: Replace `compositor-shell` (Preact) with your Odoo +20 OWL addon. + +**Where Python fits**: Not applicable. The shell is browser-side. + +**Where the Odoo 20 OWL shell hooks in**: This IS the integration point. +Create an OWL component that: + +```typescript +// Pseudocode for Odoo 20 OWL shell +import { Component, useState, onMounted } from "@odoo/owl"; +import { createCompositorSession } from "@gfld/compositor"; + +export class CompositorShell extends Component { + setup() { + this.state = useState({ + surfaces: new Map(), // id → { title, appId, active } + clients: new Map(), + }); + + onMounted(async () => { + const session = createCompositorSession(); + this.session = session; + + // Wire UserShellApi events to OWL reactive state + session.userShell.events.surfaceCreated = (surface) => { + this.state.surfaces.set(surface.id, { + title: "", appId: "", active: false + }); + }; + session.userShell.events.surfaceDestroyed = (surface) => { + this.state.surfaces.delete(surface.id); + }; + session.userShell.events.surfaceTitleUpdated = (surface, title) => { + const s = this.state.surfaces.get(surface.id); + if (s) s.title = title; + }; + session.userShell.events.surfaceActivationUpdated = (surface, active) => { + const s = this.state.surfaces.get(surface.id); + if (s) s.active = active; + }; + // ... etc for clientCreated, clientDestroyed, etc. + + // Initialize the canvas + const canvas = this.canvasRef.el; + session.userShell.actions.initScene(() => ({ + canvas, + width: canvas.clientWidth, + height: canvas.clientHeight, + })); + }); + } + + // User clicks app in launcher + async launchApp(appId) { + const response = await fetch("/web/compositor/launch", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ app_id: appId }), + }); + const { signal_url } = await response.json(); + // Browser JS opens signal_url WebSocket → compositor handles the rest + } + + // User clicks taskbar entry + activateSurface(surfaceId) { + this.session.userShell.actions.activateSurface({ id: surfaceId }); + } +} +``` + +### Odoo 20 backend addon + +The Odoo 20 backend addon acts as a thin proxy between the OWL shell and +the Python compositor-proxy. It handles: + +- App launcher config (per-user, per-tenant allowed apps list) +- Auth (Odoo session → compositor-proxy session mapping) +- Audit log (which user launched which app when) +- Multi-tenant isolation (separate Python proxy processes per tenant, + containerized) +- Per-tenant resource limits (max concurrent apps, max memory) + +```python +# Pseudocode for Odoo 20 backend controller +from odoo import http +from odoo.http import request + +class CompositorController(http.Controller): + + @http.route("/web/compositor/apps", type="json", auth="user") + def list_apps(self): + # Return user-allowed apps from applications.json + user = request.env.user + return request.env["compositor.app"].search_read( + [("allowed_group_ids", "in", user.groups_id.ids)], + ["name", "icon", "app_id"], + ) + + @http.route("/web/compositor/launch", type="json", auth="user") + def launch_app(self, app_id): + user = request.env.user + # Look up the user's compositor-proxy host (per-tenant routing) + proxy_host = user.compositor_proxy_host + proxy_port = user.compositor_proxy_port + + # Tell the Python proxy to spawn the app + # The proxy returns a signal_url + auth key + import aiohttp + async with aiohttp.ClientSession() as sess: + async with sess.post( + f"http://{proxy_host}:{proxy_port}/launch", + json={ + "app_id": app_id, + "odoo_session_id": request.session.sid, + "user_id": user.id, + }, + ) as resp: + result = await resp.json() + + # Audit log + request.env["compositor.audit.log"].create({ + "user_id": user.id, + "app_id": app_id, + "timestamp": fields.Datetime.now(), + }) + + return {"signal_url": result["signal_url"]} +``` + +### Python compositor-proxy (the big new code) + +This is the bulk of the new code. Replace Node's `compositor-proxy` + +`compositor-proxy-cli` with a Python equivalent. The key components: + +```python +# Pseudocode for Python compositor-proxy +import asyncio +import os +import signal +import subprocess +from aiohttp import web, WSMsgType +from pywayland.server import Display +from xkbcommon import xkb +import gi +gi.require_version("Gst", "1.0") +gi.require_version("GstGL", "1.0") +from gi.repository import Gst, GstGL, GLib + +class CompositorProxy: + def __init__(self, bind_ip="0.0.0.0", bind_port=8081, render_device="/dev/dri/renderD128"): + self.bind_ip = bind_ip + self.bind_port = bind_port + self.render_device = render_device + self.display = Display() + self.sessions = {} # session_id → SessionProcess + + async def start(self): + # 1. Create wl_display + add socket + self.display.create() + socket_name = self.display.add_socket_auto() + os.environ["WAYLAND_DISPLAY"] = socket_name + + # 2. Init SHM formats (YUV420, NV12, YUV444, NV21) + self.init_shm() + + # 3. Init DRM (open /dev/dri/renderD128, register dmabuf globals) + self.init_drm() + + # 4. Integrate wl_display FD into asyncio loop + loop = asyncio.get_event_loop() + loop.add_reader(self.display.get_fd(), self.dispatch_requests) + + # 5. Start GStreamer + Gst.init(None) + self.gst_loop = GLib.MainLoop() + # Run GLib loop in a separate thread (GStreamer requires it) + import threading + threading.Thread(target=self.gst_loop.run, daemon=True).start() + + # 6. Start aiohttp server + app = web.Application() + app.add_routes([ + web.get("/signal", self.handle_signal), + web.get("/channel", self.handle_channel), + web.post("/launch", self.handle_launch), + # ... 6 more endpoints + ]) + runner = web.AppRunner(app) + await runner.setup() + site = web.TCPSite(runner, self.bind_ip, self.bind_port) + await site.start() + + # Run forever + await asyncio.Event().wait() + + def dispatch_requests(self): + """Called by asyncio when wl_display FD is readable.""" + self.display.dispatch_events_pending() + + async def handle_launch(self, request): + body = await request.json() + app_id = body["app_id"] + + # Spawn the app, set WAYLAND_DISPLAY + env = {**os.environ, "WAYLAND_DISPLAY": os.environ["WAYLAND_DISPLAY"]} + proc = subprocess.Popen( + self.applications[app_id]["executable"], + env=env, + # NEW vs Greenfield: sandbox with bubblewrap + preexec_fn=lambda: self.setup_sandbox(), + ) + + return web.json_response({ + "signal_url": f"ws://{self.bind_ip}:{self.bind_port}/signal?app_id={app_id}", + "pid": proc.pid, + }) + + def setup_sandbox(self): + """NEW: bubblewrap + seccomp sandbox (Greenfield has none).""" + import ctypes + libc = ctypes.CDLL("libc.so.6") + # ... bwrap / unshare / seccomp_filter setup +``` + +### Native app bridge (Python equivalent) + +For each Wayland client that connects (one per app), spawn a Python +`NativeWaylandClientSession` equivalent that: + +1. Sets up the wire-message interceptor (using pywayland's resource + dispatch). +2. Creates a `PROTOCOL` ARQ channel (port `kcp.ts` to Python or use + `pykcp`). +3. Creates a `FRAME` ARQ channel for the surface's encoded content. +4. Manages the app's lifecycle (10-min SIGHUP grace, SIGTERM escalation). + +For XWayland, replace `westfield-xwayland.c` with a Python equivalent +that: + +1. Creates `/tmp/.X11-unix/X` listener socket. +2. Sets `DISPLAY=:N` environment variable. +3. Spawns `Xwayland :N -ac -rootless -listen -displayfd -wm `. +4. Wraps the `wm_fd` as a Python file-like object. +5. Uses `xtsb` (the existing TypeScript X11 bindings) on the browser side + to handle XWM logic — the browser side stays the same. + +### Hardware acceleration (Python equivalent) + +For the encoding pipeline: + +- Use GStreamer Python bindings (`gi.repository.Gst`): + - `appsrc` for input buffers + - `glupload`, `glcolorconvert`, `glshader` for GPU processing + - `x264enc` / `nvh264enc` / `vaapih264enc` for H.264 encoding + - `appsink` for output samples +- Same encoder selection logic as Greenfield (PNG ≤ 256×256, H.264 above). +- Run the GLib main loop in a separate thread, use `asyncio.run_in_executor` + for cross-thread encoding requests. +- For zero-copy dmabuf, use `GstGLMemoryEGL` (same as Greenfield). + +For the browser decode side: **no change**. The browser uses WebCodecs +preferred + FFmpeg Wasm fallback, same as Greenfield. + +## Key trade-offs vs staying on Node + +### Staying on Node (use Greenfield as-is, just replace compositor-shell with Odoo) + +**Pros**: +- ~90% of Greenfield's code reused as-is (browser compositor + native proxy) +- Less code to write/maintain +- Faster time-to-market +- Battle-tested in Greenfield's rc1 cycle + +**Cons**: +- Native Node addons (`.node` files) are Linux-x64 only, ABI-locked to + Node major +- GStreamer + libwayland integration is via C N-API, harder to extend +- Odoo team may have more Python expertise than Node expertise +- AGPL-3.0 license on `compositor-proxy` — viral to your modifications + +### Switching to Python + +**Pros**: +- Aligns with Odoo's Python backend stack +- `pywayland` (cffi) is easier to debug than N-API addons +- Python GStreamer bindings are well-documented +- Easier to integrate with Odoo's auth/session model +- Can avoid AGPL by reimplementing from scratch (or use only MIT libs) + +**Cons**: +- Significant new code (rewrite the proxy + native addons in Python) +- Python GIL may bottleneck encoding throughput (use multiprocessing or + run GStreamer in separate thread) +- pywayland is less mature than libwayland's native bindings +- You take on the maintenance of a forked Greenfield-equivalent + +### Recommendation: hybrid approach + +1. **Browser side**: Use Greenfield's `@gfld/compositor` library as-is + (MIT-licensed via `@gfld/compositor-protocol` + `@gfld/common`, AGPL only + on `@gfld/compositor` itself — but the shell is yours so the AGPL + virality is contained to the compositor library). +2. **Native side**: Start with Greenfield's `compositor-proxy` (AGPL) to + get to MVP fast. Then progressively port to Python as needs evolve + (sandboxing, multi-tenant, custom auth, Odoo session integration). +3. **Shell**: Build the Odoo 20 OWL addon from scratch (clean room, + MIT/Apache license). + +## Implementation phases + +### Phase 1 (1-2 weeks): MVP — display a single Linux app + +1. Run Greenfield's compositor-proxy-cli as a Docker container. +2. Build minimal Odoo 20 OWL addon that creates a `` and + `userShell.events` handlers. +3. Embed the OWL addon in an Odoo action. +4. Trigger app launch via HTTP fetch to the proxy. +5. Verify the Linux app displays in the canvas. + +### Phase 2 (2-4 weeks): Multi-app, basic window management + +1. Implement `surfaceCreated` / `surfaceDestroyed` handlers. +2. Build a taskbar OWL component listing active surfaces. +3. Implement `activateSurface` action on taskbar click. +4. Test with 2-3 concurrent apps. + +### Phase 3 (4-8 weeks): Python port of proxy + +1. Replace Node `compositor-proxy-cli` with a Python equivalent. +2. Implement WebSocket endpoints in `aiohttp` or `FastAPI`. +3. Port the KCP TypeScript implementation to Python (or use `pykcp`). +4. Replace `wayland-server-addon.node` with `pywayland`. +5. Replace `proxy-encoding-addon.node` with GStreamer Python bindings. +6. Add bubblewrap sandboxing (NEW — Greenfield has none). + +### Phase 4 (8-16 weeks): Polish & scale + +1. Multi-tenant isolation (separate Python proxy processes per tenant). +2. Per-tenant resource limits (cgroups, max memory, max CPU). +3. Audit log integration with Odoo. +4. Clipboard sync (native → DOM and reverse). +5. Touch input implementation (Greenfield has stub only). +6. Primary selection (middle-click paste). +7. Adaptive bitrate (consume `EncoderFeedback` data, Greenfield doesn't). + +## Specific replacements for each Greenfield native component + +| Greenfield native component | Python replacement | Notes | +|---|---|---| +| `wayland-server-addon.node` (1231 lines C) | `pywayland` (cffi bindings) | Same libwayland-server, Python wrapper | +| `proxy-poll-addon.node` (127 lines C) | `asyncio.loop.add_reader()` | Built-in, no addon needed | +| `proxy-encoding-addon.node` + `libproxy-encoding.so` (~1700 lines C) | `gi.repository.Gst` (GStreamer Python bindings) | Same GStreamer, Python wrapper | +| Patched `libwayland-server` (vendored at `native/wayland/src/wayland-server/`) | Use upstream libwayland-server + Greenfield's patches applied | Apply `changes.diff` to a fork of libwayland, install via system package | +| `westfield-*.c` (EGL, dmabuf, xwayland helpers) | Reimplement in Python via `pyEGL` or `python-xkbcommon` + `pygobject`'s `GstGL` | Less mature than Greenfield's C — be prepared to debug | +| `xtsb` (X11 TypeScript bindings) | Use as-is on browser side, replace Python side with `python-xlib` | Browser-side xtsb is MIT-licensed | +| Node `ws` library | `aiohttp.web.WebSocketResponse` or `starlette.websockets.WebSocket` | Mature, well-documented | +| Node `pkg` (single-binary packaging) | PyInstaller or `shiv` | Same end result | +| Node `fork()` per session | `multiprocessing.Process` or `os.fork()` | Direct equivalent | +| `compositor-proxy-cli` (CLI args) | `argparse` | Standard Python | +| `ajv` (JSON schema validation) | `jsonschema` | Standard Python | + +## Final architecture diagram (target state) + +``` +┌──────────────────────────────────────────────────────────────────────────────────┐ +│ BROWSER (Odoo 20 web client) │ +│ │ +│ ┌────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Odoo 20 OWL Shell addon (your code, MIT/Apache) │ │ +│ │ - Application launcher, window manager, keyboard picker │ │ +│ │ - Per-output │ │ +│ │ - Implements UserShellApi.events.* callbacks │ │ +│ └────────────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ userShell.events + actions │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────────────────┐ │ +│ │ @gfld/compositor (browser-side, MIT libs + AGPL compositor) │ │ +│ │ - Surface, View, Output, Loop, Seat, Keyboard, Pointer, Touch │ │ +│ │ - render/ (WebGL shaders) │ │ +│ │ - remote/ (Channel, KCP, decoders) │ │ +│ └────────────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────────────┘ + │ + │ WebSocket + KCP-over-WS + │ 9 endpoint paths per session + ▼ +┌──────────────────────────────────────────────────────────────────────────────────┐ +│ PYTHON HOST (Linux server) │ +│ │ +│ ┌────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Odoo 20 backend addon (your code, MPL-2.0/LGPL) │ │ +│ │ - /web/compositor/apps, /web/compositor/launch │ │ +│ │ - Per-user, per-tenant allowed apps list │ │ +│ │ - Audit log │ │ +│ └────────────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ subprocess / IPC │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Python compositor-proxy (your code, MIT/Apache) │ │ +│ │ - aiohttp/FastAPI WebSocket server (9 endpoints) │ │ +│ │ - pywayland (cffi bindings to libwayland-server) │ │ +│ │ - python-xkbcommon for keymap compilation │ │ +│ │ - GStreamer Python bindings for H.264 encoding │ │ +│ │ - asyncio event loop integrating libwayland FD polling │ │ +│ │ - Per-app subprocess.Popen with bubblewrap sandbox │ │ +│ │ - KCP port (pykcp or hand-rolled) │ │ +│ └────────────────────────────────────────────────────────────────────────────┘ │ +│ │ │ +│ │ WAYLAND_DISPLAY=wayland-N │ +│ ▼ │ +│ ┌────────────────────────────────────────────────────────────────────────────┐ │ +│ │ Native Linux apps (gtk4-demo, xterm, kwrite, ...) │ │ +│ │ - Use host system libraries │ │ +│ │ - Sandboxed via bubblewrap + seccomp │ │ +│ └────────────────────────────────────────────────────────────────────────────┘ │ +└──────────────────────────────────────────────────────────────────────────────────┘ +``` diff --git a/11-pitfalls.md b/11-pitfalls.md new file mode 100644 index 0000000..1a6b224 --- /dev/null +++ b/11-pitfalls.md @@ -0,0 +1,532 @@ +# 11 — Pitfalls, Anti-patterns, and What to Avoid + +This is the prominent pitfalls chapter. Greenfield's known weaknesses, +anti-patterns, and design pitfalls are catalogued here so the user can +avoid making the same mistakes in their own Wayland-to-HTML compositor +(Odoo 20 + JS/Python). Each pitfall comes with: **what Greenfield does** +(with file/directory reference), **why it's a problem** (technical +reason), and **what the user should do instead** (concrete recommendation +for their JS/Python + Odoo 20 stack). + +## Headline numbers + +- **299** `TODO|FIXME|HACK|XXX|@deprecated|workaround` hits across non-vendored source (after excluding `examples/sdk/*` upstreams and `xtsb/protocol/xproto.xml`). +- **13 packages** all pinned to `"version": "1.0.0-rc1"` — no `CHANGELOG.md`, no `SECURITY.md`, no `CONTRIBUTING.md`, no issue/PR templates. +- **1 commit** visible on `master` (`6c578f4`, "Fix typo in code snippet (#161)", 2025-10-15) — implying the public repo is a release snapshot, not the live dev branch. +- **3 native Node `.node` addons**, all Linux-x64 only. +- **0** uses of `OffscreenCanvas` or `transferControlToOffscreen` anywhere in the compositor. +- **2** Web Workers used — and only for the WASM H.264 *fallback* decoder path. The primary WebCodecs path uses none. + +## Catalogued pitfalls (A–U) + +### Critical (will bite you on day 1) + +#### A. Native Node addons + +**What Greenfield does**: Three Node-API addons: +- `wayland-server-addon.node` (1231 lines C, wraps patched libwayland-server) +- `proxy-encoding-addon.node` (290 lines C, wraps GStreamer) +- `proxy-poll-addon.node` (127 lines C, wraps libuv `uv_poll_t`) + +All Linux-x64 only, built with CMake + Ninja. ABI-locked to Node major +versions. + +**Why it's a problem**: +- Linux-x64 only — no ARM64 support +- ABI breaks on Node major upgrades +- Hard to ship — must rebuild for each target platform +- Hard to debug — C N-API debugging requires core dumps +- GStreamer integration is via C, not Python +- No Python story + +**What you should do instead**: Use `pywayland` (cffi bindings to +libwayland-server), `python-xkbcommon`, and GStreamer's Python bindings +(`gi.repository.Gst`). One Python process can replace the entire +`compositor-proxy-cli`. See `10-porting-to-odoo-js-python.md` for details. + +#### B. rc1 with no policy + +**What Greenfield does**: All 13 packages at `1.0.0-rc1`. No CHANGELOG, +no SECURITY.md, no CONTRIBUTING.md. + +**Why it's a problem**: Breaking changes can land silently. The +maintainer's roadmap is unclear. + +**What you should do instead**: Pin to a specific commit hash if you fork +it. Track upstream commits manually. Plan for the eventuality that +1.0.0 final may break APIs. + +#### C. Encoding rigidity + +**What Greenfield does**: Three H.264 GStreamer pipelines (x264/nvh264/ +vaapih264) with hand-tuned magic constants: +- `bitrate=12000 vbv-buf-capacity=160 bframes=0 rc-lookahead=0` +- VA-API is "experimental" +- AMD (`vah264enc`) absent +- PNG fallback at hardcoded ≤256×256 threshold +- Browser-side hardware decode is explicitly disabled in `Session.ts:93` + with a FIXME "hardware decoding frame offset is wrong" + +**Why it's a problem**: +- No adaptive bitrate — video plays at 12 Mbps regardless of network + conditions +- No AMD support — limits hardware encoder options +- Browser decode is software-only — wastes GPU + +**What you should do instead**: +- Build an adaptive multi-codec encoding router (PNG for ≤64×64 or static, + VP8/H.264 for mid-size, H.264/AV1 for video) +- Drive the router by the `EncoderFeedback` measurement Greenfield already + produces but doesn't consume +- Use `pykcp` or similar for congestion-aware bitrate adaptation +- Enable browser hardware decode (`prefer-hardware`) — fix the "frame + offset is wrong" bug + +#### D. Wire protocol has no schema version field + +**What Greenfield does**: No version prefix on any binary message: +- `EncodedFrame` (FRAME channel) +- `RemoteOutOfBandChannel` (PROTOCOL channel OOB) +- The FD-channel in `RemoteAppLauncher.ts:612` +- Undocumented magic close codes (`4200`, `4201`, `4403`, `4001`, `1005`) + +All break-lockstep — if you change the wire format, both sides must +upgrade simultaneously or messages will be misinterpreted. + +**Why it's a problem**: +- Cannot evolve the protocol without breaking all clients +- Cannot detect version mismatches +- Hard to debug — close codes are magic numbers + +**What you should do instead**: +- Prefix every binary message with `u16 protocolVersion` +- Document every close code in a comment +- Use JSON `{"v":1, "type":"..."}` for the control plane +- Build a feature-flag system so old clients can negotiate capabilities + +#### E. Main-thread decode + render + +**What Greenfield does**: +- `VideoDecoder` runs on main thread +- WebGL context bound to `HTMLCanvasElement` +- No `OffscreenCanvas` +- No `Worker` in render path +- 0 uses of `OffscreenCanvas` or `transferControlToOffscreen` in the + compositor +- Only 2 Web Workers used (FFmpeg Wasm fallback), shared across surfaces + +**Why it's a problem**: +- Jank on 1080p surfaces — main thread blocks on decode +- No way to recover if main thread stalls (input also blocked) +- Can't use `requestAnimationFrame` for compositing if main thread is busy + +**What you should do instead**: +- Render in a Web Worker with `OffscreenCanvas` from day one +- Ship `EncodedVideoChunk` / `ArrayBuffer` transfers (zero-copy) to the + worker +- Use `VideoFrame` directly as a WebGL texture source — no `copyTo` +- Use multiple workers (one per output) for parallel rendering + +### High severity + +#### F. xkbcommon Wasm is multi-MB + +**What Greenfield does**: `libs/compositor-wasm/xkbcommon/build.sh` +clones libxkbcommon 1.5.0, embeds the entire xkeyboard-config database +via `--embed-file`. Result: multi-MB WASM module loaded on every page. + +**Why it's a problem**: +- Slow initial load +- Layout list parsed via fragile string matching +- Hard to update keyboard layouts without rebuilding WASM + +**What you should do instead**: +- Use the browser's `navigator.keyboard.getLayoutMap()` API for basic + layout detection +- For full xkbcommon support, lazy-load the WASM only when needed +- Consider server-side keymap compilation (Python `python-xkbcommon`) + and ship only the result string to the browser + +#### G. Clipboard edge cases + +**What Greenfield does**: +- Hardcoded MIME list `['text/plain', 'text/html', 'image/png']` +- No primary selection (`wp_primary_selection_v1` not implemented) +- Change detection by blob size (false negatives on same-size different + content) +- Focus check at `DataOffer.ts:73` is a FIXME +- Browser-side clipboard polling (no `clipboardchange` event) + +**Why it's a problem**: +- Middle-click paste doesn't work (Linux users expect it) +- Large clipboard content (e.g. copied image) is poorly handled +- MIME negotiation races + +**What you should do instead**: +- Implement `zwp_primary_selection_device_manager_v1` from day one +- Use exact MIME matching, not substring +- For large clipboard content, use chunked transfer +- Watch for the `clipboardchange` event in modern browsers (currently + behind flags in Chrome) + +#### H. XWayland/xtsb complexity + +**What Greenfield does**: +- A full XCB reimplementation in TypeScript with its own 5800-line + `xproto.xml` +- The XWM is a hand-port of wlroots' `xwm.c` with FIXMEs pointing at + specific wlroots commits +- `XWindowManager.ts:746` literally says "this makes xterm close somehow" +- Title bars painted pixel-by-pixel in `XWindowFrame.ts` + +**Why it's a problem**: +- 5800 lines of X11 protocol XML to maintain +- Hand-ported XWM logic is fragile and out of sync with upstream wlroots +- Magic workarounds in code + +**What you should do instead**: +- Use `python-xlib` on the Python proxy side instead of porting xtsb +- Consider running XWayland with a minimal window manager (e.g. mutter + in kiosk mode) instead of implementing XWM yourself +- For XWayland apps only (no full X11 desktop needed), the XWM can be + much simpler than wlroots' + +#### I. Adaptive bitrate feedback loop is wired but not consumed + +**What Greenfield does**: +- `EncoderFeedback.ts` measures client decode duration at 1 Hz +- 4-byte feedback messages sent from browser to proxy +- Proxy receives them but does NOT use them to adapt bitrate +- `Channel.ts:181` has a TODO confirming the gap + +**Why it's a problem**: +- Video plays at fixed 12 Mbps regardless of network conditions +- No way to recover from congestion +- Wasted bandwidth on slow networks + +**What you should do instead**: +- Wire `EncoderFeedback` to a bitrate adaptation controller +- Use `GstElement.set_property("bitrate", ...)` to adjust x264enc/nvh264enc +- Implement `requestKeyUnit` plumbing (Greenfield has it end-to-end but + the browser never calls it) + +#### J. Single-seat, single-session model + +**What Greenfield does**: +- One seat per browser session +- One Node process per browser session (per `compositorSessionId`) +- No multi-seat support (e.g. multiple keyboards/mice for collaborative + editing) +- No multi-region isolation + +**Why it's a problem**: +- Can't support multi-user scenarios (e.g. conference room with multiple + keyboards) +- No way to isolate apps between tenants + +**What you should do instead**: +- For multi-tenant isolation: separate Python proxy processes per tenant, + containerized (Docker + bubblewrap) +- For multi-seat: implement `wl_seat` global per seat, route input + events by seat + +#### K. Custom web-bitmapbuf protocol — apps must be Greenfield-aware + +**What Greenfield does**: +- `protocol/web-bitmapbuf-unstable-v1.xml` is Greenfield's custom + extension +- Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to the + compositor +- Standard Wayland apps (even browser-resident ones) fall to slow H.264 + path + +**Why it's a problem**: +- Apps must be rebuilt against `@gfld/client-protocol` to use the fast + path +- Standard Wayland apps can't take advantage of zero-copy GPU textures +- Browser-resident WASM apps are slow unless rebuilt + +**What you should do instead**: +- Document clearly that the fast path requires Greenfield-aware apps +- For native Linux apps (your main use case), this isn't an issue — they + go through the H.264 path regardless +- For browser-resident WASM apps, encourage users to build against + `@gfld/client-protocol` + +#### L. Connection loss handling + +**What Greenfield does**: +- 10-minute zombie timeout on disconnect (`NativeAppContext.ts:152-160`) +- No state resumption on reconnect +- `eagerBufferContents` cache in `BufferStream.ts:31` is unbounded (TODO + at `:87` confirms the leak) +- App kill timers: SIGHUP@10min, SIGKILL@+10s +- Close code 1001 → immediate SIGHUP; 4001 → immediate SIGTERM + +**Why it's a problem**: +- User loses all running apps on browser refresh (unless they reconnect + within 10 min) +- Memory leak from unbounded cache +- No way to survive proxy restart + +**What you should do instead**: +- Bound the `eagerBufferContents` cache (e.g. max 100 frames) +- Persist app state on browser side (localStorage) so users can resume +- For proxy restart, checkpoint app state to disk (json file) +- Implement graceful reconnection with state sync (the KCP plumbing + exists) + +#### M. No sandboxing + +**What Greenfield does**: +- `spawn(executable, args, { env })` with no bubblewrap/seccomp/namespaces +- Basic Auth is a single replayable credential compared with `!==` +- `compositorSessionId` is client-generated + +**Why it's a problem**: +- Native apps have full access to the host filesystem +- One compromised app can read all other apps' data +- Basic Auth credentials can be replayed +- Client-generated session IDs can be spoofed + +**What you should do instead**: +- Sandbox every spawned app with bubblewrap + seccomp + per-session Linux + user namespace +- Replace Basic Auth with short-lived signed JWT bound to Odoo's session + model +- Server-generated session IDs (use `secrets.token_urlsafe(32)`) + +#### N. Heavy toolchain + +**What Greenfield does**: +- Yarn 4 + CMake + Ninja + Meson + pkg-config +- Two different Emscripten versions: 3.1.46 for FFmpeg, 3.1.60 for SDK +- SDK builds 30+ C libraries from source +- gtk4 WASM is currently broken +- Native build needs `autoconf libtool automake meson ninja-build bison + cmake build-essential gperf` + +**Why it's a problem**: +- Complex build environment setup +- Long build times +- Emscripten version skew between libs and SDK +- gtk4 WASM broken means no GTK4 browser apps + +**What you should do instead**: +- Stick with a single Emscripten version +- Pre-build WASM modules and ship them as artifacts (don't rebuild every + time) +- For Python, use `pip install` for all dependencies (simpler than CMake + + Ninja) +- Skip the WASM app SDK entirely if you only care about native Linux apps + +### Medium / low (but worth fixing) + +#### O. WebCodecs usage + +**What Greenfield does**: +- WebCodecs IS used (correcting the brief in some agents' notes) +- BUT HW-decode is disabled +- `VideoEncoder` is unused + +**Why it's a problem**: +- Wastes GPU on software decode +- Could encode in browser (for upload to other peers) but doesn't + +**What you should do instead**: +- Enable HW-decode by fixing the "frame offset is wrong" FIXME +- Consider WebCodecs `VideoEncoder` for any peer-to-peer video paths + +#### P. WebCodecs hardware decode disabled + +**What Greenfield does**: +- `Session.ts:93` has `// FIXME hardware decoding frame offset is wrong` +- `hardwareAcceleration: 'prefer-software'` is hardcoded + +**Why it's a problem**: +- Software decode at 1080p60 is ~30% CPU on a modern laptop +- Battery life impact on mobile +- Wastes GPU + +**What you should do instead**: +- Investigate and fix the frame offset bug +- Enable `prefer-hardware` for capable browsers +- Provide a runtime toggle + +#### Q. Unbounded `eagerBufferContents` cache + +**What Greenfield does**: +- `BufferStream.ts:31` has unbounded cache +- TODO at `:87` confirms the leak + +**Why it's a problem**: +- Memory leak on long-running sessions +- Can OOM the browser tab + +**What you should do instead**: +- Bound the cache to a maximum number of frames (e.g. 100) +- Use LRU eviction +- Periodic cleanup of orphaned entries + +#### R. `H264NALDecoder.worker.ts` is minimal + +**What Greenfield does**: +- 22 lines of code +- `@ts-ignore`'d +- No error boundary + +**Why it's a problem**: +- Worker crashes silently +- No retry on transient errors +- No metrics on decode failures + +**What you should do instead**: +- Add error boundary with retry logic +- Log decode failures to the proxy (for diagnostics) +- Pool workers for resilience + +#### S. Magic undocumented WebSocket close codes + +**What Greenfield does**: +- Close codes `4200`, `4201`, `4403`, `4001`, `1005` are used without + documentation +- Only codes 4000-4999 (application-level) tear down KCP + +**Why it's a problem**: +- Hard to debug connection issues +- Close codes are magic numbers + +**What you should do instead**: +- Document every close code in code comments +- Use named constants (e.g. `CLOSE_APP_TERMINATED = 4001`) +- Build a close-code → human-readable mapping + +#### T. Generated protocol files committed + +**What Greenfield does**: +- `packages/compositor-proxy/src/protocol/*.js` is generated but + committed to the repo +- Can drift from XML source + +**Why it's a problem**: +- Easy to forget to regenerate +- PRs may include stale generated code + +**What you should do instead**: +- Don't commit generated files +- Use `prebuild` script to generate them +- Add a CI check that regenerated files match committed ones + +#### U. Confirmed copy-paste bug in `webcodec-buffer-decoder.ts` + +**What Greenfield does**: +- `webcodec-buffer-decoder.ts:142-144, 156-157` has: + `this.opaqueDecoder = undefined` twice +- Should be `alphaDecoder` the second time +- Leaks the alpha decoder after errors + +**Why it's a problem**: +- Memory leak +- Wasm decoder context not cleaned up properly +- Can cause decoder pool exhaustion + +**What you should do instead**: +- Audit all `cleanup` paths in decoder code +- Add unit tests for error scenarios + +## Design wins to copy + +Not all of Greenfield's choices are bad. Here are 12 design wins worth +copying verbatim: + +1. **Generator-driven protocol stubs** (`libs/compositor-generator/`, + `client-generator/`, `compositor-proxy-generator/`) — single XML source + for both sides. Eliminates wire-format drift. +2. **Dual-buffer commit semantics** (`Surface.ts`: `state` vs + `pendingState`, atomic merge with pixman damage regions). Classic + Wayland correctness. +3. **UserShellApi decoupling** — shell sees only `{events, actions}`, + never Wayland resources. This is THE integration seam you'll use. +4. **Asynchronous browser compositor** — per-client channels, never blocks + on a slow client. Essential for multi-app scenarios. +5. **KCP-over-WebSocket** for UDP-like latency over TCP (`kcp.ts`, 1170 + lines). Better than plain TCP for real-time video. +6. **WebCodecs primary + WASM fallback + capability probe** (`Session.ts`). + Modern browsers get the fast path, older ones still work. +7. **Zero-copy dmabuf → encoder** via standard `linux-dmabuf-unstable-v1` + (`westfield-dmabuf.c`). Pixels stay on GPU. +8. **Per-surface encoder feedback channel** (`EncoderFeedback.ts`) + measuring client decode duration — even though adaptation isn't wired + up yet, the telemetry is right. +9. **`ReconnectingWebSocket`** with exponential backoff + message queue. + Survives transient network drops. +10. **Minimal shell** — 562 lines of TypeScript total in `compositor-shell`; + kiosk variant is 40 lines. Demonstrates that the shell can be tiny. +11. **Honest docs** — `docs/pages/design/` openly admits WebSocket/TCP is + wrong for real-time and waits on WebTransport. Useful as a reference + for what trade-offs were considered. +12. **AGPL-3.0** — deliberate copyleft, but be aware before integrating. + If you want to keep your modifications proprietary, fork the MIT + libs and reimplement the AGPL parts. + +## Top-5 recommendations for the user's Odoo 20 + JS/Python compositor + +1. **Skip native Node addons entirely.** Use `pywayland` (cffi), + `python-xkbcommon`, and GStreamer's `gi.repository.Gst` Python + bindings. One Python process can replace the entire + `compositor-proxy-cli`. See `10-porting-to-odoo-js-python.md` for + details. + +2. **Sandbox every spawned app** with bubblewrap + seccomp + per-session + Linux user namespace; replace Basic Auth with short-lived signed JWT + bound to Odoo's session model. + +3. **Render in a Web Worker with `OffscreenCanvas`** from day one; ship + `EncodedVideoChunk` / `ArrayBuffer` transfers (zero-copy) to the + worker; use `VideoFrame` directly as a WebGL texture source — no + `copyTo`. + +4. **Build an adaptive multi-codec encoding router** (PNG for ≤64×64 or + static, VP8/H.264 for mid-size, H.264/AV1 for video), driven by the + `EncoderFeedback` measurement Greenfield already produces but doesn't + consume. + +5. **Version your wire protocol from commit #1** — prefix every binary + message with `u16 protocolVersion`; document every close code; use + JSON `{"v":1, "type":"..."}` for the control plane. + +## License implications of AGPL-3.0 + +The four `packages/*` (compositor, compositor-proxy, compositor-proxy-cli, +compositor-shell) are **AGPL-3.0-or-later**. Implications: + +- If you host Greenfield's `compositor-proxy` as a network service and + modify it, you MUST release your modifications under AGPL-3.0+. +- The browser-side `@gfld/compositor` is also AGPL. If you modify it + (e.g. to integrate with Odoo), your modifications must be AGPL. +- The lower-level `libs/*` (common, client-protocol, compositor-protocol, + the generators, compositor-wasm, compositor-ffmpeg-h264, xtsb) are MIT + or Apache-2.0 — these are safe to use without copyleft obligations. +- **Strategy to avoid AGPL**: use the MIT libs as-is, reimplement the + AGPL parts from scratch (browser compositor + native proxy). The + Python port described in `10-porting-to-odoo-js-python.md` is a clean + way to do this — if you write your Python proxy from scratch, you can + license it however you want. + +## What Greenfield's codebase reveals about its engineering process + +The codebase reveals several engineering process insights: + +1. **One-person show** — most commits are by Erik De Rijcke. The codebase + has the strengths (consistent vision, tight integration) and weaknesses + (single point of failure, no code review diversity) of a one-person + project. +2. **Long-running project** — code references back to 2017-era commits + (WebRTC was a previous transport; the `kcp.ts` MTU comment says + "webrtc datachannel MTU"). The architecture has evolved through several + major pivots. +3. **rc1 is meaningful** — the maintainer has been working toward 1.0 for + years and rc1 represents a deliberate freeze. Don't expect 1.0 final + any time soon, but also don't expect major architecture changes. +4. **Honest about limitations** — docs/pages/design/ explicitly admits + what's wrong and what's planned. Use this to inform your own roadmap. diff --git a/12-appendix.md b/12-appendix.md new file mode 100644 index 0000000..ee869c9 --- /dev/null +++ b/12-appendix.md @@ -0,0 +1,400 @@ +# 12 — Appendix: Glossary, File Index, References + +## Glossary of Greenfield terms + +### Wayland concepts + +| Term | Meaning | +|---|---| +| `wl_display` | The Wayland display connection — top-level object for both clients and the compositor. | +| `wl_registry` | Per-client registry of available globals (compositor, seat, shm, output, etc.). | +| `wl_compositor` | Global that creates `wl_surface` and `wl_region` objects. | +| `wl_surface` | A 2D rectangular pixel buffer with state stack — the basic drawable. | +| `wl_seat` | Represents a set of input devices (keyboard, pointer, touch). | +| `wl_keyboard` | Per-seat keyboard interface — sends keymap, key events, modifiers. | +| `wl_pointer` | Per-seat pointer interface — sends enter/leave/motion/button/axis events. | +| `wl_touch` | Per-seat touch interface (declared but inert in Greenfield). | +| `wl_output` | A monitor/screen — surfaces are positioned relative to outputs. | +| `wl_shm` | Shared memory buffer mechanism — client provides raw pixel buffer. | +| `wl_shm_pool` | A pool of shared memory that can be sliced into multiple `wl_buffer`s. | +| `wl_buffer` | A pixel buffer that can be attached to a `wl_surface`. | +| `wl_callback` | One-shot frame callback — fired when the compositor presents a frame. | +| `wl_data_device_manager` | Global for clipboard / DnD. | +| `wl_data_device` | Per-seat-per-client data device (selection, DnD). | +| `wl_data_source` | The provider side of a selection (the app that has data to offer). | +| `wl_data_offer` | The receiver side of a selection (the app that wants to receive data). | +| `wl_drm` | Legacy protocol for DMA-BUF buffers (used by Mesa, XWayland). | +| `wl_subcompositor` | Global for subsurfaces (nested surfaces). | +| `wl_subsurface` | A surface nested inside another surface (sync or desync). | +| `wl_region` | A region (set of rectangles) used for input/opaque/damage. | + +### xdg-shell concepts + +| Term | Meaning | +|---|---| +| `xdg_wm_base` | The xdg-shell global — clients use it to register as desktop apps. | +| `xdg_surface` | A wrapper around `wl_surface` for xdg-shell semantics. | +| `xdg_toplevel` | A top-level window (with title, app_id, state, parent). | +| `xdg_popup` | A transient popup window (menus, tooltips, dropdowns). | +| `xdg_positioner` | A specification for how to position a popup relative to its parent. | +| `configure` | Compositor → client event telling the client what state/size to apply. | +| `ack_configure` | Client → compositor request acknowledging a configure with a serial. | + +### Greenfield-specific terms + +| Term | Meaning | +|---|---| +| `Session` | Top-level browser-side compositor session. Created by `createCompositorSession()`. | +| `Channel` | Transport abstraction. `ARQChannel` (KCP-wrapped) for reliable channels, `SimpleChannel` for raw WebSocket. | +| `ARQChannel` | Automatic Repeat reQuest channel — KCP-over-WebSocket, reliable, low-latency. | +| `SimpleChannel` | Raw WebSocket channel, no KCP. Used for FEEDBACK (4-byte messages). | +| `Globals` | Registry of Wayland globals on the compositor side (`Globals.ts`). | +| `Seat` | Browser-side `wl_seat` implementation. Aggregates keyboard/pointer/touch + focus management. | +| `Surface` | Browser-side `wl_surface` implementation. Double-buffered state (`state` vs `pendingState`). | +| `View` | A surface's position on an output, used for compositing. One view per surface per output. | +| `Output` | Browser-side `wl_output` implementation. Wraps an HTML `` with WebGL context. | +| `RenderState` | Per-view-per-scene texture + transform state. | +| `Scene` | One per output/canvas. Manages the view stack and WebGL textures. | +| `BufferContents` | Abstraction over a buffer's content (mimeType + payload). | +| `BufferImplementation` | Interface for buffer types: SHM, ImageBitmap, StreamingBuffer. | +| `StreamingBuffer` | Per-surface buffer that receives encoded frames over the FRAME channel. | +| `BufferStream` | Manages serial → frame promise resolution. | +| `EncodedFrame` | Wire format parser for encoded frames (H.264 / PNG). | +| `EncoderFeedback` | 4-byte feedback messages sent from browser to proxy at 1 Hz (refreshInterval + decode duration). | +| `FrameFeedback` | Proxy-side receiver of `EncoderFeedback` messages. Used to pace `wl_frame.done` callbacks. | +| `ProxyFD` | JSON descriptor for an FD over WebSocket: `{handle, type, host}`. | +| `webFS` | Browser-side resolver for ProxyFD references. | +| `InputOutput` | FD abstraction (browser pipes for in-browser apps, WebSocket endpoints for native apps). | +| `RemoteInputOutput` | Native-app FD operations via WebSocket endpoints. | +| `webInputOutput` | In-browser-app FD operations via `MessageChannel` ports. | +| `UserShellApi` | The integration seam — `{events, actions}` interface between compositor and shell. | +| `CompositorClient` | Opaque handle for a Wayland client (browser-side). | +| `CompositorSurface` | Opaque handle for a Wayland surface (browser-side). | +| `DesktopSurface` | Floating or AlwaysFullScreen mode wrapper. Emits `userShell.events.*`. | +| `FloatingDesktopSurface` | Default floating window mode. | +| `AlwaysFullScreenDesktopSurface` | Kiosk mode — one window per output. | +| `XdgWmBase` | `xdg_wm_base` global, with ping-based unresponsiveness detection. | +| `XdgSurface` | `xdg_surface` role wrapper. | +| `XdgToplevel` | `xdg_toplevel` role — top-level windows. | +| `XdgPopup` | `xdg_popup` role — transient windows. | +| `XdgPositioner` | `xdg_positioner` — popup positioning math. | +| `web-bitmapbuf-unstable-v1` | Greenfield's custom Wayland protocol extension for browser JS apps. | +| `wayland-server-addon` | C N-API addon wrapping patched libwayland-server. | +| `proxy-encoding-addon` | C N-API addon wrapping GStreamer H.264/PNG encoder. | +| `proxy-poll-addon` | C N-API addon wrapping libuv `uv_poll_t` for wl_display FD polling. | +| `xtsb` | X11 TypeScript Bindings — pure-TS X11 client (no libxcb dependency). | +| `westfield-*` | Greenfield's C helpers: `westfield-buffer.c`, `westfield-dmabuf.c`, `westfield-egl.c`, `westfield-xwayland.c`. | +| `wlr_drm` | `wl_drm` global implementation (in `wlr_drm.c`). | +| `wlr_linux_dmabuf_v1` | `zwp_linux_dmabuf_v1` global implementation. | + +### Encoding / rendering terms + +| Term | Meaning | +|---|---| +| `dmabuf` | Linux DMA-BUF — a buffer that can be zero-copied between processes and GPUs. | +| `EGLImage` | EGL wrapper around a dmabuf, used by GStreamer GL. | +| `GstGLMemoryEGL` | GStreamer GL memory backed by an EGLImage — zero-copy GPU buffer. | +| `I420` | YUV 4:2:0 planar format (Y, U, V as separate planes). | +| `NV12` | YUV 4:2:0 semi-planar format (Y plane + interleaved UV plane). | +| `BT.601` | Color space conversion standard used by Greenfield's YUV→RGB shaders. | +| `NAL` | H.264 Network Abstraction Layer unit — a single encoded video frame or part thereof. | +| `AU` | H.264 Access Unit — a complete frame. | +| `keyframe` | H.264 I-frame — independently decodable, used to start a stream. | +| `WebCodecs` | Browser API for native video decode/encode (`VideoDecoder`, `VideoEncoder`). | +| `WebTransport` | Future browser API for UDP-like transport over QUIC. Greenfield plans to use it. | +| `KCP` | Reliable UDP protocol implemented in user-space. Greenfield runs it over WebSocket. | + +## File-to-feature index + +### Browser side (`packages/compositor/src/`) + +| File | Implements / Role | +|---|---| +| `index.ts` | Public API: `createCompositorSession()`, `createAppLauncher()` | +| `Session.ts` | Top-level browser session, decoder setup | +| `Compositor.ts` | `wl_compositor` global | +| `Globals.ts` | Wayland globals registry | +| `Surface.ts` | `wl_surface` impl, double-buffered state, commit semantics | +| `SurfaceChild.ts` | Surface tree child abstraction | +| `SurfaceRole.ts` | SurfaceRole interface | +| `Subsurface.ts` | `wl_subsurface` role, sync/desync mode | +| `Subcompositor.ts` | `wl_subcompositor` global | +| `View.ts` | Positioned surface for compositing | +| `Output.ts` | `wl_output`, WebGL canvas binding | +| `Shm.ts` | `wl_shm` / shm pool, BMP-header hack | +| `BufferContents.ts` | Buffer abstraction (with mimeType) | +| `BufferImplementation.ts` | Buffer implementation interface | +| `ImageBitmapBuffer.ts` | ImageBitmap-backed buffer | +| `Region.ts` | `wl_region`, pixman-WASM damage regions | +| `Callback.ts` | Frame callback (`wl_callback`) | +| `Loop.ts` | Compositor main loop, RAF integration | +| `Seat.ts` | `wl_seat` global, multi-seat support, focus management | +| `Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events | +| `Pointer.ts` | `wl_pointer` impl, motion, buttons, axis | +| `Touch.ts` | `wl_touch` (46-line stub, inert) | +| `Xkb.ts` | xkbcommon WASM wrapper | +| `KeyEvent.ts` | Key event wrapper | +| `ButtonEvent.ts` | Pointer button event wrapper | +| `AxisEvent.ts` | Pointer axis (scroll) event wrapper | +| `InputQueue.ts` | Event coalescing + RAF dispatch | +| `InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) | +| `DataDeviceManager.ts` | `wl_data_device_manager` global | +| `DataSource.ts` | `wl_data_source` impl, the provider side | +| `DataOffer.ts` | `wl_data_offer` impl, the receiver side | +| `WaylandDataSource.ts` | Wrapper around native `wl_data_source` events | +| `Shell.ts` | `wl_shell` global (legacy) | +| `ShellSurface.ts` | `wl_shell_surface` role | +| `XdgWmBase.ts` | `xdg_wm_base` global, ping-based unresponsiveness detection | +| `XdgSurface.ts` | `xdg_surface` role, configure/ack round-trip | +| `XdgToplevel.ts` | `xdg_toplevel` role, window state | +| `XdgPopup.ts` | `xdg_popup` role, transient windows, grabs | +| `XdgPositioner.ts` | `xdg_positioner`, popup positioning math | +| `UserShellApi.ts` | The integration seam — events + actions | +| `render/Renderer.ts` | 4-phase render pipeline | +| `render/Scene.ts` | One per output, manages view stack + textures | +| `render/SceneShader.ts` | Final pass: textured quads to canvas | +| `render/Texture.ts` | WebGL texture wrapper | +| `render/RenderState.ts` | Per-view-per-scene texture + transform | +| `render/Program.ts` | WebGL program wrapper | +| `render/ShaderCompiler.ts` | Shader compile/link helper | +| `render/ShaderSources.ts` | GLSL source strings | +| `render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass | +| `render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader | +| `render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) | +| `render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) | +| `render/RGBandA2RGBAShader.ts` | Shader for above | +| `remote/Channel.ts` | `ARQChannel` + `SimpleChannel` | +| `remote/kcp.ts` | ~1100-line KCP implementation | +| `remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery | +| `remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch | +| `remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps | +| `remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz | +| `remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer | +| `remote/BufferStream.ts` | Manages serial → frame promise resolution | +| `remote/EncodedFrame.ts` | Wire format parser for encoded frames | +| `remote/buffer-decoder.ts` | Decoder interface | +| `remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path | +| `remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path | +| `remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder | +| `remote/XWindowManager.ts` | X11 CLIPBOARD atom watcher (XWayland) | +| `remote/XDataSource.ts` | X11 clipboard source (XWayland apps) | +| `remote/XDnDDataSource.ts` | X11 DnD source | +| `remote/api.ts` | Public exports | +| `browser/input.ts` | DOM keyboard → wl_keyboard events | +| `browser/pointer.ts` | DOM pointer → wl_pointer events | +| `browser/dnd.ts` | DnD bridge | +| `browser/selection.ts` | Clipboard bridge | +| `browser/text-selection.ts` | Firefox fallback (text-only) | +| `browser/capabilities.ts` | Browser feature detection | +| `desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes | +| `math/Mat4.ts` | 4×4 matrix math | +| `math/Rect.ts` | Rectangle math | +| `web/WebBuffersFactory.ts` | Browser-side impl of `web_bitmapbuf_factory` | +| `web/WebInputOutput.ts` | In-browser-app FD operations | +| `web/WebConnectionHandler.ts` | In-browser-app connection handling | +| `polyfill/createImageBitmap.js` | ImageBitmap polyfill | + +### Native side + +| File | Implements / Role | +|---|---| +| `packages/compositor-proxy/src/index.ts` | Exports `createSession()`, `launchApplication()` | +| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session | +| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths | +| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers | +| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor | +| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm | +| `packages/compositor-proxy/src/Channel.ts` | ARQChannel + SimpleChannel | +| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation | +| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done | +| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation | +| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration | +| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge | +| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node | +| `packages/compositor-proxy/src/io/` | WebSocket server glue | +| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration | +| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules | +| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) | +| `packages/compositor-proxy/src/config.ts` | Encoding-related configuration | +| `packages/compositor-proxy/native/CMakeLists.txt` | Build config | +| `packages/compositor-proxy/native/wayland/src/addon.c` | Node-API addon: wayland-server-addon.node | +| `packages/compositor-proxy/native/wayland/src/westfield-buffer.c` | Buffer helpers | +| `packages/compositor-proxy/native/wayland/src/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM | +| `packages/compositor-proxy/native/wayland/src/westfield-egl.c` | EGL zero-copy setup | +| `packages/compositor-proxy/native/wayland/src/westfield-xwayland.c` | XWayland spawn + bridge | +| `packages/compositor-proxy/native/wayland/src/wlr_drm.c` | `wl_drm` global impl | +| `packages/compositor-proxy/native/wayland/src/wlr_linux_dmabuf_v1.c` | `zwp_linux_dmabuf_v1` global impl | +| `packages/compositor-proxy/native/wayland/src/drm-protocol.c` | DRM protocol helpers | +| `packages/compositor-proxy/native/wayland/src/linux-dmabuf-unstable-v1-protocol.c` | linux-dmabuf-v1 protocol stubs | +| `packages/compositor-proxy/native/wayland/src/wayland-server/` | Vendored + patched libwayland-server | +| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder | +| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding | +| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon: proxy-encoding-addon.node | +| `packages/compositor-proxy/native/poll/src/poll.c` | libuv uv_poll_t wrapper | +| `packages/compositor-proxy/native/node-api/` | Vendored Node-API headers | +| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session | +| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing | +| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions | +| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler | +| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps | + +### Libs + +| File | Role | +|---|---| +| `libs/common/src/Connection.ts` | Shared wire format codec | +| `libs/common/src/index.ts` | Public exports | +| `libs/client-generator/{ProtocolParser,ProtocolArguments,index}.mjs` | Browser client bindings generator | +| `libs/compositor-generator/{ProtocolParser,ProtocolArguments,index}.mjs` | Browser compositor bindings generator | +| `libs/compositor-proxy-generator/src/{EndpointProtocolParser,EndpointProtocolArguments,index}.mjs` | Proxy interceptor generator | +| `libs/client-protocol/src/westfield-runtime-client.ts` | Client-side runtime | +| `libs/client-protocol/src/protocol/` | Generated client bindings | +| `libs/compositor-protocol/src/westfield-runtime-server.ts` | Server-side runtime | +| `libs/compositor-protocol/src/protocol/` | Generated server bindings | +| `libs/compositor-wasm/build_wasm.sh` | pixman + xkbcommon WASM build | +| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon 1.5.0 build | +| `libs/compositor-wasm/pixman/build.sh` | pixman 0.42.2 build | +| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function | +| `libs/compositor-ffmpeg-h264/build_wasm.sh` | FFmpeg n5.1.2 → WASM build | +| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API | +| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module | +| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point | +| `libs/xtsb/protocol/xproto.xml` | X11 core protocol XML (5800+ lines) | +| `libs/xtsb/protocol/shape.xml` | Shape extension XML | +| `libs/xtsb/protocol/xfixes.xml` | XFIXES extension XML | +| `libs/xtsb/protocol/render.xml` | Render extension XML | +| `libs/xtsb/protocol/composite.xml` | Composite extension XML | +| `libs/xtsb/src/` | Generated TS X11 bindings | + +### Protocol XML + +| File | Role | +|---|---| +| `protocol/wayland.xml` | Standard upstream Wayland core protocol | +| `protocol/xdg-shell.xml` | Stable xdg-shell protocol | +| `protocol/wayland-drm.xml` | Legacy wl_drm for DMA-BUF (proxy-only) | +| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom extension for browser JS apps | + +## External references + +### Wayland protocol + +- **Wayland documentation**: https://wayland.freedesktop.org/docs/html/ +- **Wayland protocol XML**: https://gitlab.freedesktop.org/wayland/wayland/-/blob/main/protocol/wayland.xml +- **xdg-shell protocol**: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/stable/xdg-shell/xdg-shell.xml +- **wayland-drm protocol**: https://gitlab.freedesktop.org/wayland/wayland/-/blob/main/protocol/wayland.xml (wayland-drm is included) +- **linux-dmabuf-unstable-v1**: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml + +### libwayland + +- **libwayland source**: https://gitlab.freedesktop.org/wayland/wayland +- **Wayland book**: https://wayland-book.com/ +- **pywayland (Python bindings)**: https://pywayland.readthedocs.io/ + +### xkbcommon + +- **xkbcommon homepage**: https://xkbcommon.org/ +- **xkbcommon source**: https://github.com/xkbcommon/libxkbcommon +- **python-xkbcommon**: https://github.com/sde1000/python-xkbcommon + +### GStreamer + +- **GStreamer homepage**: https://gstreamer.freedesktop.org/ +- **GStreamer Python bindings**: https://gstreamer.freedesktop.org/documentation/tutorials/basic/hello-world.html +- **GStreamer GL plugins**: https://gstreamer.freedesktop.org/documentation/opengl/index.html +- **x264enc docs**: https://gstreamer.freedesktop.org/documentation/x264/index.html +- **nvh264enc docs**: https://gstreamer.freedesktop.org/documentation/nvcodec/index.html + +### FFmpeg Wasm + +- **FFmpeg homepage**: https://ffmpeg.org/ +- **FFmpeg Wasm build**: https://github.com/ffmpegwasm/ffmpeg.wasm +- **Emscripten**: https://emscripten.org/ + +### WebCodecs + +- **WebCodecs API**: https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API +- **VideoDecoder**: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder +- **WebCodecs samples**: https://w3c.github.io/webcodecs/samples/ + +### WebTransport + +- **WebTransport**: https://developer.mozilla.org/en-US/docs/Web/API/WebTransport +- **WebTransport over HTTP/3**: https://datatracker.ietf.org/doc/html/draft-ietf-webtrans-http3 + +### KCP + +- **KCP protocol**: https://github.com/skywind3000/kcp +- **KCP documentation (Chinese)**: https://skywind3000.github.io/kcp/ +- **pykcp (Python bindings)**: https://github.com/Zero3K/pykcp + +### Emscripten + +- **Emscripten homepage**: https://emscripten.org/ +- **Emscripten porting guide**: https://emscripten.org/docs/porting/index.html + +### X11 / XCB + +- **XCB documentation**: https://xcb.freedesktop.org/ +- **X protocol reference**: https://www.x.org/releases/X11R7.7/doc/xproto/x11protocol.html +- **python-xlib**: https://github.com/python-xlib/python-xlib + +### XWayland + +- **XWayland docs**: https://wayland.freedesktop.org/xserver.html +- **wlroots XWM source**: https://gitlab.freedesktop.org/wlroots/wlroots/-/tree/master/xcage + +### Greenfield project + +- **Greenfield homepage**: https://www.greenfield.app +- **Greenfield documentation**: http://udev.be/greenfield +- **Greenfield source**: https://github.com/udevbe/greenfield +- **Greenfield design docs**: https://github.com/udevbe/greenfield/blob/master/docs/pages/design/index.md +- **Greenfield future plans**: https://github.com/udevbe/greenfield/blob/master/docs/pages/future_plans/index.md +- **Author's blog**: https://wayouttheresoftware.blogspot.com/2023/07/some-history-about-greenfield.html + +### Odoo 20 + +- **Odoo homepage**: https://www.odoo.com/ +- **Odoo documentation**: https://www.odoo.com/documentation/20.0/ +- **Odoo OWL framework**: https://github.com/odoo/owl +- **Odoo development tutorial**: https://www.odoo.com/documentation/20.0/developer/howtos.html + +## Pointers to the user's next steps + +1. **Read chapters in order**: `00` → `01` → `02` → `07` → `08` → `10` → `11`. + This gives you the architecture map, the protocol layer, the native app + bridge (your primary use case), the UserShellApi integration seam, the + porting blueprint, and the pitfalls to avoid. + +2. **Start with Phase 1 of the porting blueprint** (`10-porting-to-odoo-js-python.md`): + Run Greenfield's compositor-proxy-cli as a Docker container, build a + minimal Odoo 20 OWL addon that creates a ``, and verify a single + Linux app displays. + +3. **Decide on the AGPL question early**: Will you accept AGPL on the + browser compositor + native proxy (and release your modifications), or + will you reimplement them from scratch (and license them however you + want)? + +4. **Choose your Python stack**: `aiohttp` vs `FastAPI` for the WebSocket + server. `pywayland` for the Wayland bindings. `python-xkbcommon` for + keymaps. `gi.repository.Gst` for GStreamer encoding. `bubblewrap` + + `seccomp` for sandboxing. + +5. **Plan for sandboxing from day one**: Greenfield has none. If you skip + this, your Odoo users can run arbitrary Linux code on your server. + That's a security hole you can't ship. + +6. **Build the OWL shell with `UserShellApi.events` callbacks**: This is + the clean integration seam. Don't try to import Greenfield internals + into your Odoo shell — use the public `UserShellApi` interface. + +7. **Budget for the Python port**: ~4-8 weeks for a working Python proxy. + ~8-16 weeks for production quality (sandboxing, multi-tenant, adaptive + bitrate, clipboard sync, touch input, primary selection). + +8. **Track upstream Greenfield commits**: Pin to a specific commit hash + if you fork it. Watch for 1.0 final (may break APIs). + +Good luck with your Wayland-to-HTML compositor build! diff --git a/README.md b/README.md new file mode 100644 index 0000000..1c10a04 --- /dev/null +++ b/README.md @@ -0,0 +1,103 @@ +# Greenfield Architecture Analysis — for an Odoo 20 + JS/Python Wayland-to-HTML Compositor + +> **Purpose of this bundle.** A deep, source-level architectural analysis of the +> Greenfield repository (https://github.com/udevbe/greenfield, commit `6c578f4`, +> `1.0.0-rc1`) written specifically for an engineering team that is about to build +> its own Wayland-to-HTML compositor. The target stack is **JavaScript (browser) + +> Python (host) + Odoo 20 addon shell**. Greenfield is treated purely as a +> reference implementation; the actual Greenfield UI shell (`packages/compositor-shell`) +> is **explicitly out of scope** — the user will supply their own Odoo 20 OWL shell. + +## How to read this bundle + +The bundle is split into 13 Markdown files. They are designed to be read in +order, but each file is self-contained and can be read alone if you only need +one layer. Files are organized from the highest-level view down to the +implementation-level details, then the porting blueprint, then pitfalls. + +| # | File | What you'll find inside | +|---|------|-------------------------| +| 0 | `00-overview.md` | What Greenfield is, repo layout, package inventory, tech stack table, ASCII architecture diagram. | +| 1 | `01-architecture.md` | Layer-by-layer map from Wayland XML → generators → runtime → transport → rendering. | +| 2 | `02-wayland-protocol.md` | Protocol layer: 3 generators, wire format, object lifecycle, FD transport, custom `web-bitmapbuf` extension. | +| 3 | `03-html5-rendering.md` | Surface → canvas compositing pipeline: Surface state, damage, FrameLoop, WebGL shaders, multi-output. | +| 4 | `04-transport.md` | WebSocket + KCP-over-WebSocket, multi-channel design, message formats, FD virtualization, reconnection. | +| 5 | `05-input-handling.md` | Keyboard / pointer / touch / focus, xkbcommon WASM, keymap FD workaround, DOM → Wayland event mapping. | +| 6 | `06-data-clipboard.md` | `wl_data_device`, selection, drag-and-drop, MIME negotiation, XWayland bridge, primary-selection gap. | +| 7 | `07-native-app-bridge.md` | How real Linux apps get launched, the 3 native Node addons, XWayland bridge, app lifecycle & kill timers. | +| 8 | `08-app-lifecycle.md` | xdg-shell state machine, configure/ack round-trips, popup positioner math, subsurface sync, `UserShellApi`. | +| 9 | `09-hardware-accel.md` | GStreamer H.264 encoders, WASM FFmpeg fallback, WebCodecs preferred path, WebGL texture upload, alpha split. | +| 10 | `10-porting-to-odoo-js-python.md` | Concrete blueprint for porting each Greenfield layer to **JS browser compositor + Python host proxy + Odoo 20 OWL shell**. | +| 11 | `11-pitfalls.md` | 21 catalogued pitfalls + 12 design wins, with file references and what-to-do-instead for each. | +| 12 | `12-appendix.md` | Glossary, file-to-feature index, external references. | + +## Recommended reading order + +- **Architects / tech leads** — read `00` → `01` → `11` → `10`. You'll get the + lay of the land, what to copy, what to avoid, and the porting blueprint. +- **Browser-compositor engineers** — read `02` → `03` → `04` → `05` → `08` → + `09`. The core engine chapters, in dependency order. +- **Host-proxy engineers (Python)** — read `07` → `04` → `09` → `02` (proxy + generator section) → `10`. Most of the porting work is on your side. +- **Odoo 20 OWL shell engineers** — read `08` (especially the `UserShellApi` + section) → `01` → `10` (Odoo integration section). The shell is the + integration seam where Greenfield stops and your work begins. +- **Skip-it sections** — `06` (clipboard) is only needed if you care about + clipboard sync between browser and native apps. + +## Scope & exclusions + +**In scope** — every Greenfield package, lib, generator, native addon, +protocol file, transport channel, rendering path, encoding pipeline, and +the `UserShellApi` integration seam. + +**Out of scope** — the actual `packages/compositor-shell` Preact+Tailwind +SPA. The user is building their own shell as an Odoo 20 addon. We only +describe the protocol-level integration point (`UserShellApi`) so the +Odoo shell knows what to consume. + +**Not covered in depth** — the `sdk/` (Emscripten-based WASM app SDK for +browser-resident Wayland apps). Mentioned in `00-overview.md` and +`10-porting-to-odoo-js-python.md` but not analyzed layer-by-layer, +because the user's goal is running **real Linux apps** in the browser, +not browser-resident WASM apps. + +## Conventions + +- **File paths** are always relative to the repo root, e.g. + `packages/compositor/src/Surface.ts`. +- **ASCII diagrams** are used throughout — they render correctly in any + Markdown viewer, survive copy-paste, and work in code reviews. +- **No code excerpts** by user request. We describe algorithms in prose + and reference file paths so you can read the source yourself. +- **Cross-references** look like `(see §3 of `03-html5-rendering.md`)`. + +## How this analysis was produced + +This bundle was produced by 10 parallel research agents, each assigned to a +single layer of the Greenfield codebase: + +- **Agent A** — top-level architecture +- **Agent B** — Wayland protocol layer (generators + runtime) +- **Agent C** — HTML5 rendering pipeline +- **Agent D** — transport layer (WebSocket + KCP) +- **Agent E** — input handling (keyboard / pointer / touch / xkbcommon) +- **Agent F** — data device & clipboard +- **Agent G** — native app bridge (process model + native addons) +- **Agent H** — xdg-shell / app lifecycle / `UserShellApi` +- **Agent I** — hardware acceleration / encoding / WASM +- **Agent J** — pitfalls, anti-patterns, design wins + +Each agent read the relevant source files end-to-end, produced a stage +summary, and appended its findings to `/home/z/my-project/worklog.md`. +This bundle is the synthesis of those findings into a single readable +document set. + +## License of the source + +Greenfield's runtime packages (`compositor`, `compositor-proxy`, +`compositor-proxy-cli`, `compositor-shell`) are **AGPL-3.0-or-later**. +The lower-level libs (generators, `common`, `compositor-wasm`, +`compositor-ffmpeg-h264`, `xtsb`) are MIT or Apache-2.0. Anyone forking +or building on the runtime packages must respect AGPL — see +`11-pitfalls.md` §License for implications.