Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents)

This commit is contained in:
1b-agent committed 2026-10-08 08:48:25 +00:00
commit c4b45cf218
15 files changed
+5444

No files matched your search

+24
View File
@@ -0,0 +1,24 @@
# Python
__pycache__/
*.pyc
*.pyo
.venv/
venv/
# OS
.DS_Store
Thumbs.db
# Editor
.vscode/
.idea/
*.swp
# Logs
*.log
# Build
node_modules/
dist/
build/
*.egg-info/
+386
View File
@@ -0,0 +1,386 @@
# 00 — What Greenfield Is and How It's Structured
## The one-paragraph summary
Greenfield is a Wayland compositor that runs partly in the browser (the actual
compositing and rendering, written in TypeScript against WebGL / WebCodecs /
WASM) and partly on a Linux host (a Node.js process that hosts real Linux
Wayland apps via `libwayland-server` and bridges their wire traffic to the
browser over WebSocket). The two halves together let you display **real Linux
applications** (GTK3/4, Qt, xterm, X11 apps via XWayland) inside a browser tab,
as if they were native browser apps. An optional Emscripten-based SDK lets you
compile separate Wayland apps to WASM and run them entirely in the browser with
no Linux host required.
## Repo layout
```
greenfield/
├── README.md # "The in-browser wayland compositor"
├── package.json # yarn 4.5.0 workspaces: libs/*, packages/*, examples/webapps/*, examples/compositor/*
├── .yarnrc.yml # nodeLinker: pnpm, global cache off
├── yarn.lock # 9368 lines
├── publish.sh # yarn workspaces foreach npm publish
├── protocol/ # Wayland XML protocol definitions (4 files)
│ ├── wayland.xml # Core Wayland protocol (124KB)
│ ├── xdg-shell.xml # Desktop shell protocol (48KB)
│ ├── wayland-drm.xml # DRM buffer formats (8.8KB)
│ └── web-bitmapbuf-unstable-v1.xml # *** Greenfield custom extension (2KB) ***
├── libs/ # 9 internal @gfld/* libraries
│ ├── common/ # @gfld/common - shared client/server runtime (wire format)
│ ├── client-protocol/ # @gfld/client-protocol - browser-side generated Wayland bindings
│ ├── compositor-protocol/ # @gfld/compositor-protocol - Node.js-side generated Wayland bindings
│ ├── client-generator/ # CLI: XML → TS browser protocol source
│ ├── compositor-generator/ # CLI: XML → TS Node protocol source
│ ├── compositor-proxy-generator/ # CLI: XML → TS proxy protocol source
│ ├── compositor-wasm/ # pixman + xkbcommon compiled to WASM
│ ├── compositor-ffmpeg-h264/ # FFmpeg n5.1.2 compiled to WASM (H.264 software fallback decoder)
│ └── xtsb/ # X11/XCB TypeScript bindings (vendored)
├── packages/ # 4 user-facing @gfld/* packages
│ ├── compositor/ # @gfld/compositor - browser-side Wayland compositor library (AGPL)
│ ├── compositor-shell/ # @gfld/compositor-shell - Preact+Tailwind SPA shell (AGPL) *** OUT OF SCOPE ***
│ ├── compositor-proxy/ # @gfld/compositor-proxy - native Wayland compositor proxy (AGPL, Linux-only)
│ └── compositor-proxy-cli/ # @gfld/compositor-proxy-cli - CLI binary (AGPL, Linux-only)
├── sdk/ # Greenfield WebAssembly SDK (for browser-resident WASM apps)
│ ├── README.md
│ ├── build.sh # clones emsdk 3.1.60, builds 30+ sysrootlibs
│ ├── sdk_env.sh # exports PKG_CONFIG_LIBDIR + PATH=toolkit/bin
│ ├── toolkit/ # gfcc/gf++/gfar/gfnm wrappers around emcc + wayland-scanner + meson cross inis
│ └── sysrootlibs/ # 34 build scripts for cross-compiling graphics stack (wayland, pixman, cairo, pango, gtk4, etc.)
├── examples/
│ ├── compositor/experimental-fullscreen/ # vite PWA kiosk running a single remote app
│ ├── docker-compose/ # Dockerfile + applications.json + docker-compose.yml for gtk3-demo/gtk4-demo/xterm/glxgears
│ ├── sdk/simple-shm/ # Weston simple-shm.c built with gfcc + wayland-scanner
│ ├── sdk/weston/ # Weston clients (terminal, dnd, editor, keyboard, flower, ...) meson-built
│ ├── sdk/gtk4/ # GTK4 demos meson-built to WASM
│ └── webapps/ # Pure-JS browser demos: simple-shm, webgl, weston-clients (symlink), gtk4
├── docs/ # Jekyll site (CNAME: udev.be/greenfield)
│ └── pages/{components, design, future_plans, getting_started, sdk}
└── docker/ # Production Dockerfile for compositor-proxy-cli
```
## The dual-side architecture
Greenfield is split into **two cooperating halves**, connected over WebSocket
(or, in the future, WebTransport). Understanding this split is fundamental to
everything else in this bundle.
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ BROWSER SIDE (TypeScript + WASM) │
│ │
│ ┌─────────────────────────┐ ┌──────────────────────────────────────────┐ │
│ │ @gfld/compositor-shell │ │ @gfld/compositor │ │
│ │ (Preact + Tailwind SPA)│───▶│ - Session, Compositor, Surface, Seat │ │
│ │ *** OUT OF SCOPE *** │ │ - Keyboard, Pointer, Touch, Xkb (WASM) │ │
│ └─────────────────────────┘ │ - Output (WebGL canvas) │ │
│ │ - render/ (WebGL shaders: YUV→RGB, etc.) │ │
│ ┌─────────────────────────┐ │ - desktop/ (Floating vs AlwaysFullScreen) │ │
│ │ JS/WASM webapps │───▶│ - remote/ (Channel, kcp.ts, decoders) │ │
│ │ (use @gfld/client- │ │ - browser/ (dnd, selection, input) │ │
│ │ protocol + common) │ │ │ │
│ └─────────────────────────┘ └──────────────┬──────────────┬─────────────┘ │
│ │ │ │
│ depends on ▼ ▼ │
│ ┌─────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ @gfld/compositor│ │ @gfld/common │ │ @gfld/xtsb │ │
│ │ -protocol │ │ (Connection, │ │ (X11/XCB bindings,│ │
│ │ (Node-side │ │ wire format)│ │ used by both │ │
│ │ generated │ │ │ │ compositor+proxy)│ │
│ │ bindings) │ │ │ └──────────────────┘ │
│ └─────────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
│ WebSocket (TCP, perMessageDeflate=false)
│ + KCP-over-WebSocket (reliable UDP, ready for WebTransport)
│ + 9 endpoint paths per session
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ NATIVE SIDE (Linux, Node 18+, C) │
│ │
│ ┌──────────────────────────┐ ┌──────────────────────────────────────────┐ │
│ │ @gfld/compositor-proxy-cli│───▶│ @gfld/compositor-proxy │ │
│ │ - main.ts: http + ws │ │ - Session, SessionController │ │
│ │ WebSocketServer │ │ - NativeWaylandClientSession │ │
│ │ - forks SessionProcess │ │ - NativeWaylandCompositorSession │ │
│ │ per browser session │ │ - NativeAppContext (launch) │ │
│ │ - --bind-port=8081 │ │ - SurfaceBufferEncoding │ │
│ │ - --applications=<json> │ │ - Channel, FrameFeedback │ │
│ │ - --encoder=x264|nvh264 │ │ - XWaylandSession │ │
│ │ - --render-device=/dev/ │ │ - kcp.ts (1100-line TypeScript impl) │ │
│ │ dri/renderD128 │ │ - wayland-server.ts (TS wrapper) │ │
│ └──────────────────────────┘ │ - io/, encoding/, protocol/ │ │
│ └──────────────┬──────────────┬─────────────┘ │
│ │ │ │
│ depends on ▼ ▼ │
│ ┌────────────────────────┐ ┌────────────────────────────────┐ │
│ │ @gfld/compositor-proxy-│ │ native/ (C, built via CMake + │ │
│ │ generator │ │ Ninja) │ │
│ │ (generates *_interface │ │ - wayland/src/addon.c │ │
│ │ and *_interceptor.js) │ │ - encoding/src/gst_frame_ │ │
│ └────────────────────────┘ │ encoder.c │ │
│ │ - poll/src/poll.c (libuv) │ │
│ │ - wayland-server/ (patched) │ │
│ │ - westfield-*.c (EGL, dmabuf, │ │
│ │ XWayland, etc.) │ │
│ └────────────────────────────────┘ │
│ │
│ Native Wayland apps (GTK3/4, Qt, xterm, X11 via XWayland) attach to the │
│ proxy as if it were a real wl_display. The proxy encodes each surface's │
│ buffer to H.264 via GStreamer (x264 software or nvh264 hardware) using │
│ zero-copy dmabuf, sends frames over WebSocket/KCP to the browser. │
└─────────────────────────────────────────────────────────────────────────────┘
```
## Tech-stack table
| Category | Component | Tech / Version | Role |
|---|---|---|---|
| **Languages** | Browser TS | TypeScript ^5.5.2 | All `@gfld/*` TS packages |
| | Native C | C99 | `packages/compositor-proxy/native/` |
| | Shell | bash | sdk/build.sh, toolkit wrappers |
| | Python | python3 (xcbgen) | libs/xtsb protocol generation |
| **Build tools** | Workspace | Yarn 4.5.0 (Berry, pnpm nodeLinker) | Monorepo workspace mgmt |
| | Browser bundler | Vite ^5.3.1 | compositor-shell, examples |
| | TS compiler | tsc ^5.5.2 | libs/* and packages/* build |
| | Native build | CMake 3.13+ + Ninja | packages/compositor-proxy `build:native` |
| | Native pkg | @vercel/pkg (Brotli) + @vercel/ncc | compositor-proxy-cli single-binary |
| | Lint/format | ESLint ^8.57 + Prettier ^3.3.2 | All TS packages |
| | Test | Jest ^29.7.0 + ts-jest ^29.1.5 | libs/xtsb, packages/compositor-proxy |
| **WASM toolchain** | Emscripten (libs build) | emsdk 3.1.46 | libs/compositor-wasm, libs/compositor-ffmpeg-h264 |
| | Emscripten (SDK build) | emsdk 3.1.60 | sdk/build.sh (sysrootlibs) |
| | Meson cross | meson-gf-cross.ini (host_machine: wasm32/emscripten) | sdk/toolkit |
| **Wayland / GL stack** | wayland.xml / xdg-shell.xml / wayland-drm.xml | upstream freedesktop.org | protocol/ |
| | wayland-scanner | vendored at sdk/toolkit/bin/wayland-scanner | C header/codegen for native clients |
| | libwayland (server) | vendored + patched at packages/compositor-proxy/native/wayland/src/wayland-server/ | native compositor server side |
| | wl_drm + linux-dmabuf-unstable-v1 | implemented in native/wayland/src/wlr_*.c | zero-copy dmabuf to GStreamer |
| **Custom protocol** | web-bitmapbuf-unstable-v1.xml | Greenfield extension (Erik De Rijcke, 2023) | browser JS/WASM clients wrap HTML ImageBitmap as wl_buffer via fd |
| **Native GL/media runtime** | GStreamer | 1.20 (built from gitlab.freedesktop.org in docker) | H.264 encoding pipeline, plugins: base/good/bad/ugly/gl + nvcodec |
| | GStreamer encoders | x264 (software), nvh264 (NVIDIA hardware), vaapih264 (Intel, experimental) | --encoder flag on compositor-proxy-cli |
| | Mesa / OpenGL / EGL | libgbm1, libdrm2, libopengl0, libglvnd0, libegl1-mesa | GPU buffer mgmt + EGL rendering |
| | NVIDIA | libnvidia-egl-wayland1, libnvidia-egl-gbm1 | optional HW accel |
| **Browser transport** | WebSocket | ws ^8.17.1 (Node), native WebSocket (browser) | primary transport (TCP) |
| | KCP | TypeScript impl (1100+ lines per side) | planned reliable UDP — future switch to WebTransport |
| | WebTransport | mentioned in docs/design as future | not yet implemented |
| | XWayland | xwayland package + xauth + xxd | X11 backward compat on native side |
| **Browser rendering** | WebGL | WebGL1 native, `desynchronized:true` | Output.ts canvas, render/*.ts shaders |
| | WebCodecs API | @types/dom-webcodecs ^0.1.11 | H.264 GPU decode in browser (preferred path) |
| | WebAssembly (FFmpeg) | libs/compositor-ffmpeg-h264 (FFmpeg n5.1.2) | software fallback H.264 decode |
| | WebAssembly (pixman+xkbcommon) | libs/compositor-wasm (emsdk 3.1.46) | pixel ops + XKB keymap math |
| | SharedArrayBuffer + Web Workers | browser-native (COOP/COEP required) | async compositor, H.264 NAL decoder workers |
| | ImageBitmap | browser-native | web-bitmapbuf-unstable-v1 backing store |
| **Shell UI (out of scope)** | Preact | ^10.22.0 | compositor-shell SPA |
| | Tailwind CSS | ^3.4.4 | compositor-shell styling |
| | @preact/signals | ^1.2.3 | reactivity |
## Package inventory
### Browser side
**`packages/compositor`** (`@gfld/compositor`, AGPL-3.0) — The browser-side
Wayland compositor library. Pure TypeScript + WebGL + WebCodecs + WASM, no
3rd-party runtime deps. Exports `createCompositorSession()` and
`createAppLauncher(session, 'web' | 'remote')`. Internally: `Session.ts`
orchestrates, `Compositor.ts`/`Surface.ts`/`Seat.ts`/`Pointer.ts`/
`Keyboard.ts`/`Output.ts` implement Wayland semantics, `render/` holds WebGL
shaders (YUV/YUVA/RGB→RGBA conversion), `desktop/` arranges surfaces
(Floating vs AlwaysFullScreen), `web/` speaks to same-origin-iframe JS/WASM
clients, `remote/` speaks to the native proxy (with `kcp.ts`,
`websocket-events.ts`, `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`,
`H264NALDecoder.worker.ts`, `xwayland/`), `browser/` handles dnd/selection/
input. Output is a WebGL texture inside an HTML5 canvas. Async — a slow client
never blocks another.
**`libs/client-protocol`** (`@gfld/client-protocol`, MIT) — Browser-side
generated Wayland protocol bindings (wayland.xml + xdg-shell.xml +
web-bitmapbuf-unstable-v1.xml). Used by JS/WASM webapps via `@gfld/common`
for the wire format.
**`libs/compositor-protocol`** (`@gfld/compositor-protocol`, MIT) — Node-side
generated Wayland protocol bindings (same 3 XMLs). Used by
`@gfld/compositor` for the in-browser server-side marshalling of Wayland
messages.
**`libs/common`** (`@gfld/common`, MIT) — Shared client/server Wayland
wire-format runtime. Two source files: `Connection.ts` (message framing +
argument codec) + `index.ts`. Tiny core that both `client-protocol` and
`compositor-protocol` build on.
**`libs/compositor-wasm`** (`@gfld/compositor-wasm`, MIT) — pixman 0.42.2 +
xkbcommon 1.5.0 compiled to WASM via emsdk 3.1.46. Exposes
`src/index.ts` (`init` function). Used by `@gfld/compositor` for pixel-region
math and XKB keyboard layout state.
**`libs/compositor-ffmpeg-h264`** (`@gfld/compositor-ffmpeg-h264`, Apache-2.0)
— FFmpeg n5.1.2 compiled to a single WASM module. TypeScript wrapper in
`src/{H264Decoder.ts, H264Worker.ts, index.ts}` runs in a Web Worker. Used as
the software fallback H.264 decoder when the browser lacks WebCodecs.
### Native side
**`packages/compositor-proxy`** (`@gfld/compositor-proxy`, AGPL-3.0,
Linux-only, Node ≥18) — Native Wayland compositor that bridges Linux apps to
the browser. Has BOTH a TypeScript layer (`src/Session.ts`,
`NativeWaylandClientSession.ts`, `NativeWaylandCompositorSession.ts`,
`NativeAppContext.ts`, `SurfaceBufferEncoding.ts`, `XWaylandSession.ts`,
`kcp.ts`, `Channel.ts`, `FrameFeedback.ts`, `wayland-server.ts`) AND a C
layer (`native/`) built with `cmake -G Ninja`. The C layer produces:
- `wayland-server.so` — patched libwayland-server (vendored at
`native/wayland/src/wayland-server/`, with `changes.diff` and
`generate_wayland_server_patch.sh`)
- `westfield.so` — Greenfield's westfield-* helpers (buffer, dmabuf, egl,
xwayland, fdutils) + `wlr_drm.c` + `wlr_linux_dmabuf_v1.c` +
`drm-protocol.c` + `linux-dmabuf-unstable-v1-protocol.c`
- `proxy-encoding.so` — GStreamer frame encoder (`gst_frame_encoder.c` +
`gst_main_loop.c`)
- `proxy-encoding-addon.node` + `proxy-poll-addon.node` +
`wayland-server-addon.node` — Node-API addons (vendored node-api headers in
`native/node-api/`) loaded by `src/addons/`
- `poll.c` — epoll wrapper backed by libuv
CMake deps: glib-2.0, gstreamer-1.0/app/video/gl/allocators ≥1.18,
graphene-1.0, egl ≥1.5, opengl ≥3.3, libffi, gbm, libdrm. Runtime also needs
libudev, mesa, NVIDIA EGL wayland/gbm drivers, xwayland. Implements `wl_drm`
+ `linux-dmabuf-v1` for zero-copy GPU buffer transfer to GStreamer encoder
pipeline (x264 software or nvh264 hardware). Connects to browser via
WebSocket; has TypeScript KCP implementation ready for the future WebTransport
switch. Supports direct peer-to-peer copy-paste between two proxy instances
to avoid browser round-trip.
**`packages/compositor-proxy-cli`** (`@gfld/compositor-proxy-cli`, AGPL-3.0,
Linux-only, Node ≥18) — CLI binary that wraps `@gfld/compositor-proxy`.
`src/main.ts` runs an HTTP server + `ws` WebSocketServer on `--bind-port`
(default 8081), and on first connection per `compositorSessionId` query
param, `fork()`s a `SessionProcess` (src/SessionProcess.ts) — one child per
active browser session. CLI args: `--applications=<json>` (URL→executable
map), `--encoder=x264|nvh264`, `--render-device=/dev/dri/renderD128`,
`--base-url`, `--allow-origin`, `--bind-ip`, `--bind-port`, `--basic-auth`.
The `--applications` JSON maps paths like `/gtk4-demo`, `/kwrite`, `/xterm`
to executables + args + env. Packaged with `@vercel/pkg` (Brotli, target
`node18-linux`) into a single static binary.
**`libs/compositor-proxy-generator`** (`@gfld/compositor-proxy-generator`,
MIT) — CLI that converts `protocol/{wayland,xdg-shell,wayland-drm}.xml` into
TypeScript source under `packages/compositor-proxy/src/protocol/`. Used at
build time only (yarn `generate` script).
### Shell side (OUT OF SCOPE for user)
**`packages/compositor-shell`** (`@gfld/compositor-shell`, AGPL-3.0) — Preact
+ Tailwind single-page-app shell built with Vite. Depends only on
`@gfld/compositor` + `preact` at runtime. Provides application launcher,
keyboard config, window management UI. **Explicitly out of scope for the
user** per task brief. The protocol-level integration point the user needs
to know about is `UserShellApi` (see `08-app-lifecycle.md`).
### Generators (3 sibling CLIs)
All three generators share the same dependency set (`camelcase`,
`uppercamelcase`, `meow`, `xml2js`) and similar layout
(`ProtocolParser.mjs` + `ProtocolArguments.mjs` + `index.mjs`). They consume
the same `protocol/*.xml` files but emit slightly different TypeScript
flavors for each side:
- **`libs/client-generator`** — emits browser-side client bindings into
`libs/client-protocol/src/protocol/`.
- **`libs/compositor-generator`** — emits Node.js-side server bindings into
`libs/compositor-protocol/src/protocol/`.
- **`libs/compositor-proxy-generator`** — emits proxy-side bindings into
`packages/compositor-proxy/src/protocol/`. Different file layout:
`src/EndpointProtocolArguments.mjs` + `EndpointProtocolParser.mjs` +
`index.mjs`.
### XWayland support
**`libs/xtsb`** (`@gfld/xtsb`, MIT, repo: github.com/udevbe/xtsb) — Vendored
TypeScript X11/XCB bindings. Builds from `xcbgen/ts_client.py` against 5
XCB protocol XMLs in `libs/xtsb/protocol/`: `xproto.xml`, `shape.xml`,
`xfixes.xml`, `render.xml`, `composite.xml`. Dual main/module entries:
`dist/main/index.node.js` (Node) + `dist/module/index.web.js` (browser).
Used by BOTH `@gfld/compositor` (browser side,
`packages/compositor/src/remote/xwayland/`) AND `@gfld/compositor-proxy`
(native side, `src/XWaylandSession.ts`) — same TS API on both ends of the
wire. On the native C side,
`packages/compositor-proxy/native/wayland/src/westfield-xwayland.c`
actually launches XWayland and bridges it.
### Wire between browser and native
- **Transport today**: WebSocket over TCP. Browser connects to
`ws(s)://<proxy-host>:<port>/<app-path>?compositorSessionId=<uuid>`.
`compositor-proxy-cli` uses `ws` ^8.17.1 with `perMessageDeflate: false`
(compression off for latency). The `compositorSessionId` is generated by
the browser and used by the CLI to route the upgrade to the correct
per-session child process.
- **Transport future**: KCP-over-WebTransport (reliable UDP). The codebase
already has a full TS KCP implementation in both
`packages/compositor-proxy/src/kcp.ts` (~1100 lines) and
`packages/compositor/src/remote/kcp.ts`. `docs/pages/design/index.md`
explicitly calls out that WebSockets-over-TCP is ill-suited for real-time,
that WebRTC DataChannels' SCTP congestion control is too slow, and that
they're waiting for WebTransport + will pair it with KCP +
forward-error-correction.
- **Out-of-band data**: copy-paste between two proxy instances goes
peer-to-peer (proxy → proxy) without browser round-trip, configured via
`--base-url` so each proxy can advertise itself to peers.
### SDK (`sdk/`) — for browser-resident WASM apps, NOT native apps
The Greenfield SDK lets you compile existing native Wayland C/C++ apps to
WebAssembly and run them directly in the browser compositor (no proxy
needed). Based on Emscripten 3.1.60 with three Greenfield-specific
extensions:
1. `sdk/toolkit/bin/gfcc` and `gf++` — wrapper scripts around `emcc`/`em++`
that inject pthread pool size, asyncify, and the unix-socket fs shim.
2. `sdk/toolkit/bin/wayland-scanner` — symlink to the natively-built
wayland-scanner inside `build-sysroot/bin/wayland-scanner`, used to
generate C client headers/source from xdg-shell.xml during app builds.
3. `sdk/sysrootlibs/` — 34 build scripts that cross-compile a full graphics
stack (wayland, wayland-protocols, pixman, zlib, png, xml2, xkbcommon,
icu, harfbuzz, freetype, fontconfig, pcre2, glib, jpeg, cairo, fribidi,
tiff4, pango, gdk-pixbuf, egl-stub, glesv2-stub, epoxy-emscripten,
graphene, sass, sassc, uapi-stub, upoll, cygepoll, expat, libffi;
gtk4 currently broken) into `sdk/sysroot/` for static linking.
`sdk/build.sh` runs everything end-to-end and writes `meson-gf-toolchain.ini`
(with absolute `_SDK_DIR`). `sdk/sdk_env.sh` exports `PKG_CONFIG_LIBDIR`
(pointing at `sysroot/lib/pkgconfig` + `build-sysroot/lib/pkgconfig`) and
prepends `toolkit/bin` to `PATH`. Meson projects use
`--cross-file toolkit/meson-gf-cross.ini --cross-file toolkit/meson-gf-toolchain.ini`
(host_machine: cpu=mvp, cpu_family=wasm32, system=emscripten).
**Important**: the `sdk/` is **not for native apps**. It's exclusively for
**WASM apps** that run inside the browser via Emscripten. Native Linux apps
on the proxy use the host's own system libraries — the Dockerfile installs
them via `apt-get install libffi8 libgbm1 libgstreamer1.0-plugins-base …
xwayland xauth`.
## Versioning note
Every workspace package is `1.0.0-rc1`. This signals:
1. **Feature freeze for 1.0** — the architecture, protocol bindings, and
public APIs are expected to be stable; no breaking changes anticipated
before 1.0.0 final.
2. **Not yet production-blessed** — known issues remain (e.g. gtk4 WASM is
"broken for now" per `sdk/build.sh` comment; WebTransport/KCP-over-UDP
is designed but not yet active; Linux Kernel Library WASM + JuiceFS web
FS are explicitly future work).
3. **Yarn Berry semantics** — `workspace:^` resolution means all `@gfld/*`
deps are pinned to rc1 across the tree; if the user forks and bumps,
they bump the whole tree consistently.
For the user's purpose (building their own Wayland-to-HTML compositor for an
Odoo 20 + JS/Python web OS), the rc1 status means they should treat
Greenfield as a stable architectural reference but plan to pin to a
specific commit if they fork it, since 1.0.0 final may land breaking-ish
changes before the rc1 → 1.0.0 transition completes.
## License of the source
The four `packages/*` (compositor, compositor-proxy, compositor-proxy-cli,
compositor-shell) are **AGPL-3.0-or-later**. The lower-level `libs/*` are
MIT or Apache-2.0. Anyone forking or building on the runtime packages must
respect AGPL — see `11-pitfalls.md` §License for implications.
+443
View File
@@ -0,0 +1,443 @@
# 01 — Layered Architecture Map
This chapter walks Greenfield's architecture layer by layer, from the lowest
abstraction (Wayland XML) up to the highest (the user-facing shell). Each layer
is described in terms of: **responsibility**, **key abstractions**,
**dependencies**, and **files of interest**. The chapter ends with an ASCII
diagram showing how the layers stack and where data flows between them.
## The eight layers
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Layer 8: USER SHELL (Odoo 20 OWL addon, supplied by user) │
│ Hooks: UserShellApi (events + actions) │
│ ─ NOT covered in this chapter (see 08-app-lifecycle.md) │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 7: DESKTOP / LAYOUT POLICY │
│ Floating vs AlwaysFullScreen, view stacking, focus activation refcount │
│ packages/compositor/src/desktop/Desktop.ts │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 6: SURFACE LIFECYCLE / ROLES │
│ XdgWmBase, XdgSurface, XdgToplevel, XdgPopup, XdgPositioner, Shell, │
│ ShellSurface, Subcompositor, Subsurface, SurfaceRole │
│ packages/compositor/src/{Xdg*, Shell*, Sub*, SurfaceRole, SurfaceChild} │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 5: RENDERING PIPELINE │
│ Surface (state stack), View, Output (WebGL canvas), Renderer, Scene, │
│ Shaders (YUV→RGBA, YUVA→RGBA, RGB+A→RGBA), Loop (RAF), Callback │
│ packages/compositor/src/{Surface, View, Output, Loop, Callback, Region} │
│ packages/compositor/src/render/ │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 4: TRANSPORT (browser side) │
│ Channel (ARQ + Simple), ReconnectingWebSocket, RemoteAppLauncher, │
│ StreamingBuffer, BufferStream, EncodedFrame, EncoderFeedback, decoders │
│ (WebCodecs + WASM fallback) │
│ packages/compositor/src/remote/ │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 4: TRANSPORT (native side) │
│ Channel (ARQ + Simple), SessionController, kcp.ts, NativeAppContext, │
│ FrameFeedback, SurfaceBufferEncoding, wayland-server.ts │
│ packages/compositor-proxy/src/{Channel, kcp, Session*, Native*, ...} │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 3: WAYLAND RUNTIME (browser side) │
│ Connection (wire framing), Client, WlObject registry, marshallers │
│ libs/common/src/Connection.ts │
│ libs/compositor-protocol/src/{westfield-runtime-server.ts, generated/*} │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 3: WAYLAND RUNTIME (native side) │
│ wayland-server.ts + wayland-server-addon.node (N-API → libwayland-server)│
│ MessageInterceptor (per-resource request/event routing) │
│ packages/compositor-proxy/src/{wayland-server.ts, protocol/*_interceptor}│
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 2: PROTOCOL GENERATORS │
│ client-generator, compositor-generator, compositor-proxy-generator │
│ libs/{client,compositor,compositor-proxy}-generator/ │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 1: WAYLAND PROTOCOL XML │
│ protocol/{wayland, xdg-shell, wayland-drm, web-bitmapbuf-unstable-v1}.xml│
└──────────────────────────────────────────────────────────────────────────┘
```
## Layer 1 — Wayland Protocol XML
**Responsibility**: declare the protocol surface — what interfaces, requests,
events, enums, and bitfield types exist on the wire.
**Files of interest**:
- `protocol/wayland.xml` — standard upstream Wayland core. 22 interfaces,
`wl_compositor` v4, `wl_seat` v6, `wl_surface` v4 (roughly Wayland
1.20-ish). Kristian Høgsberg copyright. Used by all 3 generators.
- `protocol/xdg-shell.xml` — stable `xdg_shell` v1, 5 interfaces (xdg_wm_base,
xdg_surface, xdg_toplevel, xdg_popup, xdg_positioner). Used by all 3
generators.
- `protocol/wayland-drm.xml` — legacy `wl_drm` v2, used by native apps (Mesa,
XWayland) to advertise dmabuf buffers. **Only the proxy generator
processes this file** — so the proxy can intercept `create_buffer` /
`create_planar_buffer` / `create_prime_buffer` and turn dmabuf fds into
ProxyFDs for the browser's GStreamer encoding pipeline. Browser-side libs
do not generate `wl_drm` stubs.
- `protocol/web-bitmapbuf-unstable-v1.xml` — **Greenfield's custom
extension**. Single interface `web_bitmapbuf_factory` v1, single request
`create_buffer(id: new_id<wl_buffer>, bitmap: fd)`. Lets browser JS apps
hand a GPU-backed `ImageBitmap` directly to the compositor as a `wl_buffer`
— bypassing SHM copy, encoding, and dmabuf import. The `fd` arg is
marshalled via the same `fileDescriptor(...)` path as
`wl_shm.create_pool`'s fd, but holds an `ImageBitmap` (becomes a
transferable on the flush path). **Not processed by the proxy generator**
— native apps never use it.
**Dependencies**: none. This is the source of truth.
## Layer 2 — Protocol Generators
**Responsibility**: parse the XML with `xml2js`, walk the interface tree,
emit typed TypeScript bindings for each side.
**Key abstractions**: `ProtocolParser.mjs` (shared XML walker) +
`ProtocolArguments.mjs` (per-side argument-type dispatch table).
**Files of interest**:
- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs` —
emits TS `*Proxy` classes for browser apps that want to be Wayland clients
of the browser compositor (via `MessagePort`).
- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
— emits TS `*Resource` + `*Requests` classes for the browser-resident
compositor itself (receives requests, dispatches to `*Requests` impls,
emits events). Skips `wl_display`, `wl_registry` (hand-written in runtime).
- `libs/compositor-proxy-generator/{EndpointProtocolParser,
EndpointProtocolArguments, index}.mjs` — emits CommonJS
`*_interface.js` (wl_interface struct definitions for the N-API addon)
+ `*_interceptor.js` (per-opcode interceptors that decide routing) for
the Node.js native proxy.
**Why three generators?** Because each side has a fundamentally different job:
1. **client-generator** → browser apps that want to be Wayland clients of the
browser compositor (via `MessagePort`).
2. **compositor-generator** → the browser-resident compositor itself (receives
requests, dispatches to `*Requests` impls, emits events).
3. **compositor-proxy-generator** → the Node.js native proxy bridging real
libwayland-client apps to the browser. Emits `wl_interface` struct
definitions for the N-API addon + per-opcode interceptors that decide
routing.
**Dependencies**: `xml2js`, `camelcase`, `uppercamelcase`, `meow`.
## Layer 3 — Wayland Runtime
**Responsibility**: marshal and unmarshal the actual wire bytes. Implement
object lifecycle (creation, dispatch, destruction), new-id tracking, listener
dispatch, file descriptor handling.
**Browser side** (`libs/common/src/Connection.ts`) — the shared runtime for
both `@gfld/client-protocol` and `@gfld/compositor-protocol`:
- Wire format is **standard upstream Wayland**, byte-for-byte. 8-byte
header: `u32 object_id | u16 opcode | u16 size-incl-header`, then per-arg
bytes. `u/i/f/n/o` are 4 bytes; `s` and `a` are length-prefixed u32 +
payload padded to 4-byte alignment; `h` (FD) takes zero bytes — FDs travel
out-of-band in `wireMsg.fds[]`.
- Reading: `wlObject[opcode](message)` — the generated classes use
computed-property-key methods `[0](message) { ... }`, `[1](message) { ... }`
so opcode is the array index.
- Client side: `Proxy.marshallConstructor` calls `display.generateNextId()`
(1-indexed, recycles ids from `wl_display.delete_id`), constructs the new
Proxy (self-registers in constructor), patches the `new_id` arg's `value`,
then flushes.
- Server side: `Client.marshallConstructor` allocates from
`SERVER_OBJECT_ID_BASE = 0xff000000` (or recycles locally). When a
client-allocated id is destroyed, the server sends `wl_display.delete_id`
so the client can recycle.
- Connection registry: `wlObjects: { [id]: WlObject }`; `registerWlObject`
throws on collision.
- `Connection.message` dispatches by literal `wlObject[opcode](message)`.
Each handler returns void or a Promise; `Connection` awaits the promise
before reading the next message — this preserves wire-order while still
allowing async implementations. Per-client Connections mean a slow client
doesn't block others.
**Native side** (`packages/compositor-proxy/src/wayland-server.ts`) — a TS
wrapper around `wayland-server-addon.node` (the C N-API addon that wraps a
**patched** `libwayland-server`):
- `MessageInterceptor` — per-resource request/event dispatch. Each
`wl_*_interceptor.js` file declares `R<opcode>(...args)` and
`E<opcode>(...args)` functions that decide whether a message is routed
`{native: true}` (handle in the proxy), `{browser: true}` (forward to
browser compositor), or both.
- Special cases: `wl_display.sync` → both native and browser (must complete
before `done` event), `wl_display.get_registry` → both, `wl_registry.bind`
→ native if id is in `nativeGlobalNames[]` (e.g. `wl_drm`, `wl_shm`),
else browser. Fast-sync optimization skips browser round trip when no
prior browser-bound request occurred.
**File descriptor handling — the Greenfield innovation**:
`type FD = unknown` — it's intentionally opaque. Browser-side FDs are
**transferable JS objects**:
- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives
in browser, zero-copy transferred via `MessagePort.postMessage(...,
transferList)`).
- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg
(GPU-backed texture, zero-copy).
- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe).
Native-side (proxy) FDs are real POSIX ints.
`NativeWaylandClientSession.onWireMessageEnd` converts each to a JSON
`ProxyFD = {handle, type, host}` (host = proxy's HTTP base URL),
length-prefixed + 4-byte aligned, prepended to the wire message before
sending over WebSocket/KCP. Browser's `webFS` resolves them by fetching
data over HTTP.
**Dependencies**: Layer 2's generated bindings + `@gfld/common`.
## Layer 4 — Transport
**Responsibility**: carry wire bytes between the browser compositor and the
native proxy. Multiplex logical channels. Handle reconnection, backpressure,
and latency.
**Browser side** (`packages/compositor/src/remote/`):
- `Channel.ts` — defines `ARQChannel` (KCP-wrapped, reliable, low-latency)
and `SimpleChannel` (raw WebSocket).
- `ReconnectingWebSocket` — vendored from `pladaria/reconnecting-websocket`
with `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`, `minUptime=5s`,
`connTimeout=4s`, `retries=∞`.
- `RemoteAppLauncher.ts` — sets up the 4 logical channels per app client
(PROTOCOL, FRAME, XWM, FEEDBACK) and routes incoming messages.
- `RemoteInputOutput.ts` — implements the `InputOutput` interface for native
apps, with `mkfifo`, `mkstempMmap`, `readFd`, `writeFdAsStream`, `closeFd`
over separate WebSocket endpoints.
- `StreamingBuffer.ts`, `BufferStream.ts`, `EncodedFrame.ts`,
`EncoderFeedback.ts` — decode pipeline for incoming H.264 / PNG frames.
- `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`,
`H264NALDecoder.worker.ts` — the actual decoders.
**Native side** (`packages/compositor-proxy/src/`):
- `Channel.ts` — mirror image of the browser side. `ARQChannel` and
`SimpleChannel`.
- `kcp.ts` — ~1100-line TypeScript port of the canonical C `ikcp.c`.
- `SessionController.ts` — WS server with 9 endpoint paths:
`/signal`, `/channel`, `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
- `Session.ts`, `SessionProcess.ts`, `SessionController.ts` — process model:
one main process, one forked child per browser session.
- `NativeAppContext.ts` — per-app state, signaling WS, lifecycle (10-min
SIGHUP grace on disconnect, SIGTERM→SIGKILL escalation).
- `FrameFeedback.ts` — receives 4-byte feedback messages from the browser,
uses them to pace `wl_frame.done` callbacks.
- `SurfaceBufferEncoding.ts`, `ProxyBuffer.ts` — encoding pipeline.
- `io/` subdirectory — WebSocket server glue.
- `encoding/` subdirectory — high-level encoding orchestration.
**Wire format**: see `04-transport.md` for the full byte-level breakdown of
PROTOCOL, FRAME, XWM, and FEEDBACK channels.
**Dependencies**: Layer 3 + `ws` (Node) + browser-native WebSocket + `ReconnectingWebSocket`.
## Layer 5 — Rendering Pipeline
**Responsibility**: take incoming buffer frames (SHM or encoded), turn them
into pixels on a browser canvas/WebGL context. Manage surface state, damage
tracking, frame callbacks, and the compositing loop.
**Key abstractions**:
- `Surface.ts` — the `wl_surface` implementation. Double-buffered state
(`state` vs `pendingState`), atomic merge in `commitPending()`. Subsurface
sync mode cascades through the ancestor chain.
- `SurfaceChild.ts`, `SurfaceRole.ts`, `Subsurface.ts`, `Subcompositor.ts` —
surface tree / role system.
- `View.ts` — represents a surface's position on an output, used for
compositing.
- `Output.ts` — the `wl_output` implementation; how output is rendered to a
WebGL canvas.
- `Shm.ts` — `wl_shm` / shm pool handling. How shared memory buffers arrive
in the browser.
- `BufferContents.ts`, `BufferImplementation.ts`, `ImageBitmapBuffer.ts` —
buffer abstraction hierarchy.
- `Region.ts` — `wl_region`, damage regions (pixman-WASM backed).
- `Callback.ts` — frame callback mechanism.
- `Loop.ts` — the compositor main loop / `requestAnimationFrame` integration.
- `Compositor.ts` — the `wl_compositor` global; how it creates surfaces and
regions.
- `Session.ts` — top-level browser-side session.
- `Globals.ts` — how Wayland globals are registered.
- `render/` subdirectory — the WebGL renderer: `Renderer.ts`, `Scene.ts`,
`SceneShader.ts`, `Texture.ts`, `RenderState.ts`, `Program.ts`,
`ShaderCompiler.ts`, `ShaderSources.ts`, `YUVA2RGBA.ts`, `YUVA2RGBAShader.ts`,
`YUV2RGBShader.ts`, `RGBXandAToRGBA.ts`, `RGBandA2RGBAShader.ts`.
- `browser/` subdirectory — browser integration glue (canvas, RAF,
ResizeObserver, dnd, selection, input).
- `math/` subdirectory — matrix/region math (`Mat4.ts`, `Rect.ts`).
**Pipeline**: see `03-html5-rendering.md` for the full
commit → damage → render → present pipeline.
**Dependencies**: Layer 4 + WebGL + WebCodecs + `@gfld/compositor-wasm` (pixman + xkbcommon).
## Layer 6 — Surface Lifecycle / Roles
**Responsibility**: implement the role-specific surface state machines
(xdg-toplevel, xdg-popup, xdg-positioner, wl_shell_surface, subsurface).
Manage configure/ack-configure round-trips, popup grabs, window state.
**Files of interest**:
- `XdgWmBase.ts` — the `xdg_wm_base` global. How clients advertise they're
XDG clients. Includes ping-based unresponsiveness detection.
- `XdgSurface.ts` — `xdg_surface`. The role wrapper around a `wl_surface`.
- `XdgToplevel.ts` — `xdg_toplevel`. Top-level windows. State (maximized,
fullscreen, minimized, activated, resizing). Configure/ack-configure
round-trip.
- `XdgPopup.ts` — `xdg_popup`. Transient windows, menus, tooltips. Grab
semantics.
- `XdgPositioner.ts` — `xdg_positioner`. How popups are positioned relative
to parent.
- `Shell.ts`, `ShellSurface.ts` — the legacy `wl_shell` (deprecated but still
implemented for old clients).
- `Subcompositor.ts`, `Subsurface.ts`, `SurfaceChild.ts`, `SurfaceRole.ts`
— subsurface compositing, the surface tree.
**State machines**: see `08-app-lifecycle.md` for the full
XdgToplevel lifecycle, configure/ack round-trip semantics, popup
positioner math, and subsurface sync/desync mode.
**Dependencies**: Layer 5 + Layer 3 (Wayland runtime).
## Layer 7 — Desktop / Layout Policy
**Responsibility**: arrange surfaces on the output. Decide stacking order,
focus policy, and the high-level window management mode (Floating vs
AlwaysFullScreen).
**Files of interest**:
- `desktop/Desktop.ts` — implements both `FloatingDesktopSurface` and
`AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`.
Floating lets the user move/resize windows freely; AlwaysFullScreen
enforces one window per output (kiosk mode).
**Dependencies**: Layer 6 + Layer 5 (for View rendering).
## Layer 8 — User Shell
**Responsibility**: the actual user-facing UI. The taskbar, window
decorations, app launcher, keyboard layout picker, etc.
**Out of scope for the user** — they will build their own shell as an Odoo 20
OWL addon. The integration point is `UserShellApi.ts` (see
`08-app-lifecycle.md`).
**Greenfield's reference shell** (`packages/compositor-shell`) is a Preact +
Tailwind SPA, 562 lines of TypeScript total. Mentioned here only for
orientation; not analyzed in this bundle.
## How data flows across the layers
```
NATIVE APP (e.g. gtk4-demo, X11 via XWayland)
│
│ libwayland-client → UNIX socket + SCM_RIGHTS FDs
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ NATIVE PROXY (Node.js + C N-API addons) │
│ │
│ Layer 3 (native): wayland-server.ts + wayland-server-addon.node │
│ ↳ patched libwayland-server parses wire bytes │
│ ↳ MessageInterceptor routes each request to {native, browser} │
│ ↳ FDs wrapped as ProxyFD JSON │
│ │
│ Layer 4 (native): Channel.ts + kcp.ts + SessionController │
│ ↳ 4 logical channels: PROTOCOL, FRAME, XWM, FEEDBACK │
│ ↳ KCP-over-WebSocket for ARQ channels │
│ ↳ SimpleChannel for FEEDBACK │
│ │
│ Layer 5 (encoding): SurfaceBufferEncoding + proxy-encoding-addon │
│ ↳ GStreamer H.264 (x264 / nvh264 / vaapih264) on dedicated thread │
│ ↳ PNG fallback for ≤256×256 surfaces │
│ ↳ Zero-copy dmabuf via EGLImage → GstGLMemoryEGL │
│ │
│ Layer 1 (XML) → Layer 2 (proxy-generator) feeds Layer 3 interceptors │
└─────────────────────────────────────────────────────────────────────────┘
│
│ WebSocket (TCP) + KCP-over-WebSocket
│ 9 endpoint paths per session
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ BROWSER COMPOSITOR (TypeScript + WebGL + WebCodecs + WASM) │
│ │
│ Layer 4 (browser): Channel.ts + RemoteAppLauncher + decoders │
│ ↳ ReconnectingWebSocket with exponential backoff │
│ ↳ ARQChannel + SimpleChannel mirror the native side │
│ ↳ WebCodecs VideoDecoder preferred, FFmpeg WASM fallback │
│ ↳ H264NALDecoder.worker.ts (2 workers: opaque + alpha) │
│ │
│ Layer 3 (browser): @gfld/common Connection + compositor-protocol │
│ ↳ Standard Wayland wire format, byte-for-byte │
│ ↳ WlObject registry, opcode dispatch via wlObject[opcode](msg) │
│ ↳ Per-client Connections (slow client never blocks another) │
│ │
│ Layer 5: Surface.ts + View.ts + Output.ts + render/ │
│ ↳ Double-buffered state (state vs pendingState) │
│ ↳ Damage tracking via pixman-WASM regions │
│ ↳ WebGL1 with FBOs for YUV→RGBA, RGB+Alpha→RGBA conversion │
│ ↳ desynchronized:true canvas for low-latency presentation │
│ ↳ Frame callbacks fired on requestAnimationFrame vsync │
│ │
│ Layer 6: XdgWmBase + XdgSurface + XdgToplevel + XdgPopup + ... │
│ ↳ Configure/ack-configure round-trips │
│ ↳ Popup positioner math (anchor + gravity + constraints) │
│ ↳ Subsurface sync/desync mode cascading │
│ │
│ Layer 7: desktop/Desktop.ts │
│ ↳ Floating vs AlwaysFullScreen mode │
│ ↳ View stacking, focus refcount │
│ │
│ Layer 8: UserShellApi (events + actions) │
│ ↳ Events: clientCreated, surfaceCreated, surfaceTitleUpdated, │
│ surfaceActivationUpdated, ... │
│ ↳ Actions: initScene, refreshScene, destroyScene, │
│ closeClient, activateSurface, setUserConfiguration │
│ ↳ THE INTEGRATION SEAM — Odoo 20 OWL shell hooks here │
└─────────────────────────────────────────────────────────────────────────┘
│
│ UserShellApi.events.* fired by compositor
│ UserShellApi.actions.* called by shell
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ USER SHELL (Odoo 20 OWL addon — supplied by user) │
│ │
│ - Application launcher (lists apps from --applications JSON) │
│ - Window management UI (taskbar, decorations, minimize/maximize) │
│ - Keyboard layout picker (RMLVO) │
│ - Per-output canvas (HTMLCanvasElement, one per Output) │
│ - Trigger app launches via fetch('http://proxy:8081/<path>') │
│ - Open signalURL WebSocket to receive CONNECT_CHANNEL messages │
│ │
│ Not analyzed in this bundle (out of scope). │
└─────────────────────────────────────────────────────────────────────────┘
```
## Reading guide for the rest of the bundle
Now that you've seen the layer map, the remaining chapters drill into each
layer in detail:
- `02-wayland-protocol.md` — Layers 1, 2, 3 (XML → generators → runtime)
- `03-html5-rendering.md` — Layer 5 (rendering pipeline)
- `04-transport.md` — Layer 4 (transport, both sides)
- `05-input-handling.md` — Layer 5 input subsystem (cross-cuts Layers 3, 4, 5)
- `06-data-clipboard.md` — Layer 5 data device (cross-cuts Layers 3, 4, 5)
- `07-native-app-bridge.md` — Layer 4 native + Layer 3 native + Layer 1 XML native (proxy-only `wl_drm`)
- `08-app-lifecycle.md` — Layer 6 + Layer 8 (UserShellApi seam)
- `09-hardware-accel.md` — Layer 5 encoding + Layer 4 frame channel + Layer 3 WASM
- `10-porting-to-odoo-js-python.md` — how to rebuild all 8 layers in your stack
- `11-pitfalls.md` — what Greenfield does wrong, layer by layer
- `12-appendix.md` — glossary, file index, external references
+325
View File
@@ -0,0 +1,325 @@
# 02 — Wayland Protocol Layer
This chapter covers Greenfield's protocol layer in depth: how Wayland XML
protocol definitions get parsed, how typed TypeScript stubs get generated for
each of the three sides (browser client, browser compositor, native proxy),
how wire messages are marshalled/unmarshalled at runtime, how objects
lifecycle is managed, and how the FD-over-WebSocket problem is solved.
## The three generators
Greenfield ships **three sibling generator CLIs** that consume the same
Wayland XML files but emit slightly different TypeScript flavors:
| | `client-generator` | `compositor-generator` | `compositor-proxy-generator` |
|---|---|---|---|
| Output | TS `*Proxy` classes (browser) | TS `*Resource` + `*Requests` (Node) | CommonJS `*_interface.js` + `*_interceptor.js` (Node + N-API) |
| Listener pattern | `listener?: <Itf>Events` (optional, duck-typed, async) | `implementation: <Itf>Requests` (required) | `requestHandlers` / `eventHandlers` maps |
| Skips | none | `wl_display`, `wl_registry` (hand-written in runtime) | none — proxy must intercept display/registry/bind |
| Arg-type table | `ProtocolArguments.mjs` returns `{signature, jsType, marshallGen}` | Same shape, uses `Westfield.${resourceName}` for objects | `EndpointProtocolArguments.mjs` returns plain wayland sig letters (`u`, `?s`, `n`, `h`, ...) |
| XMLs consumed | `wayland.xml` + `xdg-shell.xml` + `web-bitmapbuf-unstable-v1.xml` | same 3 | `wayland.xml` + `xdg-shell.xml` + `wayland-drm.xml` |
All three share `xml2js` + `camelcase` + `uppercamelcase` and a near-identical
`ProtocolParser.mjs`. The XML → JSON → walk `interface[]`/`request[]`/
`event[]`/`enum[]` → per-arg type-dispatch pipeline is identical.
### Why three generators?
Because each side has a fundamentally different job:
1. **`client-generator`** → browser apps that want to be Wayland clients of
the browser compositor (via `MessagePort`). These are the same-origin
iframe JS/WASM apps that ship in `examples/webapps/`.
2. **`compositor-generator`** → the browser-resident compositor itself
(`@gfld/compositor`). It receives requests, dispatches them to
`*Requests` implementation classes, and emits events back.
3. **`compositor-proxy-generator`** → the Node.js native proxy bridging real
`libwayland-client` apps to the browser. Emits `wl_interface` struct
definitions for the N-API addon + per-opcode interceptors that decide
routing.
## Wire format (`libs/common/src/Connection.ts`)
**Standard upstream Wayland wire format, byte-for-byte.** No custom binary
encoding at the protocol layer.
```
┌─────────────┬──────────┬──────────┬─────────────────────────────────────────────┐
│ object id │ opcode │ size │ args (variable per arg type, see §4) │
│ uint32 │ uint16 │ uint16 │ u,i,f,n,o = 4 bytes; s,a = 4+len+pad; │
│ (4 bytes) │ (2 B) │ (2 B) │ h = 0 bytes, FD goes in side-band fds[] │
└─────────────┴──────────┴──────────┴─────────────────────────────────────────────┘
▲
└── size includes this 8-byte header.
Total message size = size field.
```
### Per-arg marshalling table
| Arg signature letter | Type | Size on wire | Notes |
|---|---|---|---|
| `u` | uint32 | 4 bytes | unsigned integer |
| `i` | int32 | 4 bytes | signed integer |
| `f` | fixed | 4 bytes | wl_fixed_t (24.8 signed) |
| `s` | string | 4 + len + pad | length-prefixed u32, padded to 4-byte alignment |
| `a` | array | 4 + len + pad | length-prefixed u32, padded to 4-byte alignment |
| `o` | object | 4 bytes | object id (0 means null) |
| `n` | new_id | 4 bytes | newly-allocated object id |
| `h` | file descriptor | 0 bytes | FD goes in side-band `wireMsg.fds[]` |
### Reading
`Connection.message` dispatches by literal `wlObject[opcode](message)`. The
generated classes use computed-property-key methods:
`[0](message) { ... }`, `[1](message) { ... }` so opcode is the array index.
Each handler returns void or a Promise; `Connection` awaits the promise
before reading the next message — this preserves wire-order while still
allowing async implementations. Per-client Connections mean a slow client
doesn't block others.
## Object lifecycle & new-id tracking
### Client side
`Proxy.marshallConstructor` calls `display.generateNextId()` (1-indexed,
recycles ids from `wl_display.delete_id`), constructs the new Proxy
(self-registers in constructor), patches the `new_id` arg's `value` (it was
`0` placeholder from `newObject()`), then flushes.
### Server side
`Client.marshallConstructor` allocates from `SERVER_OBJECT_ID_BASE =
0xff000000` (or recycles locally). When a client-allocated id is destroyed,
the server sends `wl_display.delete_id` so the client can recycle. On
`Client.close()`, objects are destroyed in **descending id order** (children
first).
### Connection registry
`wlObjects: { [id]: WlObject }`; `registerWlObject` throws on collision.
## Listener dispatch
`Connection.message` dispatches by literal `wlObject[opcode](message)`. Each
handler returns void or a Promise; `Connection` awaits the promise before
reading the next message — this preserves wire-order while still allowing
async implementations. Per-client Connections mean a slow client doesn't
block others.
## File descriptor handling — the Greenfield innovation
`type FD = unknown` — it's intentionally opaque. The actual FD
representation depends on which link it's traversing.
### Browser ↔ Browser (in-iframe, via `MessagePort`)
FDs are transferable JS objects:
- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives
in browser, zero-copy transferred via `MessagePort.postMessage(...,
transferList)`).
- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg
(GPU-backed texture, zero-copy).
- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe).
### Browser ↔ Proxy (over WebSocket/KCP)
FDs are virtualized as JSON `ProxyFD` descriptors:
```
ProxyFD = { handle: number, type: 'pipe-read' | 'pipe-write' | 'shm' | 'unknown', host: string }
```
Where:
- `handle` is the proxy's internal numeric fd id
- `type` is the kind of fd (so the browser knows what to do with it)
- `host` is the proxy's HTTP base URL (for proxy-to-proxy FD relay)
`NativeWaylandClientSession.onWireMessageEnd` converts each POSIX fd to a
ProxyFD JSON, length-prefixed + 4-byte aligned, prepended to the wire
message before sending over WebSocket/KCP. The browser's `webFS` resolves
them by fetching data over a separate ad-hoc WebSocket endpoint
(`/read-fd`, `/write-fd-as-stream`, etc.).
### Native App ↔ Proxy (libwayland socket)
FDs are real POSIX ints, sent via `SCM_RIGHTS` by libwayland-server.
### Most wl_buffer FDs never need to be read
Because the proxy intercepts `wl_surface.commit`, reads the buffer locally
via DRM/SHM, encodes to H.264, and ships only the encoded bytes — the
browser never needs to read the original FD.
## How the proxy translates between native and browser
**It doesn't translate wire bytes** — both sides speak identical Wayland
wire format. What it translates is:
- **FDs**: POSIX int ↔ `ProxyFD` JSON.
- **Object-id space**: native libwayland's id range vs browser's
`0xff000000`+ server range.
- **Routing per message** via `MessageInterceptor.interceptRequest` /
`interceptEvent`. Special cases:
- `wl_display.sync` → both native and browser (must complete before `done`
event)
- `wl_display.get_registry` → both
- `wl_registry.bind` → native if id is in `nativeGlobalNames[]` (e.g.
`wl_drm`, `wl_shm`), else browser
- Fast-sync optimization skips browser round trip when no prior
browser-bound request occurred
## The four protocol XMLs
### `protocol/wayland.xml`
Standard upstream Wayland core. 22 interfaces, `wl_compositor` v4, `wl_seat`
v6, `wl_surface` v4 (roughly Wayland 1.20-ish). Kristian Høgsberg copyright.
Used by all 3 generators.
### `protocol/xdg-shell.xml`
Stable `xdg_shell` v1, 5 interfaces: `xdg_wm_base`, `xdg_surface`,
`xdg_toplevel`, `xdg_popup`, `xdg_positioner`. Used by all 3 generators.
### `protocol/wayland-drm.xml`
Legacy `wl_drm` v2, used by native apps (Mesa, XWayland) to advertise dmabuf
buffers. **Only the proxy generator processes this file** — so the proxy can
intercept `create_buffer` / `create_planar_buffer` / `create_prime_buffer`
and turn dmabuf fds into ProxyFDs for the browser's GStreamer encoding
pipeline. Browser-side libs do not generate `wl_drm` stubs.
### `protocol/web-bitmapbuf-unstable-v1.xml` — Greenfield's custom extension
This is **Greenfield's own Wayland protocol extension** for in-browser
JS/WASM clients. It defines a single singleton interface:
- **`web_bitmapbuf_factory` (v1)** with one request
`create_buffer(new_id wl_buffer id, fd bitmap)` — "Create a wl_buffer
object by wrapping an HTML bitmap, so it can be used with a surface."
The `fd` argument is an HTML bitmap file descriptor (a Greenfield-specific
construct: an `ImageBitmap`-like object exposed as an FD-like handle). This
lets JavaScript or WebAssembly Wayland clients back a surface with a native
browser bitmap (`ImageBitmap`, `OffscreenCanvas` transferable) and zero-copy
it into the compositor's WebGL texture pipeline — bypassing the SHM
pixel-copy path.
The browser-side factory lives in
`packages/compositor/src/web/WebBuffersFactory.ts`. **Not processed by the
proxy generator** — native apps never use it.
This is a trade-off worth noting: the fast path is Greenfield-specific. To
get zero-copy `ImageBitmap` upload to a GPU texture, apps must be
Greenfield-aware (i.e. built against `@gfld/client-protocol`). Standard
Wayland apps fall back to the H.264-encoded path even when running in the
browser compositor.
## ASCII pipeline diagram
```
┌──────────────────────────────────────────────────────────────────────────────┐
│ PROTOCOL XML SOURCES │
│ wayland.xml xdg-shell.xml wayland-drm.xml web-bitmapbuf-unstable-v1 │
└──────────┬──────────────┬──────────────┬─────────────────┬─────────────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌───────────────────┐ ┌────────────────────┐
│ client- │ │ compositor- │ │ compositor-proxy- │ │ (no proxy gen for │
│ generator │ │ generator │ │ generator │ │ web-bitmapbuf) │
│ ProtocolParser │ │ ProtocolParser │ │ EndpointProtocol │ │ │
│ .mjs │ │ .mjs │ │ Parser.mjs │ │ │
└───────┬────────┘ └───────┬────────┘ └─────────┬─────────┘ └─────────┬──────────┘
│ xml2js │ xml2js │ xml2js │
▼ ▼ ▼ ▼
┌────────────────┐ ┌────────────────┐ ┌─────────────────────┐ ┌────────────────────┐
│ wayland.ts │ │ wayland.ts │ │ *_interface.js │ │ wayland.ts │
│ xdg_shell.ts │ │ xdg_shell.ts │ │ *_interceptor.js │ │ xdg_shell.ts │
│ web_bitmapbuf.ts│ │ web_bitmapbuf.ts│ │ (wayland,xdg,drm) │ │ web_bitmapbuf.ts │
│ *Proxy classes │ │ *Resource + │ │ wl_interface structs│ │ *Proxy classes │
│ *Events ifaces │ │ *Requests │ │ + R*/E* handlers │ │ *Events interfaces │
│ *ProtocolName │ │ *ProtocolName │ │ + requestHandlers │ │ *ProtocolName │
│ *Enum │ │ *Enum │ │ /eventHandlers │ │ *Enum │
└───────┬─────────┘ └───────┬─────────┘ └──────────┬──────────┘ └─────────┬──────────┘
│ imports │ imports │ require()'d │ imports
▼ ▼ ▼ ▼
┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐
│ @gfld/common │ │ packages/compositor-proxy/ │
│ Connection.marshallMsg() ── writes │ │ src/wayland-server.ts │
│ u32 id | u16 opcode | u16 size | args │ │ ↳ re-exports wayland-server-addon │
│ Connection.message() ── parses header, │ │ ↳ MessageInterceptor (R*/E* dispatch) │
│ dispatches wlObject[opcode](message) │ │ ↳ unmarshallArgs(message, 'usun') etc │
│ Per-arg marshallers: uint/int/fixed/ │ │ NativeWaylandClientSession.ts │
│ string/array/object/objectOptional/ │ │ ↳ onWireMessageRequest → interceptRequest │
│ newObject/fileDescriptor │ │ ↳ onWireMessageEnd → pack ProxyFD JSON │
│ Per-arg unmarshallers: u/i/f/s/a/o/n/h │ │ ↳ onWireMessageEvents → sendEvents (C) │
└──────────────┬───────────────────────────┘ └─────────────────────┬────────────────────────┘
│ │
▼ ▼
┌────────────────────────────────────────┐ ┌────────────────────────────────────────────────────┐
│ Browser (client iframe) │ │ Node (compositor-proxy) │
│ │ │ │
│ DisplayImpl.messagePort.onmessage ←───┼─────────┼─ protocolChannel (WS/KCP) sends/receives │
│ connection.message({buffer, fds}) │ │ [u32 oob-op][u32 lastSerial][u32 fdCount] ... │
│ wlObject[opcode](message) │ │ [u32 len][ProxyFD JSON, padded][wayland msg bytes]│
│ listener?.<eventName>(...unmarshalled) │ │ │
│ │ │ ← from native: wl_client wire callback (C addon) │
│ proxy.marshall(id, opcode, [args]) │ │ → to native: sendEvents(wlClient, buf, fds) (C) │
│ connection.marshallMsg writes Uint32 │ │ │
│ onFlush → postMessage(buf, transfer) │ │ Native libwayland-server │
│ ↳ buffer + ArrayBufferView/ImageBitmap│ │ accepts socket from native wayland apps │
│ /MessagePort as transferables │ │ parses wire bytes, dispatches to interceptors │
└────────────────────────────────────────┘ └────────────────────────────────────────────────────┘
WIRE FORMAT (identical on all links — standard Wayland):
┌─────────────┬──────────┬──────────┬─────────────────────────────────────────────┐
│ object id │ opcode │ size │ args (variable per arg type, see §4) │
│ uint32 │ uint16 │ uint16 │ u,i,f,n,o = 4 bytes; s,a = 4+len+pad; │
│ (4 bytes) │ (2 B) │ (2 B) │ h = 0 bytes, FD goes in side-band fds[] │
└─────────────┴──────────┴──────────┴─────────────────────────────────────────────┘
▲
└── size includes this 8-byte header. Total message size = size field.
FD transport varies per link:
Browser↔Browser (MessagePort): FD is a transferable JS object (ArrayBufferView, ImageBitmap, MessagePort).
Browser↔Proxy (WebSocket/KCP): FD is a JSON ProxyFD {handle, type, host}, length-prefixed + 4-byte aligned, prepended before the message header.
Native App↔Proxy (libwayland socket): FD is a real POSIX fd, sent via SCM_RIGHTS by libwayland-server.
```
## Build-time workflow
If you edit a protocol XML in your fork, you regenerate stubs by running
the matching `generate` script:
- `yarn workspace @gfld/client-protocol generate` — regenerates browser
client stubs.
- `yarn workspace @gfld/compositor-protocol generate` — regenerates browser
compositor stubs.
- `yarn workspace @gfld/compositor-proxy generate` — regenerates proxy
interceptor modules, then `build:native` (because `*_interface.js`
declares `wl_interface` structs the N-API addon consumes).
To add a new Greenfield-only extension: drop XML in `/protocol/`, add a
`generate:<name>` script to `client-protocol` / `compositor-protocol`
`package.json`, regenerate. Only add to `compositor-proxy` if native apps
will speak it.
## Notable file references
- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
- `libs/compositor-proxy-generator/src/{EndpointProtocolParser, EndpointProtocolArguments, index}.mjs`
- `libs/common/src/Connection.ts` — the runtime wire codec (shared)
- `libs/compositor-protocol/src/westfield-runtime-server.ts` — server-side runtime
- `libs/client-protocol/src/westfield-runtime-client.ts` — client-side runtime
- `packages/compositor-proxy/src/wayland-server.ts` — proxy's TS wrapper around the N-API addon
- `packages/compositor-proxy/src/NativeWaylandClientSession.ts` — per-client message interceptor dispatch
- `packages/compositor-proxy/src/protocol/wl_*_interceptor.js` — generated per-opcode routing decisions
- `packages/compositor/src/web/WebBuffersFactory.ts` — browser-side implementation of the custom `web_bitmapbuf_factory` global
## One stale file spotted
`libs/compositor-protocol/index.js` (CommonJS, references non-existent
`./src/Resource` / `./src/Client` / `./src/Display` / `./src/Global`). The
actual entry point is `src/index.ts`. Safe to delete in a fork.
+379
View File
@@ -0,0 +1,379 @@
# 03 — HTML5 Rendering Pipeline
This chapter covers Greenfield's browser-side rendering pipeline: how
incoming Wayland buffers (SHM or encoded) end up as pixels on a browser
canvas/WebGL context, including damage tracking, commit semantics, surface
state stacking, and the compositing loop. The browser side lives in
`packages/compositor/src/`.
## Three buffer types, one BufferImplementation interface
All incoming buffers, regardless of origin, end up as a `BufferContents<T>`
with a `mimeType` that dispatches to the right upload path. There are three
concrete buffer types:
### 1. SHM buffer (`LazyImageBitmapBuffer`)
Comes from `wl_shm_pool` — the standard Wayland shared-memory buffer
mechanism. Browser-side implementation lives in `Shm.ts`. The raw pixel data
arrives as a transferable `ArrayBufferView` (transferred zero-copy via
`MessagePort` from a browser-iframe web app, or fetched via the FD endpoint
WebSocket from the proxy). Greenfield wraps the bytes as an `ImageData` /
BMP-Blob and decodes them via `createImageBitmap()` to produce an
`ImageBitmap` ready for WebGL `texImage2D`.
**Notable hack**: for `xrgb8888` SHM buffers, Greenfield prepends a BMP
header in front of the raw pixel data so the browser's `createImageBitmap`
decodes it natively, avoiding a per-pixel JS loop. This is a clever trick to
offload pixel conversion to the browser's BMP decoder.
### 2. Web bitmap buffer (`ImageBitmapBuffer` via `web_bitmapbuf_factory`)
Comes from Greenfield's custom `web-bitmapbuf-unstable-v1` protocol
extension. Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to
the compositor as a `wl_buffer` — bypassing SHM copy, encoding, and dmabuf
import. The `fd` arg is marshalled as a transferable, so the GPU texture
never leaves VRAM.
### 3. Remote streaming buffer (`StreamingBuffer`)
Comes from the native proxy via the FRAME data channel. Receives encoded
frames (H.264 or PNG) on a separate WebSocket per surface. Includes an
out-of-band mechanism for `BufferCreation` and `CommitSerial` opcodes so
the browser can match each encoded frame to the right `wl_surface.commit`
serial.
## Surface state stack
`Surface.ts` implements classic Wayland double-buffered state:
- `Surface.state` — the committed, currently-visible state.
- `Surface.pendingState` — the in-flight state, accumulated by client
requests until `commit` is called.
- `mergeSurfaceState` — the atomic swap in `commitPending()`.
Subsurfaces add a third layer of complexity:
- `subsurfaceChildren` arrays (committed + pending, bottom→top)
- `placeAbove` / `placeBelow` reorder pending
- `effectiveSync` getter cascades through the ancestor chain (if any
ancestor is sync, this subsurface is effectively sync)
- Sync subsurfaces cache to `cachedState` and only flush when an ancestor
commits via `onParentCommit`
- `setDesync` immediately calls `onCommit` to flush any pending cache
## Commit semantics
When a Wayland client calls `wl_surface.commit`:
1. If the surface has a remote streaming buffer, Greenfield pulls the
`serial = commitSerials.shift()` (an out-of-band opcode 7 mechanism that
lets `StreamingBuffer.getContents(surface, serial)` wait for the right
encoded frame).
2. `bufferContents = await bufferImpl.getContents(surface, serial)` —
resolves when the encoded frame for this serial has arrived.
3. `role.onCommit(this)` — role-specific logic (XdgToplevel, XdgPopup,
Subsurface, etc.).
4. `Surface.commitPending()` runs:
- Commit `subsurfaceChildren` order + apply pendingPosition → cascade
`onParentCommit`.
- Calculate derived pending state (buffer transform matrix, surface size,
pixman region).
- Release old buffer if replaced.
- `mergeSurfaceState(state, pendingState)` — atomic swap.
- `damaged = true`; clear pending damage + frame callbacks.
## Damage tracking
`damageRects` (post-transform) and `bufferDamageRects` (buffer-space) are
accumulated correctly on the surface state via pixman-WASM region math. The
damage is tracked faithfully — but **vestigial at the GL level**. The
renderer doesn't consume it for scissoring — it repaints every visible view
every RAF. The only damage gate is the boolean `surface.damaged`, used to
decide whether a streaming buffer's frame needs re-upload + release.
`RenderState.visibleSceneRegion` (pixman) is used only to decide if a
`RenderState` exists at all (i.e. whether the view is on this output), not
for per-rect clipping.
## Compositing pipeline
The renderer is **WebGL1 only**, no OffscreenCanvas, no WebGL2. The
context is acquired with `getContext('webgl', {antialias:false, depth:false,
alpha:true, premultipliedAlpha:false, preserveDrawingBuffer:false,
desynchronized:true})`. FBOs are used for in-GPU YUV→RGBA and
RGB+Alpha→RGBA conversion into per-view textures.
The compositor runs in **four phases** per render pass:
### PHASE 1 — pixel upload (per view per scene)
Coalesced via `setTimeout`-based cancellable microtask. For each view, for
each scene it's visible on:
1. `view.applyTransformations()` — compute `parent.transformation *
translate(position+offset)`.
2. For each `renderState` in `view.renderStates`:
`scene[bufferContents.mimeType](bufferContents, renderState)` — dispatch
to the right upload path:
- `'image/bitmap'` → `texture.setContent(imageBitmap, size)` — direct
`texImage2D` / `texSubImage2D` upload.
- `'video/h264'` → `YUVA2RGBA` or `RGBXandA2RGBA.convertInto(...)` —
FBO pass that uploads Y/U/V/alpha-Y as 4× LUMINANCE textures, runs the
YUVA→RGBA shader (BT.601 coefficients), writes into the renderState's
RGBA texture.
- `'image/png'` → `texture.setContent(bitmap, size); bitmap.close()`.
3. Collect `surface.state.frameCallbacks` → `Renderer.frameCallbacks`.
4. `surface.damaged = false`; `streamingBuffer.release()` (remote only).
### PHASE 2 — schedule render frame
`Renderer.render()` coalesces via a `setTimeout`-based cancellable
microtask, then schedules a single `requestAnimationFrame` for PHASE 3.
### PHASE 3 — draw (per scene, inside RAF)
For each scene (one per canvas/output):
1. `sceneShader.use()`; `updateSceneData(canvas)` — viewport, clear, BLEND,
projection.
2. For each view in the view stack (flattened from `topLevelViews`
recursively via `Renderer.updateViewStack`):
- If `renderState && view.mapped`:
- `sceneShader.updateViewData(view, renderState)` — bind texture,
`u_transform`, quad buffer.
- `sceneShader.draw()` — `gl.drawArrays(TRIANGLE_STRIP, 0, 6)` of a
textured quad per scene.
### PHASE 4 — fire frame callbacks
For each callback in `Renderer.frameCallbacks`:
`callback.done(time)` → `wl_callback.done(time)`; `wl_callback.destroy()`.
Then `session.flush()` to send any pending wire messages.
The `time` value is the `requestAnimationFrame` high-resolution timestamp,
so apps see real vsync-aligned frame timing.
## Multi-output
Multi-output = one Scene per canvas. Each view has one `RenderState` (and
one WebGL texture) per scene it's visible on. So a surface visible on two
outputs (e.g. mirroring) has two textures — one per canvas — and gets
uploaded to each on PHASE 1.
## Subsurfaces stacking
Subsurfaces stack via `subsurfaceChildren` arrays (committed + pending,
bottom→top). `placeAbove` / `placeBelow` reorder pending;
`Renderer.updateViewStack` flattens `topLevelViews` recursively.
**Important**: no parent-geometry clipping — subsurfaces can render outside
their parent. The only clip is the scene's canvas region, enforced implicitly
by NDC mapping.
## Cursor and DnD bypass GL entirely
Cursor and DnD bypass GL entirely. They draw to a separate singleton 2D
canvas and use CSS `cursor: url(dataURL)` or HTML5
`dataTransfer.setDragImage`. Their frame callbacks fire immediately with
`Date.now()`, not synced to RAF.
This is a deliberate optimization: cursor updates must be low-latency, and
going through the full GL pipeline would add a frame of latency.
## Decode pipeline (for H.264 frames)
Decoding is delegated to:
### Preferred path: WebCodecs `VideoDecoder`
At session startup, `Session.create()` probes
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
hardwareAcceleration:'prefer-software'})`. If supported, an `opaqueDecoder`
and `alphaDecoder` are created per surface (lazily, on first H.264 frame).
Encoded NALs are wrapped as `EncodedVideoChunk {type:key|delta, data:nal}`
and fed to the decoder. The output `VideoFrame` is uploaded directly to a
WebGL texture.
**Important**: hardware decode is **explicitly disabled** in `Session.ts`
with a FIXME comment "hardware decoding frame offset is wrong". Only
software decoding is used today.
### Fallback path: FFmpeg Wasm in Web Workers
If WebCodecs is unavailable, two persistent Workers (`h264NALDecoderOpaque`,
`h264NALDecoderAlpha`) are created at session startup. Each worker has one
`AVCodecContext` per surface, keyed by `surfaceH264DecodeId`. Encoded NALs
are transferred zero-copy into the worker via
`worker.postMessage(nal, [nal.buffer])`. Decoded I420 planes are transferred
back as `[y.buffer, u.buffer, v.buffer]` and uploaded as 4× LUMINANCE
textures.
## PNG path (both decoders)
The PNG path is identical in both branches:
`new Blob([opaque], {type:'image/png'})` →
`createImageBitmap(blob, sx, sy, w, h)` — the `sx/sy/w/h` crops the
encoder padding back to logical size. PNG is always used for surfaces ≤
256×256 pixels because H.264 I-frame overhead dominates at small sizes.
## WebGL shaders
Three shaders in `render/`:
- `YUVA2RGBAShader` — converts YUVA (Y, U, V, alpha-Y as 4 LUMINANCE
textures) to RGBA in a single FBO pass using BT.601 coefficients. Alpha
is recovered by running the alpha-Y plane through the same YUV→RGB matrix
and taking the R channel.
- `RGBandA2RGBAShader` — for the `DualPlaneRGBA*` WebCodecs outputs
(opaque + alpha planes composited into RGBA).
- `SceneShader` — final pass: draws every view's RGBA texture to the canvas
with `gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`. Note: expects premultiplied
alpha but the YUVA shader outputs straight alpha — a minor inconsistency
that works because Wayland surfaces rarely use partial alpha.
## ASCII pipeline diagram
```
CLIENT (Wayland app / web worker / compositor-proxy)
│ wl_surface.attach / damage / frame / commit
▼
Surface.commit()
│ serial = commitSerials.shift() (remote-only)
│ bufferContents = await bufferImpl.getContents(surface, serial)
│ role.onCommit(this)
▼
Surface.commitPending()
│ 1. commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit
│ 2. calculateDerivedPendingState (bufferTransform matrix, surface.size, pixmanRegion)
│ 3. release old buffer if replaced
│ 4. mergeSurfaceState(state, pendingState) ◄── atomic swap
│ 5. damaged = true; clear pending damage/frame callbacks
▼
Renderer.render()
│ coalesce via queueCancellableMicrotask (setTimeout)
▼
PHASE 1 — pixel upload (per view per scene):
view.applyTransformations() // parent.transformation * translate(position+offset)
for renderState in view.renderStates:
scene[bufferContents.mimeType](bufferContents, renderState)
'image/bitmap' → texture.setContent(imageBitmap, size) // texImage2D / texSubImage2D
'video/h264' → YUVA2RGBA / RGBXandA2RGBA.convertInto(...) // 4 LUMINANCE or 2 RGBA tex
→ FBO + YUVA2RGBAShader (BT.601) → renderState.texture
'image/png' → texture.setContent(bitmap, size); bitmap.close()
collect surface.state.frameCallbacks → Renderer.frameCallbacks
surface.damaged = false; streamingBuffer.release() (remote only)
▼
PHASE 2 — createRenderFrame().then(time => { // = requestAnimationFrame
PHASE 3 — for scene in scenes: // one per canvas/output
sceneShader.use(); updateSceneData(canvas) // viewport, clear, BLEND, projection
for view in viewStack:
if renderState && view.mapped:
sceneShader.updateViewData(view, renderState) // bind texture, u_transform, quad buffer
sceneShader.draw() // gl.drawArrays(TRIANGLE_STRIP, 0, 6)
PHASE 4 — for callback in frameCallbacks: callback.done(time)
→ wl_callback.done(time); wl_callback.destroy()
session.flush()
})
```
## The remote streaming buffer content flow
The remote streaming buffer content arrives in parallel on a separate FRAME
WebSocket channel:
1. `deliverContentToBufferStream` (in `RemoteAppLauncher.ts`) receives
incoming frame messages.
2. `BufferStream.onBufferContents` parses the encoded frame envelope.
3. `createEncodedFrame` (in `EncodedFrame.ts`) parses `{contentSerial,
mimeType, size, encodedSize, opaque, alpha?}`.
4. Resolves the `onFrameAvailable(serial)` promise that
`StreamingBuffer.getContents` is awaiting.
5. The decoded `DecodedFrame` is then uploaded into the view's
`RenderState.texture` via the appropriate FBO+shader conversion.
## Engineering observations
1. **Vestigial damage tracking** — `damageRects` are accumulated correctly
but the renderer doesn't use them for scissoring. Could be a future
optimization for partial repaints.
2. **Multi-output texture cost** — each view has one texture per scene,
so 4 outputs means 4× texture memory for every surface.
3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion
to the browser's BMP decoder. Worth knowing if you need to support
other SHM formats.
4. **Commit-serial mechanism** — the out-of-band opcode 7 lets the
streaming buffer wait for the right encoded frame, ensuring wire-order
semantics even with async decode.
5. **Encoder feedback telemetry** — `EncoderFeedback.ts` measures client
decode duration, but the adaptation loop is wired but not consumed (a
missed opportunity for adaptive bitrate).
6. **Context-loss audit gap** — WebGL context loss is not audited in the
test suite; in production, you should handle `webglcontextlost` and
`webglcontextrestored` events.
7. **`queueCancellableMicrotask` misnomer** — the function is actually a
`setTimeout`-based coalescer, not a microtask. Naming is misleading.
8. **Frame-callback coalescing** — multiple `wl_surface.frame` requests
within one render pass fire on the same RAF tick. Correct Wayland
behavior, but worth knowing.
9. **Cursor/DnD bypass** — they bypass the GL pipeline entirely for
latency. The cursor is rendered as a CSS `cursor: url(dataURL)`.
10. **Subsurface sync cascade** — three-layer mechanism (`cachedState`,
`commitCache`, `effectiveSync`) is correct but subtle. Easy to break
in maintenance.
## File reference
| File | Role |
|---|---|
| `packages/compositor/src/Surface.ts` | `wl_surface` implementation, double-buffered state, commit semantics |
| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction |
| `packages/compositor/src/SurfaceRole.ts` | Role abstraction (SurfaceRole interface) |
| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global |
| `packages/compositor/src/View.ts` | Positioned surface for compositing |
| `packages/compositor/src/Output.ts` | `wl_output`, WebGL canvas binding |
| `packages/compositor/src/Shm.ts` | `wl_shm` / shm pool, BMP-header hack |
| `packages/compositor/src/BufferContents.ts` | Buffer abstraction (with mimeType) |
| `packages/compositor/src/BufferImplementation.ts` | Buffer implementation interface |
| `packages/compositor/src/ImageBitmapBuffer.ts` | ImageBitmap-backed buffer |
| `packages/compositor/src/Region.ts` | `wl_region`, pixman-WASM damage regions |
| `packages/compositor/src/Callback.ts` | Frame callback (wl_callback) |
| `packages/compositor/src/Loop.ts` | Compositor main loop, RAF integration |
| `packages/compositor/src/Compositor.ts` | `wl_compositor` global |
| `packages/compositor/src/Session.ts` | Top-level browser session, decoder setup |
| `packages/compositor/src/Globals.ts` | Wayland globals registry |
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/buffer-decoder.ts` | Decoder interface |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Receives 4-byte feedback messages |
| `packages/compositor/src/remote/Channel.ts` | ARQChannel + SimpleChannel |
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Sets up channels per app, frame delivery |
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer events |
| `packages/compositor/src/browser/dnd.ts` | DnD bridge |
| `packages/compositor/src/browser/selection.ts` | Clipboard bridge |
| `packages/compositor/src/browser/capabilities.ts` | Browser feature detection |
| `packages/compositor/src/browser/input.ts` | DOM keyboard → wl_keyboard events |
| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
| `packages/compositor/src/math/Mat4.ts` | 4×4 matrix math |
| `packages/compositor/src/math/Rect.ts` | Rectangle math |
| `packages/compositor/src/polyfill/createImageBitmap.js` | ImageBitmap polyfill |
+397
View File
@@ -0,0 +1,397 @@
# 04 — Transport Layer (WebSocket / KCP / Channel Layer)
This chapter covers Greenfield's transport: how Wayland wire traffic flows
between the Linux-side proxy and the browser-side compositor over WebSocket,
the binary framing, message ordering, latency/backpressure, KCP usage,
multiplexing, and connection lifecycle.
## Architecture at a glance
Greenfield uses a **multi-WebSocket, KCP-wrapped, asymmetric flow-control**
design:
- **Two-tier process model**: A `main.ts` HTTP hub forks one `SessionProcess`
per browser session (per `compositorSessionId`). Raw `net.Socket`s are
handed to children via Node IPC `sendHandle`. Each session child owns its
own `wl_display` (via `wayland-server-addon.node`).
- **9 WebSocket endpoints** per session: `/signal` (JSON control),
`/channel` (binary data channels), `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
All with `perMessageDeflate: false`, TCP `noDelay: true`.
## Channel types
There are **4 logical channel types** per app client, each implemented as a
separate WebSocket:
| Type | Class | KCP? | Purpose |
|------|-------|------|---------|
| PROTOCOL | ARQChannel | Yes | Wayland wire protocol, per `wl_client` |
| FRAME | ARQChannel | Yes | H.264/PNG encoded surface content, per `wl_surface` |
| XWM | ARQChannel | Yes | XWayland window manager protocol |
| FEEDBACK | SimpleChannel | No | 4-byte telemetry (refreshInterval + decode avg), 1 Hz |
**There is no multiplexing** of multiple Wayland clients onto one WS. Each
logical channel is its own WS, eliminating head-of-line blocking between
video, input, and control at the cost of many connections per app (10+ for
complex apps).
## KCP-over-WebSocket
`kcp.ts` is a ~1100-line TypeScript port of the canonical C `ikcp.c`.
Greenfield runs it **on top of WebSocket** (not real UDP) for:
- 30 ms min RTO vs TCP's 200 ms.
- 100 ms update tick (`setInterval(kcp.update, 100)`).
- Per-channel congestion domains independent of TCP.
- `setNoDelay(1, 100, 0, 1)` — nodelay enabled, congestion control disabled
(`nc=1`).
- 24-byte segment header: `conv|cmd|frg|wnd|ts|sn|una|len`.
- MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC
design).
### Asymmetric windows (deliberate)
- Proxy side: `snd_wnd=128, rcv_wnd=1024`
- Browser side: `snd_wnd=1024, rcv_wnd=128`
So proxy→browser (heavy video) has a smaller in-flight cap to bound latency;
browser→proxy (bursty input) has a large window to avoid stalls.
### Why KCP-over-WebSocket?
Because real-time video over plain TCP suffers from head-of-line blocking
and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in
user-space, while still riding on WebSocket (the only bidirectional binary
transport universally available in browsers today).
The long-term plan, documented in `docs/pages/design/index.md`, is to
switch to **WebTransport** (real UDP) when browser support is widespread,
and pair it with KCP for reliability. The KCP code is already in place;
only the WebSocket transport layer needs to be swapped out.
## Binary message format
All multi-byte fields are **little-endian**.
### PROTOCOL channel
```
u32 outOfBandOpcode (0 = in-band events, else 1=ResourceDestroyed,
2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial)
u32 eventSerial
u32 fdsCount
for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned)
concatenated Wayland wire messages (object_id + opcode/size + args)
```
### FRAME channel (per H.264 frame)
```
u32 bufferId
u32 bufferCreationSerial
u32 bufferContentSerial
encodedFrame:
u32 contentSerial, u32 encodingType (0=h264/1=png),
u32 width, u32 height, u32 encodedWidth, u32 encodedHeight,
u32 opaqueLength, u8[] opaqueData,
u32 alphaLength, u8[] alphaData ← alpha encoded as a SEPARATE H.264 stream
```
### FEEDBACK channel
Just 4 bytes: `u16 refreshIntervalMs + u16 avgDecodeDurationMs`. Sent at
1 Hz from browser to proxy. Used by the proxy to pace `wl_frame.done`
callbacks so the native app sees a virtual monitor refresh rate matching
the browser's actual RAF cadence.
## File descriptor handling
Can't send FDs over WebSocket — Greenfield virtualizes them:
- **ProxyFD** = `{handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown',
host:string}`.
- Serialized as JSON inline in the message stream with a u32 length prefix.
- Browser opens **separate ad-hoc WebSocket** to `/read-fd`,
`/write-fd-as-stream`, etc., only when it actually needs the bytes.
- Most `wl_buffer` FDs never need to be read because the proxy intercepts
`wl_surface.commit`, reads the buffer locally via DRM/SHM, encodes to
H.264, and ships only the encoded bytes.
- **Foreign ProxyFD** (`proxyFD.host !== local baseURL`) triggers
proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the
foreign proxy's `/write-fd-as-stream`, and pipes the data through. This
is how copy-paste between two proxy instances avoids a browser round-trip.
## Reconnection & lifecycle
### Browser side
Vendored `ReconnectingWebSocket` (pladaria) with:
- `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`
- `minUptime=5s`, `connTimeout=4s`
- `retries=∞`
Channel state preservation: only close codes 4000–4999 (application-level)
tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps
ticking and the WS reconnects underneath.
### Proxy side
No auto-reconnect (it's the server). The Channel object lives on, waiting
for a new WS upgrade with matching `id` + `key`. Outbound messages queue
in `outboundMessages` and flush on `onOpen`.
### App kill timers
When the browser disconnects from the `/signal` WebSocket:
- **10-minute SIGHUP grace period** — gives the user time to reconnect
(e.g. if they refresh the browser tab).
- After 10 minutes of no reconnect, escalating to **SIGKILL after another
10s**.
- Close code 1001 → immediate SIGHUP (user closed tab intentionally).
- Close code 4001 → immediate SIGTERM (user clicked "force quit" in the
shell).
### Session close
When all `wl_clients` disconnect, 1-second grace then `session.close()` →
`process.exit()` in the session child. Main process deletes the entry
from its `sessionProcesses` map.
## Process model — three layers
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
A single Node.js process listens on `:8081`. Per browser session, it
`fork()`s one `SessionProcess` child — one child per `compositorSessionId`,
**not** per app. Multiple apps in the same browser tab share one child.
WebSocket upgrades are handed off to the matching child via
`childProcess.send(wsUpgrade, socket)`.
### Layer 2: Session Process (`SessionProcess.ts`)
The forked child. Receives `start` (creates `Session` +
`SessionController`), `launchApp` (spawns the actual app binary), and
`wsUpgrade` (hands WS to the controller) IPC messages. On `launchApp` it
calls `launchApplication()` from `@gfld/compositor-proxy`, which `spawn()`s
the executable, captures `pid`, replies to main with `{pid, key}`. On
`session.close()` the process exits.
### Layer 3: Proxy Core (`compositor-proxy/src/`)
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
wl_display FD into Node's libuv loop via `startPoll()`.
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
Uses `MessageInterceptor` + auto-generated per-resource interceptor
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
`{native, browser, neverReplies?}` routing decision. Handles
`wl_display.sync` fast-path (skips browser round-trip when no reply
expected). Marshalls FDs as `ProxyFD` JSON.
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
(10 s) on force-quit, auto-destroy on app exit.
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
connect). Xwayland itself becomes a Wayland client of the proxy. The
`wm_fd` is bridged via xtsb to the browser, which implements the actual
XWM logic.
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
## ASCII diagram of the full transport stack
```
LINUX SIDE | BROWSER SIDE
|
┌─────────────────────────┐ | ┌──────────────────────────────┐
│ gtk4-demo (process) │ | │ Browser tab │
│ libwayland-client.so │ | │ (Greenfield compositor) │
└───────────┬─────────────┘ | └──────────────┬───────────────┘
│ WAYLAND_DISPLAY=wayland-0 │
│ Unix socket + SCM_RIGHTS FD passing │ DOM events → input
▼ │ WebSocket API
┌──────────────────────────────────────────┐ │
│ SessionProcess (fork'd Node child) │ │
│ ┌────────────────────────────────────┐ │ │
│ │ wayland-server-addon.node │ │ │
│ │ (libwayland-server N-API) │ │ │
│ │ - wl_display + wl_client │ │ │
│ │ - epoll via proxy-poll-addon │ │ │
│ └─────────────┬──────────────────────┘ │ │
│ │ wire msg + FDs │ │
│ ▼ │ │
│ ┌─────────────────────────────────────┐ │ │
│ │ NativeWaylandClientSession │ │ │
│ │ - interceptRequest / interceptEvent│ │ │
│ │ - onWireMessageEnd: serialize │ │ │
│ │ FDs as JSON ProxyFDs inline │ │ │
│ │ - fastSync optimization │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ │ outbound buffer │ │
│ ▼ │ │
│ ┌─────────────────────────────────────┐ │ │
│ │ ARQChannel (KCP-wrapped) │ │ │
│ │ MTU=64000, snd_wnd=128, rcv_wnd=1024│ │ │
│ │ KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data │
│ │ kcp.update() every 100ms │ │ │
│ │ - PROTOCOL (per wl_client) │ │ │
│ │ - FRAME (per wl_surface) │ │ │
│ │ - XWM (per XWayland) │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ ┌─────────────┴───────────────────────┐ │ │
│ │ SimpleChannel (FEEDBACK, no KCP) │ │ │
│ │ 4 bytes: refreshIntervalMs + │ │ │
│ │ avgDecodeDurationMs │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ ┌─────────────▼───────────────────────┐ │ │
│ │ NativeAppContext │ │ │
│ │ - signalingWebSocket (JSON msgs) │ │ │
│ │ - kill timers: SIGHUP@10min, │ │ │
│ │ SIGKILL@+10s │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ ┌─────────────▼───────────────────────┐ │ │
│ │ SessionController │ │ │
│ │ ws.WebSocketServer (noServer) │ │ │
│ │ perMessageDeflate: false │ │ │
│ │ Endpoints: /signal /channel /mkfifo│ │ │
│ │ /mkstemp-mmap /read-fd /write-fd-as-stream│ │
│ │ /read-fd-as-stream /close-fd /request-keyframe│ │
│ └─────────────┬───────────────────────┘ │ │
└────────────────┼─────────────────────────────┘ │
│ │
│ ┌──────────────────────────┐ │
│ │ main.ts (HTTP+WS hub) │ │
│ │ - socket.pause() + IPC │ │
│ │ send to session proc │ │
│ │ - noDelay:true (TCP) │ │
│ └────────────┬────────────┘ │
│ │ │
│ ◄──── 1 WebSocket per channel ────► │
│ ◄── 1 signaling WS per app ──► │
│ ◄── ad-hoc WebFD streams ─► │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────┐ │
│ │ ReconnectingWebSocket │ │
│ │ min 1-5s, factor 1.3, max 10s │ │
│ │ minUptime 5s, connTimeout 4s │ │
│ └────────────┬───────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ ARQChannel / SimpleChannel (mirror image) │ │
│ │ snd_wnd=1024, rcv_wnd=128 │ │
│ │ kcp.input → peekSize/recv → onMessage │ │
│ └────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ RemoteAppLauncher.onChannel │ │
│ │ PROTOCOL → session.display.createClient │ │
│ │ FRAME → deliverContentToBufferStream │ │
│ │ XWM → XWindowManagerConnection │ │
│ │ FEEDBACK → ClientEncodersFeedback │ │
│ └────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ @gfld/compositor-protocol Client │ │
│ │ - parses Wayland wire format │ │
│ │ - dispatches to wl_*_resource impls │ │
│ │ - H264NALDecoder.worker.ts decodes frames │ │
│ └────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ Surface + Scene rendering │ │
│ │ - ImageBitmap / VideoFrame / Canvas │ │
│ │ - DOM input → wl_keyboard/pointer reqs │ │
│ └────────────┬───────────────────────────────┘ │
│ │ input events flow back via the │
│ │ PROTOCOL channel as ordinary │
│ │ Wayland requests (same framing) │
└──────────────────┘ │
Browser DOM ◄─┘
```
## Key takeaways for building your own transport
1. **Separate control plane from data plane** — Greenfield uses `/signal`
(JSON) vs `/channel` (binary). The control plane handles app launch,
termination, kill timers; the data plane handles Wayland wire + encoded
frames.
2. **One WS per logical stream** — simpler, no HoL blocking, but many TCP
connections.
3. **KCP-over-WS** trades double-reliability overhead for fast (30 ms)
retransmits; reasonable for real-time video.
4. **Virtualize FDs as JSON tokens** with lazy separate-WS endpoints for
actual byte transfer.
5. **`bufferedAmount` check ≠ backpressure** — Greenfield's check is a drop
gate, not real flow control (acknowledged TODO).
6. **Distinguish transient vs permanent close** — codes 4000-4999 tear
down; lower codes auto-reconnect.
7. **Piggyback out-of-band opcodes** on the data channel via a leading u32
to avoid extra control WS.
8. **Asymmetric KCP windows** match each direction's bandwidth profile.
9. **Decode feedback drives frame pacing** — the proxy synthesizes
`wl_frame.done` callbacks at the browser's reported cadence.
10. **One Node child per browser session** for fault isolation and
per-session `wl_display` ownership.
## File reference
### Native side
| File | Role |
|---|---|
| `packages/compositor-proxy/src/Channel.ts` | `ARQChannel` + `SimpleChannel` |
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
### Browser side
| File | Role |
|---|---|
| `packages/compositor/src/remote/Channel.ts` | `ARQChannel` + `SimpleChannel` (mirror) |
| `packages/compositor/src/remote/kcp.ts` | KCP mirror |
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/api.ts` | Public exports |
### CLI
| File | Role |
|---|---|
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
| `packages/compositor-proxy-cli/wait-until-ready.sh` | Startup readiness check |
+366
View File
@@ -0,0 +1,366 @@
# 05 — Input Handling (Keyboard, Pointer, Touch, Focus)
This chapter covers Greenfield's input handling: keyboard (with xkbcommon in
browser), pointer, touch, focus management, and how input events travel from
the browser to native Linux apps via the proxy.
## Headline
Greenfield is a **browser-native Wayland compositor**. All seat logic —
keymap compilation, modifier state, pointer grabs, button/axis dispatch —
runs **in the browser**, against a **WASM build of libxkbcommon 1.5.0 +
pixman**. The `compositor-proxy` is *not* a compositor — it is a thin
libwayland-server bridge that relays Wayland wire messages between the
browser and native clients and provides file-descriptor shims.
There is no second xkbcommon on the proxy: the keymap is generated in the
browser from RMLVO and shipped to the proxy as a real native shm fd via a
custom `/mkstemp-mmap` WebSocket endpoint. `wl_touch` is advertised but
**never populated with events** — touch input is collapsed into Pointer
Events.
## The Seat abstraction
`Seat.ts` (in `packages/compositor/src/`) is the `wl_seat` global. It
aggregates keyboard, pointer, and touch, manages focus, and runs the input
queue. Each `Seat` has:
- `Keyboard` (always present)
- `Pointer` (always present)
- `Touch` (advertised only if `'ontouchstart' in document.documentElement`,
but inert — see §Touch below)
- `InputQueue` — coalesces events and dispatches them on the next RAF
- `InputOutput` — abstracts FD operations (browser pipes for in-browser
apps, WebSocket endpoints for native apps via proxy)
Each seat also tracks:
- The currently focused `Surface` for keyboard input
- The currently focused `Surface` for pointer input (transient)
- The active pointer grab (default, popup, drag, or resize)
- The active keyboard grab (default or popup)
## Keyboard — xkbcommon in the browser
### WASM xkbcommon build
`libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`,
applies a patch (removes tests, shims `strdup`), builds with emscripten
3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`,
exports the keymap/state C API as a single-file ES module.
The same build approach is used for pixman (for region math).
### Keymap compilation flow
The browser, not the proxy, is the source of truth for the keymap. The
flow is:
1. Browser receives a keymap config (either default or user-supplied via
`userShell.actions.setUserConfiguration({ keyboardLayoutName })`).
2. `Seat.notifyUpdateKeymap(nrmlvo)` is called.
3. `xkb_context_new` + `xkb_keymap_new_from_names` (WASM calls) compile the
RMLVO into a keymap.
4. `xkb_keymap_get_as_string` serializes the keymap as a long string.
5. `Keyboard.sendKeymap(resource)` is called for each `wl_keyboard`
resource:
- `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a
`/mkstemp-mmap` WebSocket, sends the blob, and receives a `ProxyFD {
handle: <native_fd>, type: 'shm', host: baseURL }`.
- `keyboardResource.keymap(xkbV1, ProxyFD, size)` is sent to the
native client.
6. The proxy deserializes the ProxyFD, looks up the native fd via
`proxyFDtoNativeFD` (identity when `host === baseURL`), ships it via
`sendEvents(wlClient, msg, fds)` from the N-API addon —
`libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain
socket.
7. The native client's libwayland receives the keymap fd, mmaps it, and
calls native `xkb_keymap_new_from_string` to compile its own copy.
So the keymap flows **browser → proxy → native**, not the reverse. This
is the opposite of what you might expect (since the proxy is on the Linux
host where xkbcommon natively lives), but it makes sense: the browser is
the actual compositor and is the one that needs the keymap to translate
DOM key events to Wayland keycodes.
### DOM → Wayland key event translation
Browser DOM `KeyboardEvent` has `event.code` (e.g. `"KeyA"`) and
`event.key` (the resolved glyph). Greenfield uses **only the physical
`.code`**, mirroring evdev:
1. `event.code` is looked up in `EvDevKeyCode` reverse-name-map → evdev
keycode (e.g. `30` for `KeyA`).
2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset).
3. `capsLock` / `numLock` are read from `event.getModifierState(...)`.
The DOM `.key` glyph is **ignored** — only physical `.code` is used, so
the same physical key always produces the same evdev keycode regardless
of layout. This is correct Wayland behavior.
### Modifier tracking
Modifier state is tracked entirely via the WASM libxkbcommon state
machine:
1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or
`xkb.keyUp(x11)`.
2. Internally, `_xkb_state_update_key` is invoked.
3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` /
`modsLocked` / `group` (each a `_xkb_state_serialize_mods` call).
4. High-level `KeyboardModifier` bits are computed by checking xkb
mod-indices resolved at keymap-compile time.
5. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)`
sends the `wl_keyboard.modifiers` event to the native client.
### Notable bug
`Seat.ts` lines 511-513 contain a copy/paste typo:
`modsLatched = modsLocked` (should be `= modsLatched`). The cached
`keyboard.modifiers` struct is unreliable but doesn't affect protocol
output because `notifyModifiers` reads from `xkb` directly each time.
## Pointer
### DOM → Wayland pointer events
Browser DOM `MouseEvent` is converted to Greenfield's internal
`ButtonEvent` / `AxisEvent`:
- `pointermove` → `createButtonEventFromMouseEvent` → `ButtonEvent`
(motion-only)
- `pointerdown` / `pointerup` → `ButtonEvent` (with button + state)
- `wheel` → `createAxisEventFromWheelEvent` → `AxisEvent`
### Pointer enter / leave between surfaces
`DefaultPointerGrab.focus()` runs on every motion event:
1. `Renderer.pickView` walks the view stack front-to-back.
2. Returns the first view whose `Surface.isWithinInputRegion(point)`
(pixman-WASM region) is true.
3. On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to
the old surface, then `wl_pointer.enter` + `frame` (+ modifiers if
keyboard is grabbed) to the new surface.
### Pointer capture / window leave
- `canvas.setPointerCapture` is called on `pointerdown`.
- `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus,
releases depressed keys, sets `needFocusInit=true` for re-sync),
resets `buttonCount=0`.
- DnD suppresses this handler mid-drag.
### Pointer acceleration / cursor images
**No acceleration** — verbatim `offsetX/Y`.
Cursor images are **CSS `cursor: url(data:...) x y, auto`**:
1. Native client calls `wl_pointer.set_cursor`.
2. The cursor surface commits a buffer.
3. `CursorRole` draws the bitmap to a hidden canvas.
4. `toDataURL()` produces a data URL.
5. Set on `document.body.style.cursor`.
OS composites the cursor. No rotation/scale transforms.
### Scroll wheel / touchpad
`WheelEvent.deltaMode`:
- `DOM_DELTA_LINE × 12` — discrete wheel clicks
- `DOM_DELTA_PAGE × surface size` — page-sized scrolls
- `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad
Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version
≥5 also gets `axis_source(wheel)` + `frame()` (groups events).
No `axis_discrete` / `value120` / `axis_stop` events are sent.
### Linux button codes
DOM `event.button` (0=left, 1=middle, 2=right, 3=back, 4=forward) is
mapped to Linux input protocol button codes via the `linuxInput` table
in `browser/input.ts`.
## Touch — declared but inert
`Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners.
`wl_touch` capability is advertised if `'ontouchstart' in
document.documentElement`, but no events are ever sent. Touch input flows
through Pointer Events as mouse input.
**For the user**: if you need real touch support (e.g. for tablets), this
is a feature you'll have to implement from scratch. The protocol stubs
exist (via generated `wl_touch` bindings), but no event source populates
them.
## Focus management
### DOM focus
The `<canvas>` owns keyboard focus permanently (`tabIndex=1`).
`Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between
native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface`
activation handlers.
### DOM → Wayland focus
- Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores
saved focus, re-syncs modifiers).
- Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves
focus, releases depressed keys, sets `needFocusInit=true`).
### Surface-level focus
- Click on a surface → `XdgToplevel` activation request →
`desktopSurface.activate()` → `gainFocus()` (refcounted via
`focusCount`, fires `surfaceActivationUpdated` event to the user shell
on 0↔1 transitions).
- Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`,
validating the serial against the keyboard/pointer `grabSerial`.
## Input queue
`InputQueue.ts` coalesces events and dispatches them on the next RAF
(`InputQueue.drainQueues`). This batches bursts of input events (e.g. fast
mouse motion) into a single dispatch, avoiding redundant work.
The queue is FIFO per event type (motion, axis, key, button). Each
dispatch calls the appropriate `seat.notify*` method.
## ASCII event-flow diagram
```
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER (DOM) │
│ │
│ <canvas tabIndex=1> ─┐ │
│ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │
│ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │
│ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │
│ │ • evdevKeyCode = EvDevKeyCode[event.code] │
│ │ • x11KeyCode = evdevKeyCode + 8 │
│ │ • capsLock/numLock = event.getModifierState(...) │
│ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │
│ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ session.inputQueue.queue{Motion,Axis,Key,Button}
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │
│ │
│ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame │
│ │ │
│ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │
│ │ └─ Keyboard.sendKey(event) │
│ │ └─ keyboardResource.key(serial,time,evdev,st) │
│ │ │
│ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │
│ │ └─ Pointer.sendButton │
│ │ └─ pointerResource.button(serial,time, │
│ │ linuxInput[btn], state) │
│ │ │
│ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │
│ │ └─ pointerResource.motion(time, sx_f, sy_f) │
│ │ │
│ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │
│ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │
│ │ └─ pointerResource.axis(time, axis, Fixed) │
│ │ + axisSource(wheel) + frame() if v>=5 │
│ │ │
│ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │
│ │ │
│ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │
│ └─ Keyboard.sendModifiers(modsDepressed,Latched, │
│ Locked,group) │
│ │
│ Keymap path (once per WlKeyboardResource): │
│ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │
│ → xkb_context_new + xkb_keymap_new_from_names (libxkbcommon WASM) │
│ → xkb_keymap_get_as_string ──► keymapString │
│ Keyboard.sendKeymap(resource): │
│ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │
│ keyboardResource.keymap(xkbV1, ProxyFD, size) │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ client.connection.onFlush(wireMessages)
│ ─ one big Uint32Array, FDs serialized as ProxyFD JSON
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ WIRE (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts) │
│ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ ws on proxy :SessionController │
│ Message layout (Uint32Array): │
│ [0] outOfBandOpcode (0 = in-band) │
│ [1] eventSerial │
│ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │
│ then concatenated Wayland wire messages │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ NativeWaylandClientSession.onMessage
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ PROXY (Node.js, compositor-proxy) │
│ │
│ SessionController wss paths: /mkfifo /mkstemp-mmap /read-fd /close-fd │
│ /read-fd-as-stream /write-fd-as-stream │
│ /signal /channel /request-keyframe │
│ │
│ mkstempMmap(ws): │
│ receives Blob → Buffer.concat → createMemoryMappedFile(buf) ← native addon │
│ returns ProxyFD { handle: <native fd>, type: 'shm', host: baseURL } │
│ │
│ onWireMessageEvents: │
│ for each ProxyFD in message: │
│ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) // just returns .handle │
│ └─ if proxyFD.host === baseURL → identity │
│ └─ if foreign host → spawn a bridging pipe + WebSocket │
│ messageInterceptor.interceptEvent(...) // empty stubs for kbd/ptr/touch │
│ sendEvents(wlClient, messageBuffer, fdsBuffer) ← libwayland-server addon │
│ flush(wlClient) │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ libwayland-server sends wire msg + SCM_RIGHTS FDs
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ NATIVE (Wayland client e.g. weston-terminal, Xwayland app) │
│ wl_display.fd readable → libwayland-client demarshals │
│ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │
│ wl_keyboard.key / enter / leave / modifiers → client app │
│ wl_pointer.enter / motion / button / axis / frame → client app │
└──────────────────────────────────────────────────────────────────────────────────────┘
```
## Key answers (quick reference)
| Question | Answer |
|---|---|
| DOM→Wayland key event translation | `event.code` (e.g. `"KeyA"`) is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30`). `x11KeyCode = evdevKeyCode + 8`. DOM `.key` glyph is **ignored** — only physical `.code` is used, mirroring evdev. |
| Modifier tracking | Entirely via WASM libxkbcommon state machine. `Seat.updateModifierState` calls `xkb.keyDown/Up(x11)` → `_xkb_state_update_key`, then `notifyModifiers` reads `xkb.modsDepressed/Latched/Locked/group` (each a `_xkb_state_serialize_mods` call). High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. |
| Focus: DOM ↔ native surfaces | The `<canvas>` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave/enter` between native surfaces. Activation is driven by `XdgToplevel`/`ShellSurface` activation handlers. |
| Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` calls `notifyKeyboardFocusOut` (saves kbd focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. DnD suppresses this handler mid-drag. |
| Touch event mapping | **Not implemented.** `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. |
| Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView` walks the view stack front-to-back, returns first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. On change, `Pointer.setFocus` emits `leave+frame` to old surface, `enter+frame` (+modifiers) to new. |
| Pointer acceleration / cursor images | **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: client calls `wl_pointer.set_cursor` → `CursorRole` → draw bitmap to hidden canvas → `toDataURL()` → set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. |
| Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: `DOM_DELTA_LINE × 12`, `DOM_DELTA_PAGE × surface size`, `DOM_DELTA_PIXEL ÷ 3`. Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. v≥5 also gets `axis_source(wheel)` + `frame()`. No `axis_discrete/value120/stop`. |
| WASM xkbcommon? | Yes. `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch, builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. Sibling build does the same for `pixman`. |
| Keymap FD workaround for WebSocket | Direction is **browser → proxy**, not the reverse. Browser calls `inputOutput.mkstempMmap(Blob)` → opens `/mkstemp-mmap` WebSocket → proxy creates real native shm fd via `createMemoryMappedFile` addon → returns `ProxyFD {handle: <native_fd>, type: 'shm', host: baseURL}`. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`; the ProxyFD is **serialized as JSON** in the wire message's `fds[]` array (WebSockets can't pass FDs natively). Proxy deserializes, maps `handle` → native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. |
## File reference
| File | Role |
|---|---|
| `packages/compositor/src/Seat.ts` | `wl_seat` global, multi-seat support, focus management |
| `packages/compositor/src/Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events |
| `packages/compositor/src/Pointer.ts` | `wl_pointer` impl, motion, buttons, axis |
| `packages/compositor/src/Touch.ts` | `wl_touch` (46-line stub, inert) |
| `packages/compositor/src/Xkb.ts` | xkbcommon WASM wrapper |
| `packages/compositor/src/KeyEvent.ts` | Key event wrapper |
| `packages/compositor/src/ButtonEvent.ts` | Pointer button event wrapper |
| `packages/compositor/src/AxisEvent.ts` | Pointer axis (scroll) event wrapper |
| `packages/compositor/src/InputQueue.ts` | Event coalescing + RAF dispatch |
| `packages/compositor/src/InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) |
| `packages/compositor/src/browser/input.ts` | DOM event listeners → Greenfield events |
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer |
| `libs/compositor-wasm/` | pixman + xkbcommon WASM build |
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
| `packages/compositor-proxy/src/io/` | WebSocket server glue (mkstemp-mmap endpoint etc.) |
+314
View File
@@ -0,0 +1,314 @@
# 06 — Data Device, Selection, DnD (`wl_data_device`)
This chapter covers Greenfield's data device / clipboard / drag-and-drop
implementation: how `wl_data_device_manager`, `wl_data_source`,
`wl_data_offer`, `wl_data_device` are implemented, how clipboard contents
are transferred between browser and native apps, and how DnD is coordinated.
## Faithful Wayland model
`DataDeviceManager` is a v3 global singleton. `wl_data_device` is per
seat-per-client and the implementation lives directly on `Seat` (which
implements `WlDataDeviceRequests`). `WaylandDataSource` / `DataOffer` are
the standard source/offer pair, ported almost verbatim from weston,
including the `chooseAction()` bit-trick for DnD action negotiation.
## The FD-over-WebSocket problem
Wayland's `wl_data_offer.receive(mimeType, fd)` /
`wl_data_source.send(mimeType, fd)` carry a real UNIX fd. Greenfield solves
this with a **`ProxyFD` JSON descriptor**:
```
ProxyFD = {
handle: number,
type: 'pipe-read' | 'pipe-write' | 'shm' | 'unknown',
host: string // proxy's HTTP base URL
}
```
Three concrete `InputOutput` / `InputOutputFD` implementations:
### `webInputOutput` — for in-browser web apps
Pipes are `MessageChannel` ports, transferred as `ArrayBuffer`s. All
"fd" operations are in-browser zero-copy transfers.
### `RemoteInputOutput` — for native apps
All fd operations go over a secondary WebSocket to endpoints:
`/mkfifo`, `/mkstemp-mmap`, `/read-fd-as-stream`, `/write-fd-as-stream`,
`/read-fd`, `/close-fd`.
### `ProxyInputOutput` (proxy-side) — owns the real OS pipes and memfd files
`proxyFDtoNativeFD(proxyFD)` returns the raw int fd because every ProxyFD
has `host === proxy.baseURL`.
The proxy interceptors for `wl_data_source` / `wl_data_offer` /
`wl_data_device` / `wl_data_device_manager` are auto-generated no-ops that
only handle resource construction and lifetime — all actual semantics live
in the browser compositor.
### Crucial consequence
For native→native clipboard transfers (e.g. user copies in one Linux app,
pastes in another), the browser compositor never sees a single byte of the
data — it only relays the fd reference. The actual content stays inside
the proxy host's OS kernel pipe.
## MIME-type negotiation
At the Wayland level, mime types are just relayed strings. Mapping to the
browser's hard-coded allow-list `['text/plain', 'text/html', 'image/png']`
happens only at the DOM boundary:
### Native → browser
`selection.ts:handleWaylandDataSourceUpdate`: each source mime type is
substring-matched against the allow-list (`text/plain;charset=utf-8`
contains `text/plain`, so it maps), survivors are pulled as separate blobs
via `mkfifo + send + readBlob`, and finally
`navigator.clipboard.write([new ClipboardItem(...)])`.
### Browser → native
`BrowserDataSource.ts`: `ClipboardItem.types` become the source mime types,
plus a synthetic `text/plain;charset=utf-8` alias is appended (Linux apps
expect the charset form). On `send`, the alias is normalised back to
`text/plain` for the lookup.
### Caveat: substring matching is fragile
The substring match `dataSourceMimeType.indexOf(browserMimeType) !== -1`
can collide (`image/png` matching `image/png;base64` is fine, but
`text/plain;charset=iso-8859-1` silently becomes `text/plain`). Consider
an explicit prefix or exact-match policy in your implementation.
## Clipboard mirroring is pull-based
The browser has no `clipboardchange` event (Chromium issue 933608 is
cited in the code). Greenfield polls `navigator.clipboard.read()`:
- on init
- on every `window.focus` event
De-duplicates offers by mime-type set + total blob size, then calls
`seat.setSelectionInternal(createBrowserDataSource(offers), serial)`.
The feedback loop is broken because `handleWaylandDataSourceUpdate` returns
early when `dataSource instanceof BrowserDataSource`.
### Firefox fallback
`text-selection.ts` uses only `readText` / `writeText` and a
`BrowserTextDataSource` that serves `text/plain;charset=utf-8` — no images
or HTML.
## Drag-and-drop
### Wayland-side (Seat.startDrag)
1. Validates the implicit pointer grab.
2. Attaches `DragIconRole` to the optional icon surface.
3. Installs a `PointerDrag` grab on both pointer and keyboard.
`PointerDrag` tracks the focused data device, sends
`wl_data_device.enter` / `leave` / `motion` / `drop`, creates fresh
`wl_data_offer`s via `sendOffer`, and runs the v3 action negotiation.
### Browser-side (`BrowserDnD` class)
Creates a hidden full-canvas `dragHelper` div with `draggable=true` when
the Wayland drag-icon surface commits, so the browser shows the Wayland icon
via `event.dataTransfer.setDragImage(cursorCanvas, ...)`. While the user
drags, `dragover` events on the dragHelper are converted back to
`ButtonEvent`s and pushed through `session.inputQueue.queueMotion`, keeping
`PointerDrag` fed.
### Not implemented: dragging data FROM a Wayland app INTO a real DOM target
The FIXME in `dnd.ts` admits: dragging data FROM a Wayland app INTO a real
DOM drop target (file drag-out, etc.) is **not implemented**.
## XWayland bridge
`XWindowManager` listens on the X11 `CLIPBOARD` atom via `XFixes` and
builds `XDataSource` (clipboard) or `XDnDDataSource` (XDnD). When the
receiver pulls data:
1. `XDataSource.send` calls `XConvertSelection` to have the X server push
the bytes into a property on `selectionWindow`.
2. `writeProperty` copies them into the receiver's `InputOutputFD`.
3. INCR transfers are handled (`getIncrChunk`, `sendIncrChunk`).
## Primary selection is NOT implemented
A grep for `primary_selection`, `gtk_primary`, `zwp_primary_selection`
across the entire `packages/` tree returns zero hits. XWayland only
watches the `CLIPBOARD` atom, not `PRIMARY`. **Middle-click paste does
not work in Greenfield today.**
**For the user**: if your users expect Linux-style middle-click paste,
you'll need to implement `zwp_primary_selection_device_manager_v1` (or
`gtk_primary_selection`) on the compositor side and forward `PRIMARY`
notifications in `XWindowManager`.
## ASCII diagram — clipboard flow in both directions
```
CLIPBOARD / SELECTION FLOW
=================================================================
A) NATIVE APP COPIES ──► BROWSER DOM CLIPBOARD (Ctrl+C in Linux app)
=================================================================
┌──────────────┐ wl_data_device.set_selection(src,serial) ┌────────────────────────┐
│ Native app │ ─────────────────────────────────────────────► │ compositor │
│ (Wayland or │ │ (browser, in DOM) │
│ XWayland) │ ◄────── wl_data_offer.offer("text/plain") ─── │ │
└──────────────┘ (only if this app has keyboard focus)└───────────┬────────────┘
│ selectionListeners fire
▼
┌──────────────────────────┐
│ handleWaylandDataSource │
│ Update() (selection.ts)│
└──────────┬──────────────┘
mkfifo() ──────────────────────────► │ proxy /mkfifo
ProxyFD read+write (real OS pipe) ◄───── returns 2 ProxyFDs
▼
dataSource.send("text/plain", writeFD)
│
▼
wl_data_source.send("text/plain", writeFDProxyFD)
─────────────────────────────────────────► │ Native source
│ receives real write fd
│ (proxyFDtoNativeFD
│ is a passthrough)
▼
Native app writes bytes
──────────────► OS pipe ───┐
│
readFD.readBlob() ◄──────────── read from /read-fd-as-stream ◄─┘
▼ (proxy streams over WS)
Blob {type:"text/plain"}
▼
navigator.clipboard.write([ new ClipboardItem({"text/plain": blob}) ])
▼
┌────────────────────────────┐
│ DOM clipboard populated │
│ (Ctrl+V anywhere now works) │
└────────────────────────────┘
=================================================================
B) USER COPIES IN BROWSER ──► NATIVE APP GETS SELECTION (Ctrl+C in DOM)
=================================================================
┌─────────────────────────────┐
│ User copies in DOM │
│ (browser tab, textarea...) │
└──────────────┬──────────────┘
│ on window 'focus' event (or initial load)
▼
navigator.clipboard.read() ──► ClipboardItems[]
▼
handleBrowserDataSourceUpdate() — de-dup vs previous offers
▼
createBrowserDataSource(offers)
.mimeTypes = offer.types + ['text/plain;charset=utf-8'] alias
▼
seat.setSelectionInternal(browserDataSource, serial)
│
│ 1) selectionListeners fire — handleWaylandDataSourceUpdate
│ returns early (instanceof BrowserDataSource), breaking loop
│
│ 2) sendSelection(focusedClient)
▼
┌────────────────────────┐ wl_data_offer.offer("text/plain"), ┌──────────────┐
│ compositor │ ─ wl_data_device.selection(offer) ──► │ Native app │
│ (browser, in DOM) │ │ (focused) │
└───────────┬────────────┘ └──────┬───────┘
│ │
│ When the native app calls wl_data_offer.receive("text/plain", fd):
│ │
│ ◄──────── wl_data_offer.receive(mimeType, writeFDProxyFD) ◄── native pipe
│ (fd was created by the native app's libwayland; proxy
│ wrapped it as ProxyFD{host:proxy.baseURL, type:'unknown'}
│ then compositor's DataOffer.receive calls wrapFD → RemoteInputOutputFD)
▼
BrowserDataSource.send(mimeType, ioFD)
│
▼
matchingOffer.getType(mimeType) ─► Promise<Blob> (from ClipboardItem)
│
▼
ioFD.write(blob) ─► POST /write-fd-as-stream?fd=<handle> ─► proxy
│
▼
createWriteStream({fd, autoClose:true})
ws stream ──► native write fd
│
▼
Bytes flow through OS pipe into native app
│
▼
ioFD.close() ─► /close-fd ─► EOF ─► native app's read returns 0
=================================================================
PROXY FD TRANSLATION (works the same way in B and A)
=================================================================
Native fd (in proxy process)
│
│ wrap into ProxyFD JSON: {handle, type, host: proxy.baseURL}
│ JSON-encode, length-prefix, send over WS protocol channel
▼
┌─────────────────────────────────────────────────┐
│ Browser compositor │
│ client.userData.inputOutput.wrapFD(fd, type) │
│ → RemoteInputOutputFD (RemoteInputOutput) │
│ → WebPipeInputOutputFD (webInputOutput, │
│ for in-browser apps)│
└─────────────────────────────────────────────────┘
```
## File reference
| File | Role |
|---|---|
| `packages/compositor/src/DataDeviceManager.ts` | `wl_data_device_manager` global |
| `packages/compositor/src/DataSource.ts` | `wl_data_source` impl, the provider side |
| `packages/compositor/src/DataOffer.ts` | `wl_data_offer` impl, the receiver side |
| `packages/compositor/src/WaylandDataSource.ts` | Wrapper around native `wl_data_source` events |
| `packages/compositor/src/Seat.ts` | `wl_data_device` impl (Seat implements `WlDataDeviceRequests`) |
| `packages/compositor/src/browser/selection.ts` | Browser clipboard bridge (native → DOM) |
| `packages/compositor/src/browser/text-selection.ts` | Firefox fallback (text-only) |
| `packages/compositor/src/browser/dnd.ts` | Browser DnD bridge (Wayland → DOM) |
| `packages/compositor/src/InputOutput.ts` | FD abstraction interface |
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
| `packages/compositor/src/remote/XWindowManager.ts` | X11 CLIPBOARD atom watcher |
| `packages/compositor/src/remote/XDataSource.ts` | X11 clipboard source (XWayland apps) |
| `packages/compositor/src/remote/XDnDDataSource.ts` | X11 DnD source |
## Next actions / suggested improvements
1. **Primary selection** — implement `zwp_primary_selection_device_manager`
v1 (or `gtk_primary_selection`) on the compositor side and forward
`PRIMARY` notifications in `XWindowManager`. This unlocks middle-click
paste from native apps.
2. **`clipboardchange` event** — when browsers ship it, replace the
`window.focus`-based polling in `selection.ts` with a proper event-driven
subscription.
3. **Native→DOM DnD** — implement the dataTransfer side of `BrowserDnD`
so a Wayland drag can deliver bytes to a real DOM drop target (the
`dnd.ts` FIXME).
4. **MIME allow-list discovery** — replace the hard-coded
`['text/plain', 'text/html', 'image/png']` with a runtime check once
browsers expose the supported types.
5. **MIME-type strict matching** — the substring match
`dataSourceMimeType.indexOf(browserMimeType) !== -1` can collide
(`image/png` matching `image/png;base64` is fine, but
`text/plain;charset=iso-8859-1` silently becomes `text/plain`).
Consider an explicit prefix or exact-match policy.
+404
View File
@@ -0,0 +1,404 @@
# 07 — Native App Bridge (How Real Linux Apps Get Connected)
This chapter covers Greenfield's native app bridge: how real Linux
applications get launched, connected to the proxy, and routed to the
browser. This is the bridge that makes a Linux Wayland app appear inside
the browser. **This is the most important chapter for the user** — they
want to run real Linux apps in their Odoo 20 web app.
## Two app models — don't conflate them
Greenfield has **two app models**:
### Model 1: WASM web apps (browser-resident)
Built with the Greenfield SDK (Emscripten), run inside the browser via
`MessageChannel`. The `sdk/` directory serves only this model. Apps are
compiled to WASM, run in the same browser tab as the compositor, and
speak Wayland over a `MessagePort` instead of a UNIX socket.
### Model 2: Native Linux apps (the focus of this chapter)
Real ELF binaries (`gtk4-demo`, `kwrite`, `xterm`) running as child
processes of the Node.js proxy. Apps use the host's own system libraries
— **no sysroot, no chroot, no container**.
## Process model — three layers
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
A single Node.js process listens on `:8081`. Per browser session, it
`fork()`s one `SessionProcess` child — **one child per `compositorSessionId`**,
not per app. Multiple apps in the same browser tab share one child.
WebSocket upgrades are handed off to the matching child via
`childProcess.send(wsUpgrade, socket)`.
CLI args:
- `--bind-port=8081` (default)
- `--applications=<json>` (URL → executable map)
- `--encoder=x264|nvh264|vaapih264`
- `--render-device=/dev/dri/renderD128`
- `--base-url`, `--allow-origin`, `--bind-ip`, `--basic-auth`
### Layer 2: Session Process (`SessionProcess.ts`)
The forked child. Receives IPC messages:
- `start` — creates `Session` + `SessionController`
- `launchApp` — spawns the actual app binary
- `wsUpgrade` — hands WS to the controller
On `launchApp` it calls `launchApplication()` from `@gfld/compositor-proxy`,
which `spawn()`s the executable, captures `pid`, replies to main with
`{pid, key}`. On `session.close()` the process exits.
### Layer 3: Proxy Core (`compositor-proxy/src/`)
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
wl_display FD into Node's libuv loop via `startPoll()`.
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
Uses `MessageInterceptor` + auto-generated per-resource interceptor
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
`{native, browser, neverReplies?}` routing decision. Handles
`wl_display.sync` fast-path (skips browser round-trip when no reply
expected). Marshalls FDs as `ProxyFD` JSON.
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
(10 s) on force-quit, auto-destroy on app exit.
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
connect). Xwayland itself becomes a Wayland client of the proxy. The
`wm_fd` is bridged via xtsb to the browser, which implements the actual
XWM logic.
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
## App launch flow (definitive answer)
1. Browser sends `GET http://proxy:8081/<path>` with
`X-Compositor-Session-Id: <uuid>` header.
2. Main process forks a `SessionProcess` if none exists for that session
ID, sends `start` IPC.
3. Main sends `launchApp` IPC with `{name, executable, args, env, serial}`.
4. `SessionProcess` calls `launchApplication()`:
- `appEnv = {...process.env, ...env, WAYLAND_DISPLAY:
session.nativeWaylandCompositorSession.waylandDisplay}`
- `child_process.spawn(executable, args, {env: appEnv})` — **plain
`spawn`, no chroot/container**
5. On `spawn` event, captures `pid`, creates `NativeAppContext`, registers
`exit` listener.
6. Replies to main with `{pid, key}` → main returns HTTP 201 with
`{baseURL, signalURL, key, pid, name}`.
7. Browser opens `signalURL` as WebSocket → control channel established.
8. App connects to `$XDG_RUNTIME_DIR/wayland-N` → libwayland fires
`on_client_created` → proxy looks up PID via `getCredentials()`, walks
`/proc/<pid>/status` PPid chain to match a `NativeAppContext`.
9. For each new `wl_client`, creates a `PROTOCOL` ARQ channel and sends
`CONNECT_CHANNEL` over the signaling WS; browser opens a new WS at
`/channel?id=…`.
10. Wayland wire messages now flow: app → libwayland →
`setWireMessageCallback` → JS interceptor → routed to `{native,
browser}`.
## Native addon architecture
**Important**: The "native" code is **C, not C++** (one GLSL shader string +
GObject-style). Uses Node-API (N-API).
### `wayland-server-addon.node` (1231 lines C)
Wraps a **patched** `libwayland-server` (`libwayland-server.so.0` with
Greenfield additions in `native/wayland/changes.diff`):
- `wl_client_set_wire_message_cb`
- `wl_client_set_registry_created_cb`
- `wl_client_set_sync_done_cb`
- `wl_client_add_resource_created_listener`
- `wl_display_set_global_created_cb`
- `wl_resource_destroy_silently`
- `wl_get_server_object_ids_batch`
- `wl_connection_copy_fds_in`
Plus the `westfield` shared lib (EGL/GBM init, `wlr_linux_dmabuf_v1`,
`wlr_drm`, `westfield-xwayland`). **31 N-API functions** exposed. This is
what makes the proxy a real Wayland compositor.
### `proxy-poll-addon.node` (127 lines C)
Tiny wrapper around `uv_poll_t` (libuv, not raw epoll — though libuv uses
epoll under the hood). `startPoll(fd, cb)` integrates the wl_display
event-loop FD into Node's libuv loop so wayland events fire as Node
callbacks without blocking.
### `proxy-encoding-addon.node` (290 lines C) + `libproxy-encoding.so` (~1400 lines C)
GStreamer H.264/PNG encoder. Runs on a **dedicated GMainLoop thread** with
an `eventfd`-based message queue (`SyncSource` GSource). Two pipelines per
surface (opaque + alpha) for translucent windows. Supports `x264`
(software), `nvh264` (Nvidia NVENC), `vaapih264` (Intel, experimental).
Zero-copy DMA-BUF path via EGLImage → GStreamer GL. Uses
`napi_threadsafe_function` to marshal encoded samples back to Node.
## XWayland bridge
### Native side (`westfield-xwayland.c`, 466 lines C)
1. Creates `/tmp/.X11-unix/X<N>` listener socket
2. `setenv("DISPLAY", ":<N>", 1)`
3. Waits for X client connect
4. On connect: `socketpair(AF_UNIX)` × 3 (wayland / wm / display)
5. `wl_client_create(wl_display, sv[0])` — **Xwayland itself becomes a
Wayland client of the proxy!**
6. `fork()` + `execlp("Xwayland", "Xwayland", ":<N>", "-ac", "-rootless",
"-listen", "<unix_fd>", "-displayfd", "<display_fd>", "-wm", "<wm_fd>",
NULL)`
### xtsb (X11 TypeScript Bindings)
`libs/xtsb/`: pure-TS X11 client, no libxcb. Dual-environment (Node
`net.Socket` + browser WebSocket). Auto-generated from XCB XML protocol
files via `xcbgen/ts_client.py`. Handles MIT-MAGIC-COOKIE-1 auth from
`~/.Xauthority`. The browser-side xtsb implements the actual XWM logic
(translates X `CreateNotify` events → `xdg_toplevel` Wayland surfaces).
`XWaylandSession.ts` uses `nodeFDConnectionSetup(wmFd)` to wrap the wm_fd
as a `net.Socket`, performs X11 auth, and pipes the X11 wire protocol
through the XWM data channel to the browser.
## SDK misconception corrected
The `sdk/` directory is **NOT for native apps**. It's exclusively for
**WASM apps** that run inside the browser via Emscripten. The
`sysrootlibs/` cross-compiles wayland/gtk/cairo/etc. to wasm32. Native
Linux apps on the proxy use the host's own system libraries — the
Dockerfile installs them via `apt-get install libffi8 libgbm1
libgstreamer1.0-plugins-base … xwayland xauth`. The proxy's `pkg` config
only bundles three libs as assets (`libproxy-encoding.so`,
`libwayland-server.so.0`, `libwestfield.so`) — these are the proxy's OWN
addon dependencies, not the apps'.
## ASCII diagram — native app launch & connection
```
Browser (compositor-shell) Proxy host (Linux) App host (often same as proxy)
========================== ==================== ==============================
[1] User clicks app in shell $ compositor-proxy-cli \
│ └─── main.ts listens on :8081
│ sessionProcesses: Record<id, ChildProcess>
│
│ HTTP GET /gtk4-demo
│ Header: X-Compositor-Session-Id: <uuid>
├──────────────────────────────► │
│ │ if no child for <uuid>: fork(SessionProcess.ts)
│ │ └── IPC: {type:'start', payload:{config, sessionId}}
│ │ └── createSession() → NativeWaylandCompositorSession
│ │ ├── createDisplay()
│ │ │ └── wl_display_create() (libwayland-server)
│ │ ├── addSocketAuto()
│ │ │ └── creates $XDG_RUNTIME_DIR/wayland-N
│ │ │ returns "wayland-N" ◄── this becomes WAYLAND_DISPLAY
│ │ ├── initShm() + YUV420/NV12/YUV444/NV21
│ │ ├── initDrm("/dev/dri/renderD128")
│ │ │ └── westfield_egl_new() → EGLDisplay via GBM
│ │ │ wlr_linux_dmabuf_v1 global (zero-copy GPU buffers)
│ │ │ wlr_drm global (legacy dmabuf)
│ │ └── startPoll(wlDisplayFd, dispatchRequests)
│ │ └── uv_poll_t on EPOLLIN
│ │ (proxy-poll-addon.node)
│ │
│ │ IPC: {type:'launchApp', payload:{name, executable, args, env, serial}}
│ │ └── launchApplication() (NativeAppContext.ts)
│ │ ├── appEnv = {...process.env, ...env,
│ │ │ WAYLAND_DISPLAY: "wayland-N"}
│ │ ├── child_process.spawn("gtk4-demo", [], {env: appEnv})
│ │ │ ◄── posix_spawn/execvp, NO chroot, NO container
│ │ │ ──────────────────────────────► [2] gtk4-demo process starts
│ │ │ │
│ │ │ │ libwayland-client.so
│ │ │ │ connects to
│ │ │ │ $XDG_RUNTIME_DIR/wayland-N
│ │ │ │
│ │ │ ▼
│ │ │ ┌──────────────────────────┐
│ │ │ │ wl_display event loop │
│ │ │ │ (in proxy, libwayland- │
│ │ │ │ server.so.0) │
│ │ │ │ │
│ │ │ ◄── uv_poll fires, │ on_client_created → │
│ │ │ dispatchRequests() │ getCredentials(pid) │
│ │ │ wl_event_loop_dispatch() │ findMatchingNativeApp │
│ │ │ │ Context(pid) by PPID │
│ │ │ │ walk │
│ │ │ │ │
│ │ │ │ setWireMessageCallback │
│ │ │ │ setBufferCreated... │
│ │ │ │ setSyncDoneCallback │
│ │ │ │ setClientDestroyed... │
│ │ │ │ │
│ │ │ │ NativeWaylandClientSession
│ │ │ │ .messageInterceptor │
│ │ │ │ dispatches R<opcode> │
│ │ │ │ → {native, browser} │
│ │ │ └──────────────────────────┘
│ │ │
│ │ ├── child.on('spawn') → capture pid
│ │ │ createNativeAppContext(pid, name, external:false)
│ │ │ nativeAppContext.onDisconnect() // start 10-min SIGHUP timer
│ │ │
│ │ └── child.on('exit') → nativeAppContext.onExit({exitCode|signal})
│ │
│ │ IPC reply: {type:'launchAppSuccess', payload:{pid, key}}
│
│ HTTP 201 Created
│ Body: { baseURL, signalURL:"ws://…/signal?key=…", key, pid, name, internal:false }
◄────────────────────────────── │
│ │
│ Open WS: ws://…/signal?key=<16-hex>
├──────────────────────────────► │
│ │ SessionController demux /signal → nativeAppContext.onConnect(ws)
│ │ clears 10-min SIGHUP timer
│ │
│ │ For each new wl_client (one per wayland connection):
│ │ createProtocolChannel() → ARQChannel (KCP over WS)
│ │ sends CONNECT_CHANNEL {url:"ws://…/channel?id=…&key=…", desc}
│ ◄──── CONNECT_CHANNEL JSON ───┤
│ │
│ Open WS: ws://…/channel?id=…&key=…
├──────────────────────────────► │ binds WS to the ARQChannel
│ │
│ Wayland wire protocol flows over the ARQ channel (KCP reliable, 100ms interval):
│
│ App → wl_surface.commit ──── proxy intercepts R1 (attach) + R6 (commit) ────► browser compositor
│ App ← wl_surface.frame ──── proxy intercepts E1 (frame callback) ◄────── FrameFeedback paces callback
│ App → wl_display.sync ────── proxy fast-path sync (no roundtrip if no reply expected)
│ App ← wl_keyboard.enter ─── browser → proxy → sendEvents(wlClient, …) → flush(wlClient)
│
│ Buffer path (per wl_surface):
│ - App submits wl_buffer (dmabuf or shm)
│ - setBufferCreatedCallback → ProxyBuffer with creationSerial
│ - On commit: wl_surface_interceptor.R6 → Encoder.encodeFrame()
│ └── proxy-encoding-addon.encodeFrame(bufferId, contentSerial, creationSerial)
│ └── GStreamer pipeline (x264/nvh264/vaapih264) on dedicated GMainLoop thread
│ └── encoded H.264 sample → napi_threadsafe_function → Node main thread
│ └── sent over the surface's FRAME data channel (separate WS, also CONNECT_CHANNEL'd)
│
│ X11 path (xterm etc.):
│ - App connects to /tmp/.X11-unix/X1 (DISPLAY=:1, set by westfield_xwayland.c)
│ - First X connect → spawn_xserver() forks: execlp("Xwayland", …, "-rootless", "-wm", wm_fd)
│ - Xwayland itself becomes a Wayland client of the proxy (wl_client_create)
│ - When Xwayland ready (displayFd readable):
│ └── xtsb.nodeFDConnectionSetup(wmFd) → wraps as net.Socket
│ └── pipes through XWM data channel (4th channel type) to browser
│ └── browser-side xtsb speaks X11 → translates X windows → xdg_toplevel surfaces
│
│ Lifecycle:
│ - User closes browser tab → /signal WS closes with code 1001 → kill('SIGHUP')
│ - User clicks "force quit" in shell → /signal WS sends KILL_APP {signal:'SIGTERM'} → kill('SIGTERM')
│ └── if not exited in 10s → SIGKILL
│ - App crashes → child 'exit' event → nativeAppContext.onExit() → APP_TERMINATED msg → browser UI updates
│ - All clients gone for >1s → session.close() → process.exit() in SessionProcess.ts → main deletes entry
│
│ Native addon role summary:
│ - wayland-server-addon.node: full Wayland compositor (libwayland-server fork + westfield EGL/GBM/dmabuf/XWayland)
│ - proxy-poll-addon.node: libuv uv_poll_t hook so wayland FD events fire as Node callbacks
│ - proxy-encoding-addon.node: GStreamer H.264/PNG encoder on a dedicated GMainLoop thread, threadsafe_function back to Node
```
## File reference
### CLI (`packages/compositor-proxy-cli/src/`)
| File | Role |
|---|---|
| `main.ts` | HTTP + WS hub, forks per session |
| `main-controller.ts` | CLI arg parsing |
| `main-args.ts` | CLI arg definitions |
| `SessionProcess.ts` | Per-session child, IPC handler |
| `example-applications.json` | Pre-configured apps |
| `wait-until-ready.sh` | Startup readiness check |
### Proxy TS layer (`packages/compositor-proxy/src/`)
| File | Role |
|---|---|
| `index.ts` | Exports `createSession()`, `launchApplication()` |
| `Session.ts` | Top-level proxy session |
| `SessionController.ts` | WS server with 9 endpoint paths |
| `NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
| `NativeWaylandClientSession.ts` | Per-client message interceptor |
| `NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
| `Channel.ts` | ARQChannel + SimpleChannel |
| `kcp.ts` | ~1100-line KCP implementation |
| `FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
| `ProxyBuffer.ts` | Proxy-side buffer representation |
| `SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
| `XWaylandSession.ts` | XWayland spawn + bridge |
| `wayland-server.ts` | TS wrapper around wayland-server-addon.node |
| `io/` | WebSocket server glue |
| `encoding/` | High-level encoding orchestration |
| `protocol/` | Generated interceptor modules |
| `addons/` | .node file shims (Linux-x64) |
| `config.ts` | Encoding-related configuration |
### Proxy C layer (`packages/compositor-proxy/native/`)
| Path | Role |
|---|---|
| `CMakeLists.txt` | Build config |
| `wayland/src/addon.c` | Node-API addon: wayland-server-addon.node |
| `wayland/src/westfield-buffer.c` | Buffer helpers |
| `wayland/src/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM |
| `wayland/src/westfield-egl.c` | EGL zero-copy setup |
| `wayland/src/westfield-xwayland.c` | XWayland spawn + bridge |
| `wayland/src/wlr_drm.c` | wl_drm global impl |
| `wayland/src/wlr_linux_dmabuf_v1.c` | zwp_linux_dmabuf_v1 global impl |
| `wayland/src/drm-protocol.c` | DRM protocol helpers |
| `wayland/src/linux-dmabuf-unstable-v1-protocol.c` | linux-dmabuf-v1 protocol stubs |
| `wayland/src/wayland-server/` | Vendored + patched libwayland-server (with changes.diff) |
| `encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder |
| `encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
| `encoding/src/node_encoder.c` | Node-API addon: proxy-encoding-addon.node |
| `poll/src/poll.c` | libuv uv_poll_t wrapper |
| `node-api/` | Vendored Node-API headers |
### XWayland support (`libs/xtsb/`)
| File | Role |
|---|---|
| `README.md` | Project intro |
| `package.json` | @gfld/xtsb package config |
| `protocol/xproto.xml` | X11 core protocol XML (5800+ lines) |
| `protocol/shape.xml` | Shape extension XML |
| `protocol/xfixes.xml` | XFIXES extension XML |
| `protocol/render.xml` | Render extension XML |
| `protocol/composite.xml` | Composite extension XML |
| `src/` | Generated TS bindings |
## Next actions for user (Odoo 20 integration)
1. Build the proxy on a Linux host (or use `udevbe/compositor-proxy-cli`
Docker image).
2. Write an `applications.json` mapping desired apps to HTTP paths.
3. Start proxy with `--allow-origin=https://your-odoo.example.com`.
4. Embed the Greenfield `compositor-shell` Preact SPA in Odoo (iframe or
QWeb template) — OR replace it with your Odoo 20 OWL shell (see
`10-porting-to-odoo-js-python.md`).
5. Trigger app launches from Odoo via
`fetch('http://<proxy>:8081/<path>', { headers: { 'X-Compositor-Session-Id': sessionId }})`
— Odoo frontend JS uses the returned `signalURL` to open the per-app
control WS.
6. For multi-tenant isolation: one `SessionProcess` per browser session is
automatic, but file-system isolation between tenants requires separate
proxy instances or containers per tenant.
7. For GPU acceleration: pass `--device /dev/dri:/dev/dri` to Docker; use
`--encoder=nvh264|vaapih264|x264` based on GPU.
+304
View File
@@ -0,0 +1,304 @@
# 08 — App Lifecycle (XDG Shell, Toplevels, Popups, Subsurfaces)
This chapter covers Greenfield's application lifecycle layer: how xdg-shell
(toplevels, popups, positioners), wl_shell (legacy), and subsurface
compositing are implemented, and how window state transitions are handled.
EXCLUDE the actual UI shell (`packages/compositor-shell`) — focus on the
protocol-level lifecycle.
## Three role families, one shared abstraction
All surface roles implement `SurfaceRole` (`{ view, optional
desktopSurface, onCommit(surface) }`). `XdgToplevel`, `XdgPopup`, and
`ShellSurface` additionally implement `DesktopSurfaceRole`, which bridges
them to a `DesktopSurface` (`FloatingDesktopSurface` or
`AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`). The
`DesktopSurface` is where **every** `userShell.events.*` emission
originates — making it the true integration boundary.
## XdgToplevel lifecycle — the headline diagram
```
┌───────────────────────────────────────────────┐
│ wl_surface created (no role, no buffer yet) │
└─────────────────────┬─────────────────────────┘
│ client: xdg_wm_base.get_xdg_surface
│ (errors if surface already has buffer)
▼
┌───────────────────────────────────────────────┐
│ XdgSurface created │
│ • surface.hasKeyboard/Pointer/Touch = true │
│ • configured=false, configureList=[] │
└─────────────────────┬─────────────────────────┘
│ client: xdg_surface.get_toplevel
│ (errors if surface has another role)
▼
┌───────────────────────────────────────────────┐
│ XdgToplevel created │
│ • surface.role = xdgToplevel │
│ • View created │
│ • DesktopSurface created + init() │
│ • ackConfigure override installed │
└─────────────────────┬─────────────────────────┘
│ client: xdg_toplevel.set_title /
│ set_app_id / set_min/max_size /
│ set_parent (stored, propagated)
▼
┌───────────────────────────────────────────────┐
│ First wl_surface.commit (buffer === undefined) │
│ XdgToplevel.onCommit() → ensureAdded() │
│ • desktopSurface.add() │
│ → userShell.events.surfaceCreated(...) │
│ • scheduleConfigure(first) │
│ → microtask → sendConfigure() │
│ → resource.configure(w,h,states) │
│ • added=true │
└─────────────────────┬─────────────────────────┘
│ client receives configure
│ client: xdg_surface.ack_configure(serial)
▼
┌───────────────────────────────────────────────┐
│ ackConfigure(serial, configure) │
│ • handleAckConfigure trims older serials │
│ • configured=true │
│ • next.state = configure.state │
│ • next.size = configure.size │
└─────────────────────┬─────────────────────────┘
│ client: wl_surface.commit (with buffer)
▼
┌───────────────────────────────────────────────┐
│ ★ MAPPED (steady state) ★ │
│ onCommit(): │
│ • surface.commitPending() (state latch) │
│ • xdgSurface.commit() (geometry latch) │
│ • current.state ← next.state │
│ • current.min/max ← next.min/max │
│ • desktopSurface.commit() → render() │
└─────────────────────┬─────────────────────────┘
│
┌────────────────────────────────────────┼────────────────────────────┐
│ │ │
user/shell requests client requests destruction
│ │ │
▼ ▼ ▼
┌──────────────────────────┐ ┌─────────────────────────────┐ ┌──────────────────────┐
│ configureMaximized(b) │ │ setMaximized / unsetMaximized│ │ wl_surface.destroy │
│ configureFullscreen(b) │ │ setFullscreen/unsetFullscreen│ │ or xdg_toplevel. │
│ configureActivated(b) │ │ setMinimized │ │ destroy │
│ configureResizing(b) │ │ setParent / setTitle / │ ├──────────────────────┤
│ configureSize(w,h) │ │ setAppId │ │ destroy listener: │
└────────────┬─────────────┘ │ move / resize / showWindowMenu│ │ • surface.unmap() │
│ └──────────────┬──────────────┘ │ • desktopSurface. │
│ each mutates │ │ removed() │
│ pending.state / pending.size │ │ → surfaceDestroyed │
▼ │ │ • view.destroy() │
┌──────────────────────────┐ │ └──────────────────────┘
│ scheduleConfigure( │ │
│ stateCompareEquals(), │ ┌─────────────────────────────┐
│ sendConfigure) │ │ setMinimized → │
│ • coalesce via µtask │ │ desktopSurface.minimize() │
│ • if pending==current → │ │ • view.mapped=false │
│ skip │ │ • seat drops keyboard focus │
│ • else: push serial, │ │ if this was the focused │
│ emit configure event │ │ surface │
│ │ └─────────────────────────────┘
└────────────┬─────────────┘
│
│ cycle repeats: ack_configure → commit → render
└────────────────────────────────────► back to ★ MAPPED ★
```
## Configure / ack-configure round-trip
This is the **most important mechanism** to understand in xdg-shell. It's
an asynchronous protocol-level sync between the compositor and the
client.
### Why it's needed
The compositor wants to tell the client "please resize to (800, 600)
because the user maximized you". But the client may not be ready to
resize, or may want a different size (e.g. the app enforces a 16:9 aspect
ratio). The configure/ack round-trip lets the client acknowledge the
configure with a serial, then on the next commit, the compositor knows
which configure the client has applied.
### How Greenfield implements it
1. Compositor mutates `pending.{state,size}`.
2. Coalesces via `queueCancellableMicrotask` with a `stateCompareEquals`
short-circuit (if pending == current, skip).
3. Emits `resource.configure(w,h,states)` with a fresh serial.
4. Client acks: `xdg_surface.ack_configure(serial)`.
5. `handleAckConfigure` trims older serials, sets `configured=true`,
copies `configure.{state,size}` into `next`.
6. Next commit latches `next → current`.
The `configured` flag gates buffer attachment: an `unconfiguredBuffer`
error is raised if the client tries to attach a buffer before acking.
## XdgPositioner math
`anchor` selects a point on the anchor rect (in parent window-geometry
space); `gravity` selects which corner/edge of the popup is placed at
that point; `offset` is applied last.
### Constraint adjustment cascade
Runs as X-then-Y cascade:
1. Try **flip** — swap anchor+gravity via `inverseX` / `inverseY` tables,
revert if still violating.
2. Try **slide** — shift offset by violation delta.
3. Try **resize** — shrink `sizeRect`.
Slides/flips are skipped if popup size on that axis exceeds viewport.
## Popup grabs
`XdgPopup.grab` rejects if:
- Surface is already mapped, OR
- Parent isn't either:
- a toplevel (root of new chain), OR
- the topmost popup of an existing grab (`notTheTopmostPopup` error)
`seat.popupGrabStart` validates the serial against the keyboard/pointer
`grabSerial`, then starts a `PopupGrab` on both devices. `popupGrabEnd`
walks the chain in reverse calling `requestClose()` on each (sends
`popup_done` / `xdg_popup.popup_done`).
## Subsurfaces sync/desync
Three-layer mechanism:
1. **`cachedState`** — full pending-state shadow on the subsurface role.
2. **`commitCache`** — flush on parent commit.
3. **`effectiveSync`** getter — cascades: if any ancestor is sync, this
subsurface is effectively sync.
### Decision matrix
| Sync mode | Dirty | Action |
|---|---|---|
| Sync | yes | Cache to `cachedState`, flush when parent commits |
| Sync | no | No-op (already cached) |
| Desync | yes | Flush immediately to parent's pending state |
| Desync | no | Acts like a normal surface (commit directly to its own pendingState) |
`setDesync` immediately calls `onCommit` to flush any pending cache.
## wl_shell (legacy path)
Legacy path. Reduced feature set:
- `configureActivated` / `configureResizing` are explicit no-ops.
- State machine via `SurfaceStates` enum (MAXIMIZED / FULLSCREEN / POPUP /
TRANSIENT / TOP_LEVEL) and `changeState()`.
- Per-surface ping (separate from `XdgWmBase`'s per-client ping).
## The UserShellApi seam — critical for the Odoo 20 OWL shell
Located in `UserShellApi.ts`, exposed as `session.userShell: UserShellApi =
{ events, actions }`. The default `createUserShellApi(session)` returns
`events: {}` (empty) and a populated `actions` object. **The shell
assigns the event callbacks; the compositor always invokes them with
`?.()` so unhandled events are silent no-ops.** This is the single
integration seam — the protocol layer above, and the shell UI below.
### Events (compositor → shell) — all optional
| Event | Payload | Fired from | When |
|---|---|---|---|
| `clientCreated` | `CompositorClient` | `WebConnectionHandler`, `RemoteAppLauncher` | A new app client connects. |
| `clientDestroyed` | `CompositorClient` | same | A client disconnects. |
| `clientUnresponsiveUpdated` | `(client, unresponsive: boolean)` | `XdgWmBase`, `ShellSurface` | Ping (2s timeout, every 5s) — `true` = hung, `false` = recovered. |
| `surfaceCreated` | `CompositorSurface` | `FloatingDesktopSurface.add()`, `AlwaysFullScreenDesktopSurface` | Toplevel first added to desktop. |
| `surfaceDestroyed` | `CompositorSurface` | `.removed()` | Window destroyed. |
| `surfaceTitleUpdated` | `(surface, title)` | `.setTitle()` | `xdg_toplevel.set_title` / `wl_shell_surface.set_title`. |
| `surfaceAppIdUpdated` | `(surface, appId)` | `.setAppId()` | `xdg_toplevel.set_app_id` / `wl_shell_surface.set_class`. |
| `surfaceActivationUpdated` | `(surface, active: boolean)` | `gainFocus()` / `loseFocus()` | Refcounted via `focusCount` — only fires on 0↔1 transitions. |
| `notify` | `(variant, message)` | (declared, no direct emissions) | Free-form notification channel. |
| `sceneRefreshed` | `sceneId: string` | `Scene:75` | Render finished. |
### Actions (shell → compositor) — provided by default
| Action | Effect |
|---|---|
| `initScene(canvasCreator)` | Bootstraps a WebGL canvas + input/output. One per output. |
| `refreshScene()` | `session.renderer.render()` — manual redraw. |
| `destroyScene(sceneId)` | Destroys a specific scene. |
| `setUserConfiguration(config)` | Updates `scrollFactor`, `keyboardLayoutName`. |
| `closeClient({ id })` | Closes a client connection. |
| `activateSurface(compositorSurface)` | **Click-to-focus entry point from the shell's taskbar.** Reverse-lookups the surface via `session.display.clients[clientId].connection.wlObjects[surfaceId]` and calls `desktopSurface.activate()`. |
### Opaque handle types
```
interface CompositorSurface { id: number; client: CompositorClient }
interface CompositorClient { id: string }
```
`toCompositorSurface(desktopSurface)` adapts `{ surface.resource.id,
surface.resource.client.id }`; `lookupSurface(session, compositorSurface)`
is the reverse walk.
## Notable gaps for the Odoo OWL shell
1. **No maximize/fullscreen/resize/minimize events** in
`UserShellApiEvents`. The shell can't directly observe window-state
transitions through this API — only focus (`surfaceActivationUpdated`)
and metadata (title/appId). For a taskbar this is fine; for
window-decorations rendering it is not. Options: (a) extend
`UserShellApiEvents` with `surfaceStateChanged`; (b) the OWL shell
imports `DesktopSurface` directly and introspects; (c) add a
`querySurfaceState` action.
2. **No geometry/position event** — the shell can't render a frame that
hugs surface bounds from the shell side without taking over rendering.
3. **`notify` declared but unused** in compositor core — dead interface
slot.
4. **`surfaceCreated` fires once on first commit; no separate map/unmap
signal** — minimization is internal (`view.mapped=false`) and not
propagated.
## Risks flagged
- `XdgWmBase.doPing` uses serial 0 (`// FIXME use a proper serial`) — works
only because there's a single outstanding ping per client.
- `XdgToplevel.onCommit` has commented-out buffer-size-vs-configured-state
validation (lines 140–158) — the compositor trusts clients to honor
configured sizes.
- `XdgSurface.scheduleConfigure`'s `pendingSame` argument inverts meaning
depending on whether a microtask is already queued — subtle and easy to
break in maintenance.
## File reference
| File | Role |
|---|---|
| `packages/compositor/src/Shell.ts` | `wl_shell` global (legacy) |
| `packages/compositor/src/ShellSurface.ts` | `wl_shell_surface` role |
| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global |
| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
| `packages/compositor/src/SurfaceRole.ts` | SurfaceRole interface |
| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction |
| `packages/compositor/src/UserShellApi.ts` | The integration seam — events + actions |
| `packages/compositor/src/XdgWmBase.ts` | `xdg_wm_base` global, ping-based unresponsiveness detection |
| `packages/compositor/src/XdgSurface.ts` | `xdg_surface` role, configure/ack round-trip |
| `packages/compositor/src/XdgToplevel.ts` | `xdg_toplevel` role, window state |
| `packages/compositor/src/XdgPopup.ts` | `xdg_popup` role, transient windows, grabs |
| `packages/compositor/src/XdgPositioner.ts` | `xdg_positioner`, popup positioning math |
| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
## Next actions (for Odoo 20 OWL shell)
1. Design `createOdooUserShellApi(session)` returning the default
`actions` and OWL-signal-bound callbacks for the 8 events.
2. Decide how to surface maximize/fullscreen/resize/minimize state (extend
events vs. introspect `DesktopSurface`).
3. Wire `clientCreated` / `clientDestroyed` to an Odoo `AppContext`-style
registry for the app launcher UI.
4. Feed `surfaceActivationUpdated` into an OWL reactive store as the
single source of truth for active-window focus.
5. `initScene(canvasCreator)` — OWL shell owns the `HTMLCanvasElement`
lifecycle, one per output.
+423
View File
@@ -0,0 +1,423 @@
# 09 — Hardware Acceleration, Encoding, WASM, GPU Textures
This chapter covers Greenfield's hardware acceleration / encoding / WASM
pipeline: how native buffers get encoded on the proxy side, transported,
decoded in the browser, and turned into WebGL textures or canvas images.
## What Greenfield actually ships
### Two Wasm libraries under `libs/`
**`compositor-wasm`** builds **pixman 0.42.2 + xkbcommon 1.5.0**
(Emscripten 3.1.46, `-msimd128 -O3 -flto`). Used by the browser for
**region math** (damage tracking) and **keyboard layout** — *not* for
pixel format conversion or H.264 decode.
**`compositor-ffmpeg-h264`** is a **stripped-down FFmpeg n5.1.2**
(`--disable-everything --enable-decoder=h264`) compiled to a 1.7 MB
single-file Wasm worker. A 30-line `decoder.c` wrapper exposes
`create_codec_context` / `decode` / `close_frame` / `destroy_codec_context`.
Decodes to **I420 (YUV 4:2:0 planar)** and returns plane pointers; planes
are `postMessage`'d back to the main thread as **transferable
`ArrayBuffer`s**. No threads (single-threaded, SIMD only).
### Native encoder addon (`packages/compositor-proxy/native/encoding/`)
A **GStreamer wrapper** exposed via Node-API. The TS side (`Encoder.ts`)
calls `appEndpointNative.encodeFrame(encoder, bufferId, contentSerial,
creationSerial)`. The C side (`node_encoder.c`) sniffs the wl_resource:
- `zwp_linux_dmabuf_v1` → DMA path
- Legacy `wl_drm` → DMA path
- `wl_shm_buffer` → SHM path
For DMA-BUFs the proxy wraps the dma-buf as an `EGLImage` and then as a
`GstGLMemoryEGL`, so pixels **stay on GPU** through the color-convert
stages.
### Threading
Encoding runs on a dedicated GLib main-loop thread fed by an
`eventfd`-signalled `GAsyncQueue`. Output returns to Node via a
`napi_threadsafe_function` with **zero-copy** external Buffers (finalised
back to C on GC).
## Encoder selection (per-frame, deterministic)
`gst_frame_encoder.c::frame_encoder_descriptions[]` declares **four**
pipelines:
| name | element | when used | split_alpha | coded-size multiple |
|---|---|---|---|---|
| `x264` | `x264enc` (software, default) | >256×256 + ARGB/XRGB/dmabuf | true (parallel alpha H.264 stream) | 128 |
| `nvh264` | `nvh264enc` (Nvidia NVENC) | same, `--encoder=nvh264` | true | 128 |
| `vaapih264` | `vaapih264enc` (Intel/AMD VA-API, "experimental") | same, `--encoder=vaapih264` | true | 128 |
| `png` | `pngenc` (fallback) | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 |
**Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264
encoder. YUV420/NV12/NV21/Y444 SHM formats are *not* supported on the
H.264 path and only by PNG at small sizes — effectively unsupported for
video-class content.
### Canonical H.264 pipeline
```
appsrc → glupload → glcolorconvert → glshader(scale→coded_size) →
glcolorconvert → video/x-raw(memory:GLMemory),format=NV12 → gldownload →
queue → x264enc tune=zerolatency bitrate=12000 vbv-buf-capacity=160
bframes=0 rc-lookahead=0 sliced-threads byte-stream=true →
video/x-h264,profile=high,stream-format=byte-stream,alignment=au → appsink
```
The `glshader` element does two jobs:
1. Pad up to the encoder's macroblock multiple (128 for H.264, 16 for PNG)
using a `u_transformation` matrix.
2. For the **alpha pipeline**, run `vec4(pix.a, pix.a, pix.a, 0.)` so
alpha is encoded as a grayscale H.264 stream — H.264 has no alpha, so
Greenfield ships two parallel AU streams.
Bitrates: opaque 12000 / alpha 1200 (x264), 12000 / 1200 (nvh264).
## Wire format
Single contiguous blob, little-endian:
```
struct packed props {
u32 buffer_id, creation_serial, content_serial,
u32 frame_encoding_type (0=h264, 1=png),
u32 width, height, coded_width, coded_height
}
u32 opaque_length
u8 opaque[opaque_length] // H.264 AU or PNG blob
u32 alpha_length // 0 if no alpha split
u8 alpha[alpha_length] // H.264 AU of alpha-only stream
```
Parsed by `remote/EncodedFrame.ts::createEncodedFrame`.
## Transport (`Channel.ts`)
Four logical channels per client multiplexed over WebSockets:
| Channel | Type | Purpose |
|---|---|---|
| PROTOCOL | ARQ (KCP) | Wayland wire protocol + out-of-band ops |
| FRAME | ARQ (KCP) | Encoded surface frames |
| XWM | ARQ (KCP) | XWayland WM messages |
| FEEDBACK | SIMPLE (raw WS) | 4-byte feedback messages |
ARQ = a JS KCP implementation layered over WebSocket. Tuning:
`MTU=64000`, `MAX_BUFFERED_AMOUNT=2949120` (~2.8 MB — the back-pressure
ceiling), `SND_WINDOW=128`, `RCV_WINDOW=1024`,
`setNoDelay(1, 100, 0, 1)`.
## Browser decode (`packages/compositor/src/remote/`)
At session startup `Session.create()` probes
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
hardwareAcceleration:'prefer-software'})`:
### WebCodecs preferred
Software-only; the `prefer-hardware` path is implemented but commented out
with `// FIXME hardware decoding frame offset is wrong`.
### FFmpeg Wasm fallback
If WebCodecs unavailable: two persistent Workers (`h264NALDecoderOpaque`,
`h264NALDecoderAlpha`), one `AVCodecContext` per surface per worker,
keyed by `surfaceH264DecodeId`. Encoded NALs are transferred zero-copy
into the worker; decoded I420 planes are transferred back.
Per-surface decoder context is lazily created on first H.264 frame.
### PNG path (both decoders)
Identical in both branches: `new Blob([opaque], {type:'image/png'})` →
`createImageBitmap(blob, sx, sy, w, h)` (the `sx/sy` crops the encoder
padding back to logical size).
## WebGL upload (`render/`)
WebGL 1 with `desynchronized:true` for low-latency presentation. Per Scene
(canvas/output), three GL helpers:
### `YUVA2RGBA`
FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures**, renders into
the surface's RGBA texture via `YUVA2RGBAShader` (BT.601 coefficients,
alpha recovered by running the alpha-Y plane through the same YUV→RGB
matrix and taking R).
### `RGBXandA2RGBA`
FBO pass for the `DualPlaneRGBA*` WebCodecs outputs: opaque + alpha planes
composited via `RGBandA2RGBAShader`.
### `SceneShader`
Final pass: draws every view's RGBA texture to the canvas with
`gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)` (note: expects premultiplied
alpha but the YUVA shader outputs straight alpha — a minor inconsistency).
PNG and web-bitmapbuf paths skip the FBO and call
`texImage2D(ImageBitmap)` directly.
### Alpha pre-multiplication
Not done explicitly anywhere. The proxy extracts alpha as a Y plane; the
browser shader recombines; the canvas blend mode assumes premultiplied but
receives straight alpha — works acceptably because Wayland surfaces rarely
use partial alpha.
## End-to-end pipeline
```
┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
│ App renders via GTK/Qt/raw EGL into │
│ ┌────────────────────────────────────┐ │
│ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │
│ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │
│ │ wl_drm create_prime_buffer │ │ │
│ └────────────────────────────────────┘ │ │
└────────────────────────┬──────────────────────────────────┼────┘
│ Wayland socket (UNIX domain) │
▼ │
┌────────────────────── COMPOSITOR PROXY (Node + N-API addons)─┐
│ wayland-server-addon.node │
│ └─ wl_display → wl_client → wl_resource (buffer) │
│ └─ initShm(), initDrm("/dev/dri/renderD128") │
│ └─ westfield-egl.c: EGLDevice + EGLDisplay + EGLContext │
│ │
│ proxy-encoding-addon.node (per-surface Encoder) │
│ ┌─ node_encoder.c → frame_buffer {type: SHM|DMA, w, h, fmt} │
│ │ │
│ │ gst_main_loop.c (own GLib thread, eventfd work queue) │
│ │ │ │
│ │ ▼ │
│ │ gst_frame_encoder.c │
│ │ ┌─ Encoder selection (per-frame): │
│ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │
│ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │
│ │ │ │
│ │ │ H.264 pipelines (parallel): │
│ │ │ opaque: appsrc → glupload → glcolorconvert → │
│ │ │ glshader(scale→coded_size) → │
│ │ │ glcolorconvert(NV12) → gldownload → │
│ │ │ queue → x264enc|nvh264enc|vaapih264enc → │
│ │ │ byte-stream AU → appsink │
│ │ │ alpha: same source, fragment shader │
│ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │
│ │ │ second H.264 stream │
│ │ │ │
│ │ │ PNG pipeline (single): │
│ │ │ appsrc → glupload → glcolorconvert → glshader → │
│ │ │ RGBA → gldownload → pngenc → appsink │
│ │ │ │
│ │ └─ frame_encoding_result_to_encoded_frame(): │
│ │ packed struct {buffer_id, creation_serial, │
│ │ content_serial, type(0=h264,1=png), w, h, │
│ │ coded_w, coded_h} + opaque_len + opaque + │
│ │ alpha_len + alpha → Node Buffer (zero-copy ext.) │
│ │ │
│ └─ SurfaceBufferEncoding.ts │
│ └─ frameDataChannel.send(buffer) │
│ │
│ Channel.ts: │
│ ARQChannel (KCP-reliable over WebSocket, MTU 64 KB) │
│ SimpleChannel (raw WebSocket, for FEEDBACK only) │
│ ▲ │
│ │ FrameFeedback.ts: 16-bit rAF interval + 16-bit avg │
│ │ decode duration → used only to pace wl_frame.done │
└───────┼───────────────────────────────────────────────────────┘
│ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
▼
┌────────────────────── BROWSER (compositor) ────────────────────┐
│ WebSocket → ReconnectingWebSocket │
│ ↓ │
│ Channel.ts: ARQChannel.input(kcp) → onMessage │
│ ↓ │
│ RemoteAppLauncher.setupFrameDataChannel → │
│ BufferStream.deliverContentToBufferStream (parse serials) │
│ ↓ │
│ StreamingBuffer.getContents(surface, serial) │
│ ↓ │
│ Session.frameDecoder.decode(surface, encodedFrame) │
│ │ │
│ ├─ PNG path (both decoders): │
│ │ new Blob([opaque], {type:'image/png'}) → │
│ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │
│ │ → SinglePlane { bitmap: ImageBitmap } │
│ │ │
│ └─ H.264 path: │
│ (preferred) WebCodecs VideoDecoder │
│ config: avc1.64001f, prefer-software, optimizeForLatency│
│ opaqueDecoder + alphaDecoder per surface │
│ EncodedVideoChunk {type:key|delta, data:nal} │
│ → VideoFrame.output → │
│ if I420: copyTo → DualPlaneYUVAArrayBuffer │
│ else: keep VideoFrame → DualPlaneRGBAVideoFrame │
│ (fallback) FFmpeg Wasm in 2 Web Workers │
│ worker.postMessage(nal, [nal.buffer]) // transferable │
│ libavH264._decode → I420 planes │
│ postMessage back with [y.buffer, u.buffer, v.buffer] │
│ → DualPlaneYUVASplitBuffer │
│ │
│ Renderer.render() (rAF-driven, queueCancellableMicrotask) │
│ ↓ │
│ Scene.render(viewStack) │
│ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │
│ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │
│ └─ Scene['video/h264']: │
│ YUVA2RGBA.convertYUVASplitBufferInto / │
│ convertYUVAArrayBufferInto: │
│ Y/U/V/alphaY → 4× LUMINANCE textures │
│ bind FBO → renderState.texture (RGBA) as color attach │
│ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │
│ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │
│ VideoFrame/ImageBitmap → texImage2D directly │
│ ↓ │
│ SceneShader.draw(): drawArrays with surface's RGBA texture, │
│ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │
│ ↓ │
│ Browser composites canvas into the page │
└─────────────────────────────────────────────────────────────────┘
Same-frame web app fast path (web-bitmapbuf):
┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐
│ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │
│ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
└───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘
(no encoding, no WebSocket, no GStreamer)
```
## Key answers
### Why H.264 over PNG?
H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB.
PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split
doubling dominate at small sizes.
### WebCodecs?
Yes — preferred path, software-only (`prefer-hardware` is wired but
disabled with FIXME "frame offset is wrong"). FFmpeg Wasm is the fallback.
### Trade-offs
- H.264 = small wire / decode cost
- PNG = medium wire / browser-native fast decode (`createImageBitmap`)
- raw SHM = **not used** between proxy and browser — every surface commit
is encoded.
### Per-surface policy?
No — per-frame deterministic. Only configurable knob is
`--encoder=x264|nvh264|vaapih264`. PNG is always automatic fallback.
### Cursors?
Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS
cursor via `setBrowserCursor` — never enters the WebGL compositor.
### Alpha pre-multiplication?
Implicit in shaders. Proxy extracts alpha as a Y plane (no premult);
browser recombines via BT.601 YUVA→RGBA shader outputting straight alpha;
canvas blend is `ONE, ONE_MINUS_SRC_ALPHA` (expects premult) — minor
inconsistency, not visually problematic for typical Wayland content.
### OffscreenCanvas / Web Workers?
Workers yes (two for Wasm decoder, shared across surfaces). OffscreenCanvas
**no** — renderer is main-thread WebGL with `desynchronized:true`.
WebGPU/WebGL2 not used.
## Notable design trade-offs for your compositor
1. Always-encode policy (no raw SHM escape hatch even on LAN).
2. One encoder per surface, EOS-and-recreate on resize/format change.
3. Alpha as a separate H.264 stream — doubles cost for transparent windows.
4. Per-surface decoder context, no decoder pooling.
5. Only aggregate feedback (1 Hz rAF interval + decode duration); no
per-frame ACKs, no bitrate adaptation; `requestKeyUnit` plumbing exists
end-to-end but the browser never calls it.
6. KCP over WebSocket (not WebRTC datachannels) — simpler infra, no
STUN/TURN, but no real congestion control.
7. Emscripten SIMD (not threads) — avoids COOP/COEP requirements.
8. Two parallel GL pipelines (opaque + alpha) for H.264.
9. PNG fallback boundary empirically at 256×256 pixels.
10. WebGL 1, not 2/WebGPU — maximum browser compat at the cost of manual
LUMINANCE-texture YUV juggling and a hidden FBO intermediate.
## File reference
### Wasm libraries
| Path | Role |
|---|---|
| `libs/compositor-wasm/build_wasm.sh` | Builds pixman 0.42.2 + xkbcommon 1.5.0 to WASM (emsdk 3.1.46) |
| `libs/compositor-wasm/emscripten-build.ini` | Emscripten build config |
| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function |
| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon build (clones 1.5.0, embeds xkeyboard-config DB) |
| `libs/compositor-wasm/pixman/build.sh` | pixman build |
| `libs/compositor-ffmpeg-h264/build_wasm.sh` | Builds FFmpeg n5.1.2 to WASM (single-file, decoder-only) |
| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API |
| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module |
| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point |
| `libs/compositor-ffmpeg-h264/src/index.ts` | Exports |
### Native encoder addon
| Path | Role |
|---|---|
| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon (290 lines) |
| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder (~1400 lines) |
| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
| `packages/compositor-proxy/native/CMakeLists.txt` | Build config |
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | TS-side encoding orchestration |
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback |
| `packages/compositor-proxy/src/encoding/` | High-level encoding files |
### Browser decode
| Path | Role |
|---|---|
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs VideoDecoder path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
### WebGL rendering
| Path | Role |
|---|---|
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
### Protocol extensions
| Path | Role |
|---|---|
| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom Wayland extension for browser JS apps |
| `protocol/wayland-drm.xml` | Standard wl_drm for DMA-BUF (proxy-only) |
| `packages/compositor/src/web/WebBuffersFactory.ts` | Browser-side impl of web_bitmapbuf_factory |
+644
View File
@@ -0,0 +1,644 @@
# 10 — Porting Blueprint (JS Browser + Python Host + Odoo 20 OWL Shell)
This is the standalone porting chapter. It maps every Greenfield layer
analyzed in chapters 02-09 to concrete guidance for building a
Wayland-to-HTML compositor that integrates with an Odoo 20 addon shell and
a JS + Python backend.
## Target architecture
The recommended architecture:
```
┌──────────────────────────────────────────────────────────────────────────────┐
│ BROWSER (Odoo 20 web client) │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Odoo 20 OWL Shell addon (your code) │ │
│ │ - Application launcher (lists apps from /api/compositor/apps) │ │
│ │ - Window management UI (taskbar, decorations) │ │
│ │ - Keyboard layout picker (RMLVO) │ │
│ │ - Per-output <canvas> element │ │
│ │ - Triggers app launches via Odoo /api/compositor/launch │ │
│ │ - Implements UserShellApi.events.* callbacks │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ userShell.events + userShell.actions │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ @gfld/compositor (browser-side compositor library) │ │
│ │ - Use as-is from npm: @gfld/compositor │ │
│ │ - Replaces compositor-shell with createOdooUserShellApi(session) │ │
│ │ - Renders surfaces to <canvas> via WebGL │ │
│ │ - Manages keyboard/pointer/touch │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘
│
│ WebSocket (TCP, KCP-over-WS)
│ 9 endpoint paths per session
▼
┌──────────────────────────────────────────────────────────────────────────────┐
│ PYTHON HOST (Linux server) │
│ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Odoo 20 backend addon (your code) │ │
│ │ - /api/compositor/apps → list of available apps │ │
│ │ - /api/compositor/launch → triggers proxy to spawn app │ │
│ │ - Session management (one Odoo session ↔ one proxy session) │ │
│ │ - Per-tenant isolation, ACL, audit log │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ spawn / IPC │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Python compositor-proxy (your code — replaces @gfld/compositor-proxy) │ │
│ │ - aiohttp/FastAPI WebSocket server (replaces ws + ws-server) │ │
│ │ - pywayland (cffi bindings to libwayland-server) │ │
│ │ - python-xkbcommon for keymap compilation │ │
│ │ - GStreamer Python bindings (gi.repository.Gst) for H.264 encoding │ │
│ │ - asyncio event loop integrating libwayland FD polling │ │
│ │ - Per-app child_process spawning (subprocess.Popen with env) │ │
│ │ - Sandboxing via bubblewrap + seccomp (NEW — Greenfield has none) │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ WAYLAND_DISPLAY=wayland-N │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │ Native Linux apps (gtk4-demo, xterm, kwrite, ...) │ │
│ │ - Use host system libraries (no sysroot) │ │
│ │ - Sandboxed via bubblewrap + seccomp │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────┘
```
## Per-layer porting guidance
For each Greenfield layer analyzed in chapters 02-09:
### Layer 1: Wayland Protocol XML
**What to copy as-is**: All four protocol XML files (`wayland.xml`,
`xdg-shell.xml`, `wayland-drm.xml`, `web-bitmapbuf-unstable-v1.xml`).
They are upstream standard + Greenfield's custom extension. No need to
rewrite.
**What to replace**: Nothing. The XML is the source of truth.
**Where Python fits**: Python doesn't need to parse the XML directly —
use `pywayland` which already ships with parsed bindings for the standard
protocols. For Greenfield's custom `web-bitmapbuf-unstable-v1`, you can
either skip it (native apps don't use it) or generate Python bindings via
`pywayland-protocol-scanner`.
**Where the Odoo 20 OWL shell hooks in**: Not applicable. This layer is
below the shell.
### Layer 2: Protocol Generators
**What to copy as-is**: The TypeScript generators for the browser side
(`client-generator`, `compositor-generator`). They produce stubs for the
browser compositor and browser-resident JS apps.
**What to replace**: The `compositor-proxy-generator`. Greenfield's proxy
generates CommonJS `*_interface.js` + `*_interceptor.js` files for the
Node N-API addon. For Python, you don't need a generator — use
`pywayland`'s built-in protocol bindings.
**Where Python fits**: `pywayland` provides `wl_interface` structs and
request/event dispatch in pure Python via cffi. You write Python classes
that implement `WlCompositorRequests`, `WlSeatRequests`, etc.
**Where the Odoo 20 OWL shell hooks in**: Not applicable.
### Layer 3: Wayland Runtime
**What to copy as-is**: `@gfld/common` (`Connection.ts`) and
`@gfld/compositor-protocol` for the browser side. These implement the
standard Wayland wire format and object lifecycle, and they're MIT-licensed.
**What to replace**: The proxy's `wayland-server.ts` (which wraps the
`wayland-server-addon.node` N-API addon). Replace with `pywayland`'s
`Display` and `Client` classes.
**Where Python fits**: `pywayland` provides:
- `pywayland.server.Display` — wraps `wl_display`
- `pywayland.server.Client` — wraps `wl_client`
- `pywayland.server.Resource` — wraps `wl_resource`
- `pywayland.server.Global` — wraps `wl_global`
- Per-resource event/request dispatch via Python method overrides
You write Python classes that subclass `WlCompositor`, `WlSeat`, etc. —
similar to how Greenfield's TypeScript classes implement `*Requests`
interfaces.
**Where the Odoo 20 OWL shell hooks in**: Not applicable. The runtime
layer is below the shell.
### Layer 4: Transport
**What to copy as-is**: The browser side of `Channel.ts` (`ARQChannel`
+ `SimpleChannel` + `ReconnectingWebSocket`) and `kcp.ts`. The TypeScript
KCP implementation is solid and battle-tested.
**What to replace**: The native side of `Channel.ts`. Replace Node's `ws`
library with `aiohttp` WebSocket or `FastAPI` WebSocket.
**Where Python fits**:
- Use `aiohttp.web.WebSocketResponse` or `starlette.websockets.WebSocket`
for the WebSocket server.
- Port the KCP implementation from TypeScript to Python (or wrap a C
implementation via cffi — `pykcp` exists).
- Use `asyncio` event loop integration: `pywayland` exposes the
libwayland event-loop FD, which you can integrate via
`asyncio.loop.add_reader()`.
- For multiplexing 9 endpoints, use `aiohttp.web.Application` with 9
routes, or `FastAPI` with 9 WebSocket endpoints.
**Where the Odoo 20 OWL shell hooks in**: The shell connects to the
proxy via the same WebSocket URLs. No change needed.
### Layer 5: Rendering Pipeline (browser side)
**What to copy as-is**: The entire `packages/compositor/src/` rendering
pipeline. This is the heart of Greenfield and it's where most of the
engineering effort went. Reuse it wholesale:
- `Surface.ts`, `View.ts`, `Output.ts`, `Loop.ts`, `Callback.ts`,
`Region.ts`, `Shm.ts`, `BufferContents.ts`, `BufferImplementation.ts`,
`ImageBitmapBuffer.ts`
- `render/` directory (Renderer, Scene, SceneShader, Texture, RenderState,
shaders)
- `remote/` directory (StreamingBuffer, BufferStream, EncodedFrame,
EncoderFeedback, decoders)
**What to replace**: Nothing. This layer is pure browser TypeScript and
works as-is.
**Where Python fits**: Not applicable. The browser does all rendering.
**Where the Odoo 20 OWL shell hooks in**: The shell owns the
`<canvas>` element and passes it to
`userShell.actions.initScene(canvasCreator)`. The compositor creates a
`Scene` per canvas.
### Layer 6: Surface Lifecycle / Roles
**What to copy as-is**: All of `XdgWmBase.ts`, `XdgSurface.ts`,
`XdgToplevel.ts`, `XdgPopup.ts`, `XdgPositioner.ts`, `Shell.ts`,
`ShellSurface.ts`, `Subcompositor.ts`, `Subsurface.ts`, `SurfaceRole.ts`,
`SurfaceChild.ts`. These implement Wayland's role state machines correctly.
**What to replace**: Nothing.
**Where Python fits**: Not applicable (this is browser-side).
**Where the Odoo 20 OWL shell hooks in**: The shell implements
`UserShellApi.events` callbacks (see `08-app-lifecycle.md`). Specifically:
- `surfaceCreated` → add a window to the OWL taskbar
- `surfaceDestroyed` → remove from taskbar
- `surfaceTitleUpdated` → update window title
- `surfaceActivationUpdated` → mark window as active
- `clientCreated` / `clientDestroyed` → update app registry
- `clientUnresponsiveUpdated` → show "not responding" indicator
### Layer 7: Desktop / Layout Policy
**What to copy as-is**: `desktop/Desktop.ts` provides Floating and
AlwaysFullScreen modes. Floating is what most users want (windows can be
moved/resized freely).
**What to replace**: If your Odoo shell needs custom layout policies
(e.g. tiling, tabbed), you'll need to extend `DesktopSurface` with new
modes. But you can start with Floating.
**Where Python fits**: Not applicable.
**Where the Odoo 20 OWL shell hooks in**: The shell calls
`userShell.actions.activateSurface(compositorSurface)` when the user
clicks a taskbar entry.
### Layer 8: User Shell (the integration seam)
**What to copy as-is**: `UserShellApi.ts`. The interface is clean and
shell-agnostic.
**What to replace**: Replace `compositor-shell` (Preact) with your Odoo
20 OWL addon.
**Where Python fits**: Not applicable. The shell is browser-side.
**Where the Odoo 20 OWL shell hooks in**: This IS the integration point.
Create an OWL component that:
```typescript
// Pseudocode for Odoo 20 OWL shell
import { Component, useState, onMounted } from "@odoo/owl";
import { createCompositorSession } from "@gfld/compositor";
export class CompositorShell extends Component {
setup() {
this.state = useState({
surfaces: new Map(), // id → { title, appId, active }
clients: new Map(),
});
onMounted(async () => {
const session = createCompositorSession();
this.session = session;
// Wire UserShellApi events to OWL reactive state
session.userShell.events.surfaceCreated = (surface) => {
this.state.surfaces.set(surface.id, {
title: "", appId: "", active: false
});
};
session.userShell.events.surfaceDestroyed = (surface) => {
this.state.surfaces.delete(surface.id);
};
session.userShell.events.surfaceTitleUpdated = (surface, title) => {
const s = this.state.surfaces.get(surface.id);
if (s) s.title = title;
};
session.userShell.events.surfaceActivationUpdated = (surface, active) => {
const s = this.state.surfaces.get(surface.id);
if (s) s.active = active;
};
// ... etc for clientCreated, clientDestroyed, etc.
// Initialize the canvas
const canvas = this.canvasRef.el;
session.userShell.actions.initScene(() => ({
canvas,
width: canvas.clientWidth,
height: canvas.clientHeight,
}));
});
}
// User clicks app in launcher
async launchApp(appId) {
const response = await fetch("/web/compositor/launch", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ app_id: appId }),
});
const { signal_url } = await response.json();
// Browser JS opens signal_url WebSocket → compositor handles the rest
}
// User clicks taskbar entry
activateSurface(surfaceId) {
this.session.userShell.actions.activateSurface({ id: surfaceId });
}
}
```
### Odoo 20 backend addon
The Odoo 20 backend addon acts as a thin proxy between the OWL shell and
the Python compositor-proxy. It handles:
- App launcher config (per-user, per-tenant allowed apps list)
- Auth (Odoo session → compositor-proxy session mapping)
- Audit log (which user launched which app when)
- Multi-tenant isolation (separate Python proxy processes per tenant,
containerized)
- Per-tenant resource limits (max concurrent apps, max memory)
```python
# Pseudocode for Odoo 20 backend controller
from odoo import http
from odoo.http import request
class CompositorController(http.Controller):
@http.route("/web/compositor/apps", type="json", auth="user")
def list_apps(self):
# Return user-allowed apps from applications.json
user = request.env.user
return request.env["compositor.app"].search_read(
[("allowed_group_ids", "in", user.groups_id.ids)],
["name", "icon", "app_id"],
)
@http.route("/web/compositor/launch", type="json", auth="user")
def launch_app(self, app_id):
user = request.env.user
# Look up the user's compositor-proxy host (per-tenant routing)
proxy_host = user.compositor_proxy_host
proxy_port = user.compositor_proxy_port
# Tell the Python proxy to spawn the app
# The proxy returns a signal_url + auth key
import aiohttp
async with aiohttp.ClientSession() as sess:
async with sess.post(
f"http://{proxy_host}:{proxy_port}/launch",
json={
"app_id": app_id,
"odoo_session_id": request.session.sid,
"user_id": user.id,
},
) as resp:
result = await resp.json()
# Audit log
request.env["compositor.audit.log"].create({
"user_id": user.id,
"app_id": app_id,
"timestamp": fields.Datetime.now(),
})
return {"signal_url": result["signal_url"]}
```
### Python compositor-proxy (the big new code)
This is the bulk of the new code. Replace Node's `compositor-proxy` +
`compositor-proxy-cli` with a Python equivalent. The key components:
```python
# Pseudocode for Python compositor-proxy
import asyncio
import os
import signal
import subprocess
from aiohttp import web, WSMsgType
from pywayland.server import Display
from xkbcommon import xkb
import gi
gi.require_version("Gst", "1.0")
gi.require_version("GstGL", "1.0")
from gi.repository import Gst, GstGL, GLib
class CompositorProxy:
def __init__(self, bind_ip="0.0.0.0", bind_port=8081, render_device="/dev/dri/renderD128"):
self.bind_ip = bind_ip
self.bind_port = bind_port
self.render_device = render_device
self.display = Display()
self.sessions = {} # session_id → SessionProcess
async def start(self):
# 1. Create wl_display + add socket
self.display.create()
socket_name = self.display.add_socket_auto()
os.environ["WAYLAND_DISPLAY"] = socket_name
# 2. Init SHM formats (YUV420, NV12, YUV444, NV21)
self.init_shm()
# 3. Init DRM (open /dev/dri/renderD128, register dmabuf globals)
self.init_drm()
# 4. Integrate wl_display FD into asyncio loop
loop = asyncio.get_event_loop()
loop.add_reader(self.display.get_fd(), self.dispatch_requests)
# 5. Start GStreamer
Gst.init(None)
self.gst_loop = GLib.MainLoop()
# Run GLib loop in a separate thread (GStreamer requires it)
import threading
threading.Thread(target=self.gst_loop.run, daemon=True).start()
# 6. Start aiohttp server
app = web.Application()
app.add_routes([
web.get("/signal", self.handle_signal),
web.get("/channel", self.handle_channel),
web.post("/launch", self.handle_launch),
# ... 6 more endpoints
])
runner = web.AppRunner(app)
await runner.setup()
site = web.TCPSite(runner, self.bind_ip, self.bind_port)
await site.start()
# Run forever
await asyncio.Event().wait()
def dispatch_requests(self):
"""Called by asyncio when wl_display FD is readable."""
self.display.dispatch_events_pending()
async def handle_launch(self, request):
body = await request.json()
app_id = body["app_id"]
# Spawn the app, set WAYLAND_DISPLAY
env = {**os.environ, "WAYLAND_DISPLAY": os.environ["WAYLAND_DISPLAY"]}
proc = subprocess.Popen(
self.applications[app_id]["executable"],
env=env,
# NEW vs Greenfield: sandbox with bubblewrap
preexec_fn=lambda: self.setup_sandbox(),
)
return web.json_response({
"signal_url": f"ws://{self.bind_ip}:{self.bind_port}/signal?app_id={app_id}",
"pid": proc.pid,
})
def setup_sandbox(self):
"""NEW: bubblewrap + seccomp sandbox (Greenfield has none)."""
import ctypes
libc = ctypes.CDLL("libc.so.6")
# ... bwrap / unshare / seccomp_filter setup
```
### Native app bridge (Python equivalent)
For each Wayland client that connects (one per app), spawn a Python
`NativeWaylandClientSession` equivalent that:
1. Sets up the wire-message interceptor (using pywayland's resource
dispatch).
2. Creates a `PROTOCOL` ARQ channel (port `kcp.ts` to Python or use
`pykcp`).
3. Creates a `FRAME` ARQ channel for the surface's encoded content.
4. Manages the app's lifecycle (10-min SIGHUP grace, SIGTERM escalation).
For XWayland, replace `westfield-xwayland.c` with a Python equivalent
that:
1. Creates `/tmp/.X11-unix/X<N>` listener socket.
2. Sets `DISPLAY=:N` environment variable.
3. Spawns `Xwayland :N -ac -rootless -listen <fd> -displayfd <fd> -wm <fd>`.
4. Wraps the `wm_fd` as a Python file-like object.
5. Uses `xtsb` (the existing TypeScript X11 bindings) on the browser side
to handle XWM logic — the browser side stays the same.
### Hardware acceleration (Python equivalent)
For the encoding pipeline:
- Use GStreamer Python bindings (`gi.repository.Gst`):
- `appsrc` for input buffers
- `glupload`, `glcolorconvert`, `glshader` for GPU processing
- `x264enc` / `nvh264enc` / `vaapih264enc` for H.264 encoding
- `appsink` for output samples
- Same encoder selection logic as Greenfield (PNG ≤ 256×256, H.264 above).
- Run the GLib main loop in a separate thread, use `asyncio.run_in_executor`
for cross-thread encoding requests.
- For zero-copy dmabuf, use `GstGLMemoryEGL` (same as Greenfield).
For the browser decode side: **no change**. The browser uses WebCodecs
preferred + FFmpeg Wasm fallback, same as Greenfield.
## Key trade-offs vs staying on Node
### Staying on Node (use Greenfield as-is, just replace compositor-shell with Odoo)
**Pros**:
- ~90% of Greenfield's code reused as-is (browser compositor + native proxy)
- Less code to write/maintain
- Faster time-to-market
- Battle-tested in Greenfield's rc1 cycle
**Cons**:
- Native Node addons (`.node` files) are Linux-x64 only, ABI-locked to
Node major
- GStreamer + libwayland integration is via C N-API, harder to extend
- Odoo team may have more Python expertise than Node expertise
- AGPL-3.0 license on `compositor-proxy` — viral to your modifications
### Switching to Python
**Pros**:
- Aligns with Odoo's Python backend stack
- `pywayland` (cffi) is easier to debug than N-API addons
- Python GStreamer bindings are well-documented
- Easier to integrate with Odoo's auth/session model
- Can avoid AGPL by reimplementing from scratch (or use only MIT libs)
**Cons**:
- Significant new code (rewrite the proxy + native addons in Python)
- Python GIL may bottleneck encoding throughput (use multiprocessing or
run GStreamer in separate thread)
- pywayland is less mature than libwayland's native bindings
- You take on the maintenance of a forked Greenfield-equivalent
### Recommendation: hybrid approach
1. **Browser side**: Use Greenfield's `@gfld/compositor` library as-is
(MIT-licensed via `@gfld/compositor-protocol` + `@gfld/common`, AGPL only
on `@gfld/compositor` itself — but the shell is yours so the AGPL
virality is contained to the compositor library).
2. **Native side**: Start with Greenfield's `compositor-proxy` (AGPL) to
get to MVP fast. Then progressively port to Python as needs evolve
(sandboxing, multi-tenant, custom auth, Odoo session integration).
3. **Shell**: Build the Odoo 20 OWL addon from scratch (clean room,
MIT/Apache license).
## Implementation phases
### Phase 1 (1-2 weeks): MVP — display a single Linux app
1. Run Greenfield's compositor-proxy-cli as a Docker container.
2. Build minimal Odoo 20 OWL addon that creates a `<canvas>` and
`userShell.events` handlers.
3. Embed the OWL addon in an Odoo action.
4. Trigger app launch via HTTP fetch to the proxy.
5. Verify the Linux app displays in the canvas.
### Phase 2 (2-4 weeks): Multi-app, basic window management
1. Implement `surfaceCreated` / `surfaceDestroyed` handlers.
2. Build a taskbar OWL component listing active surfaces.
3. Implement `activateSurface` action on taskbar click.
4. Test with 2-3 concurrent apps.
### Phase 3 (4-8 weeks): Python port of proxy
1. Replace Node `compositor-proxy-cli` with a Python equivalent.
2. Implement WebSocket endpoints in `aiohttp` or `FastAPI`.
3. Port the KCP TypeScript implementation to Python (or use `pykcp`).
4. Replace `wayland-server-addon.node` with `pywayland`.
5. Replace `proxy-encoding-addon.node` with GStreamer Python bindings.
6. Add bubblewrap sandboxing (NEW — Greenfield has none).
### Phase 4 (8-16 weeks): Polish & scale
1. Multi-tenant isolation (separate Python proxy processes per tenant).
2. Per-tenant resource limits (cgroups, max memory, max CPU).
3. Audit log integration with Odoo.
4. Clipboard sync (native → DOM and reverse).
5. Touch input implementation (Greenfield has stub only).
6. Primary selection (middle-click paste).
7. Adaptive bitrate (consume `EncoderFeedback` data, Greenfield doesn't).
## Specific replacements for each Greenfield native component
| Greenfield native component | Python replacement | Notes |
|---|---|---|
| `wayland-server-addon.node` (1231 lines C) | `pywayland` (cffi bindings) | Same libwayland-server, Python wrapper |
| `proxy-poll-addon.node` (127 lines C) | `asyncio.loop.add_reader()` | Built-in, no addon needed |
| `proxy-encoding-addon.node` + `libproxy-encoding.so` (~1700 lines C) | `gi.repository.Gst` (GStreamer Python bindings) | Same GStreamer, Python wrapper |
| Patched `libwayland-server` (vendored at `native/wayland/src/wayland-server/`) | Use upstream libwayland-server + Greenfield's patches applied | Apply `changes.diff` to a fork of libwayland, install via system package |
| `westfield-*.c` (EGL, dmabuf, xwayland helpers) | Reimplement in Python via `pyEGL` or `python-xkbcommon` + `pygobject`'s `GstGL` | Less mature than Greenfield's C — be prepared to debug |
| `xtsb` (X11 TypeScript bindings) | Use as-is on browser side, replace Python side with `python-xlib` | Browser-side xtsb is MIT-licensed |
| Node `ws` library | `aiohttp.web.WebSocketResponse` or `starlette.websockets.WebSocket` | Mature, well-documented |
| Node `pkg` (single-binary packaging) | PyInstaller or `shiv` | Same end result |
| Node `fork()` per session | `multiprocessing.Process` or `os.fork()` | Direct equivalent |
| `compositor-proxy-cli` (CLI args) | `argparse` | Standard Python |
| `ajv` (JSON schema validation) | `jsonschema` | Standard Python |
## Final architecture diagram (target state)
```
┌──────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER (Odoo 20 web client) │
│ │
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
│ │ Odoo 20 OWL Shell addon (your code, MIT/Apache) │ │
│ │ - Application launcher, window manager, keyboard picker │ │
│ │ - Per-output <canvas> │ │
│ │ - Implements UserShellApi.events.* callbacks │ │
│ └────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ userShell.events + actions │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
│ │ @gfld/compositor (browser-side, MIT libs + AGPL compositor) │ │
│ │ - Surface, View, Output, Loop, Seat, Keyboard, Pointer, Touch │ │
│ │ - render/ (WebGL shaders) │ │
│ │ - remote/ (Channel, KCP, decoders) │ │
│ └────────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
│
│ WebSocket + KCP-over-WS
│ 9 endpoint paths per session
▼
┌──────────────────────────────────────────────────────────────────────────────────┐
│ PYTHON HOST (Linux server) │
│ │
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
│ │ Odoo 20 backend addon (your code, MPL-2.0/LGPL) │ │
│ │ - /web/compositor/apps, /web/compositor/launch │ │
│ │ - Per-user, per-tenant allowed apps list │ │
│ │ - Audit log │ │
│ └────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ subprocess / IPC │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
│ │ Python compositor-proxy (your code, MIT/Apache) │ │
│ │ - aiohttp/FastAPI WebSocket server (9 endpoints) │ │
│ │ - pywayland (cffi bindings to libwayland-server) │ │
│ │ - python-xkbcommon for keymap compilation │ │
│ │ - GStreamer Python bindings for H.264 encoding │ │
│ │ - asyncio event loop integrating libwayland FD polling │ │
│ │ - Per-app subprocess.Popen with bubblewrap sandbox │ │
│ │ - KCP port (pykcp or hand-rolled) │ │
│ └────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ │ WAYLAND_DISPLAY=wayland-N │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
│ │ Native Linux apps (gtk4-demo, xterm, kwrite, ...) │ │
│ │ - Use host system libraries │ │
│ │ - Sandboxed via bubblewrap + seccomp │ │
│ └────────────────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────────────┘
```
+532
View File
@@ -0,0 +1,532 @@
# 11 — Pitfalls, Anti-patterns, and What to Avoid
This is the prominent pitfalls chapter. Greenfield's known weaknesses,
anti-patterns, and design pitfalls are catalogued here so the user can
avoid making the same mistakes in their own Wayland-to-HTML compositor
(Odoo 20 + JS/Python). Each pitfall comes with: **what Greenfield does**
(with file/directory reference), **why it's a problem** (technical
reason), and **what the user should do instead** (concrete recommendation
for their JS/Python + Odoo 20 stack).
## Headline numbers
- **299** `TODO|FIXME|HACK|XXX|@deprecated|workaround` hits across non-vendored source (after excluding `examples/sdk/*` upstreams and `xtsb/protocol/xproto.xml`).
- **13 packages** all pinned to `"version": "1.0.0-rc1"` — no `CHANGELOG.md`, no `SECURITY.md`, no `CONTRIBUTING.md`, no issue/PR templates.
- **1 commit** visible on `master` (`6c578f4`, "Fix typo in code snippet (#161)", 2025-10-15) — implying the public repo is a release snapshot, not the live dev branch.
- **3 native Node `.node` addons**, all Linux-x64 only.
- **0** uses of `OffscreenCanvas` or `transferControlToOffscreen` anywhere in the compositor.
- **2** Web Workers used — and only for the WASM H.264 *fallback* decoder path. The primary WebCodecs path uses none.
## Catalogued pitfalls (A–U)
### Critical (will bite you on day 1)
#### A. Native Node addons
**What Greenfield does**: Three Node-API addons:
- `wayland-server-addon.node` (1231 lines C, wraps patched libwayland-server)
- `proxy-encoding-addon.node` (290 lines C, wraps GStreamer)
- `proxy-poll-addon.node` (127 lines C, wraps libuv `uv_poll_t`)
All Linux-x64 only, built with CMake + Ninja. ABI-locked to Node major
versions.
**Why it's a problem**:
- Linux-x64 only — no ARM64 support
- ABI breaks on Node major upgrades
- Hard to ship — must rebuild for each target platform
- Hard to debug — C N-API debugging requires core dumps
- GStreamer integration is via C, not Python
- No Python story
**What you should do instead**: Use `pywayland` (cffi bindings to
libwayland-server), `python-xkbcommon`, and GStreamer's Python bindings
(`gi.repository.Gst`). One Python process can replace the entire
`compositor-proxy-cli`. See `10-porting-to-odoo-js-python.md` for details.
#### B. rc1 with no policy
**What Greenfield does**: All 13 packages at `1.0.0-rc1`. No CHANGELOG,
no SECURITY.md, no CONTRIBUTING.md.
**Why it's a problem**: Breaking changes can land silently. The
maintainer's roadmap is unclear.
**What you should do instead**: Pin to a specific commit hash if you fork
it. Track upstream commits manually. Plan for the eventuality that
1.0.0 final may break APIs.
#### C. Encoding rigidity
**What Greenfield does**: Three H.264 GStreamer pipelines (x264/nvh264/
vaapih264) with hand-tuned magic constants:
- `bitrate=12000 vbv-buf-capacity=160 bframes=0 rc-lookahead=0`
- VA-API is "experimental"
- AMD (`vah264enc`) absent
- PNG fallback at hardcoded ≤256×256 threshold
- Browser-side hardware decode is explicitly disabled in `Session.ts:93`
with a FIXME "hardware decoding frame offset is wrong"
**Why it's a problem**:
- No adaptive bitrate — video plays at 12 Mbps regardless of network
conditions
- No AMD support — limits hardware encoder options
- Browser decode is software-only — wastes GPU
**What you should do instead**:
- Build an adaptive multi-codec encoding router (PNG for ≤64×64 or static,
VP8/H.264 for mid-size, H.264/AV1 for video)
- Drive the router by the `EncoderFeedback` measurement Greenfield already
produces but doesn't consume
- Use `pykcp` or similar for congestion-aware bitrate adaptation
- Enable browser hardware decode (`prefer-hardware`) — fix the "frame
offset is wrong" bug
#### D. Wire protocol has no schema version field
**What Greenfield does**: No version prefix on any binary message:
- `EncodedFrame` (FRAME channel)
- `RemoteOutOfBandChannel` (PROTOCOL channel OOB)
- The FD-channel in `RemoteAppLauncher.ts:612`
- Undocumented magic close codes (`4200`, `4201`, `4403`, `4001`, `1005`)
All break-lockstep — if you change the wire format, both sides must
upgrade simultaneously or messages will be misinterpreted.
**Why it's a problem**:
- Cannot evolve the protocol without breaking all clients
- Cannot detect version mismatches
- Hard to debug — close codes are magic numbers
**What you should do instead**:
- Prefix every binary message with `u16 protocolVersion`
- Document every close code in a comment
- Use JSON `{"v":1, "type":"..."}` for the control plane
- Build a feature-flag system so old clients can negotiate capabilities
#### E. Main-thread decode + render
**What Greenfield does**:
- `VideoDecoder` runs on main thread
- WebGL context bound to `HTMLCanvasElement`
- No `OffscreenCanvas`
- No `Worker` in render path
- 0 uses of `OffscreenCanvas` or `transferControlToOffscreen` in the
compositor
- Only 2 Web Workers used (FFmpeg Wasm fallback), shared across surfaces
**Why it's a problem**:
- Jank on 1080p surfaces — main thread blocks on decode
- No way to recover if main thread stalls (input also blocked)
- Can't use `requestAnimationFrame` for compositing if main thread is busy
**What you should do instead**:
- Render in a Web Worker with `OffscreenCanvas` from day one
- Ship `EncodedVideoChunk` / `ArrayBuffer` transfers (zero-copy) to the
worker
- Use `VideoFrame` directly as a WebGL texture source — no `copyTo`
- Use multiple workers (one per output) for parallel rendering
### High severity
#### F. xkbcommon Wasm is multi-MB
**What Greenfield does**: `libs/compositor-wasm/xkbcommon/build.sh`
clones libxkbcommon 1.5.0, embeds the entire xkeyboard-config database
via `--embed-file`. Result: multi-MB WASM module loaded on every page.
**Why it's a problem**:
- Slow initial load
- Layout list parsed via fragile string matching
- Hard to update keyboard layouts without rebuilding WASM
**What you should do instead**:
- Use the browser's `navigator.keyboard.getLayoutMap()` API for basic
layout detection
- For full xkbcommon support, lazy-load the WASM only when needed
- Consider server-side keymap compilation (Python `python-xkbcommon`)
and ship only the result string to the browser
#### G. Clipboard edge cases
**What Greenfield does**:
- Hardcoded MIME list `['text/plain', 'text/html', 'image/png']`
- No primary selection (`wp_primary_selection_v1` not implemented)
- Change detection by blob size (false negatives on same-size different
content)
- Focus check at `DataOffer.ts:73` is a FIXME
- Browser-side clipboard polling (no `clipboardchange` event)
**Why it's a problem**:
- Middle-click paste doesn't work (Linux users expect it)
- Large clipboard content (e.g. copied image) is poorly handled
- MIME negotiation races
**What you should do instead**:
- Implement `zwp_primary_selection_device_manager_v1` from day one
- Use exact MIME matching, not substring
- For large clipboard content, use chunked transfer
- Watch for the `clipboardchange` event in modern browsers (currently
behind flags in Chrome)
#### H. XWayland/xtsb complexity
**What Greenfield does**:
- A full XCB reimplementation in TypeScript with its own 5800-line
`xproto.xml`
- The XWM is a hand-port of wlroots' `xwm.c` with FIXMEs pointing at
specific wlroots commits
- `XWindowManager.ts:746` literally says "this makes xterm close somehow"
- Title bars painted pixel-by-pixel in `XWindowFrame.ts`
**Why it's a problem**:
- 5800 lines of X11 protocol XML to maintain
- Hand-ported XWM logic is fragile and out of sync with upstream wlroots
- Magic workarounds in code
**What you should do instead**:
- Use `python-xlib` on the Python proxy side instead of porting xtsb
- Consider running XWayland with a minimal window manager (e.g. mutter
in kiosk mode) instead of implementing XWM yourself
- For XWayland apps only (no full X11 desktop needed), the XWM can be
much simpler than wlroots'
#### I. Adaptive bitrate feedback loop is wired but not consumed
**What Greenfield does**:
- `EncoderFeedback.ts` measures client decode duration at 1 Hz
- 4-byte feedback messages sent from browser to proxy
- Proxy receives them but does NOT use them to adapt bitrate
- `Channel.ts:181` has a TODO confirming the gap
**Why it's a problem**:
- Video plays at fixed 12 Mbps regardless of network conditions
- No way to recover from congestion
- Wasted bandwidth on slow networks
**What you should do instead**:
- Wire `EncoderFeedback` to a bitrate adaptation controller
- Use `GstElement.set_property("bitrate", ...)` to adjust x264enc/nvh264enc
- Implement `requestKeyUnit` plumbing (Greenfield has it end-to-end but
the browser never calls it)
#### J. Single-seat, single-session model
**What Greenfield does**:
- One seat per browser session
- One Node process per browser session (per `compositorSessionId`)
- No multi-seat support (e.g. multiple keyboards/mice for collaborative
editing)
- No multi-region isolation
**Why it's a problem**:
- Can't support multi-user scenarios (e.g. conference room with multiple
keyboards)
- No way to isolate apps between tenants
**What you should do instead**:
- For multi-tenant isolation: separate Python proxy processes per tenant,
containerized (Docker + bubblewrap)
- For multi-seat: implement `wl_seat` global per seat, route input
events by seat
#### K. Custom web-bitmapbuf protocol — apps must be Greenfield-aware
**What Greenfield does**:
- `protocol/web-bitmapbuf-unstable-v1.xml` is Greenfield's custom
extension
- Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to the
compositor
- Standard Wayland apps (even browser-resident ones) fall to slow H.264
path
**Why it's a problem**:
- Apps must be rebuilt against `@gfld/client-protocol` to use the fast
path
- Standard Wayland apps can't take advantage of zero-copy GPU textures
- Browser-resident WASM apps are slow unless rebuilt
**What you should do instead**:
- Document clearly that the fast path requires Greenfield-aware apps
- For native Linux apps (your main use case), this isn't an issue — they
go through the H.264 path regardless
- For browser-resident WASM apps, encourage users to build against
`@gfld/client-protocol`
#### L. Connection loss handling
**What Greenfield does**:
- 10-minute zombie timeout on disconnect (`NativeAppContext.ts:152-160`)
- No state resumption on reconnect
- `eagerBufferContents` cache in `BufferStream.ts:31` is unbounded (TODO
at `:87` confirms the leak)
- App kill timers: SIGHUP@10min, SIGKILL@+10s
- Close code 1001 → immediate SIGHUP; 4001 → immediate SIGTERM
**Why it's a problem**:
- User loses all running apps on browser refresh (unless they reconnect
within 10 min)
- Memory leak from unbounded cache
- No way to survive proxy restart
**What you should do instead**:
- Bound the `eagerBufferContents` cache (e.g. max 100 frames)
- Persist app state on browser side (localStorage) so users can resume
- For proxy restart, checkpoint app state to disk (json file)
- Implement graceful reconnection with state sync (the KCP plumbing
exists)
#### M. No sandboxing
**What Greenfield does**:
- `spawn(executable, args, { env })` with no bubblewrap/seccomp/namespaces
- Basic Auth is a single replayable credential compared with `!==`
- `compositorSessionId` is client-generated
**Why it's a problem**:
- Native apps have full access to the host filesystem
- One compromised app can read all other apps' data
- Basic Auth credentials can be replayed
- Client-generated session IDs can be spoofed
**What you should do instead**:
- Sandbox every spawned app with bubblewrap + seccomp + per-session Linux
user namespace
- Replace Basic Auth with short-lived signed JWT bound to Odoo's session
model
- Server-generated session IDs (use `secrets.token_urlsafe(32)`)
#### N. Heavy toolchain
**What Greenfield does**:
- Yarn 4 + CMake + Ninja + Meson + pkg-config
- Two different Emscripten versions: 3.1.46 for FFmpeg, 3.1.60 for SDK
- SDK builds 30+ C libraries from source
- gtk4 WASM is currently broken
- Native build needs `autoconf libtool automake meson ninja-build bison
cmake build-essential gperf`
**Why it's a problem**:
- Complex build environment setup
- Long build times
- Emscripten version skew between libs and SDK
- gtk4 WASM broken means no GTK4 browser apps
**What you should do instead**:
- Stick with a single Emscripten version
- Pre-build WASM modules and ship them as artifacts (don't rebuild every
time)
- For Python, use `pip install` for all dependencies (simpler than CMake
+ Ninja)
- Skip the WASM app SDK entirely if you only care about native Linux apps
### Medium / low (but worth fixing)
#### O. WebCodecs usage
**What Greenfield does**:
- WebCodecs IS used (correcting the brief in some agents' notes)
- BUT HW-decode is disabled
- `VideoEncoder` is unused
**Why it's a problem**:
- Wastes GPU on software decode
- Could encode in browser (for upload to other peers) but doesn't
**What you should do instead**:
- Enable HW-decode by fixing the "frame offset is wrong" FIXME
- Consider WebCodecs `VideoEncoder` for any peer-to-peer video paths
#### P. WebCodecs hardware decode disabled
**What Greenfield does**:
- `Session.ts:93` has `// FIXME hardware decoding frame offset is wrong`
- `hardwareAcceleration: 'prefer-software'` is hardcoded
**Why it's a problem**:
- Software decode at 1080p60 is ~30% CPU on a modern laptop
- Battery life impact on mobile
- Wastes GPU
**What you should do instead**:
- Investigate and fix the frame offset bug
- Enable `prefer-hardware` for capable browsers
- Provide a runtime toggle
#### Q. Unbounded `eagerBufferContents` cache
**What Greenfield does**:
- `BufferStream.ts:31` has unbounded cache
- TODO at `:87` confirms the leak
**Why it's a problem**:
- Memory leak on long-running sessions
- Can OOM the browser tab
**What you should do instead**:
- Bound the cache to a maximum number of frames (e.g. 100)
- Use LRU eviction
- Periodic cleanup of orphaned entries
#### R. `H264NALDecoder.worker.ts` is minimal
**What Greenfield does**:
- 22 lines of code
- `@ts-ignore`'d
- No error boundary
**Why it's a problem**:
- Worker crashes silently
- No retry on transient errors
- No metrics on decode failures
**What you should do instead**:
- Add error boundary with retry logic
- Log decode failures to the proxy (for diagnostics)
- Pool workers for resilience
#### S. Magic undocumented WebSocket close codes
**What Greenfield does**:
- Close codes `4200`, `4201`, `4403`, `4001`, `1005` are used without
documentation
- Only codes 4000-4999 (application-level) tear down KCP
**Why it's a problem**:
- Hard to debug connection issues
- Close codes are magic numbers
**What you should do instead**:
- Document every close code in code comments
- Use named constants (e.g. `CLOSE_APP_TERMINATED = 4001`)
- Build a close-code → human-readable mapping
#### T. Generated protocol files committed
**What Greenfield does**:
- `packages/compositor-proxy/src/protocol/*.js` is generated but
committed to the repo
- Can drift from XML source
**Why it's a problem**:
- Easy to forget to regenerate
- PRs may include stale generated code
**What you should do instead**:
- Don't commit generated files
- Use `prebuild` script to generate them
- Add a CI check that regenerated files match committed ones
#### U. Confirmed copy-paste bug in `webcodec-buffer-decoder.ts`
**What Greenfield does**:
- `webcodec-buffer-decoder.ts:142-144, 156-157` has:
`this.opaqueDecoder = undefined` twice
- Should be `alphaDecoder` the second time
- Leaks the alpha decoder after errors
**Why it's a problem**:
- Memory leak
- Wasm decoder context not cleaned up properly
- Can cause decoder pool exhaustion
**What you should do instead**:
- Audit all `cleanup` paths in decoder code
- Add unit tests for error scenarios
## Design wins to copy
Not all of Greenfield's choices are bad. Here are 12 design wins worth
copying verbatim:
1. **Generator-driven protocol stubs** (`libs/compositor-generator/`,
`client-generator/`, `compositor-proxy-generator/`) — single XML source
for both sides. Eliminates wire-format drift.
2. **Dual-buffer commit semantics** (`Surface.ts`: `state` vs
`pendingState`, atomic merge with pixman damage regions). Classic
Wayland correctness.
3. **UserShellApi decoupling** — shell sees only `{events, actions}`,
never Wayland resources. This is THE integration seam you'll use.
4. **Asynchronous browser compositor** — per-client channels, never blocks
on a slow client. Essential for multi-app scenarios.
5. **KCP-over-WebSocket** for UDP-like latency over TCP (`kcp.ts`, 1170
lines). Better than plain TCP for real-time video.
6. **WebCodecs primary + WASM fallback + capability probe** (`Session.ts`).
Modern browsers get the fast path, older ones still work.
7. **Zero-copy dmabuf → encoder** via standard `linux-dmabuf-unstable-v1`
(`westfield-dmabuf.c`). Pixels stay on GPU.
8. **Per-surface encoder feedback channel** (`EncoderFeedback.ts`)
measuring client decode duration — even though adaptation isn't wired
up yet, the telemetry is right.
9. **`ReconnectingWebSocket`** with exponential backoff + message queue.
Survives transient network drops.
10. **Minimal shell** — 562 lines of TypeScript total in `compositor-shell`;
kiosk variant is 40 lines. Demonstrates that the shell can be tiny.
11. **Honest docs** — `docs/pages/design/` openly admits WebSocket/TCP is
wrong for real-time and waits on WebTransport. Useful as a reference
for what trade-offs were considered.
12. **AGPL-3.0** — deliberate copyleft, but be aware before integrating.
If you want to keep your modifications proprietary, fork the MIT
libs and reimplement the AGPL parts.
## Top-5 recommendations for the user's Odoo 20 + JS/Python compositor
1. **Skip native Node addons entirely.** Use `pywayland` (cffi),
`python-xkbcommon`, and GStreamer's `gi.repository.Gst` Python
bindings. One Python process can replace the entire
`compositor-proxy-cli`. See `10-porting-to-odoo-js-python.md` for
details.
2. **Sandbox every spawned app** with bubblewrap + seccomp + per-session
Linux user namespace; replace Basic Auth with short-lived signed JWT
bound to Odoo's session model.
3. **Render in a Web Worker with `OffscreenCanvas`** from day one; ship
`EncodedVideoChunk` / `ArrayBuffer` transfers (zero-copy) to the
worker; use `VideoFrame` directly as a WebGL texture source — no
`copyTo`.
4. **Build an adaptive multi-codec encoding router** (PNG for ≤64×64 or
static, VP8/H.264 for mid-size, H.264/AV1 for video), driven by the
`EncoderFeedback` measurement Greenfield already produces but doesn't
consume.
5. **Version your wire protocol from commit #1** — prefix every binary
message with `u16 protocolVersion`; document every close code; use
JSON `{"v":1, "type":"..."}` for the control plane.
## License implications of AGPL-3.0
The four `packages/*` (compositor, compositor-proxy, compositor-proxy-cli,
compositor-shell) are **AGPL-3.0-or-later**. Implications:
- If you host Greenfield's `compositor-proxy` as a network service and
modify it, you MUST release your modifications under AGPL-3.0+.
- The browser-side `@gfld/compositor` is also AGPL. If you modify it
(e.g. to integrate with Odoo), your modifications must be AGPL.
- The lower-level `libs/*` (common, client-protocol, compositor-protocol,
the generators, compositor-wasm, compositor-ffmpeg-h264, xtsb) are MIT
or Apache-2.0 — these are safe to use without copyleft obligations.
- **Strategy to avoid AGPL**: use the MIT libs as-is, reimplement the
AGPL parts from scratch (browser compositor + native proxy). The
Python port described in `10-porting-to-odoo-js-python.md` is a clean
way to do this — if you write your Python proxy from scratch, you can
license it however you want.
## What Greenfield's codebase reveals about its engineering process
The codebase reveals several engineering process insights:
1. **One-person show** — most commits are by Erik De Rijcke. The codebase
has the strengths (consistent vision, tight integration) and weaknesses
(single point of failure, no code review diversity) of a one-person
project.
2. **Long-running project** — code references back to 2017-era commits
(WebRTC was a previous transport; the `kcp.ts` MTU comment says
"webrtc datachannel MTU"). The architecture has evolved through several
major pivots.
3. **rc1 is meaningful** — the maintainer has been working toward 1.0 for
years and rc1 represents a deliberate freeze. Don't expect 1.0 final
any time soon, but also don't expect major architecture changes.
4. **Honest about limitations** — docs/pages/design/ explicitly admits
what's wrong and what's planned. Use this to inform your own roadmap.
+400
View File
@@ -0,0 +1,400 @@
# 12 — Appendix: Glossary, File Index, References
## Glossary of Greenfield terms
### Wayland concepts
| Term | Meaning |
|---|---|
| `wl_display` | The Wayland display connection — top-level object for both clients and the compositor. |
| `wl_registry` | Per-client registry of available globals (compositor, seat, shm, output, etc.). |
| `wl_compositor` | Global that creates `wl_surface` and `wl_region` objects. |
| `wl_surface` | A 2D rectangular pixel buffer with state stack — the basic drawable. |
| `wl_seat` | Represents a set of input devices (keyboard, pointer, touch). |
| `wl_keyboard` | Per-seat keyboard interface — sends keymap, key events, modifiers. |
| `wl_pointer` | Per-seat pointer interface — sends enter/leave/motion/button/axis events. |
| `wl_touch` | Per-seat touch interface (declared but inert in Greenfield). |
| `wl_output` | A monitor/screen — surfaces are positioned relative to outputs. |
| `wl_shm` | Shared memory buffer mechanism — client provides raw pixel buffer. |
| `wl_shm_pool` | A pool of shared memory that can be sliced into multiple `wl_buffer`s. |
| `wl_buffer` | A pixel buffer that can be attached to a `wl_surface`. |
| `wl_callback` | One-shot frame callback — fired when the compositor presents a frame. |
| `wl_data_device_manager` | Global for clipboard / DnD. |
| `wl_data_device` | Per-seat-per-client data device (selection, DnD). |
| `wl_data_source` | The provider side of a selection (the app that has data to offer). |
| `wl_data_offer` | The receiver side of a selection (the app that wants to receive data). |
| `wl_drm` | Legacy protocol for DMA-BUF buffers (used by Mesa, XWayland). |
| `wl_subcompositor` | Global for subsurfaces (nested surfaces). |
| `wl_subsurface` | A surface nested inside another surface (sync or desync). |
| `wl_region` | A region (set of rectangles) used for input/opaque/damage. |
### xdg-shell concepts
| Term | Meaning |
|---|---|
| `xdg_wm_base` | The xdg-shell global — clients use it to register as desktop apps. |
| `xdg_surface` | A wrapper around `wl_surface` for xdg-shell semantics. |
| `xdg_toplevel` | A top-level window (with title, app_id, state, parent). |
| `xdg_popup` | A transient popup window (menus, tooltips, dropdowns). |
| `xdg_positioner` | A specification for how to position a popup relative to its parent. |
| `configure` | Compositor → client event telling the client what state/size to apply. |
| `ack_configure` | Client → compositor request acknowledging a configure with a serial. |
### Greenfield-specific terms
| Term | Meaning |
|---|---|
| `Session` | Top-level browser-side compositor session. Created by `createCompositorSession()`. |
| `Channel` | Transport abstraction. `ARQChannel` (KCP-wrapped) for reliable channels, `SimpleChannel` for raw WebSocket. |
| `ARQChannel` | Automatic Repeat reQuest channel — KCP-over-WebSocket, reliable, low-latency. |
| `SimpleChannel` | Raw WebSocket channel, no KCP. Used for FEEDBACK (4-byte messages). |
| `Globals` | Registry of Wayland globals on the compositor side (`Globals.ts`). |
| `Seat` | Browser-side `wl_seat` implementation. Aggregates keyboard/pointer/touch + focus management. |
| `Surface` | Browser-side `wl_surface` implementation. Double-buffered state (`state` vs `pendingState`). |
| `View` | A surface's position on an output, used for compositing. One view per surface per output. |
| `Output` | Browser-side `wl_output` implementation. Wraps an HTML `<canvas>` with WebGL context. |
| `RenderState` | Per-view-per-scene texture + transform state. |
| `Scene` | One per output/canvas. Manages the view stack and WebGL textures. |
| `BufferContents` | Abstraction over a buffer's content (mimeType + payload). |
| `BufferImplementation` | Interface for buffer types: SHM, ImageBitmap, StreamingBuffer. |
| `StreamingBuffer` | Per-surface buffer that receives encoded frames over the FRAME channel. |
| `BufferStream` | Manages serial → frame promise resolution. |
| `EncodedFrame` | Wire format parser for encoded frames (H.264 / PNG). |
| `EncoderFeedback` | 4-byte feedback messages sent from browser to proxy at 1 Hz (refreshInterval + decode duration). |
| `FrameFeedback` | Proxy-side receiver of `EncoderFeedback` messages. Used to pace `wl_frame.done` callbacks. |
| `ProxyFD` | JSON descriptor for an FD over WebSocket: `{handle, type, host}`. |
| `webFS` | Browser-side resolver for ProxyFD references. |
| `InputOutput` | FD abstraction (browser pipes for in-browser apps, WebSocket endpoints for native apps). |
| `RemoteInputOutput` | Native-app FD operations via WebSocket endpoints. |
| `webInputOutput` | In-browser-app FD operations via `MessageChannel` ports. |
| `UserShellApi` | The integration seam — `{events, actions}` interface between compositor and shell. |
| `CompositorClient` | Opaque handle for a Wayland client (browser-side). |
| `CompositorSurface` | Opaque handle for a Wayland surface (browser-side). |
| `DesktopSurface` | Floating or AlwaysFullScreen mode wrapper. Emits `userShell.events.*`. |
| `FloatingDesktopSurface` | Default floating window mode. |
| `AlwaysFullScreenDesktopSurface` | Kiosk mode — one window per output. |
| `XdgWmBase` | `xdg_wm_base` global, with ping-based unresponsiveness detection. |
| `XdgSurface` | `xdg_surface` role wrapper. |
| `XdgToplevel` | `xdg_toplevel` role — top-level windows. |
| `XdgPopup` | `xdg_popup` role — transient windows. |
| `XdgPositioner` | `xdg_positioner` — popup positioning math. |
| `web-bitmapbuf-unstable-v1` | Greenfield's custom Wayland protocol extension for browser JS apps. |
| `wayland-server-addon` | C N-API addon wrapping patched libwayland-server. |
| `proxy-encoding-addon` | C N-API addon wrapping GStreamer H.264/PNG encoder. |
| `proxy-poll-addon` | C N-API addon wrapping libuv `uv_poll_t` for wl_display FD polling. |
| `xtsb` | X11 TypeScript Bindings — pure-TS X11 client (no libxcb dependency). |
| `westfield-*` | Greenfield's C helpers: `westfield-buffer.c`, `westfield-dmabuf.c`, `westfield-egl.c`, `westfield-xwayland.c`. |
| `wlr_drm` | `wl_drm` global implementation (in `wlr_drm.c`). |
| `wlr_linux_dmabuf_v1` | `zwp_linux_dmabuf_v1` global implementation. |
### Encoding / rendering terms
| Term | Meaning |
|---|---|
| `dmabuf` | Linux DMA-BUF — a buffer that can be zero-copied between processes and GPUs. |
| `EGLImage` | EGL wrapper around a dmabuf, used by GStreamer GL. |
| `GstGLMemoryEGL` | GStreamer GL memory backed by an EGLImage — zero-copy GPU buffer. |
| `I420` | YUV 4:2:0 planar format (Y, U, V as separate planes). |
| `NV12` | YUV 4:2:0 semi-planar format (Y plane + interleaved UV plane). |
| `BT.601` | Color space conversion standard used by Greenfield's YUV→RGB shaders. |
| `NAL` | H.264 Network Abstraction Layer unit — a single encoded video frame or part thereof. |
| `AU` | H.264 Access Unit — a complete frame. |
| `keyframe` | H.264 I-frame — independently decodable, used to start a stream. |
| `WebCodecs` | Browser API for native video decode/encode (`VideoDecoder`, `VideoEncoder`). |
| `WebTransport` | Future browser API for UDP-like transport over QUIC. Greenfield plans to use it. |
| `KCP` | Reliable UDP protocol implemented in user-space. Greenfield runs it over WebSocket. |
## File-to-feature index
### Browser side (`packages/compositor/src/`)
| File | Implements / Role |
|---|---|
| `index.ts` | Public API: `createCompositorSession()`, `createAppLauncher()` |
| `Session.ts` | Top-level browser session, decoder setup |
| `Compositor.ts` | `wl_compositor` global |
| `Globals.ts` | Wayland globals registry |
| `Surface.ts` | `wl_surface` impl, double-buffered state, commit semantics |
| `SurfaceChild.ts` | Surface tree child abstraction |
| `SurfaceRole.ts` | SurfaceRole interface |
| `Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
| `Subcompositor.ts` | `wl_subcompositor` global |
| `View.ts` | Positioned surface for compositing |
| `Output.ts` | `wl_output`, WebGL canvas binding |
| `Shm.ts` | `wl_shm` / shm pool, BMP-header hack |
| `BufferContents.ts` | Buffer abstraction (with mimeType) |
| `BufferImplementation.ts` | Buffer implementation interface |
| `ImageBitmapBuffer.ts` | ImageBitmap-backed buffer |
| `Region.ts` | `wl_region`, pixman-WASM damage regions |
| `Callback.ts` | Frame callback (`wl_callback`) |
| `Loop.ts` | Compositor main loop, RAF integration |
| `Seat.ts` | `wl_seat` global, multi-seat support, focus management |
| `Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events |
| `Pointer.ts` | `wl_pointer` impl, motion, buttons, axis |
| `Touch.ts` | `wl_touch` (46-line stub, inert) |
| `Xkb.ts` | xkbcommon WASM wrapper |
| `KeyEvent.ts` | Key event wrapper |
| `ButtonEvent.ts` | Pointer button event wrapper |
| `AxisEvent.ts` | Pointer axis (scroll) event wrapper |
| `InputQueue.ts` | Event coalescing + RAF dispatch |
| `InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) |
| `DataDeviceManager.ts` | `wl_data_device_manager` global |
| `DataSource.ts` | `wl_data_source` impl, the provider side |
| `DataOffer.ts` | `wl_data_offer` impl, the receiver side |
| `WaylandDataSource.ts` | Wrapper around native `wl_data_source` events |
| `Shell.ts` | `wl_shell` global (legacy) |
| `ShellSurface.ts` | `wl_shell_surface` role |
| `XdgWmBase.ts` | `xdg_wm_base` global, ping-based unresponsiveness detection |
| `XdgSurface.ts` | `xdg_surface` role, configure/ack round-trip |
| `XdgToplevel.ts` | `xdg_toplevel` role, window state |
| `XdgPopup.ts` | `xdg_popup` role, transient windows, grabs |
| `XdgPositioner.ts` | `xdg_positioner`, popup positioning math |
| `UserShellApi.ts` | The integration seam — events + actions |
| `render/Renderer.ts` | 4-phase render pipeline |
| `render/Scene.ts` | One per output, manages view stack + textures |
| `render/SceneShader.ts` | Final pass: textured quads to canvas |
| `render/Texture.ts` | WebGL texture wrapper |
| `render/RenderState.ts` | Per-view-per-scene texture + transform |
| `render/Program.ts` | WebGL program wrapper |
| `render/ShaderCompiler.ts` | Shader compile/link helper |
| `render/ShaderSources.ts` | GLSL source strings |
| `render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
| `render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
| `render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
| `render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
| `render/RGBandA2RGBAShader.ts` | Shader for above |
| `remote/Channel.ts` | `ARQChannel` + `SimpleChannel` |
| `remote/kcp.ts` | ~1100-line KCP implementation |
| `remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
| `remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
| `remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
| `remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
| `remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `remote/BufferStream.ts` | Manages serial → frame promise resolution |
| `remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `remote/buffer-decoder.ts` | Decoder interface |
| `remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
| `remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `remote/XWindowManager.ts` | X11 CLIPBOARD atom watcher (XWayland) |
| `remote/XDataSource.ts` | X11 clipboard source (XWayland apps) |
| `remote/XDnDDataSource.ts` | X11 DnD source |
| `remote/api.ts` | Public exports |
| `browser/input.ts` | DOM keyboard → wl_keyboard events |
| `browser/pointer.ts` | DOM pointer → wl_pointer events |
| `browser/dnd.ts` | DnD bridge |
| `browser/selection.ts` | Clipboard bridge |
| `browser/text-selection.ts` | Firefox fallback (text-only) |
| `browser/capabilities.ts` | Browser feature detection |
| `desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
| `math/Mat4.ts` | 4×4 matrix math |
| `math/Rect.ts` | Rectangle math |
| `web/WebBuffersFactory.ts` | Browser-side impl of `web_bitmapbuf_factory` |
| `web/WebInputOutput.ts` | In-browser-app FD operations |
| `web/WebConnectionHandler.ts` | In-browser-app connection handling |
| `polyfill/createImageBitmap.js` | ImageBitmap polyfill |
### Native side
| File | Implements / Role |
|---|---|
| `packages/compositor-proxy/src/index.ts` | Exports `createSession()`, `launchApplication()` |
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
| `packages/compositor-proxy/src/Channel.ts` | ARQChannel + SimpleChannel |
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
| `packages/compositor-proxy/src/config.ts` | Encoding-related configuration |
| `packages/compositor-proxy/native/CMakeLists.txt` | Build config |
| `packages/compositor-proxy/native/wayland/src/addon.c` | Node-API addon: wayland-server-addon.node |
| `packages/compositor-proxy/native/wayland/src/westfield-buffer.c` | Buffer helpers |
| `packages/compositor-proxy/native/wayland/src/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM |
| `packages/compositor-proxy/native/wayland/src/westfield-egl.c` | EGL zero-copy setup |
| `packages/compositor-proxy/native/wayland/src/westfield-xwayland.c` | XWayland spawn + bridge |
| `packages/compositor-proxy/native/wayland/src/wlr_drm.c` | `wl_drm` global impl |
| `packages/compositor-proxy/native/wayland/src/wlr_linux_dmabuf_v1.c` | `zwp_linux_dmabuf_v1` global impl |
| `packages/compositor-proxy/native/wayland/src/drm-protocol.c` | DRM protocol helpers |
| `packages/compositor-proxy/native/wayland/src/linux-dmabuf-unstable-v1-protocol.c` | linux-dmabuf-v1 protocol stubs |
| `packages/compositor-proxy/native/wayland/src/wayland-server/` | Vendored + patched libwayland-server |
| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder |
| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon: proxy-encoding-addon.node |
| `packages/compositor-proxy/native/poll/src/poll.c` | libuv uv_poll_t wrapper |
| `packages/compositor-proxy/native/node-api/` | Vendored Node-API headers |
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
### Libs
| File | Role |
|---|---|
| `libs/common/src/Connection.ts` | Shared wire format codec |
| `libs/common/src/index.ts` | Public exports |
| `libs/client-generator/{ProtocolParser,ProtocolArguments,index}.mjs` | Browser client bindings generator |
| `libs/compositor-generator/{ProtocolParser,ProtocolArguments,index}.mjs` | Browser compositor bindings generator |
| `libs/compositor-proxy-generator/src/{EndpointProtocolParser,EndpointProtocolArguments,index}.mjs` | Proxy interceptor generator |
| `libs/client-protocol/src/westfield-runtime-client.ts` | Client-side runtime |
| `libs/client-protocol/src/protocol/` | Generated client bindings |
| `libs/compositor-protocol/src/westfield-runtime-server.ts` | Server-side runtime |
| `libs/compositor-protocol/src/protocol/` | Generated server bindings |
| `libs/compositor-wasm/build_wasm.sh` | pixman + xkbcommon WASM build |
| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon 1.5.0 build |
| `libs/compositor-wasm/pixman/build.sh` | pixman 0.42.2 build |
| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function |
| `libs/compositor-ffmpeg-h264/build_wasm.sh` | FFmpeg n5.1.2 → WASM build |
| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API |
| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module |
| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point |
| `libs/xtsb/protocol/xproto.xml` | X11 core protocol XML (5800+ lines) |
| `libs/xtsb/protocol/shape.xml` | Shape extension XML |
| `libs/xtsb/protocol/xfixes.xml` | XFIXES extension XML |
| `libs/xtsb/protocol/render.xml` | Render extension XML |
| `libs/xtsb/protocol/composite.xml` | Composite extension XML |
| `libs/xtsb/src/` | Generated TS X11 bindings |
### Protocol XML
| File | Role |
|---|---|
| `protocol/wayland.xml` | Standard upstream Wayland core protocol |
| `protocol/xdg-shell.xml` | Stable xdg-shell protocol |
| `protocol/wayland-drm.xml` | Legacy wl_drm for DMA-BUF (proxy-only) |
| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom extension for browser JS apps |
## External references
### Wayland protocol
- **Wayland documentation**: https://wayland.freedesktop.org/docs/html/
- **Wayland protocol XML**: https://gitlab.freedesktop.org/wayland/wayland/-/blob/main/protocol/wayland.xml
- **xdg-shell protocol**: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/stable/xdg-shell/xdg-shell.xml
- **wayland-drm protocol**: https://gitlab.freedesktop.org/wayland/wayland/-/blob/main/protocol/wayland.xml (wayland-drm is included)
- **linux-dmabuf-unstable-v1**: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml
### libwayland
- **libwayland source**: https://gitlab.freedesktop.org/wayland/wayland
- **Wayland book**: https://wayland-book.com/
- **pywayland (Python bindings)**: https://pywayland.readthedocs.io/
### xkbcommon
- **xkbcommon homepage**: https://xkbcommon.org/
- **xkbcommon source**: https://github.com/xkbcommon/libxkbcommon
- **python-xkbcommon**: https://github.com/sde1000/python-xkbcommon
### GStreamer
- **GStreamer homepage**: https://gstreamer.freedesktop.org/
- **GStreamer Python bindings**: https://gstreamer.freedesktop.org/documentation/tutorials/basic/hello-world.html
- **GStreamer GL plugins**: https://gstreamer.freedesktop.org/documentation/opengl/index.html
- **x264enc docs**: https://gstreamer.freedesktop.org/documentation/x264/index.html
- **nvh264enc docs**: https://gstreamer.freedesktop.org/documentation/nvcodec/index.html
### FFmpeg Wasm
- **FFmpeg homepage**: https://ffmpeg.org/
- **FFmpeg Wasm build**: https://github.com/ffmpegwasm/ffmpeg.wasm
- **Emscripten**: https://emscripten.org/
### WebCodecs
- **WebCodecs API**: https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API
- **VideoDecoder**: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder
- **WebCodecs samples**: https://w3c.github.io/webcodecs/samples/
### WebTransport
- **WebTransport**: https://developer.mozilla.org/en-US/docs/Web/API/WebTransport
- **WebTransport over HTTP/3**: https://datatracker.ietf.org/doc/html/draft-ietf-webtrans-http3
### KCP
- **KCP protocol**: https://github.com/skywind3000/kcp
- **KCP documentation (Chinese)**: https://skywind3000.github.io/kcp/
- **pykcp (Python bindings)**: https://github.com/Zero3K/pykcp
### Emscripten
- **Emscripten homepage**: https://emscripten.org/
- **Emscripten porting guide**: https://emscripten.org/docs/porting/index.html
### X11 / XCB
- **XCB documentation**: https://xcb.freedesktop.org/
- **X protocol reference**: https://www.x.org/releases/X11R7.7/doc/xproto/x11protocol.html
- **python-xlib**: https://github.com/python-xlib/python-xlib
### XWayland
- **XWayland docs**: https://wayland.freedesktop.org/xserver.html
- **wlroots XWM source**: https://gitlab.freedesktop.org/wlroots/wlroots/-/tree/master/xcage
### Greenfield project
- **Greenfield homepage**: https://www.greenfield.app
- **Greenfield documentation**: http://udev.be/greenfield
- **Greenfield source**: https://github.com/udevbe/greenfield
- **Greenfield design docs**: https://github.com/udevbe/greenfield/blob/master/docs/pages/design/index.md
- **Greenfield future plans**: https://github.com/udevbe/greenfield/blob/master/docs/pages/future_plans/index.md
- **Author's blog**: https://wayouttheresoftware.blogspot.com/2023/07/some-history-about-greenfield.html
### Odoo 20
- **Odoo homepage**: https://www.odoo.com/
- **Odoo documentation**: https://www.odoo.com/documentation/20.0/
- **Odoo OWL framework**: https://github.com/odoo/owl
- **Odoo development tutorial**: https://www.odoo.com/documentation/20.0/developer/howtos.html
## Pointers to the user's next steps
1. **Read chapters in order**: `00` → `01` → `02` → `07` → `08` → `10` → `11`.
This gives you the architecture map, the protocol layer, the native app
bridge (your primary use case), the UserShellApi integration seam, the
porting blueprint, and the pitfalls to avoid.
2. **Start with Phase 1 of the porting blueprint** (`10-porting-to-odoo-js-python.md`):
Run Greenfield's compositor-proxy-cli as a Docker container, build a
minimal Odoo 20 OWL addon that creates a `<canvas>`, and verify a single
Linux app displays.
3. **Decide on the AGPL question early**: Will you accept AGPL on the
browser compositor + native proxy (and release your modifications), or
will you reimplement them from scratch (and license them however you
want)?
4. **Choose your Python stack**: `aiohttp` vs `FastAPI` for the WebSocket
server. `pywayland` for the Wayland bindings. `python-xkbcommon` for
keymaps. `gi.repository.Gst` for GStreamer encoding. `bubblewrap` +
`seccomp` for sandboxing.
5. **Plan for sandboxing from day one**: Greenfield has none. If you skip
this, your Odoo users can run arbitrary Linux code on your server.
That's a security hole you can't ship.
6. **Build the OWL shell with `UserShellApi.events` callbacks**: This is
the clean integration seam. Don't try to import Greenfield internals
into your Odoo shell — use the public `UserShellApi` interface.
7. **Budget for the Python port**: ~4-8 weeks for a working Python proxy.
~8-16 weeks for production quality (sandboxing, multi-tenant, adaptive
bitrate, clipboard sync, touch input, primary selection).
8. **Track upstream Greenfield commits**: Pin to a specific commit hash
if you fork it. Watch for 1.0 final (may break APIs).
Good luck with your Wayland-to-HTML compositor build!
+103
View File
@@ -0,0 +1,103 @@
# Greenfield Architecture Analysis — for an Odoo 20 + JS/Python Wayland-to-HTML Compositor
> **Purpose of this bundle.** A deep, source-level architectural analysis of the
> Greenfield repository (https://github.com/udevbe/greenfield, commit `6c578f4`,
> `1.0.0-rc1`) written specifically for an engineering team that is about to build
> its own Wayland-to-HTML compositor. The target stack is **JavaScript (browser) +
> Python (host) + Odoo 20 addon shell**. Greenfield is treated purely as a
> reference implementation; the actual Greenfield UI shell (`packages/compositor-shell`)
> is **explicitly out of scope** — the user will supply their own Odoo 20 OWL shell.
## How to read this bundle
The bundle is split into 13 Markdown files. They are designed to be read in
order, but each file is self-contained and can be read alone if you only need
one layer. Files are organized from the highest-level view down to the
implementation-level details, then the porting blueprint, then pitfalls.
| # | File | What you'll find inside |
|---|------|-------------------------|
| 0 | `00-overview.md` | What Greenfield is, repo layout, package inventory, tech stack table, ASCII architecture diagram. |
| 1 | `01-architecture.md` | Layer-by-layer map from Wayland XML → generators → runtime → transport → rendering. |
| 2 | `02-wayland-protocol.md` | Protocol layer: 3 generators, wire format, object lifecycle, FD transport, custom `web-bitmapbuf` extension. |
| 3 | `03-html5-rendering.md` | Surface → canvas compositing pipeline: Surface state, damage, FrameLoop, WebGL shaders, multi-output. |
| 4 | `04-transport.md` | WebSocket + KCP-over-WebSocket, multi-channel design, message formats, FD virtualization, reconnection. |
| 5 | `05-input-handling.md` | Keyboard / pointer / touch / focus, xkbcommon WASM, keymap FD workaround, DOM → Wayland event mapping. |
| 6 | `06-data-clipboard.md` | `wl_data_device`, selection, drag-and-drop, MIME negotiation, XWayland bridge, primary-selection gap. |
| 7 | `07-native-app-bridge.md` | How real Linux apps get launched, the 3 native Node addons, XWayland bridge, app lifecycle & kill timers. |
| 8 | `08-app-lifecycle.md` | xdg-shell state machine, configure/ack round-trips, popup positioner math, subsurface sync, `UserShellApi`. |
| 9 | `09-hardware-accel.md` | GStreamer H.264 encoders, WASM FFmpeg fallback, WebCodecs preferred path, WebGL texture upload, alpha split. |
| 10 | `10-porting-to-odoo-js-python.md` | Concrete blueprint for porting each Greenfield layer to **JS browser compositor + Python host proxy + Odoo 20 OWL shell**. |
| 11 | `11-pitfalls.md` | 21 catalogued pitfalls + 12 design wins, with file references and what-to-do-instead for each. |
| 12 | `12-appendix.md` | Glossary, file-to-feature index, external references. |
## Recommended reading order
- **Architects / tech leads** — read `00` → `01` → `11` → `10`. You'll get the
lay of the land, what to copy, what to avoid, and the porting blueprint.
- **Browser-compositor engineers** — read `02` → `03` → `04` → `05` → `08` →
`09`. The core engine chapters, in dependency order.
- **Host-proxy engineers (Python)** — read `07` → `04` → `09` → `02` (proxy
generator section) → `10`. Most of the porting work is on your side.
- **Odoo 20 OWL shell engineers** — read `08` (especially the `UserShellApi`
section) → `01` → `10` (Odoo integration section). The shell is the
integration seam where Greenfield stops and your work begins.
- **Skip-it sections** — `06` (clipboard) is only needed if you care about
clipboard sync between browser and native apps.
## Scope & exclusions
**In scope** — every Greenfield package, lib, generator, native addon,
protocol file, transport channel, rendering path, encoding pipeline, and
the `UserShellApi` integration seam.
**Out of scope** — the actual `packages/compositor-shell` Preact+Tailwind
SPA. The user is building their own shell as an Odoo 20 addon. We only
describe the protocol-level integration point (`UserShellApi`) so the
Odoo shell knows what to consume.
**Not covered in depth** — the `sdk/` (Emscripten-based WASM app SDK for
browser-resident Wayland apps). Mentioned in `00-overview.md` and
`10-porting-to-odoo-js-python.md` but not analyzed layer-by-layer,
because the user's goal is running **real Linux apps** in the browser,
not browser-resident WASM apps.
## Conventions
- **File paths** are always relative to the repo root, e.g.
`packages/compositor/src/Surface.ts`.
- **ASCII diagrams** are used throughout — they render correctly in any
Markdown viewer, survive copy-paste, and work in code reviews.
- **No code excerpts** by user request. We describe algorithms in prose
and reference file paths so you can read the source yourself.
- **Cross-references** look like `(see §3 of `03-html5-rendering.md`)`.
## How this analysis was produced
This bundle was produced by 10 parallel research agents, each assigned to a
single layer of the Greenfield codebase:
- **Agent A** — top-level architecture
- **Agent B** — Wayland protocol layer (generators + runtime)
- **Agent C** — HTML5 rendering pipeline
- **Agent D** — transport layer (WebSocket + KCP)
- **Agent E** — input handling (keyboard / pointer / touch / xkbcommon)
- **Agent F** — data device & clipboard
- **Agent G** — native app bridge (process model + native addons)
- **Agent H** — xdg-shell / app lifecycle / `UserShellApi`
- **Agent I** — hardware acceleration / encoding / WASM
- **Agent J** — pitfalls, anti-patterns, design wins
Each agent read the relevant source files end-to-end, produced a stage
summary, and appended its findings to `/home/z/my-project/worklog.md`.
This bundle is the synthesis of those findings into a single readable
document set.
## License of the source
Greenfield's runtime packages (`compositor`, `compositor-proxy`,
`compositor-proxy-cli`, `compositor-shell`) are **AGPL-3.0-or-later**.
The lower-level libs (generators, `common`, `compositor-wasm`,
`compositor-ffmpeg-h264`, `xtsb`) are MIT or Apache-2.0. Anyone forking
or building on the runtime packages must respect AGPL — see
`11-pitfalls.md` §License for implications.