Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents)
This commit is contained in:
commit
c4b45cf218
15 files changed
+5444
No files matched your search
+24
@@ -0,0 +1,24 @@
|
|||||||
|
# Python
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.pyo
|
||||||
|
.venv/
|
||||||
|
venv/
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# Editor
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
|
||||||
|
# Logs
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# Build
|
||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
build/
|
||||||
|
*.egg-info/
|
||||||
+386
@@ -0,0 +1,386 @@
|
|||||||
|
# 00 — What Greenfield Is and How It's Structured
|
||||||
|
|
||||||
|
## The one-paragraph summary
|
||||||
|
|
||||||
|
Greenfield is a Wayland compositor that runs partly in the browser (the actual
|
||||||
|
compositing and rendering, written in TypeScript against WebGL / WebCodecs /
|
||||||
|
WASM) and partly on a Linux host (a Node.js process that hosts real Linux
|
||||||
|
Wayland apps via `libwayland-server` and bridges their wire traffic to the
|
||||||
|
browser over WebSocket). The two halves together let you display **real Linux
|
||||||
|
applications** (GTK3/4, Qt, xterm, X11 apps via XWayland) inside a browser tab,
|
||||||
|
as if they were native browser apps. An optional Emscripten-based SDK lets you
|
||||||
|
compile separate Wayland apps to WASM and run them entirely in the browser with
|
||||||
|
no Linux host required.
|
||||||
|
|
||||||
|
## Repo layout
|
||||||
|
|
||||||
|
```
|
||||||
|
greenfield/
|
||||||
|
├── README.md # "The in-browser wayland compositor"
|
||||||
|
├── package.json # yarn 4.5.0 workspaces: libs/*, packages/*, examples/webapps/*, examples/compositor/*
|
||||||
|
├── .yarnrc.yml # nodeLinker: pnpm, global cache off
|
||||||
|
├── yarn.lock # 9368 lines
|
||||||
|
├── publish.sh # yarn workspaces foreach npm publish
|
||||||
|
├── protocol/ # Wayland XML protocol definitions (4 files)
|
||||||
|
│ ├── wayland.xml # Core Wayland protocol (124KB)
|
||||||
|
│ ├── xdg-shell.xml # Desktop shell protocol (48KB)
|
||||||
|
│ ├── wayland-drm.xml # DRM buffer formats (8.8KB)
|
||||||
|
│ └── web-bitmapbuf-unstable-v1.xml # *** Greenfield custom extension (2KB) ***
|
||||||
|
├── libs/ # 9 internal @gfld/* libraries
|
||||||
|
│ ├── common/ # @gfld/common - shared client/server runtime (wire format)
|
||||||
|
│ ├── client-protocol/ # @gfld/client-protocol - browser-side generated Wayland bindings
|
||||||
|
│ ├── compositor-protocol/ # @gfld/compositor-protocol - Node.js-side generated Wayland bindings
|
||||||
|
│ ├── client-generator/ # CLI: XML → TS browser protocol source
|
||||||
|
│ ├── compositor-generator/ # CLI: XML → TS Node protocol source
|
||||||
|
│ ├── compositor-proxy-generator/ # CLI: XML → TS proxy protocol source
|
||||||
|
│ ├── compositor-wasm/ # pixman + xkbcommon compiled to WASM
|
||||||
|
│ ├── compositor-ffmpeg-h264/ # FFmpeg n5.1.2 compiled to WASM (H.264 software fallback decoder)
|
||||||
|
│ └── xtsb/ # X11/XCB TypeScript bindings (vendored)
|
||||||
|
├── packages/ # 4 user-facing @gfld/* packages
|
||||||
|
│ ├── compositor/ # @gfld/compositor - browser-side Wayland compositor library (AGPL)
|
||||||
|
│ ├── compositor-shell/ # @gfld/compositor-shell - Preact+Tailwind SPA shell (AGPL) *** OUT OF SCOPE ***
|
||||||
|
│ ├── compositor-proxy/ # @gfld/compositor-proxy - native Wayland compositor proxy (AGPL, Linux-only)
|
||||||
|
│ └── compositor-proxy-cli/ # @gfld/compositor-proxy-cli - CLI binary (AGPL, Linux-only)
|
||||||
|
├── sdk/ # Greenfield WebAssembly SDK (for browser-resident WASM apps)
|
||||||
|
│ ├── README.md
|
||||||
|
│ ├── build.sh # clones emsdk 3.1.60, builds 30+ sysrootlibs
|
||||||
|
│ ├── sdk_env.sh # exports PKG_CONFIG_LIBDIR + PATH=toolkit/bin
|
||||||
|
│ ├── toolkit/ # gfcc/gf++/gfar/gfnm wrappers around emcc + wayland-scanner + meson cross inis
|
||||||
|
│ └── sysrootlibs/ # 34 build scripts for cross-compiling graphics stack (wayland, pixman, cairo, pango, gtk4, etc.)
|
||||||
|
├── examples/
|
||||||
|
│ ├── compositor/experimental-fullscreen/ # vite PWA kiosk running a single remote app
|
||||||
|
│ ├── docker-compose/ # Dockerfile + applications.json + docker-compose.yml for gtk3-demo/gtk4-demo/xterm/glxgears
|
||||||
|
│ ├── sdk/simple-shm/ # Weston simple-shm.c built with gfcc + wayland-scanner
|
||||||
|
│ ├── sdk/weston/ # Weston clients (terminal, dnd, editor, keyboard, flower, ...) meson-built
|
||||||
|
│ ├── sdk/gtk4/ # GTK4 demos meson-built to WASM
|
||||||
|
│ └── webapps/ # Pure-JS browser demos: simple-shm, webgl, weston-clients (symlink), gtk4
|
||||||
|
├── docs/ # Jekyll site (CNAME: udev.be/greenfield)
|
||||||
|
│ └── pages/{components, design, future_plans, getting_started, sdk}
|
||||||
|
└── docker/ # Production Dockerfile for compositor-proxy-cli
|
||||||
|
```
|
||||||
|
|
||||||
|
## The dual-side architecture
|
||||||
|
|
||||||
|
Greenfield is split into **two cooperating halves**, connected over WebSocket
|
||||||
|
(or, in the future, WebTransport). Understanding this split is fundamental to
|
||||||
|
everything else in this bundle.
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ BROWSER SIDE (TypeScript + WASM) │
|
||||||
|
│ │
|
||||||
|
│ ┌─────────────────────────┐ ┌──────────────────────────────────────────┐ │
|
||||||
|
│ │ @gfld/compositor-shell │ │ @gfld/compositor │ │
|
||||||
|
│ │ (Preact + Tailwind SPA)│───▶│ - Session, Compositor, Surface, Seat │ │
|
||||||
|
│ │ *** OUT OF SCOPE *** │ │ - Keyboard, Pointer, Touch, Xkb (WASM) │ │
|
||||||
|
│ └─────────────────────────┘ │ - Output (WebGL canvas) │ │
|
||||||
|
│ │ - render/ (WebGL shaders: YUV→RGB, etc.) │ │
|
||||||
|
│ ┌─────────────────────────┐ │ - desktop/ (Floating vs AlwaysFullScreen) │ │
|
||||||
|
│ │ JS/WASM webapps │───▶│ - remote/ (Channel, kcp.ts, decoders) │ │
|
||||||
|
│ │ (use @gfld/client- │ │ - browser/ (dnd, selection, input) │ │
|
||||||
|
│ │ protocol + common) │ │ │ │
|
||||||
|
│ └─────────────────────────┘ └──────────────┬──────────────┬─────────────┘ │
|
||||||
|
│ │ │ │
|
||||||
|
│ depends on ▼ ▼ │
|
||||||
|
│ ┌─────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
|
||||||
|
│ │ @gfld/compositor│ │ @gfld/common │ │ @gfld/xtsb │ │
|
||||||
|
│ │ -protocol │ │ (Connection, │ │ (X11/XCB bindings,│ │
|
||||||
|
│ │ (Node-side │ │ wire format)│ │ used by both │ │
|
||||||
|
│ │ generated │ │ │ │ compositor+proxy)│ │
|
||||||
|
│ │ bindings) │ │ │ └──────────────────┘ │
|
||||||
|
│ └─────────────────┘ └──────────────┘ │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
│ WebSocket (TCP, perMessageDeflate=false)
|
||||||
|
│ + KCP-over-WebSocket (reliable UDP, ready for WebTransport)
|
||||||
|
│ + 9 endpoint paths per session
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ NATIVE SIDE (Linux, Node 18+, C) │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────┐ ┌──────────────────────────────────────────┐ │
|
||||||
|
│ │ @gfld/compositor-proxy-cli│───▶│ @gfld/compositor-proxy │ │
|
||||||
|
│ │ - main.ts: http + ws │ │ - Session, SessionController │ │
|
||||||
|
│ │ WebSocketServer │ │ - NativeWaylandClientSession │ │
|
||||||
|
│ │ - forks SessionProcess │ │ - NativeWaylandCompositorSession │ │
|
||||||
|
│ │ per browser session │ │ - NativeAppContext (launch) │ │
|
||||||
|
│ │ - --bind-port=8081 │ │ - SurfaceBufferEncoding │ │
|
||||||
|
│ │ - --applications=<json> │ │ - Channel, FrameFeedback │ │
|
||||||
|
│ │ - --encoder=x264|nvh264 │ │ - XWaylandSession │ │
|
||||||
|
│ │ - --render-device=/dev/ │ │ - kcp.ts (1100-line TypeScript impl) │ │
|
||||||
|
│ │ dri/renderD128 │ │ - wayland-server.ts (TS wrapper) │ │
|
||||||
|
│ └──────────────────────────┘ │ - io/, encoding/, protocol/ │ │
|
||||||
|
│ └──────────────┬──────────────┬─────────────┘ │
|
||||||
|
│ │ │ │
|
||||||
|
│ depends on ▼ ▼ │
|
||||||
|
│ ┌────────────────────────┐ ┌────────────────────────────────┐ │
|
||||||
|
│ │ @gfld/compositor-proxy-│ │ native/ (C, built via CMake + │ │
|
||||||
|
│ │ generator │ │ Ninja) │ │
|
||||||
|
│ │ (generates *_interface │ │ - wayland/src/addon.c │ │
|
||||||
|
│ │ and *_interceptor.js) │ │ - encoding/src/gst_frame_ │ │
|
||||||
|
│ └────────────────────────┘ │ encoder.c │ │
|
||||||
|
│ │ - poll/src/poll.c (libuv) │ │
|
||||||
|
│ │ - wayland-server/ (patched) │ │
|
||||||
|
│ │ - westfield-*.c (EGL, dmabuf, │ │
|
||||||
|
│ │ XWayland, etc.) │ │
|
||||||
|
│ └────────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ Native Wayland apps (GTK3/4, Qt, xterm, X11 via XWayland) attach to the │
|
||||||
|
│ proxy as if it were a real wl_display. The proxy encodes each surface's │
|
||||||
|
│ buffer to H.264 via GStreamer (x264 software or nvh264 hardware) using │
|
||||||
|
│ zero-copy dmabuf, sends frames over WebSocket/KCP to the browser. │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Tech-stack table
|
||||||
|
|
||||||
|
| Category | Component | Tech / Version | Role |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **Languages** | Browser TS | TypeScript ^5.5.2 | All `@gfld/*` TS packages |
|
||||||
|
| | Native C | C99 | `packages/compositor-proxy/native/` |
|
||||||
|
| | Shell | bash | sdk/build.sh, toolkit wrappers |
|
||||||
|
| | Python | python3 (xcbgen) | libs/xtsb protocol generation |
|
||||||
|
| **Build tools** | Workspace | Yarn 4.5.0 (Berry, pnpm nodeLinker) | Monorepo workspace mgmt |
|
||||||
|
| | Browser bundler | Vite ^5.3.1 | compositor-shell, examples |
|
||||||
|
| | TS compiler | tsc ^5.5.2 | libs/* and packages/* build |
|
||||||
|
| | Native build | CMake 3.13+ + Ninja | packages/compositor-proxy `build:native` |
|
||||||
|
| | Native pkg | @vercel/pkg (Brotli) + @vercel/ncc | compositor-proxy-cli single-binary |
|
||||||
|
| | Lint/format | ESLint ^8.57 + Prettier ^3.3.2 | All TS packages |
|
||||||
|
| | Test | Jest ^29.7.0 + ts-jest ^29.1.5 | libs/xtsb, packages/compositor-proxy |
|
||||||
|
| **WASM toolchain** | Emscripten (libs build) | emsdk 3.1.46 | libs/compositor-wasm, libs/compositor-ffmpeg-h264 |
|
||||||
|
| | Emscripten (SDK build) | emsdk 3.1.60 | sdk/build.sh (sysrootlibs) |
|
||||||
|
| | Meson cross | meson-gf-cross.ini (host_machine: wasm32/emscripten) | sdk/toolkit |
|
||||||
|
| **Wayland / GL stack** | wayland.xml / xdg-shell.xml / wayland-drm.xml | upstream freedesktop.org | protocol/ |
|
||||||
|
| | wayland-scanner | vendored at sdk/toolkit/bin/wayland-scanner | C header/codegen for native clients |
|
||||||
|
| | libwayland (server) | vendored + patched at packages/compositor-proxy/native/wayland/src/wayland-server/ | native compositor server side |
|
||||||
|
| | wl_drm + linux-dmabuf-unstable-v1 | implemented in native/wayland/src/wlr_*.c | zero-copy dmabuf to GStreamer |
|
||||||
|
| **Custom protocol** | web-bitmapbuf-unstable-v1.xml | Greenfield extension (Erik De Rijcke, 2023) | browser JS/WASM clients wrap HTML ImageBitmap as wl_buffer via fd |
|
||||||
|
| **Native GL/media runtime** | GStreamer | 1.20 (built from gitlab.freedesktop.org in docker) | H.264 encoding pipeline, plugins: base/good/bad/ugly/gl + nvcodec |
|
||||||
|
| | GStreamer encoders | x264 (software), nvh264 (NVIDIA hardware), vaapih264 (Intel, experimental) | --encoder flag on compositor-proxy-cli |
|
||||||
|
| | Mesa / OpenGL / EGL | libgbm1, libdrm2, libopengl0, libglvnd0, libegl1-mesa | GPU buffer mgmt + EGL rendering |
|
||||||
|
| | NVIDIA | libnvidia-egl-wayland1, libnvidia-egl-gbm1 | optional HW accel |
|
||||||
|
| **Browser transport** | WebSocket | ws ^8.17.1 (Node), native WebSocket (browser) | primary transport (TCP) |
|
||||||
|
| | KCP | TypeScript impl (1100+ lines per side) | planned reliable UDP — future switch to WebTransport |
|
||||||
|
| | WebTransport | mentioned in docs/design as future | not yet implemented |
|
||||||
|
| | XWayland | xwayland package + xauth + xxd | X11 backward compat on native side |
|
||||||
|
| **Browser rendering** | WebGL | WebGL1 native, `desynchronized:true` | Output.ts canvas, render/*.ts shaders |
|
||||||
|
| | WebCodecs API | @types/dom-webcodecs ^0.1.11 | H.264 GPU decode in browser (preferred path) |
|
||||||
|
| | WebAssembly (FFmpeg) | libs/compositor-ffmpeg-h264 (FFmpeg n5.1.2) | software fallback H.264 decode |
|
||||||
|
| | WebAssembly (pixman+xkbcommon) | libs/compositor-wasm (emsdk 3.1.46) | pixel ops + XKB keymap math |
|
||||||
|
| | SharedArrayBuffer + Web Workers | browser-native (COOP/COEP required) | async compositor, H.264 NAL decoder workers |
|
||||||
|
| | ImageBitmap | browser-native | web-bitmapbuf-unstable-v1 backing store |
|
||||||
|
| **Shell UI (out of scope)** | Preact | ^10.22.0 | compositor-shell SPA |
|
||||||
|
| | Tailwind CSS | ^3.4.4 | compositor-shell styling |
|
||||||
|
| | @preact/signals | ^1.2.3 | reactivity |
|
||||||
|
|
||||||
|
## Package inventory
|
||||||
|
|
||||||
|
### Browser side
|
||||||
|
|
||||||
|
**`packages/compositor`** (`@gfld/compositor`, AGPL-3.0) — The browser-side
|
||||||
|
Wayland compositor library. Pure TypeScript + WebGL + WebCodecs + WASM, no
|
||||||
|
3rd-party runtime deps. Exports `createCompositorSession()` and
|
||||||
|
`createAppLauncher(session, 'web' | 'remote')`. Internally: `Session.ts`
|
||||||
|
orchestrates, `Compositor.ts`/`Surface.ts`/`Seat.ts`/`Pointer.ts`/
|
||||||
|
`Keyboard.ts`/`Output.ts` implement Wayland semantics, `render/` holds WebGL
|
||||||
|
shaders (YUV/YUVA/RGB→RGBA conversion), `desktop/` arranges surfaces
|
||||||
|
(Floating vs AlwaysFullScreen), `web/` speaks to same-origin-iframe JS/WASM
|
||||||
|
clients, `remote/` speaks to the native proxy (with `kcp.ts`,
|
||||||
|
`websocket-events.ts`, `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`,
|
||||||
|
`H264NALDecoder.worker.ts`, `xwayland/`), `browser/` handles dnd/selection/
|
||||||
|
input. Output is a WebGL texture inside an HTML5 canvas. Async — a slow client
|
||||||
|
never blocks another.
|
||||||
|
|
||||||
|
**`libs/client-protocol`** (`@gfld/client-protocol`, MIT) — Browser-side
|
||||||
|
generated Wayland protocol bindings (wayland.xml + xdg-shell.xml +
|
||||||
|
web-bitmapbuf-unstable-v1.xml). Used by JS/WASM webapps via `@gfld/common`
|
||||||
|
for the wire format.
|
||||||
|
|
||||||
|
**`libs/compositor-protocol`** (`@gfld/compositor-protocol`, MIT) — Node-side
|
||||||
|
generated Wayland protocol bindings (same 3 XMLs). Used by
|
||||||
|
`@gfld/compositor` for the in-browser server-side marshalling of Wayland
|
||||||
|
messages.
|
||||||
|
|
||||||
|
**`libs/common`** (`@gfld/common`, MIT) — Shared client/server Wayland
|
||||||
|
wire-format runtime. Two source files: `Connection.ts` (message framing +
|
||||||
|
argument codec) + `index.ts`. Tiny core that both `client-protocol` and
|
||||||
|
`compositor-protocol` build on.
|
||||||
|
|
||||||
|
**`libs/compositor-wasm`** (`@gfld/compositor-wasm`, MIT) — pixman 0.42.2 +
|
||||||
|
xkbcommon 1.5.0 compiled to WASM via emsdk 3.1.46. Exposes
|
||||||
|
`src/index.ts` (`init` function). Used by `@gfld/compositor` for pixel-region
|
||||||
|
math and XKB keyboard layout state.
|
||||||
|
|
||||||
|
**`libs/compositor-ffmpeg-h264`** (`@gfld/compositor-ffmpeg-h264`, Apache-2.0)
|
||||||
|
— FFmpeg n5.1.2 compiled to a single WASM module. TypeScript wrapper in
|
||||||
|
`src/{H264Decoder.ts, H264Worker.ts, index.ts}` runs in a Web Worker. Used as
|
||||||
|
the software fallback H.264 decoder when the browser lacks WebCodecs.
|
||||||
|
|
||||||
|
### Native side
|
||||||
|
|
||||||
|
**`packages/compositor-proxy`** (`@gfld/compositor-proxy`, AGPL-3.0,
|
||||||
|
Linux-only, Node ≥18) — Native Wayland compositor that bridges Linux apps to
|
||||||
|
the browser. Has BOTH a TypeScript layer (`src/Session.ts`,
|
||||||
|
`NativeWaylandClientSession.ts`, `NativeWaylandCompositorSession.ts`,
|
||||||
|
`NativeAppContext.ts`, `SurfaceBufferEncoding.ts`, `XWaylandSession.ts`,
|
||||||
|
`kcp.ts`, `Channel.ts`, `FrameFeedback.ts`, `wayland-server.ts`) AND a C
|
||||||
|
layer (`native/`) built with `cmake -G Ninja`. The C layer produces:
|
||||||
|
|
||||||
|
- `wayland-server.so` — patched libwayland-server (vendored at
|
||||||
|
`native/wayland/src/wayland-server/`, with `changes.diff` and
|
||||||
|
`generate_wayland_server_patch.sh`)
|
||||||
|
- `westfield.so` — Greenfield's westfield-* helpers (buffer, dmabuf, egl,
|
||||||
|
xwayland, fdutils) + `wlr_drm.c` + `wlr_linux_dmabuf_v1.c` +
|
||||||
|
`drm-protocol.c` + `linux-dmabuf-unstable-v1-protocol.c`
|
||||||
|
- `proxy-encoding.so` — GStreamer frame encoder (`gst_frame_encoder.c` +
|
||||||
|
`gst_main_loop.c`)
|
||||||
|
- `proxy-encoding-addon.node` + `proxy-poll-addon.node` +
|
||||||
|
`wayland-server-addon.node` — Node-API addons (vendored node-api headers in
|
||||||
|
`native/node-api/`) loaded by `src/addons/`
|
||||||
|
- `poll.c` — epoll wrapper backed by libuv
|
||||||
|
|
||||||
|
CMake deps: glib-2.0, gstreamer-1.0/app/video/gl/allocators ≥1.18,
|
||||||
|
graphene-1.0, egl ≥1.5, opengl ≥3.3, libffi, gbm, libdrm. Runtime also needs
|
||||||
|
libudev, mesa, NVIDIA EGL wayland/gbm drivers, xwayland. Implements `wl_drm`
|
||||||
|
+ `linux-dmabuf-v1` for zero-copy GPU buffer transfer to GStreamer encoder
|
||||||
|
pipeline (x264 software or nvh264 hardware). Connects to browser via
|
||||||
|
WebSocket; has TypeScript KCP implementation ready for the future WebTransport
|
||||||
|
switch. Supports direct peer-to-peer copy-paste between two proxy instances
|
||||||
|
to avoid browser round-trip.
|
||||||
|
|
||||||
|
**`packages/compositor-proxy-cli`** (`@gfld/compositor-proxy-cli`, AGPL-3.0,
|
||||||
|
Linux-only, Node ≥18) — CLI binary that wraps `@gfld/compositor-proxy`.
|
||||||
|
`src/main.ts` runs an HTTP server + `ws` WebSocketServer on `--bind-port`
|
||||||
|
(default 8081), and on first connection per `compositorSessionId` query
|
||||||
|
param, `fork()`s a `SessionProcess` (src/SessionProcess.ts) — one child per
|
||||||
|
active browser session. CLI args: `--applications=<json>` (URL→executable
|
||||||
|
map), `--encoder=x264|nvh264`, `--render-device=/dev/dri/renderD128`,
|
||||||
|
`--base-url`, `--allow-origin`, `--bind-ip`, `--bind-port`, `--basic-auth`.
|
||||||
|
The `--applications` JSON maps paths like `/gtk4-demo`, `/kwrite`, `/xterm`
|
||||||
|
to executables + args + env. Packaged with `@vercel/pkg` (Brotli, target
|
||||||
|
`node18-linux`) into a single static binary.
|
||||||
|
|
||||||
|
**`libs/compositor-proxy-generator`** (`@gfld/compositor-proxy-generator`,
|
||||||
|
MIT) — CLI that converts `protocol/{wayland,xdg-shell,wayland-drm}.xml` into
|
||||||
|
TypeScript source under `packages/compositor-proxy/src/protocol/`. Used at
|
||||||
|
build time only (yarn `generate` script).
|
||||||
|
|
||||||
|
### Shell side (OUT OF SCOPE for user)
|
||||||
|
|
||||||
|
**`packages/compositor-shell`** (`@gfld/compositor-shell`, AGPL-3.0) — Preact
|
||||||
|
+ Tailwind single-page-app shell built with Vite. Depends only on
|
||||||
|
`@gfld/compositor` + `preact` at runtime. Provides application launcher,
|
||||||
|
keyboard config, window management UI. **Explicitly out of scope for the
|
||||||
|
user** per task brief. The protocol-level integration point the user needs
|
||||||
|
to know about is `UserShellApi` (see `08-app-lifecycle.md`).
|
||||||
|
|
||||||
|
### Generators (3 sibling CLIs)
|
||||||
|
|
||||||
|
All three generators share the same dependency set (`camelcase`,
|
||||||
|
`uppercamelcase`, `meow`, `xml2js`) and similar layout
|
||||||
|
(`ProtocolParser.mjs` + `ProtocolArguments.mjs` + `index.mjs`). They consume
|
||||||
|
the same `protocol/*.xml` files but emit slightly different TypeScript
|
||||||
|
flavors for each side:
|
||||||
|
|
||||||
|
- **`libs/client-generator`** — emits browser-side client bindings into
|
||||||
|
`libs/client-protocol/src/protocol/`.
|
||||||
|
- **`libs/compositor-generator`** — emits Node.js-side server bindings into
|
||||||
|
`libs/compositor-protocol/src/protocol/`.
|
||||||
|
- **`libs/compositor-proxy-generator`** — emits proxy-side bindings into
|
||||||
|
`packages/compositor-proxy/src/protocol/`. Different file layout:
|
||||||
|
`src/EndpointProtocolArguments.mjs` + `EndpointProtocolParser.mjs` +
|
||||||
|
`index.mjs`.
|
||||||
|
|
||||||
|
### XWayland support
|
||||||
|
|
||||||
|
**`libs/xtsb`** (`@gfld/xtsb`, MIT, repo: github.com/udevbe/xtsb) — Vendored
|
||||||
|
TypeScript X11/XCB bindings. Builds from `xcbgen/ts_client.py` against 5
|
||||||
|
XCB protocol XMLs in `libs/xtsb/protocol/`: `xproto.xml`, `shape.xml`,
|
||||||
|
`xfixes.xml`, `render.xml`, `composite.xml`. Dual main/module entries:
|
||||||
|
`dist/main/index.node.js` (Node) + `dist/module/index.web.js` (browser).
|
||||||
|
Used by BOTH `@gfld/compositor` (browser side,
|
||||||
|
`packages/compositor/src/remote/xwayland/`) AND `@gfld/compositor-proxy`
|
||||||
|
(native side, `src/XWaylandSession.ts`) — same TS API on both ends of the
|
||||||
|
wire. On the native C side,
|
||||||
|
`packages/compositor-proxy/native/wayland/src/westfield-xwayland.c`
|
||||||
|
actually launches XWayland and bridges it.
|
||||||
|
|
||||||
|
### Wire between browser and native
|
||||||
|
|
||||||
|
- **Transport today**: WebSocket over TCP. Browser connects to
|
||||||
|
`ws(s)://<proxy-host>:<port>/<app-path>?compositorSessionId=<uuid>`.
|
||||||
|
`compositor-proxy-cli` uses `ws` ^8.17.1 with `perMessageDeflate: false`
|
||||||
|
(compression off for latency). The `compositorSessionId` is generated by
|
||||||
|
the browser and used by the CLI to route the upgrade to the correct
|
||||||
|
per-session child process.
|
||||||
|
- **Transport future**: KCP-over-WebTransport (reliable UDP). The codebase
|
||||||
|
already has a full TS KCP implementation in both
|
||||||
|
`packages/compositor-proxy/src/kcp.ts` (~1100 lines) and
|
||||||
|
`packages/compositor/src/remote/kcp.ts`. `docs/pages/design/index.md`
|
||||||
|
explicitly calls out that WebSockets-over-TCP is ill-suited for real-time,
|
||||||
|
that WebRTC DataChannels' SCTP congestion control is too slow, and that
|
||||||
|
they're waiting for WebTransport + will pair it with KCP +
|
||||||
|
forward-error-correction.
|
||||||
|
- **Out-of-band data**: copy-paste between two proxy instances goes
|
||||||
|
peer-to-peer (proxy → proxy) without browser round-trip, configured via
|
||||||
|
`--base-url` so each proxy can advertise itself to peers.
|
||||||
|
|
||||||
|
### SDK (`sdk/`) — for browser-resident WASM apps, NOT native apps
|
||||||
|
|
||||||
|
The Greenfield SDK lets you compile existing native Wayland C/C++ apps to
|
||||||
|
WebAssembly and run them directly in the browser compositor (no proxy
|
||||||
|
needed). Based on Emscripten 3.1.60 with three Greenfield-specific
|
||||||
|
extensions:
|
||||||
|
|
||||||
|
1. `sdk/toolkit/bin/gfcc` and `gf++` — wrapper scripts around `emcc`/`em++`
|
||||||
|
that inject pthread pool size, asyncify, and the unix-socket fs shim.
|
||||||
|
2. `sdk/toolkit/bin/wayland-scanner` — symlink to the natively-built
|
||||||
|
wayland-scanner inside `build-sysroot/bin/wayland-scanner`, used to
|
||||||
|
generate C client headers/source from xdg-shell.xml during app builds.
|
||||||
|
3. `sdk/sysrootlibs/` — 34 build scripts that cross-compile a full graphics
|
||||||
|
stack (wayland, wayland-protocols, pixman, zlib, png, xml2, xkbcommon,
|
||||||
|
icu, harfbuzz, freetype, fontconfig, pcre2, glib, jpeg, cairo, fribidi,
|
||||||
|
tiff4, pango, gdk-pixbuf, egl-stub, glesv2-stub, epoxy-emscripten,
|
||||||
|
graphene, sass, sassc, uapi-stub, upoll, cygepoll, expat, libffi;
|
||||||
|
gtk4 currently broken) into `sdk/sysroot/` for static linking.
|
||||||
|
|
||||||
|
`sdk/build.sh` runs everything end-to-end and writes `meson-gf-toolchain.ini`
|
||||||
|
(with absolute `_SDK_DIR`). `sdk/sdk_env.sh` exports `PKG_CONFIG_LIBDIR`
|
||||||
|
(pointing at `sysroot/lib/pkgconfig` + `build-sysroot/lib/pkgconfig`) and
|
||||||
|
prepends `toolkit/bin` to `PATH`. Meson projects use
|
||||||
|
`--cross-file toolkit/meson-gf-cross.ini --cross-file toolkit/meson-gf-toolchain.ini`
|
||||||
|
(host_machine: cpu=mvp, cpu_family=wasm32, system=emscripten).
|
||||||
|
|
||||||
|
**Important**: the `sdk/` is **not for native apps**. It's exclusively for
|
||||||
|
**WASM apps** that run inside the browser via Emscripten. Native Linux apps
|
||||||
|
on the proxy use the host's own system libraries — the Dockerfile installs
|
||||||
|
them via `apt-get install libffi8 libgbm1 libgstreamer1.0-plugins-base …
|
||||||
|
xwayland xauth`.
|
||||||
|
|
||||||
|
## Versioning note
|
||||||
|
|
||||||
|
Every workspace package is `1.0.0-rc1`. This signals:
|
||||||
|
|
||||||
|
1. **Feature freeze for 1.0** — the architecture, protocol bindings, and
|
||||||
|
public APIs are expected to be stable; no breaking changes anticipated
|
||||||
|
before 1.0.0 final.
|
||||||
|
2. **Not yet production-blessed** — known issues remain (e.g. gtk4 WASM is
|
||||||
|
"broken for now" per `sdk/build.sh` comment; WebTransport/KCP-over-UDP
|
||||||
|
is designed but not yet active; Linux Kernel Library WASM + JuiceFS web
|
||||||
|
FS are explicitly future work).
|
||||||
|
3. **Yarn Berry semantics** — `workspace:^` resolution means all `@gfld/*`
|
||||||
|
deps are pinned to rc1 across the tree; if the user forks and bumps,
|
||||||
|
they bump the whole tree consistently.
|
||||||
|
|
||||||
|
For the user's purpose (building their own Wayland-to-HTML compositor for an
|
||||||
|
Odoo 20 + JS/Python web OS), the rc1 status means they should treat
|
||||||
|
Greenfield as a stable architectural reference but plan to pin to a
|
||||||
|
specific commit if they fork it, since 1.0.0 final may land breaking-ish
|
||||||
|
changes before the rc1 → 1.0.0 transition completes.
|
||||||
|
|
||||||
|
## License of the source
|
||||||
|
|
||||||
|
The four `packages/*` (compositor, compositor-proxy, compositor-proxy-cli,
|
||||||
|
compositor-shell) are **AGPL-3.0-or-later**. The lower-level `libs/*` are
|
||||||
|
MIT or Apache-2.0. Anyone forking or building on the runtime packages must
|
||||||
|
respect AGPL — see `11-pitfalls.md` §License for implications.
|
||||||
@@ -0,0 +1,443 @@
|
|||||||
|
# 01 — Layered Architecture Map
|
||||||
|
|
||||||
|
This chapter walks Greenfield's architecture layer by layer, from the lowest
|
||||||
|
abstraction (Wayland XML) up to the highest (the user-facing shell). Each layer
|
||||||
|
is described in terms of: **responsibility**, **key abstractions**,
|
||||||
|
**dependencies**, and **files of interest**. The chapter ends with an ASCII
|
||||||
|
diagram showing how the layers stack and where data flows between them.
|
||||||
|
|
||||||
|
## The eight layers
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Layer 8: USER SHELL (Odoo 20 OWL addon, supplied by user) │
|
||||||
|
│ Hooks: UserShellApi (events + actions) │
|
||||||
|
│ ─ NOT covered in this chapter (see 08-app-lifecycle.md) │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 7: DESKTOP / LAYOUT POLICY │
|
||||||
|
│ Floating vs AlwaysFullScreen, view stacking, focus activation refcount │
|
||||||
|
│ packages/compositor/src/desktop/Desktop.ts │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 6: SURFACE LIFECYCLE / ROLES │
|
||||||
|
│ XdgWmBase, XdgSurface, XdgToplevel, XdgPopup, XdgPositioner, Shell, │
|
||||||
|
│ ShellSurface, Subcompositor, Subsurface, SurfaceRole │
|
||||||
|
│ packages/compositor/src/{Xdg*, Shell*, Sub*, SurfaceRole, SurfaceChild} │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 5: RENDERING PIPELINE │
|
||||||
|
│ Surface (state stack), View, Output (WebGL canvas), Renderer, Scene, │
|
||||||
|
│ Shaders (YUV→RGBA, YUVA→RGBA, RGB+A→RGBA), Loop (RAF), Callback │
|
||||||
|
│ packages/compositor/src/{Surface, View, Output, Loop, Callback, Region} │
|
||||||
|
│ packages/compositor/src/render/ │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 4: TRANSPORT (browser side) │
|
||||||
|
│ Channel (ARQ + Simple), ReconnectingWebSocket, RemoteAppLauncher, │
|
||||||
|
│ StreamingBuffer, BufferStream, EncodedFrame, EncoderFeedback, decoders │
|
||||||
|
│ (WebCodecs + WASM fallback) │
|
||||||
|
│ packages/compositor/src/remote/ │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 4: TRANSPORT (native side) │
|
||||||
|
│ Channel (ARQ + Simple), SessionController, kcp.ts, NativeAppContext, │
|
||||||
|
│ FrameFeedback, SurfaceBufferEncoding, wayland-server.ts │
|
||||||
|
│ packages/compositor-proxy/src/{Channel, kcp, Session*, Native*, ...} │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 3: WAYLAND RUNTIME (browser side) │
|
||||||
|
│ Connection (wire framing), Client, WlObject registry, marshallers │
|
||||||
|
│ libs/common/src/Connection.ts │
|
||||||
|
│ libs/compositor-protocol/src/{westfield-runtime-server.ts, generated/*} │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 3: WAYLAND RUNTIME (native side) │
|
||||||
|
│ wayland-server.ts + wayland-server-addon.node (N-API → libwayland-server)│
|
||||||
|
│ MessageInterceptor (per-resource request/event routing) │
|
||||||
|
│ packages/compositor-proxy/src/{wayland-server.ts, protocol/*_interceptor}│
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 2: PROTOCOL GENERATORS │
|
||||||
|
│ client-generator, compositor-generator, compositor-proxy-generator │
|
||||||
|
│ libs/{client,compositor,compositor-proxy}-generator/ │
|
||||||
|
├──────────────────────────────────────────────────────────────────────────┤
|
||||||
|
│ Layer 1: WAYLAND PROTOCOL XML │
|
||||||
|
│ protocol/{wayland, xdg-shell, wayland-drm, web-bitmapbuf-unstable-v1}.xml│
|
||||||
|
└──────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layer 1 — Wayland Protocol XML
|
||||||
|
|
||||||
|
**Responsibility**: declare the protocol surface — what interfaces, requests,
|
||||||
|
events, enums, and bitfield types exist on the wire.
|
||||||
|
|
||||||
|
**Files of interest**:
|
||||||
|
|
||||||
|
- `protocol/wayland.xml` — standard upstream Wayland core. 22 interfaces,
|
||||||
|
`wl_compositor` v4, `wl_seat` v6, `wl_surface` v4 (roughly Wayland
|
||||||
|
1.20-ish). Kristian Høgsberg copyright. Used by all 3 generators.
|
||||||
|
- `protocol/xdg-shell.xml` — stable `xdg_shell` v1, 5 interfaces (xdg_wm_base,
|
||||||
|
xdg_surface, xdg_toplevel, xdg_popup, xdg_positioner). Used by all 3
|
||||||
|
generators.
|
||||||
|
- `protocol/wayland-drm.xml` — legacy `wl_drm` v2, used by native apps (Mesa,
|
||||||
|
XWayland) to advertise dmabuf buffers. **Only the proxy generator
|
||||||
|
processes this file** — so the proxy can intercept `create_buffer` /
|
||||||
|
`create_planar_buffer` / `create_prime_buffer` and turn dmabuf fds into
|
||||||
|
ProxyFDs for the browser's GStreamer encoding pipeline. Browser-side libs
|
||||||
|
do not generate `wl_drm` stubs.
|
||||||
|
- `protocol/web-bitmapbuf-unstable-v1.xml` — **Greenfield's custom
|
||||||
|
extension**. Single interface `web_bitmapbuf_factory` v1, single request
|
||||||
|
`create_buffer(id: new_id<wl_buffer>, bitmap: fd)`. Lets browser JS apps
|
||||||
|
hand a GPU-backed `ImageBitmap` directly to the compositor as a `wl_buffer`
|
||||||
|
— bypassing SHM copy, encoding, and dmabuf import. The `fd` arg is
|
||||||
|
marshalled via the same `fileDescriptor(...)` path as
|
||||||
|
`wl_shm.create_pool`'s fd, but holds an `ImageBitmap` (becomes a
|
||||||
|
transferable on the flush path). **Not processed by the proxy generator**
|
||||||
|
— native apps never use it.
|
||||||
|
|
||||||
|
**Dependencies**: none. This is the source of truth.
|
||||||
|
|
||||||
|
## Layer 2 — Protocol Generators
|
||||||
|
|
||||||
|
**Responsibility**: parse the XML with `xml2js`, walk the interface tree,
|
||||||
|
emit typed TypeScript bindings for each side.
|
||||||
|
|
||||||
|
**Key abstractions**: `ProtocolParser.mjs` (shared XML walker) +
|
||||||
|
`ProtocolArguments.mjs` (per-side argument-type dispatch table).
|
||||||
|
|
||||||
|
**Files of interest**:
|
||||||
|
|
||||||
|
- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs` —
|
||||||
|
emits TS `*Proxy` classes for browser apps that want to be Wayland clients
|
||||||
|
of the browser compositor (via `MessagePort`).
|
||||||
|
- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
|
||||||
|
— emits TS `*Resource` + `*Requests` classes for the browser-resident
|
||||||
|
compositor itself (receives requests, dispatches to `*Requests` impls,
|
||||||
|
emits events). Skips `wl_display`, `wl_registry` (hand-written in runtime).
|
||||||
|
- `libs/compositor-proxy-generator/{EndpointProtocolParser,
|
||||||
|
EndpointProtocolArguments, index}.mjs` — emits CommonJS
|
||||||
|
`*_interface.js` (wl_interface struct definitions for the N-API addon)
|
||||||
|
+ `*_interceptor.js` (per-opcode interceptors that decide routing) for
|
||||||
|
the Node.js native proxy.
|
||||||
|
|
||||||
|
**Why three generators?** Because each side has a fundamentally different job:
|
||||||
|
|
||||||
|
1. **client-generator** → browser apps that want to be Wayland clients of the
|
||||||
|
browser compositor (via `MessagePort`).
|
||||||
|
2. **compositor-generator** → the browser-resident compositor itself (receives
|
||||||
|
requests, dispatches to `*Requests` impls, emits events).
|
||||||
|
3. **compositor-proxy-generator** → the Node.js native proxy bridging real
|
||||||
|
libwayland-client apps to the browser. Emits `wl_interface` struct
|
||||||
|
definitions for the N-API addon + per-opcode interceptors that decide
|
||||||
|
routing.
|
||||||
|
|
||||||
|
**Dependencies**: `xml2js`, `camelcase`, `uppercamelcase`, `meow`.
|
||||||
|
|
||||||
|
## Layer 3 — Wayland Runtime
|
||||||
|
|
||||||
|
**Responsibility**: marshal and unmarshal the actual wire bytes. Implement
|
||||||
|
object lifecycle (creation, dispatch, destruction), new-id tracking, listener
|
||||||
|
dispatch, file descriptor handling.
|
||||||
|
|
||||||
|
**Browser side** (`libs/common/src/Connection.ts`) — the shared runtime for
|
||||||
|
both `@gfld/client-protocol` and `@gfld/compositor-protocol`:
|
||||||
|
|
||||||
|
- Wire format is **standard upstream Wayland**, byte-for-byte. 8-byte
|
||||||
|
header: `u32 object_id | u16 opcode | u16 size-incl-header`, then per-arg
|
||||||
|
bytes. `u/i/f/n/o` are 4 bytes; `s` and `a` are length-prefixed u32 +
|
||||||
|
payload padded to 4-byte alignment; `h` (FD) takes zero bytes — FDs travel
|
||||||
|
out-of-band in `wireMsg.fds[]`.
|
||||||
|
- Reading: `wlObject[opcode](message)` — the generated classes use
|
||||||
|
computed-property-key methods `[0](message) { ... }`, `[1](message) { ... }`
|
||||||
|
so opcode is the array index.
|
||||||
|
- Client side: `Proxy.marshallConstructor` calls `display.generateNextId()`
|
||||||
|
(1-indexed, recycles ids from `wl_display.delete_id`), constructs the new
|
||||||
|
Proxy (self-registers in constructor), patches the `new_id` arg's `value`,
|
||||||
|
then flushes.
|
||||||
|
- Server side: `Client.marshallConstructor` allocates from
|
||||||
|
`SERVER_OBJECT_ID_BASE = 0xff000000` (or recycles locally). When a
|
||||||
|
client-allocated id is destroyed, the server sends `wl_display.delete_id`
|
||||||
|
so the client can recycle.
|
||||||
|
- Connection registry: `wlObjects: { [id]: WlObject }`; `registerWlObject`
|
||||||
|
throws on collision.
|
||||||
|
- `Connection.message` dispatches by literal `wlObject[opcode](message)`.
|
||||||
|
Each handler returns void or a Promise; `Connection` awaits the promise
|
||||||
|
before reading the next message — this preserves wire-order while still
|
||||||
|
allowing async implementations. Per-client Connections mean a slow client
|
||||||
|
doesn't block others.
|
||||||
|
|
||||||
|
**Native side** (`packages/compositor-proxy/src/wayland-server.ts`) — a TS
|
||||||
|
wrapper around `wayland-server-addon.node` (the C N-API addon that wraps a
|
||||||
|
**patched** `libwayland-server`):
|
||||||
|
|
||||||
|
- `MessageInterceptor` — per-resource request/event dispatch. Each
|
||||||
|
`wl_*_interceptor.js` file declares `R<opcode>(...args)` and
|
||||||
|
`E<opcode>(...args)` functions that decide whether a message is routed
|
||||||
|
`{native: true}` (handle in the proxy), `{browser: true}` (forward to
|
||||||
|
browser compositor), or both.
|
||||||
|
- Special cases: `wl_display.sync` → both native and browser (must complete
|
||||||
|
before `done` event), `wl_display.get_registry` → both, `wl_registry.bind`
|
||||||
|
→ native if id is in `nativeGlobalNames[]` (e.g. `wl_drm`, `wl_shm`),
|
||||||
|
else browser. Fast-sync optimization skips browser round trip when no
|
||||||
|
prior browser-bound request occurred.
|
||||||
|
|
||||||
|
**File descriptor handling — the Greenfield innovation**:
|
||||||
|
|
||||||
|
`type FD = unknown` — it's intentionally opaque. Browser-side FDs are
|
||||||
|
**transferable JS objects**:
|
||||||
|
|
||||||
|
- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives
|
||||||
|
in browser, zero-copy transferred via `MessagePort.postMessage(...,
|
||||||
|
transferList)`).
|
||||||
|
- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg
|
||||||
|
(GPU-backed texture, zero-copy).
|
||||||
|
- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe).
|
||||||
|
|
||||||
|
Native-side (proxy) FDs are real POSIX ints.
|
||||||
|
`NativeWaylandClientSession.onWireMessageEnd` converts each to a JSON
|
||||||
|
`ProxyFD = {handle, type, host}` (host = proxy's HTTP base URL),
|
||||||
|
length-prefixed + 4-byte aligned, prepended to the wire message before
|
||||||
|
sending over WebSocket/KCP. Browser's `webFS` resolves them by fetching
|
||||||
|
data over HTTP.
|
||||||
|
|
||||||
|
**Dependencies**: Layer 2's generated bindings + `@gfld/common`.
|
||||||
|
|
||||||
|
## Layer 4 — Transport
|
||||||
|
|
||||||
|
**Responsibility**: carry wire bytes between the browser compositor and the
|
||||||
|
native proxy. Multiplex logical channels. Handle reconnection, backpressure,
|
||||||
|
and latency.
|
||||||
|
|
||||||
|
**Browser side** (`packages/compositor/src/remote/`):
|
||||||
|
|
||||||
|
- `Channel.ts` — defines `ARQChannel` (KCP-wrapped, reliable, low-latency)
|
||||||
|
and `SimpleChannel` (raw WebSocket).
|
||||||
|
- `ReconnectingWebSocket` — vendored from `pladaria/reconnecting-websocket`
|
||||||
|
with `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`, `minUptime=5s`,
|
||||||
|
`connTimeout=4s`, `retries=∞`.
|
||||||
|
- `RemoteAppLauncher.ts` — sets up the 4 logical channels per app client
|
||||||
|
(PROTOCOL, FRAME, XWM, FEEDBACK) and routes incoming messages.
|
||||||
|
- `RemoteInputOutput.ts` — implements the `InputOutput` interface for native
|
||||||
|
apps, with `mkfifo`, `mkstempMmap`, `readFd`, `writeFdAsStream`, `closeFd`
|
||||||
|
over separate WebSocket endpoints.
|
||||||
|
- `StreamingBuffer.ts`, `BufferStream.ts`, `EncodedFrame.ts`,
|
||||||
|
`EncoderFeedback.ts` — decode pipeline for incoming H.264 / PNG frames.
|
||||||
|
- `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`,
|
||||||
|
`H264NALDecoder.worker.ts` — the actual decoders.
|
||||||
|
|
||||||
|
**Native side** (`packages/compositor-proxy/src/`):
|
||||||
|
|
||||||
|
- `Channel.ts` — mirror image of the browser side. `ARQChannel` and
|
||||||
|
`SimpleChannel`.
|
||||||
|
- `kcp.ts` — ~1100-line TypeScript port of the canonical C `ikcp.c`.
|
||||||
|
- `SessionController.ts` — WS server with 9 endpoint paths:
|
||||||
|
`/signal`, `/channel`, `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
|
||||||
|
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
|
||||||
|
- `Session.ts`, `SessionProcess.ts`, `SessionController.ts` — process model:
|
||||||
|
one main process, one forked child per browser session.
|
||||||
|
- `NativeAppContext.ts` — per-app state, signaling WS, lifecycle (10-min
|
||||||
|
SIGHUP grace on disconnect, SIGTERM→SIGKILL escalation).
|
||||||
|
- `FrameFeedback.ts` — receives 4-byte feedback messages from the browser,
|
||||||
|
uses them to pace `wl_frame.done` callbacks.
|
||||||
|
- `SurfaceBufferEncoding.ts`, `ProxyBuffer.ts` — encoding pipeline.
|
||||||
|
- `io/` subdirectory — WebSocket server glue.
|
||||||
|
- `encoding/` subdirectory — high-level encoding orchestration.
|
||||||
|
|
||||||
|
**Wire format**: see `04-transport.md` for the full byte-level breakdown of
|
||||||
|
PROTOCOL, FRAME, XWM, and FEEDBACK channels.
|
||||||
|
|
||||||
|
**Dependencies**: Layer 3 + `ws` (Node) + browser-native WebSocket + `ReconnectingWebSocket`.
|
||||||
|
|
||||||
|
## Layer 5 — Rendering Pipeline
|
||||||
|
|
||||||
|
**Responsibility**: take incoming buffer frames (SHM or encoded), turn them
|
||||||
|
into pixels on a browser canvas/WebGL context. Manage surface state, damage
|
||||||
|
tracking, frame callbacks, and the compositing loop.
|
||||||
|
|
||||||
|
**Key abstractions**:
|
||||||
|
|
||||||
|
- `Surface.ts` — the `wl_surface` implementation. Double-buffered state
|
||||||
|
(`state` vs `pendingState`), atomic merge in `commitPending()`. Subsurface
|
||||||
|
sync mode cascades through the ancestor chain.
|
||||||
|
- `SurfaceChild.ts`, `SurfaceRole.ts`, `Subsurface.ts`, `Subcompositor.ts` —
|
||||||
|
surface tree / role system.
|
||||||
|
- `View.ts` — represents a surface's position on an output, used for
|
||||||
|
compositing.
|
||||||
|
- `Output.ts` — the `wl_output` implementation; how output is rendered to a
|
||||||
|
WebGL canvas.
|
||||||
|
- `Shm.ts` — `wl_shm` / shm pool handling. How shared memory buffers arrive
|
||||||
|
in the browser.
|
||||||
|
- `BufferContents.ts`, `BufferImplementation.ts`, `ImageBitmapBuffer.ts` —
|
||||||
|
buffer abstraction hierarchy.
|
||||||
|
- `Region.ts` — `wl_region`, damage regions (pixman-WASM backed).
|
||||||
|
- `Callback.ts` — frame callback mechanism.
|
||||||
|
- `Loop.ts` — the compositor main loop / `requestAnimationFrame` integration.
|
||||||
|
- `Compositor.ts` — the `wl_compositor` global; how it creates surfaces and
|
||||||
|
regions.
|
||||||
|
- `Session.ts` — top-level browser-side session.
|
||||||
|
- `Globals.ts` — how Wayland globals are registered.
|
||||||
|
- `render/` subdirectory — the WebGL renderer: `Renderer.ts`, `Scene.ts`,
|
||||||
|
`SceneShader.ts`, `Texture.ts`, `RenderState.ts`, `Program.ts`,
|
||||||
|
`ShaderCompiler.ts`, `ShaderSources.ts`, `YUVA2RGBA.ts`, `YUVA2RGBAShader.ts`,
|
||||||
|
`YUV2RGBShader.ts`, `RGBXandAToRGBA.ts`, `RGBandA2RGBAShader.ts`.
|
||||||
|
- `browser/` subdirectory — browser integration glue (canvas, RAF,
|
||||||
|
ResizeObserver, dnd, selection, input).
|
||||||
|
- `math/` subdirectory — matrix/region math (`Mat4.ts`, `Rect.ts`).
|
||||||
|
|
||||||
|
**Pipeline**: see `03-html5-rendering.md` for the full
|
||||||
|
commit → damage → render → present pipeline.
|
||||||
|
|
||||||
|
**Dependencies**: Layer 4 + WebGL + WebCodecs + `@gfld/compositor-wasm` (pixman + xkbcommon).
|
||||||
|
|
||||||
|
## Layer 6 — Surface Lifecycle / Roles
|
||||||
|
|
||||||
|
**Responsibility**: implement the role-specific surface state machines
|
||||||
|
(xdg-toplevel, xdg-popup, xdg-positioner, wl_shell_surface, subsurface).
|
||||||
|
Manage configure/ack-configure round-trips, popup grabs, window state.
|
||||||
|
|
||||||
|
**Files of interest**:
|
||||||
|
|
||||||
|
- `XdgWmBase.ts` — the `xdg_wm_base` global. How clients advertise they're
|
||||||
|
XDG clients. Includes ping-based unresponsiveness detection.
|
||||||
|
- `XdgSurface.ts` — `xdg_surface`. The role wrapper around a `wl_surface`.
|
||||||
|
- `XdgToplevel.ts` — `xdg_toplevel`. Top-level windows. State (maximized,
|
||||||
|
fullscreen, minimized, activated, resizing). Configure/ack-configure
|
||||||
|
round-trip.
|
||||||
|
- `XdgPopup.ts` — `xdg_popup`. Transient windows, menus, tooltips. Grab
|
||||||
|
semantics.
|
||||||
|
- `XdgPositioner.ts` — `xdg_positioner`. How popups are positioned relative
|
||||||
|
to parent.
|
||||||
|
- `Shell.ts`, `ShellSurface.ts` — the legacy `wl_shell` (deprecated but still
|
||||||
|
implemented for old clients).
|
||||||
|
- `Subcompositor.ts`, `Subsurface.ts`, `SurfaceChild.ts`, `SurfaceRole.ts`
|
||||||
|
— subsurface compositing, the surface tree.
|
||||||
|
|
||||||
|
**State machines**: see `08-app-lifecycle.md` for the full
|
||||||
|
XdgToplevel lifecycle, configure/ack round-trip semantics, popup
|
||||||
|
positioner math, and subsurface sync/desync mode.
|
||||||
|
|
||||||
|
**Dependencies**: Layer 5 + Layer 3 (Wayland runtime).
|
||||||
|
|
||||||
|
## Layer 7 — Desktop / Layout Policy
|
||||||
|
|
||||||
|
**Responsibility**: arrange surfaces on the output. Decide stacking order,
|
||||||
|
focus policy, and the high-level window management mode (Floating vs
|
||||||
|
AlwaysFullScreen).
|
||||||
|
|
||||||
|
**Files of interest**:
|
||||||
|
|
||||||
|
- `desktop/Desktop.ts` — implements both `FloatingDesktopSurface` and
|
||||||
|
`AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`.
|
||||||
|
Floating lets the user move/resize windows freely; AlwaysFullScreen
|
||||||
|
enforces one window per output (kiosk mode).
|
||||||
|
|
||||||
|
**Dependencies**: Layer 6 + Layer 5 (for View rendering).
|
||||||
|
|
||||||
|
## Layer 8 — User Shell
|
||||||
|
|
||||||
|
**Responsibility**: the actual user-facing UI. The taskbar, window
|
||||||
|
decorations, app launcher, keyboard layout picker, etc.
|
||||||
|
|
||||||
|
**Out of scope for the user** — they will build their own shell as an Odoo 20
|
||||||
|
OWL addon. The integration point is `UserShellApi.ts` (see
|
||||||
|
`08-app-lifecycle.md`).
|
||||||
|
|
||||||
|
**Greenfield's reference shell** (`packages/compositor-shell`) is a Preact +
|
||||||
|
Tailwind SPA, 562 lines of TypeScript total. Mentioned here only for
|
||||||
|
orientation; not analyzed in this bundle.
|
||||||
|
|
||||||
|
## How data flows across the layers
|
||||||
|
|
||||||
|
```
|
||||||
|
NATIVE APP (e.g. gtk4-demo, X11 via XWayland)
|
||||||
|
│
|
||||||
|
│ libwayland-client → UNIX socket + SCM_RIGHTS FDs
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ NATIVE PROXY (Node.js + C N-API addons) │
|
||||||
|
│ │
|
||||||
|
│ Layer 3 (native): wayland-server.ts + wayland-server-addon.node │
|
||||||
|
│ ↳ patched libwayland-server parses wire bytes │
|
||||||
|
│ ↳ MessageInterceptor routes each request to {native, browser} │
|
||||||
|
│ ↳ FDs wrapped as ProxyFD JSON │
|
||||||
|
│ │
|
||||||
|
│ Layer 4 (native): Channel.ts + kcp.ts + SessionController │
|
||||||
|
│ ↳ 4 logical channels: PROTOCOL, FRAME, XWM, FEEDBACK │
|
||||||
|
│ ↳ KCP-over-WebSocket for ARQ channels │
|
||||||
|
│ ↳ SimpleChannel for FEEDBACK │
|
||||||
|
│ │
|
||||||
|
│ Layer 5 (encoding): SurfaceBufferEncoding + proxy-encoding-addon │
|
||||||
|
│ ↳ GStreamer H.264 (x264 / nvh264 / vaapih264) on dedicated thread │
|
||||||
|
│ ↳ PNG fallback for ≤256×256 surfaces │
|
||||||
|
│ ↳ Zero-copy dmabuf via EGLImage → GstGLMemoryEGL │
|
||||||
|
│ │
|
||||||
|
│ Layer 1 (XML) → Layer 2 (proxy-generator) feeds Layer 3 interceptors │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
│ WebSocket (TCP) + KCP-over-WebSocket
|
||||||
|
│ 9 endpoint paths per session
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ BROWSER COMPOSITOR (TypeScript + WebGL + WebCodecs + WASM) │
|
||||||
|
│ │
|
||||||
|
│ Layer 4 (browser): Channel.ts + RemoteAppLauncher + decoders │
|
||||||
|
│ ↳ ReconnectingWebSocket with exponential backoff │
|
||||||
|
│ ↳ ARQChannel + SimpleChannel mirror the native side │
|
||||||
|
│ ↳ WebCodecs VideoDecoder preferred, FFmpeg WASM fallback │
|
||||||
|
│ ↳ H264NALDecoder.worker.ts (2 workers: opaque + alpha) │
|
||||||
|
│ │
|
||||||
|
│ Layer 3 (browser): @gfld/common Connection + compositor-protocol │
|
||||||
|
│ ↳ Standard Wayland wire format, byte-for-byte │
|
||||||
|
│ ↳ WlObject registry, opcode dispatch via wlObject[opcode](msg) │
|
||||||
|
│ ↳ Per-client Connections (slow client never blocks another) │
|
||||||
|
│ │
|
||||||
|
│ Layer 5: Surface.ts + View.ts + Output.ts + render/ │
|
||||||
|
│ ↳ Double-buffered state (state vs pendingState) │
|
||||||
|
│ ↳ Damage tracking via pixman-WASM regions │
|
||||||
|
│ ↳ WebGL1 with FBOs for YUV→RGBA, RGB+Alpha→RGBA conversion │
|
||||||
|
│ ↳ desynchronized:true canvas for low-latency presentation │
|
||||||
|
│ ↳ Frame callbacks fired on requestAnimationFrame vsync │
|
||||||
|
│ │
|
||||||
|
│ Layer 6: XdgWmBase + XdgSurface + XdgToplevel + XdgPopup + ... │
|
||||||
|
│ ↳ Configure/ack-configure round-trips │
|
||||||
|
│ ↳ Popup positioner math (anchor + gravity + constraints) │
|
||||||
|
│ ↳ Subsurface sync/desync mode cascading │
|
||||||
|
│ │
|
||||||
|
│ Layer 7: desktop/Desktop.ts │
|
||||||
|
│ ↳ Floating vs AlwaysFullScreen mode │
|
||||||
|
│ ↳ View stacking, focus refcount │
|
||||||
|
│ │
|
||||||
|
│ Layer 8: UserShellApi (events + actions) │
|
||||||
|
│ ↳ Events: clientCreated, surfaceCreated, surfaceTitleUpdated, │
|
||||||
|
│ surfaceActivationUpdated, ... │
|
||||||
|
│ ↳ Actions: initScene, refreshScene, destroyScene, │
|
||||||
|
│ closeClient, activateSurface, setUserConfiguration │
|
||||||
|
│ ↳ THE INTEGRATION SEAM — Odoo 20 OWL shell hooks here │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
│ UserShellApi.events.* fired by compositor
|
||||||
|
│ UserShellApi.actions.* called by shell
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ USER SHELL (Odoo 20 OWL addon — supplied by user) │
|
||||||
|
│ │
|
||||||
|
│ - Application launcher (lists apps from --applications JSON) │
|
||||||
|
│ - Window management UI (taskbar, decorations, minimize/maximize) │
|
||||||
|
│ - Keyboard layout picker (RMLVO) │
|
||||||
|
│ - Per-output canvas (HTMLCanvasElement, one per Output) │
|
||||||
|
│ - Trigger app launches via fetch('http://proxy:8081/<path>') │
|
||||||
|
│ - Open signalURL WebSocket to receive CONNECT_CHANNEL messages │
|
||||||
|
│ │
|
||||||
|
│ Not analyzed in this bundle (out of scope). │
|
||||||
|
└─────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reading guide for the rest of the bundle
|
||||||
|
|
||||||
|
Now that you've seen the layer map, the remaining chapters drill into each
|
||||||
|
layer in detail:
|
||||||
|
|
||||||
|
- `02-wayland-protocol.md` — Layers 1, 2, 3 (XML → generators → runtime)
|
||||||
|
- `03-html5-rendering.md` — Layer 5 (rendering pipeline)
|
||||||
|
- `04-transport.md` — Layer 4 (transport, both sides)
|
||||||
|
- `05-input-handling.md` — Layer 5 input subsystem (cross-cuts Layers 3, 4, 5)
|
||||||
|
- `06-data-clipboard.md` — Layer 5 data device (cross-cuts Layers 3, 4, 5)
|
||||||
|
- `07-native-app-bridge.md` — Layer 4 native + Layer 3 native + Layer 1 XML native (proxy-only `wl_drm`)
|
||||||
|
- `08-app-lifecycle.md` — Layer 6 + Layer 8 (UserShellApi seam)
|
||||||
|
- `09-hardware-accel.md` — Layer 5 encoding + Layer 4 frame channel + Layer 3 WASM
|
||||||
|
- `10-porting-to-odoo-js-python.md` — how to rebuild all 8 layers in your stack
|
||||||
|
- `11-pitfalls.md` — what Greenfield does wrong, layer by layer
|
||||||
|
- `12-appendix.md` — glossary, file index, external references
|
||||||
@@ -0,0 +1,325 @@
|
|||||||
|
# 02 — Wayland Protocol Layer
|
||||||
|
|
||||||
|
This chapter covers Greenfield's protocol layer in depth: how Wayland XML
|
||||||
|
protocol definitions get parsed, how typed TypeScript stubs get generated for
|
||||||
|
each of the three sides (browser client, browser compositor, native proxy),
|
||||||
|
how wire messages are marshalled/unmarshalled at runtime, how objects
|
||||||
|
lifecycle is managed, and how the FD-over-WebSocket problem is solved.
|
||||||
|
|
||||||
|
## The three generators
|
||||||
|
|
||||||
|
Greenfield ships **three sibling generator CLIs** that consume the same
|
||||||
|
Wayland XML files but emit slightly different TypeScript flavors:
|
||||||
|
|
||||||
|
| | `client-generator` | `compositor-generator` | `compositor-proxy-generator` |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Output | TS `*Proxy` classes (browser) | TS `*Resource` + `*Requests` (Node) | CommonJS `*_interface.js` + `*_interceptor.js` (Node + N-API) |
|
||||||
|
| Listener pattern | `listener?: <Itf>Events` (optional, duck-typed, async) | `implementation: <Itf>Requests` (required) | `requestHandlers` / `eventHandlers` maps |
|
||||||
|
| Skips | none | `wl_display`, `wl_registry` (hand-written in runtime) | none — proxy must intercept display/registry/bind |
|
||||||
|
| Arg-type table | `ProtocolArguments.mjs` returns `{signature, jsType, marshallGen}` | Same shape, uses `Westfield.${resourceName}` for objects | `EndpointProtocolArguments.mjs` returns plain wayland sig letters (`u`, `?s`, `n`, `h`, ...) |
|
||||||
|
| XMLs consumed | `wayland.xml` + `xdg-shell.xml` + `web-bitmapbuf-unstable-v1.xml` | same 3 | `wayland.xml` + `xdg-shell.xml` + `wayland-drm.xml` |
|
||||||
|
|
||||||
|
All three share `xml2js` + `camelcase` + `uppercamelcase` and a near-identical
|
||||||
|
`ProtocolParser.mjs`. The XML → JSON → walk `interface[]`/`request[]`/
|
||||||
|
`event[]`/`enum[]` → per-arg type-dispatch pipeline is identical.
|
||||||
|
|
||||||
|
### Why three generators?
|
||||||
|
|
||||||
|
Because each side has a fundamentally different job:
|
||||||
|
|
||||||
|
1. **`client-generator`** → browser apps that want to be Wayland clients of
|
||||||
|
the browser compositor (via `MessagePort`). These are the same-origin
|
||||||
|
iframe JS/WASM apps that ship in `examples/webapps/`.
|
||||||
|
2. **`compositor-generator`** → the browser-resident compositor itself
|
||||||
|
(`@gfld/compositor`). It receives requests, dispatches them to
|
||||||
|
`*Requests` implementation classes, and emits events back.
|
||||||
|
3. **`compositor-proxy-generator`** → the Node.js native proxy bridging real
|
||||||
|
`libwayland-client` apps to the browser. Emits `wl_interface` struct
|
||||||
|
definitions for the N-API addon + per-opcode interceptors that decide
|
||||||
|
routing.
|
||||||
|
|
||||||
|
## Wire format (`libs/common/src/Connection.ts`)
|
||||||
|
|
||||||
|
**Standard upstream Wayland wire format, byte-for-byte.** No custom binary
|
||||||
|
encoding at the protocol layer.
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────┬──────────┬──────────┬─────────────────────────────────────────────┐
|
||||||
|
│ object id │ opcode │ size │ args (variable per arg type, see §4) │
|
||||||
|
│ uint32 │ uint16 │ uint16 │ u,i,f,n,o = 4 bytes; s,a = 4+len+pad; │
|
||||||
|
│ (4 bytes) │ (2 B) │ (2 B) │ h = 0 bytes, FD goes in side-band fds[] │
|
||||||
|
└─────────────┴──────────┴──────────┴─────────────────────────────────────────────┘
|
||||||
|
▲
|
||||||
|
└── size includes this 8-byte header.
|
||||||
|
Total message size = size field.
|
||||||
|
```
|
||||||
|
|
||||||
|
### Per-arg marshalling table
|
||||||
|
|
||||||
|
| Arg signature letter | Type | Size on wire | Notes |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `u` | uint32 | 4 bytes | unsigned integer |
|
||||||
|
| `i` | int32 | 4 bytes | signed integer |
|
||||||
|
| `f` | fixed | 4 bytes | wl_fixed_t (24.8 signed) |
|
||||||
|
| `s` | string | 4 + len + pad | length-prefixed u32, padded to 4-byte alignment |
|
||||||
|
| `a` | array | 4 + len + pad | length-prefixed u32, padded to 4-byte alignment |
|
||||||
|
| `o` | object | 4 bytes | object id (0 means null) |
|
||||||
|
| `n` | new_id | 4 bytes | newly-allocated object id |
|
||||||
|
| `h` | file descriptor | 0 bytes | FD goes in side-band `wireMsg.fds[]` |
|
||||||
|
|
||||||
|
### Reading
|
||||||
|
|
||||||
|
`Connection.message` dispatches by literal `wlObject[opcode](message)`. The
|
||||||
|
generated classes use computed-property-key methods:
|
||||||
|
`[0](message) { ... }`, `[1](message) { ... }` so opcode is the array index.
|
||||||
|
Each handler returns void or a Promise; `Connection` awaits the promise
|
||||||
|
before reading the next message — this preserves wire-order while still
|
||||||
|
allowing async implementations. Per-client Connections mean a slow client
|
||||||
|
doesn't block others.
|
||||||
|
|
||||||
|
## Object lifecycle & new-id tracking
|
||||||
|
|
||||||
|
### Client side
|
||||||
|
|
||||||
|
`Proxy.marshallConstructor` calls `display.generateNextId()` (1-indexed,
|
||||||
|
recycles ids from `wl_display.delete_id`), constructs the new Proxy
|
||||||
|
(self-registers in constructor), patches the `new_id` arg's `value` (it was
|
||||||
|
`0` placeholder from `newObject()`), then flushes.
|
||||||
|
|
||||||
|
### Server side
|
||||||
|
|
||||||
|
`Client.marshallConstructor` allocates from `SERVER_OBJECT_ID_BASE =
|
||||||
|
0xff000000` (or recycles locally). When a client-allocated id is destroyed,
|
||||||
|
the server sends `wl_display.delete_id` so the client can recycle. On
|
||||||
|
`Client.close()`, objects are destroyed in **descending id order** (children
|
||||||
|
first).
|
||||||
|
|
||||||
|
### Connection registry
|
||||||
|
|
||||||
|
`wlObjects: { [id]: WlObject }`; `registerWlObject` throws on collision.
|
||||||
|
|
||||||
|
## Listener dispatch
|
||||||
|
|
||||||
|
`Connection.message` dispatches by literal `wlObject[opcode](message)`. Each
|
||||||
|
handler returns void or a Promise; `Connection` awaits the promise before
|
||||||
|
reading the next message — this preserves wire-order while still allowing
|
||||||
|
async implementations. Per-client Connections mean a slow client doesn't
|
||||||
|
block others.
|
||||||
|
|
||||||
|
## File descriptor handling — the Greenfield innovation
|
||||||
|
|
||||||
|
`type FD = unknown` — it's intentionally opaque. The actual FD
|
||||||
|
representation depends on which link it's traversing.
|
||||||
|
|
||||||
|
### Browser ↔ Browser (in-iframe, via `MessagePort`)
|
||||||
|
|
||||||
|
FDs are transferable JS objects:
|
||||||
|
|
||||||
|
- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives
|
||||||
|
in browser, zero-copy transferred via `MessagePort.postMessage(...,
|
||||||
|
transferList)`).
|
||||||
|
- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg
|
||||||
|
(GPU-backed texture, zero-copy).
|
||||||
|
- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe).
|
||||||
|
|
||||||
|
### Browser ↔ Proxy (over WebSocket/KCP)
|
||||||
|
|
||||||
|
FDs are virtualized as JSON `ProxyFD` descriptors:
|
||||||
|
|
||||||
|
```
|
||||||
|
ProxyFD = { handle: number, type: 'pipe-read' | 'pipe-write' | 'shm' | 'unknown', host: string }
|
||||||
|
```
|
||||||
|
|
||||||
|
Where:
|
||||||
|
- `handle` is the proxy's internal numeric fd id
|
||||||
|
- `type` is the kind of fd (so the browser knows what to do with it)
|
||||||
|
- `host` is the proxy's HTTP base URL (for proxy-to-proxy FD relay)
|
||||||
|
|
||||||
|
`NativeWaylandClientSession.onWireMessageEnd` converts each POSIX fd to a
|
||||||
|
ProxyFD JSON, length-prefixed + 4-byte aligned, prepended to the wire
|
||||||
|
message before sending over WebSocket/KCP. The browser's `webFS` resolves
|
||||||
|
them by fetching data over a separate ad-hoc WebSocket endpoint
|
||||||
|
(`/read-fd`, `/write-fd-as-stream`, etc.).
|
||||||
|
|
||||||
|
### Native App ↔ Proxy (libwayland socket)
|
||||||
|
|
||||||
|
FDs are real POSIX ints, sent via `SCM_RIGHTS` by libwayland-server.
|
||||||
|
|
||||||
|
### Most wl_buffer FDs never need to be read
|
||||||
|
|
||||||
|
Because the proxy intercepts `wl_surface.commit`, reads the buffer locally
|
||||||
|
via DRM/SHM, encodes to H.264, and ships only the encoded bytes — the
|
||||||
|
browser never needs to read the original FD.
|
||||||
|
|
||||||
|
## How the proxy translates between native and browser
|
||||||
|
|
||||||
|
**It doesn't translate wire bytes** — both sides speak identical Wayland
|
||||||
|
wire format. What it translates is:
|
||||||
|
|
||||||
|
- **FDs**: POSIX int ↔ `ProxyFD` JSON.
|
||||||
|
- **Object-id space**: native libwayland's id range vs browser's
|
||||||
|
`0xff000000`+ server range.
|
||||||
|
- **Routing per message** via `MessageInterceptor.interceptRequest` /
|
||||||
|
`interceptEvent`. Special cases:
|
||||||
|
- `wl_display.sync` → both native and browser (must complete before `done`
|
||||||
|
event)
|
||||||
|
- `wl_display.get_registry` → both
|
||||||
|
- `wl_registry.bind` → native if id is in `nativeGlobalNames[]` (e.g.
|
||||||
|
`wl_drm`, `wl_shm`), else browser
|
||||||
|
- Fast-sync optimization skips browser round trip when no prior
|
||||||
|
browser-bound request occurred
|
||||||
|
|
||||||
|
## The four protocol XMLs
|
||||||
|
|
||||||
|
### `protocol/wayland.xml`
|
||||||
|
|
||||||
|
Standard upstream Wayland core. 22 interfaces, `wl_compositor` v4, `wl_seat`
|
||||||
|
v6, `wl_surface` v4 (roughly Wayland 1.20-ish). Kristian Høgsberg copyright.
|
||||||
|
Used by all 3 generators.
|
||||||
|
|
||||||
|
### `protocol/xdg-shell.xml`
|
||||||
|
|
||||||
|
Stable `xdg_shell` v1, 5 interfaces: `xdg_wm_base`, `xdg_surface`,
|
||||||
|
`xdg_toplevel`, `xdg_popup`, `xdg_positioner`. Used by all 3 generators.
|
||||||
|
|
||||||
|
### `protocol/wayland-drm.xml`
|
||||||
|
|
||||||
|
Legacy `wl_drm` v2, used by native apps (Mesa, XWayland) to advertise dmabuf
|
||||||
|
buffers. **Only the proxy generator processes this file** — so the proxy can
|
||||||
|
intercept `create_buffer` / `create_planar_buffer` / `create_prime_buffer`
|
||||||
|
and turn dmabuf fds into ProxyFDs for the browser's GStreamer encoding
|
||||||
|
pipeline. Browser-side libs do not generate `wl_drm` stubs.
|
||||||
|
|
||||||
|
### `protocol/web-bitmapbuf-unstable-v1.xml` — Greenfield's custom extension
|
||||||
|
|
||||||
|
This is **Greenfield's own Wayland protocol extension** for in-browser
|
||||||
|
JS/WASM clients. It defines a single singleton interface:
|
||||||
|
|
||||||
|
- **`web_bitmapbuf_factory` (v1)** with one request
|
||||||
|
`create_buffer(new_id wl_buffer id, fd bitmap)` — "Create a wl_buffer
|
||||||
|
object by wrapping an HTML bitmap, so it can be used with a surface."
|
||||||
|
|
||||||
|
The `fd` argument is an HTML bitmap file descriptor (a Greenfield-specific
|
||||||
|
construct: an `ImageBitmap`-like object exposed as an FD-like handle). This
|
||||||
|
lets JavaScript or WebAssembly Wayland clients back a surface with a native
|
||||||
|
browser bitmap (`ImageBitmap`, `OffscreenCanvas` transferable) and zero-copy
|
||||||
|
it into the compositor's WebGL texture pipeline — bypassing the SHM
|
||||||
|
pixel-copy path.
|
||||||
|
|
||||||
|
The browser-side factory lives in
|
||||||
|
`packages/compositor/src/web/WebBuffersFactory.ts`. **Not processed by the
|
||||||
|
proxy generator** — native apps never use it.
|
||||||
|
|
||||||
|
This is a trade-off worth noting: the fast path is Greenfield-specific. To
|
||||||
|
get zero-copy `ImageBitmap` upload to a GPU texture, apps must be
|
||||||
|
Greenfield-aware (i.e. built against `@gfld/client-protocol`). Standard
|
||||||
|
Wayland apps fall back to the H.264-encoded path even when running in the
|
||||||
|
browser compositor.
|
||||||
|
|
||||||
|
## ASCII pipeline diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ PROTOCOL XML SOURCES │
|
||||||
|
│ wayland.xml xdg-shell.xml wayland-drm.xml web-bitmapbuf-unstable-v1 │
|
||||||
|
└──────────┬──────────────┬──────────────┬─────────────────┬─────────────────────┘
|
||||||
|
│ │ │ │
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
┌────────────────┐ ┌────────────────┐ ┌───────────────────┐ ┌────────────────────┐
|
||||||
|
│ client- │ │ compositor- │ │ compositor-proxy- │ │ (no proxy gen for │
|
||||||
|
│ generator │ │ generator │ │ generator │ │ web-bitmapbuf) │
|
||||||
|
│ ProtocolParser │ │ ProtocolParser │ │ EndpointProtocol │ │ │
|
||||||
|
│ .mjs │ │ .mjs │ │ Parser.mjs │ │ │
|
||||||
|
└───────┬────────┘ └───────┬────────┘ └─────────┬─────────┘ └─────────┬──────────┘
|
||||||
|
│ xml2js │ xml2js │ xml2js │
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
┌────────────────┐ ┌────────────────┐ ┌─────────────────────┐ ┌────────────────────┐
|
||||||
|
│ wayland.ts │ │ wayland.ts │ │ *_interface.js │ │ wayland.ts │
|
||||||
|
│ xdg_shell.ts │ │ xdg_shell.ts │ │ *_interceptor.js │ │ xdg_shell.ts │
|
||||||
|
│ web_bitmapbuf.ts│ │ web_bitmapbuf.ts│ │ (wayland,xdg,drm) │ │ web_bitmapbuf.ts │
|
||||||
|
│ *Proxy classes │ │ *Resource + │ │ wl_interface structs│ │ *Proxy classes │
|
||||||
|
│ *Events ifaces │ │ *Requests │ │ + R*/E* handlers │ │ *Events interfaces │
|
||||||
|
│ *ProtocolName │ │ *ProtocolName │ │ + requestHandlers │ │ *ProtocolName │
|
||||||
|
│ *Enum │ │ *Enum │ │ /eventHandlers │ │ *Enum │
|
||||||
|
└───────┬─────────┘ └───────┬─────────┘ └──────────┬──────────┘ └─────────┬──────────┘
|
||||||
|
│ imports │ imports │ require()'d │ imports
|
||||||
|
▼ ▼ ▼ ▼
|
||||||
|
┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────────┐
|
||||||
|
│ @gfld/common │ │ packages/compositor-proxy/ │
|
||||||
|
│ Connection.marshallMsg() ── writes │ │ src/wayland-server.ts │
|
||||||
|
│ u32 id | u16 opcode | u16 size | args │ │ ↳ re-exports wayland-server-addon │
|
||||||
|
│ Connection.message() ── parses header, │ │ ↳ MessageInterceptor (R*/E* dispatch) │
|
||||||
|
│ dispatches wlObject[opcode](message) │ │ ↳ unmarshallArgs(message, 'usun') etc │
|
||||||
|
│ Per-arg marshallers: uint/int/fixed/ │ │ NativeWaylandClientSession.ts │
|
||||||
|
│ string/array/object/objectOptional/ │ │ ↳ onWireMessageRequest → interceptRequest │
|
||||||
|
│ newObject/fileDescriptor │ │ ↳ onWireMessageEnd → pack ProxyFD JSON │
|
||||||
|
│ Per-arg unmarshallers: u/i/f/s/a/o/n/h │ │ ↳ onWireMessageEvents → sendEvents (C) │
|
||||||
|
└──────────────┬───────────────────────────┘ └─────────────────────┬────────────────────────┘
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌────────────────────────────────────────┐ ┌────────────────────────────────────────────────────┐
|
||||||
|
│ Browser (client iframe) │ │ Node (compositor-proxy) │
|
||||||
|
│ │ │ │
|
||||||
|
│ DisplayImpl.messagePort.onmessage ←───┼─────────┼─ protocolChannel (WS/KCP) sends/receives │
|
||||||
|
│ connection.message({buffer, fds}) │ │ [u32 oob-op][u32 lastSerial][u32 fdCount] ... │
|
||||||
|
│ wlObject[opcode](message) │ │ [u32 len][ProxyFD JSON, padded][wayland msg bytes]│
|
||||||
|
│ listener?.<eventName>(...unmarshalled) │ │ │
|
||||||
|
│ │ │ ← from native: wl_client wire callback (C addon) │
|
||||||
|
│ proxy.marshall(id, opcode, [args]) │ │ → to native: sendEvents(wlClient, buf, fds) (C) │
|
||||||
|
│ connection.marshallMsg writes Uint32 │ │ │
|
||||||
|
│ onFlush → postMessage(buf, transfer) │ │ Native libwayland-server │
|
||||||
|
│ ↳ buffer + ArrayBufferView/ImageBitmap│ │ accepts socket from native wayland apps │
|
||||||
|
│ /MessagePort as transferables │ │ parses wire bytes, dispatches to interceptors │
|
||||||
|
└────────────────────────────────────────┘ └────────────────────────────────────────────────────┘
|
||||||
|
|
||||||
|
WIRE FORMAT (identical on all links — standard Wayland):
|
||||||
|
┌─────────────┬──────────┬──────────┬─────────────────────────────────────────────┐
|
||||||
|
│ object id │ opcode │ size │ args (variable per arg type, see §4) │
|
||||||
|
│ uint32 │ uint16 │ uint16 │ u,i,f,n,o = 4 bytes; s,a = 4+len+pad; │
|
||||||
|
│ (4 bytes) │ (2 B) │ (2 B) │ h = 0 bytes, FD goes in side-band fds[] │
|
||||||
|
└─────────────┴──────────┴──────────┴─────────────────────────────────────────────┘
|
||||||
|
▲
|
||||||
|
└── size includes this 8-byte header. Total message size = size field.
|
||||||
|
|
||||||
|
FD transport varies per link:
|
||||||
|
Browser↔Browser (MessagePort): FD is a transferable JS object (ArrayBufferView, ImageBitmap, MessagePort).
|
||||||
|
Browser↔Proxy (WebSocket/KCP): FD is a JSON ProxyFD {handle, type, host}, length-prefixed + 4-byte aligned, prepended before the message header.
|
||||||
|
Native App↔Proxy (libwayland socket): FD is a real POSIX fd, sent via SCM_RIGHTS by libwayland-server.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Build-time workflow
|
||||||
|
|
||||||
|
If you edit a protocol XML in your fork, you regenerate stubs by running
|
||||||
|
the matching `generate` script:
|
||||||
|
|
||||||
|
- `yarn workspace @gfld/client-protocol generate` — regenerates browser
|
||||||
|
client stubs.
|
||||||
|
- `yarn workspace @gfld/compositor-protocol generate` — regenerates browser
|
||||||
|
compositor stubs.
|
||||||
|
- `yarn workspace @gfld/compositor-proxy generate` — regenerates proxy
|
||||||
|
interceptor modules, then `build:native` (because `*_interface.js`
|
||||||
|
declares `wl_interface` structs the N-API addon consumes).
|
||||||
|
|
||||||
|
To add a new Greenfield-only extension: drop XML in `/protocol/`, add a
|
||||||
|
`generate:<name>` script to `client-protocol` / `compositor-protocol`
|
||||||
|
`package.json`, regenerate. Only add to `compositor-proxy` if native apps
|
||||||
|
will speak it.
|
||||||
|
|
||||||
|
## Notable file references
|
||||||
|
|
||||||
|
- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
|
||||||
|
- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
|
||||||
|
- `libs/compositor-proxy-generator/src/{EndpointProtocolParser, EndpointProtocolArguments, index}.mjs`
|
||||||
|
- `libs/common/src/Connection.ts` — the runtime wire codec (shared)
|
||||||
|
- `libs/compositor-protocol/src/westfield-runtime-server.ts` — server-side runtime
|
||||||
|
- `libs/client-protocol/src/westfield-runtime-client.ts` — client-side runtime
|
||||||
|
- `packages/compositor-proxy/src/wayland-server.ts` — proxy's TS wrapper around the N-API addon
|
||||||
|
- `packages/compositor-proxy/src/NativeWaylandClientSession.ts` — per-client message interceptor dispatch
|
||||||
|
- `packages/compositor-proxy/src/protocol/wl_*_interceptor.js` — generated per-opcode routing decisions
|
||||||
|
- `packages/compositor/src/web/WebBuffersFactory.ts` — browser-side implementation of the custom `web_bitmapbuf_factory` global
|
||||||
|
|
||||||
|
## One stale file spotted
|
||||||
|
|
||||||
|
`libs/compositor-protocol/index.js` (CommonJS, references non-existent
|
||||||
|
`./src/Resource` / `./src/Client` / `./src/Display` / `./src/Global`). The
|
||||||
|
actual entry point is `src/index.ts`. Safe to delete in a fork.
|
||||||
@@ -0,0 +1,379 @@
|
|||||||
|
# 03 — HTML5 Rendering Pipeline
|
||||||
|
|
||||||
|
This chapter covers Greenfield's browser-side rendering pipeline: how
|
||||||
|
incoming Wayland buffers (SHM or encoded) end up as pixels on a browser
|
||||||
|
canvas/WebGL context, including damage tracking, commit semantics, surface
|
||||||
|
state stacking, and the compositing loop. The browser side lives in
|
||||||
|
`packages/compositor/src/`.
|
||||||
|
|
||||||
|
## Three buffer types, one BufferImplementation interface
|
||||||
|
|
||||||
|
All incoming buffers, regardless of origin, end up as a `BufferContents<T>`
|
||||||
|
with a `mimeType` that dispatches to the right upload path. There are three
|
||||||
|
concrete buffer types:
|
||||||
|
|
||||||
|
### 1. SHM buffer (`LazyImageBitmapBuffer`)
|
||||||
|
|
||||||
|
Comes from `wl_shm_pool` — the standard Wayland shared-memory buffer
|
||||||
|
mechanism. Browser-side implementation lives in `Shm.ts`. The raw pixel data
|
||||||
|
arrives as a transferable `ArrayBufferView` (transferred zero-copy via
|
||||||
|
`MessagePort` from a browser-iframe web app, or fetched via the FD endpoint
|
||||||
|
WebSocket from the proxy). Greenfield wraps the bytes as an `ImageData` /
|
||||||
|
BMP-Blob and decodes them via `createImageBitmap()` to produce an
|
||||||
|
`ImageBitmap` ready for WebGL `texImage2D`.
|
||||||
|
|
||||||
|
**Notable hack**: for `xrgb8888` SHM buffers, Greenfield prepends a BMP
|
||||||
|
header in front of the raw pixel data so the browser's `createImageBitmap`
|
||||||
|
decodes it natively, avoiding a per-pixel JS loop. This is a clever trick to
|
||||||
|
offload pixel conversion to the browser's BMP decoder.
|
||||||
|
|
||||||
|
### 2. Web bitmap buffer (`ImageBitmapBuffer` via `web_bitmapbuf_factory`)
|
||||||
|
|
||||||
|
Comes from Greenfield's custom `web-bitmapbuf-unstable-v1` protocol
|
||||||
|
extension. Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to
|
||||||
|
the compositor as a `wl_buffer` — bypassing SHM copy, encoding, and dmabuf
|
||||||
|
import. The `fd` arg is marshalled as a transferable, so the GPU texture
|
||||||
|
never leaves VRAM.
|
||||||
|
|
||||||
|
### 3. Remote streaming buffer (`StreamingBuffer`)
|
||||||
|
|
||||||
|
Comes from the native proxy via the FRAME data channel. Receives encoded
|
||||||
|
frames (H.264 or PNG) on a separate WebSocket per surface. Includes an
|
||||||
|
out-of-band mechanism for `BufferCreation` and `CommitSerial` opcodes so
|
||||||
|
the browser can match each encoded frame to the right `wl_surface.commit`
|
||||||
|
serial.
|
||||||
|
|
||||||
|
## Surface state stack
|
||||||
|
|
||||||
|
`Surface.ts` implements classic Wayland double-buffered state:
|
||||||
|
|
||||||
|
- `Surface.state` — the committed, currently-visible state.
|
||||||
|
- `Surface.pendingState` — the in-flight state, accumulated by client
|
||||||
|
requests until `commit` is called.
|
||||||
|
- `mergeSurfaceState` — the atomic swap in `commitPending()`.
|
||||||
|
|
||||||
|
Subsurfaces add a third layer of complexity:
|
||||||
|
|
||||||
|
- `subsurfaceChildren` arrays (committed + pending, bottom→top)
|
||||||
|
- `placeAbove` / `placeBelow` reorder pending
|
||||||
|
- `effectiveSync` getter cascades through the ancestor chain (if any
|
||||||
|
ancestor is sync, this subsurface is effectively sync)
|
||||||
|
- Sync subsurfaces cache to `cachedState` and only flush when an ancestor
|
||||||
|
commits via `onParentCommit`
|
||||||
|
- `setDesync` immediately calls `onCommit` to flush any pending cache
|
||||||
|
|
||||||
|
## Commit semantics
|
||||||
|
|
||||||
|
When a Wayland client calls `wl_surface.commit`:
|
||||||
|
|
||||||
|
1. If the surface has a remote streaming buffer, Greenfield pulls the
|
||||||
|
`serial = commitSerials.shift()` (an out-of-band opcode 7 mechanism that
|
||||||
|
lets `StreamingBuffer.getContents(surface, serial)` wait for the right
|
||||||
|
encoded frame).
|
||||||
|
2. `bufferContents = await bufferImpl.getContents(surface, serial)` —
|
||||||
|
resolves when the encoded frame for this serial has arrived.
|
||||||
|
3. `role.onCommit(this)` — role-specific logic (XdgToplevel, XdgPopup,
|
||||||
|
Subsurface, etc.).
|
||||||
|
4. `Surface.commitPending()` runs:
|
||||||
|
- Commit `subsurfaceChildren` order + apply pendingPosition → cascade
|
||||||
|
`onParentCommit`.
|
||||||
|
- Calculate derived pending state (buffer transform matrix, surface size,
|
||||||
|
pixman region).
|
||||||
|
- Release old buffer if replaced.
|
||||||
|
- `mergeSurfaceState(state, pendingState)` — atomic swap.
|
||||||
|
- `damaged = true`; clear pending damage + frame callbacks.
|
||||||
|
|
||||||
|
## Damage tracking
|
||||||
|
|
||||||
|
`damageRects` (post-transform) and `bufferDamageRects` (buffer-space) are
|
||||||
|
accumulated correctly on the surface state via pixman-WASM region math. The
|
||||||
|
damage is tracked faithfully — but **vestigial at the GL level**. The
|
||||||
|
renderer doesn't consume it for scissoring — it repaints every visible view
|
||||||
|
every RAF. The only damage gate is the boolean `surface.damaged`, used to
|
||||||
|
decide whether a streaming buffer's frame needs re-upload + release.
|
||||||
|
|
||||||
|
`RenderState.visibleSceneRegion` (pixman) is used only to decide if a
|
||||||
|
`RenderState` exists at all (i.e. whether the view is on this output), not
|
||||||
|
for per-rect clipping.
|
||||||
|
|
||||||
|
## Compositing pipeline
|
||||||
|
|
||||||
|
The renderer is **WebGL1 only**, no OffscreenCanvas, no WebGL2. The
|
||||||
|
context is acquired with `getContext('webgl', {antialias:false, depth:false,
|
||||||
|
alpha:true, premultipliedAlpha:false, preserveDrawingBuffer:false,
|
||||||
|
desynchronized:true})`. FBOs are used for in-GPU YUV→RGBA and
|
||||||
|
RGB+Alpha→RGBA conversion into per-view textures.
|
||||||
|
|
||||||
|
The compositor runs in **four phases** per render pass:
|
||||||
|
|
||||||
|
### PHASE 1 — pixel upload (per view per scene)
|
||||||
|
|
||||||
|
Coalesced via `setTimeout`-based cancellable microtask. For each view, for
|
||||||
|
each scene it's visible on:
|
||||||
|
|
||||||
|
1. `view.applyTransformations()` — compute `parent.transformation *
|
||||||
|
translate(position+offset)`.
|
||||||
|
2. For each `renderState` in `view.renderStates`:
|
||||||
|
`scene[bufferContents.mimeType](bufferContents, renderState)` — dispatch
|
||||||
|
to the right upload path:
|
||||||
|
- `'image/bitmap'` → `texture.setContent(imageBitmap, size)` — direct
|
||||||
|
`texImage2D` / `texSubImage2D` upload.
|
||||||
|
- `'video/h264'` → `YUVA2RGBA` or `RGBXandA2RGBA.convertInto(...)` —
|
||||||
|
FBO pass that uploads Y/U/V/alpha-Y as 4× LUMINANCE textures, runs the
|
||||||
|
YUVA→RGBA shader (BT.601 coefficients), writes into the renderState's
|
||||||
|
RGBA texture.
|
||||||
|
- `'image/png'` → `texture.setContent(bitmap, size); bitmap.close()`.
|
||||||
|
3. Collect `surface.state.frameCallbacks` → `Renderer.frameCallbacks`.
|
||||||
|
4. `surface.damaged = false`; `streamingBuffer.release()` (remote only).
|
||||||
|
|
||||||
|
### PHASE 2 — schedule render frame
|
||||||
|
|
||||||
|
`Renderer.render()` coalesces via a `setTimeout`-based cancellable
|
||||||
|
microtask, then schedules a single `requestAnimationFrame` for PHASE 3.
|
||||||
|
|
||||||
|
### PHASE 3 — draw (per scene, inside RAF)
|
||||||
|
|
||||||
|
For each scene (one per canvas/output):
|
||||||
|
|
||||||
|
1. `sceneShader.use()`; `updateSceneData(canvas)` — viewport, clear, BLEND,
|
||||||
|
projection.
|
||||||
|
2. For each view in the view stack (flattened from `topLevelViews`
|
||||||
|
recursively via `Renderer.updateViewStack`):
|
||||||
|
- If `renderState && view.mapped`:
|
||||||
|
- `sceneShader.updateViewData(view, renderState)` — bind texture,
|
||||||
|
`u_transform`, quad buffer.
|
||||||
|
- `sceneShader.draw()` — `gl.drawArrays(TRIANGLE_STRIP, 0, 6)` of a
|
||||||
|
textured quad per scene.
|
||||||
|
|
||||||
|
### PHASE 4 — fire frame callbacks
|
||||||
|
|
||||||
|
For each callback in `Renderer.frameCallbacks`:
|
||||||
|
`callback.done(time)` → `wl_callback.done(time)`; `wl_callback.destroy()`.
|
||||||
|
Then `session.flush()` to send any pending wire messages.
|
||||||
|
|
||||||
|
The `time` value is the `requestAnimationFrame` high-resolution timestamp,
|
||||||
|
so apps see real vsync-aligned frame timing.
|
||||||
|
|
||||||
|
## Multi-output
|
||||||
|
|
||||||
|
Multi-output = one Scene per canvas. Each view has one `RenderState` (and
|
||||||
|
one WebGL texture) per scene it's visible on. So a surface visible on two
|
||||||
|
outputs (e.g. mirroring) has two textures — one per canvas — and gets
|
||||||
|
uploaded to each on PHASE 1.
|
||||||
|
|
||||||
|
## Subsurfaces stacking
|
||||||
|
|
||||||
|
Subsurfaces stack via `subsurfaceChildren` arrays (committed + pending,
|
||||||
|
bottom→top). `placeAbove` / `placeBelow` reorder pending;
|
||||||
|
`Renderer.updateViewStack` flattens `topLevelViews` recursively.
|
||||||
|
|
||||||
|
**Important**: no parent-geometry clipping — subsurfaces can render outside
|
||||||
|
their parent. The only clip is the scene's canvas region, enforced implicitly
|
||||||
|
by NDC mapping.
|
||||||
|
|
||||||
|
## Cursor and DnD bypass GL entirely
|
||||||
|
|
||||||
|
Cursor and DnD bypass GL entirely. They draw to a separate singleton 2D
|
||||||
|
canvas and use CSS `cursor: url(dataURL)` or HTML5
|
||||||
|
`dataTransfer.setDragImage`. Their frame callbacks fire immediately with
|
||||||
|
`Date.now()`, not synced to RAF.
|
||||||
|
|
||||||
|
This is a deliberate optimization: cursor updates must be low-latency, and
|
||||||
|
going through the full GL pipeline would add a frame of latency.
|
||||||
|
|
||||||
|
## Decode pipeline (for H.264 frames)
|
||||||
|
|
||||||
|
Decoding is delegated to:
|
||||||
|
|
||||||
|
### Preferred path: WebCodecs `VideoDecoder`
|
||||||
|
|
||||||
|
At session startup, `Session.create()` probes
|
||||||
|
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
|
||||||
|
hardwareAcceleration:'prefer-software'})`. If supported, an `opaqueDecoder`
|
||||||
|
and `alphaDecoder` are created per surface (lazily, on first H.264 frame).
|
||||||
|
Encoded NALs are wrapped as `EncodedVideoChunk {type:key|delta, data:nal}`
|
||||||
|
and fed to the decoder. The output `VideoFrame` is uploaded directly to a
|
||||||
|
WebGL texture.
|
||||||
|
|
||||||
|
**Important**: hardware decode is **explicitly disabled** in `Session.ts`
|
||||||
|
with a FIXME comment "hardware decoding frame offset is wrong". Only
|
||||||
|
software decoding is used today.
|
||||||
|
|
||||||
|
### Fallback path: FFmpeg Wasm in Web Workers
|
||||||
|
|
||||||
|
If WebCodecs is unavailable, two persistent Workers (`h264NALDecoderOpaque`,
|
||||||
|
`h264NALDecoderAlpha`) are created at session startup. Each worker has one
|
||||||
|
`AVCodecContext` per surface, keyed by `surfaceH264DecodeId`. Encoded NALs
|
||||||
|
are transferred zero-copy into the worker via
|
||||||
|
`worker.postMessage(nal, [nal.buffer])`. Decoded I420 planes are transferred
|
||||||
|
back as `[y.buffer, u.buffer, v.buffer]` and uploaded as 4× LUMINANCE
|
||||||
|
textures.
|
||||||
|
|
||||||
|
## PNG path (both decoders)
|
||||||
|
|
||||||
|
The PNG path is identical in both branches:
|
||||||
|
`new Blob([opaque], {type:'image/png'})` →
|
||||||
|
`createImageBitmap(blob, sx, sy, w, h)` — the `sx/sy/w/h` crops the
|
||||||
|
encoder padding back to logical size. PNG is always used for surfaces ≤
|
||||||
|
256×256 pixels because H.264 I-frame overhead dominates at small sizes.
|
||||||
|
|
||||||
|
## WebGL shaders
|
||||||
|
|
||||||
|
Three shaders in `render/`:
|
||||||
|
|
||||||
|
- `YUVA2RGBAShader` — converts YUVA (Y, U, V, alpha-Y as 4 LUMINANCE
|
||||||
|
textures) to RGBA in a single FBO pass using BT.601 coefficients. Alpha
|
||||||
|
is recovered by running the alpha-Y plane through the same YUV→RGB matrix
|
||||||
|
and taking the R channel.
|
||||||
|
- `RGBandA2RGBAShader` — for the `DualPlaneRGBA*` WebCodecs outputs
|
||||||
|
(opaque + alpha planes composited into RGBA).
|
||||||
|
- `SceneShader` — final pass: draws every view's RGBA texture to the canvas
|
||||||
|
with `gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)`. Note: expects premultiplied
|
||||||
|
alpha but the YUVA shader outputs straight alpha — a minor inconsistency
|
||||||
|
that works because Wayland surfaces rarely use partial alpha.
|
||||||
|
|
||||||
|
## ASCII pipeline diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
CLIENT (Wayland app / web worker / compositor-proxy)
|
||||||
|
│ wl_surface.attach / damage / frame / commit
|
||||||
|
▼
|
||||||
|
Surface.commit()
|
||||||
|
│ serial = commitSerials.shift() (remote-only)
|
||||||
|
│ bufferContents = await bufferImpl.getContents(surface, serial)
|
||||||
|
│ role.onCommit(this)
|
||||||
|
▼
|
||||||
|
Surface.commitPending()
|
||||||
|
│ 1. commit subsurfaceChildren order + apply pendingPosition → cascade onParentCommit
|
||||||
|
│ 2. calculateDerivedPendingState (bufferTransform matrix, surface.size, pixmanRegion)
|
||||||
|
│ 3. release old buffer if replaced
|
||||||
|
│ 4. mergeSurfaceState(state, pendingState) ◄── atomic swap
|
||||||
|
│ 5. damaged = true; clear pending damage/frame callbacks
|
||||||
|
▼
|
||||||
|
Renderer.render()
|
||||||
|
│ coalesce via queueCancellableMicrotask (setTimeout)
|
||||||
|
▼
|
||||||
|
PHASE 1 — pixel upload (per view per scene):
|
||||||
|
view.applyTransformations() // parent.transformation * translate(position+offset)
|
||||||
|
for renderState in view.renderStates:
|
||||||
|
scene[bufferContents.mimeType](bufferContents, renderState)
|
||||||
|
'image/bitmap' → texture.setContent(imageBitmap, size) // texImage2D / texSubImage2D
|
||||||
|
'video/h264' → YUVA2RGBA / RGBXandA2RGBA.convertInto(...) // 4 LUMINANCE or 2 RGBA tex
|
||||||
|
→ FBO + YUVA2RGBAShader (BT.601) → renderState.texture
|
||||||
|
'image/png' → texture.setContent(bitmap, size); bitmap.close()
|
||||||
|
collect surface.state.frameCallbacks → Renderer.frameCallbacks
|
||||||
|
surface.damaged = false; streamingBuffer.release() (remote only)
|
||||||
|
▼
|
||||||
|
PHASE 2 — createRenderFrame().then(time => { // = requestAnimationFrame
|
||||||
|
PHASE 3 — for scene in scenes: // one per canvas/output
|
||||||
|
sceneShader.use(); updateSceneData(canvas) // viewport, clear, BLEND, projection
|
||||||
|
for view in viewStack:
|
||||||
|
if renderState && view.mapped:
|
||||||
|
sceneShader.updateViewData(view, renderState) // bind texture, u_transform, quad buffer
|
||||||
|
sceneShader.draw() // gl.drawArrays(TRIANGLE_STRIP, 0, 6)
|
||||||
|
PHASE 4 — for callback in frameCallbacks: callback.done(time)
|
||||||
|
→ wl_callback.done(time); wl_callback.destroy()
|
||||||
|
session.flush()
|
||||||
|
})
|
||||||
|
```
|
||||||
|
|
||||||
|
## The remote streaming buffer content flow
|
||||||
|
|
||||||
|
The remote streaming buffer content arrives in parallel on a separate FRAME
|
||||||
|
WebSocket channel:
|
||||||
|
|
||||||
|
1. `deliverContentToBufferStream` (in `RemoteAppLauncher.ts`) receives
|
||||||
|
incoming frame messages.
|
||||||
|
2. `BufferStream.onBufferContents` parses the encoded frame envelope.
|
||||||
|
3. `createEncodedFrame` (in `EncodedFrame.ts`) parses `{contentSerial,
|
||||||
|
mimeType, size, encodedSize, opaque, alpha?}`.
|
||||||
|
4. Resolves the `onFrameAvailable(serial)` promise that
|
||||||
|
`StreamingBuffer.getContents` is awaiting.
|
||||||
|
5. The decoded `DecodedFrame` is then uploaded into the view's
|
||||||
|
`RenderState.texture` via the appropriate FBO+shader conversion.
|
||||||
|
|
||||||
|
## Engineering observations
|
||||||
|
|
||||||
|
1. **Vestigial damage tracking** — `damageRects` are accumulated correctly
|
||||||
|
but the renderer doesn't use them for scissoring. Could be a future
|
||||||
|
optimization for partial repaints.
|
||||||
|
2. **Multi-output texture cost** — each view has one texture per scene,
|
||||||
|
so 4 outputs means 4× texture memory for every surface.
|
||||||
|
3. **SHM xrgb8888 BMP-header hack** — cleverly offloads pixel conversion
|
||||||
|
to the browser's BMP decoder. Worth knowing if you need to support
|
||||||
|
other SHM formats.
|
||||||
|
4. **Commit-serial mechanism** — the out-of-band opcode 7 lets the
|
||||||
|
streaming buffer wait for the right encoded frame, ensuring wire-order
|
||||||
|
semantics even with async decode.
|
||||||
|
5. **Encoder feedback telemetry** — `EncoderFeedback.ts` measures client
|
||||||
|
decode duration, but the adaptation loop is wired but not consumed (a
|
||||||
|
missed opportunity for adaptive bitrate).
|
||||||
|
6. **Context-loss audit gap** — WebGL context loss is not audited in the
|
||||||
|
test suite; in production, you should handle `webglcontextlost` and
|
||||||
|
`webglcontextrestored` events.
|
||||||
|
7. **`queueCancellableMicrotask` misnomer** — the function is actually a
|
||||||
|
`setTimeout`-based coalescer, not a microtask. Naming is misleading.
|
||||||
|
8. **Frame-callback coalescing** — multiple `wl_surface.frame` requests
|
||||||
|
within one render pass fire on the same RAF tick. Correct Wayland
|
||||||
|
behavior, but worth knowing.
|
||||||
|
9. **Cursor/DnD bypass** — they bypass the GL pipeline entirely for
|
||||||
|
latency. The cursor is rendered as a CSS `cursor: url(dataURL)`.
|
||||||
|
10. **Subsurface sync cascade** — three-layer mechanism (`cachedState`,
|
||||||
|
`commitCache`, `effectiveSync`) is correct but subtle. Easy to break
|
||||||
|
in maintenance.
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/Surface.ts` | `wl_surface` implementation, double-buffered state, commit semantics |
|
||||||
|
| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction |
|
||||||
|
| `packages/compositor/src/SurfaceRole.ts` | Role abstraction (SurfaceRole interface) |
|
||||||
|
| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
|
||||||
|
| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global |
|
||||||
|
| `packages/compositor/src/View.ts` | Positioned surface for compositing |
|
||||||
|
| `packages/compositor/src/Output.ts` | `wl_output`, WebGL canvas binding |
|
||||||
|
| `packages/compositor/src/Shm.ts` | `wl_shm` / shm pool, BMP-header hack |
|
||||||
|
| `packages/compositor/src/BufferContents.ts` | Buffer abstraction (with mimeType) |
|
||||||
|
| `packages/compositor/src/BufferImplementation.ts` | Buffer implementation interface |
|
||||||
|
| `packages/compositor/src/ImageBitmapBuffer.ts` | ImageBitmap-backed buffer |
|
||||||
|
| `packages/compositor/src/Region.ts` | `wl_region`, pixman-WASM damage regions |
|
||||||
|
| `packages/compositor/src/Callback.ts` | Frame callback (wl_callback) |
|
||||||
|
| `packages/compositor/src/Loop.ts` | Compositor main loop, RAF integration |
|
||||||
|
| `packages/compositor/src/Compositor.ts` | `wl_compositor` global |
|
||||||
|
| `packages/compositor/src/Session.ts` | Top-level browser session, decoder setup |
|
||||||
|
| `packages/compositor/src/Globals.ts` | Wayland globals registry |
|
||||||
|
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
|
||||||
|
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
|
||||||
|
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
|
||||||
|
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
|
||||||
|
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
|
||||||
|
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
|
||||||
|
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
|
||||||
|
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
|
||||||
|
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
|
||||||
|
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
|
||||||
|
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
|
||||||
|
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
|
||||||
|
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
|
||||||
|
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||||||
|
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||||||
|
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||||||
|
| `packages/compositor/src/remote/buffer-decoder.ts` | Decoder interface |
|
||||||
|
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
|
||||||
|
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||||||
|
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||||||
|
| `packages/compositor/src/remote/EncoderFeedback.ts` | Receives 4-byte feedback messages |
|
||||||
|
| `packages/compositor/src/remote/Channel.ts` | ARQChannel + SimpleChannel |
|
||||||
|
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
|
||||||
|
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Sets up channels per app, frame delivery |
|
||||||
|
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||||||
|
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer events |
|
||||||
|
| `packages/compositor/src/browser/dnd.ts` | DnD bridge |
|
||||||
|
| `packages/compositor/src/browser/selection.ts` | Clipboard bridge |
|
||||||
|
| `packages/compositor/src/browser/capabilities.ts` | Browser feature detection |
|
||||||
|
| `packages/compositor/src/browser/input.ts` | DOM keyboard → wl_keyboard events |
|
||||||
|
| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
|
||||||
|
| `packages/compositor/src/math/Mat4.ts` | 4×4 matrix math |
|
||||||
|
| `packages/compositor/src/math/Rect.ts` | Rectangle math |
|
||||||
|
| `packages/compositor/src/polyfill/createImageBitmap.js` | ImageBitmap polyfill |
|
||||||
+397
@@ -0,0 +1,397 @@
|
|||||||
|
# 04 — Transport Layer (WebSocket / KCP / Channel Layer)
|
||||||
|
|
||||||
|
This chapter covers Greenfield's transport: how Wayland wire traffic flows
|
||||||
|
between the Linux-side proxy and the browser-side compositor over WebSocket,
|
||||||
|
the binary framing, message ordering, latency/backpressure, KCP usage,
|
||||||
|
multiplexing, and connection lifecycle.
|
||||||
|
|
||||||
|
## Architecture at a glance
|
||||||
|
|
||||||
|
Greenfield uses a **multi-WebSocket, KCP-wrapped, asymmetric flow-control**
|
||||||
|
design:
|
||||||
|
|
||||||
|
- **Two-tier process model**: A `main.ts` HTTP hub forks one `SessionProcess`
|
||||||
|
per browser session (per `compositorSessionId`). Raw `net.Socket`s are
|
||||||
|
handed to children via Node IPC `sendHandle`. Each session child owns its
|
||||||
|
own `wl_display` (via `wayland-server-addon.node`).
|
||||||
|
- **9 WebSocket endpoints** per session: `/signal` (JSON control),
|
||||||
|
`/channel` (binary data channels), `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
|
||||||
|
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
|
||||||
|
All with `perMessageDeflate: false`, TCP `noDelay: true`.
|
||||||
|
|
||||||
|
## Channel types
|
||||||
|
|
||||||
|
There are **4 logical channel types** per app client, each implemented as a
|
||||||
|
separate WebSocket:
|
||||||
|
|
||||||
|
| Type | Class | KCP? | Purpose |
|
||||||
|
|------|-------|------|---------|
|
||||||
|
| PROTOCOL | ARQChannel | Yes | Wayland wire protocol, per `wl_client` |
|
||||||
|
| FRAME | ARQChannel | Yes | H.264/PNG encoded surface content, per `wl_surface` |
|
||||||
|
| XWM | ARQChannel | Yes | XWayland window manager protocol |
|
||||||
|
| FEEDBACK | SimpleChannel | No | 4-byte telemetry (refreshInterval + decode avg), 1 Hz |
|
||||||
|
|
||||||
|
**There is no multiplexing** of multiple Wayland clients onto one WS. Each
|
||||||
|
logical channel is its own WS, eliminating head-of-line blocking between
|
||||||
|
video, input, and control at the cost of many connections per app (10+ for
|
||||||
|
complex apps).
|
||||||
|
|
||||||
|
## KCP-over-WebSocket
|
||||||
|
|
||||||
|
`kcp.ts` is a ~1100-line TypeScript port of the canonical C `ikcp.c`.
|
||||||
|
Greenfield runs it **on top of WebSocket** (not real UDP) for:
|
||||||
|
|
||||||
|
- 30 ms min RTO vs TCP's 200 ms.
|
||||||
|
- 100 ms update tick (`setInterval(kcp.update, 100)`).
|
||||||
|
- Per-channel congestion domains independent of TCP.
|
||||||
|
- `setNoDelay(1, 100, 0, 1)` — nodelay enabled, congestion control disabled
|
||||||
|
(`nc=1`).
|
||||||
|
- 24-byte segment header: `conv|cmd|frg|wnd|ts|sn|una|len`.
|
||||||
|
- MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC
|
||||||
|
design).
|
||||||
|
|
||||||
|
### Asymmetric windows (deliberate)
|
||||||
|
|
||||||
|
- Proxy side: `snd_wnd=128, rcv_wnd=1024`
|
||||||
|
- Browser side: `snd_wnd=1024, rcv_wnd=128`
|
||||||
|
|
||||||
|
So proxy→browser (heavy video) has a smaller in-flight cap to bound latency;
|
||||||
|
browser→proxy (bursty input) has a large window to avoid stalls.
|
||||||
|
|
||||||
|
### Why KCP-over-WebSocket?
|
||||||
|
|
||||||
|
Because real-time video over plain TCP suffers from head-of-line blocking
|
||||||
|
and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in
|
||||||
|
user-space, while still riding on WebSocket (the only bidirectional binary
|
||||||
|
transport universally available in browsers today).
|
||||||
|
|
||||||
|
The long-term plan, documented in `docs/pages/design/index.md`, is to
|
||||||
|
switch to **WebTransport** (real UDP) when browser support is widespread,
|
||||||
|
and pair it with KCP for reliability. The KCP code is already in place;
|
||||||
|
only the WebSocket transport layer needs to be swapped out.
|
||||||
|
|
||||||
|
## Binary message format
|
||||||
|
|
||||||
|
All multi-byte fields are **little-endian**.
|
||||||
|
|
||||||
|
### PROTOCOL channel
|
||||||
|
|
||||||
|
```
|
||||||
|
u32 outOfBandOpcode (0 = in-band events, else 1=ResourceDestroyed,
|
||||||
|
2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial)
|
||||||
|
u32 eventSerial
|
||||||
|
u32 fdsCount
|
||||||
|
for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned)
|
||||||
|
concatenated Wayland wire messages (object_id + opcode/size + args)
|
||||||
|
```
|
||||||
|
|
||||||
|
### FRAME channel (per H.264 frame)
|
||||||
|
|
||||||
|
```
|
||||||
|
u32 bufferId
|
||||||
|
u32 bufferCreationSerial
|
||||||
|
u32 bufferContentSerial
|
||||||
|
encodedFrame:
|
||||||
|
u32 contentSerial, u32 encodingType (0=h264/1=png),
|
||||||
|
u32 width, u32 height, u32 encodedWidth, u32 encodedHeight,
|
||||||
|
u32 opaqueLength, u8[] opaqueData,
|
||||||
|
u32 alphaLength, u8[] alphaData ← alpha encoded as a SEPARATE H.264 stream
|
||||||
|
```
|
||||||
|
|
||||||
|
### FEEDBACK channel
|
||||||
|
|
||||||
|
Just 4 bytes: `u16 refreshIntervalMs + u16 avgDecodeDurationMs`. Sent at
|
||||||
|
1 Hz from browser to proxy. Used by the proxy to pace `wl_frame.done`
|
||||||
|
callbacks so the native app sees a virtual monitor refresh rate matching
|
||||||
|
the browser's actual RAF cadence.
|
||||||
|
|
||||||
|
## File descriptor handling
|
||||||
|
|
||||||
|
Can't send FDs over WebSocket — Greenfield virtualizes them:
|
||||||
|
|
||||||
|
- **ProxyFD** = `{handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown',
|
||||||
|
host:string}`.
|
||||||
|
- Serialized as JSON inline in the message stream with a u32 length prefix.
|
||||||
|
- Browser opens **separate ad-hoc WebSocket** to `/read-fd`,
|
||||||
|
`/write-fd-as-stream`, etc., only when it actually needs the bytes.
|
||||||
|
- Most `wl_buffer` FDs never need to be read because the proxy intercepts
|
||||||
|
`wl_surface.commit`, reads the buffer locally via DRM/SHM, encodes to
|
||||||
|
H.264, and ships only the encoded bytes.
|
||||||
|
- **Foreign ProxyFD** (`proxyFD.host !== local baseURL`) triggers
|
||||||
|
proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the
|
||||||
|
foreign proxy's `/write-fd-as-stream`, and pipes the data through. This
|
||||||
|
is how copy-paste between two proxy instances avoids a browser round-trip.
|
||||||
|
|
||||||
|
## Reconnection & lifecycle
|
||||||
|
|
||||||
|
### Browser side
|
||||||
|
|
||||||
|
Vendored `ReconnectingWebSocket` (pladaria) with:
|
||||||
|
|
||||||
|
- `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`
|
||||||
|
- `minUptime=5s`, `connTimeout=4s`
|
||||||
|
- `retries=∞`
|
||||||
|
|
||||||
|
Channel state preservation: only close codes 4000–4999 (application-level)
|
||||||
|
tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps
|
||||||
|
ticking and the WS reconnects underneath.
|
||||||
|
|
||||||
|
### Proxy side
|
||||||
|
|
||||||
|
No auto-reconnect (it's the server). The Channel object lives on, waiting
|
||||||
|
for a new WS upgrade with matching `id` + `key`. Outbound messages queue
|
||||||
|
in `outboundMessages` and flush on `onOpen`.
|
||||||
|
|
||||||
|
### App kill timers
|
||||||
|
|
||||||
|
When the browser disconnects from the `/signal` WebSocket:
|
||||||
|
|
||||||
|
- **10-minute SIGHUP grace period** — gives the user time to reconnect
|
||||||
|
(e.g. if they refresh the browser tab).
|
||||||
|
- After 10 minutes of no reconnect, escalating to **SIGKILL after another
|
||||||
|
10s**.
|
||||||
|
- Close code 1001 → immediate SIGHUP (user closed tab intentionally).
|
||||||
|
- Close code 4001 → immediate SIGTERM (user clicked "force quit" in the
|
||||||
|
shell).
|
||||||
|
|
||||||
|
### Session close
|
||||||
|
|
||||||
|
When all `wl_clients` disconnect, 1-second grace then `session.close()` →
|
||||||
|
`process.exit()` in the session child. Main process deletes the entry
|
||||||
|
from its `sessionProcesses` map.
|
||||||
|
|
||||||
|
## Process model — three layers
|
||||||
|
|
||||||
|
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
|
||||||
|
|
||||||
|
A single Node.js process listens on `:8081`. Per browser session, it
|
||||||
|
`fork()`s one `SessionProcess` child — one child per `compositorSessionId`,
|
||||||
|
**not** per app. Multiple apps in the same browser tab share one child.
|
||||||
|
WebSocket upgrades are handed off to the matching child via
|
||||||
|
`childProcess.send(wsUpgrade, socket)`.
|
||||||
|
|
||||||
|
### Layer 2: Session Process (`SessionProcess.ts`)
|
||||||
|
|
||||||
|
The forked child. Receives `start` (creates `Session` +
|
||||||
|
`SessionController`), `launchApp` (spawns the actual app binary), and
|
||||||
|
`wsUpgrade` (hands WS to the controller) IPC messages. On `launchApp` it
|
||||||
|
calls `launchApplication()` from `@gfld/compositor-proxy`, which `spawn()`s
|
||||||
|
the executable, captures `pid`, replies to main with `{pid, key}`. On
|
||||||
|
`session.close()` the process exits.
|
||||||
|
|
||||||
|
### Layer 3: Proxy Core (`compositor-proxy/src/`)
|
||||||
|
|
||||||
|
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
|
||||||
|
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
|
||||||
|
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
|
||||||
|
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
|
||||||
|
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
|
||||||
|
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
|
||||||
|
wl_display FD into Node's libuv loop via `startPoll()`.
|
||||||
|
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
|
||||||
|
Uses `MessageInterceptor` + auto-generated per-resource interceptor
|
||||||
|
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
|
||||||
|
`{native, browser, neverReplies?}` routing decision. Handles
|
||||||
|
`wl_display.sync` fast-path (skips browser round-trip when no reply
|
||||||
|
expected). Marshalls FDs as `ProxyFD` JSON.
|
||||||
|
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
|
||||||
|
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
|
||||||
|
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
|
||||||
|
(10 s) on force-quit, auto-destroy on app exit.
|
||||||
|
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
|
||||||
|
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
|
||||||
|
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
|
||||||
|
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
|
||||||
|
connect). Xwayland itself becomes a Wayland client of the proxy. The
|
||||||
|
`wm_fd` is bridged via xtsb to the browser, which implements the actual
|
||||||
|
XWM logic.
|
||||||
|
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
|
||||||
|
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
|
||||||
|
|
||||||
|
## ASCII diagram of the full transport stack
|
||||||
|
|
||||||
|
```
|
||||||
|
LINUX SIDE | BROWSER SIDE
|
||||||
|
|
|
||||||
|
┌─────────────────────────┐ | ┌──────────────────────────────┐
|
||||||
|
│ gtk4-demo (process) │ | │ Browser tab │
|
||||||
|
│ libwayland-client.so │ | │ (Greenfield compositor) │
|
||||||
|
└───────────┬─────────────┘ | └──────────────┬───────────────┘
|
||||||
|
│ WAYLAND_DISPLAY=wayland-0 │
|
||||||
|
│ Unix socket + SCM_RIGHTS FD passing │ DOM events → input
|
||||||
|
▼ │ WebSocket API
|
||||||
|
┌──────────────────────────────────────────┐ │
|
||||||
|
│ SessionProcess (fork'd Node child) │ │
|
||||||
|
│ ┌────────────────────────────────────┐ │ │
|
||||||
|
│ │ wayland-server-addon.node │ │ │
|
||||||
|
│ │ (libwayland-server N-API) │ │ │
|
||||||
|
│ │ - wl_display + wl_client │ │ │
|
||||||
|
│ │ - epoll via proxy-poll-addon │ │ │
|
||||||
|
│ └─────────────┬──────────────────────┘ │ │
|
||||||
|
│ │ wire msg + FDs │ │
|
||||||
|
│ ▼ │ │
|
||||||
|
│ ┌─────────────────────────────────────┐ │ │
|
||||||
|
│ │ NativeWaylandClientSession │ │ │
|
||||||
|
│ │ - interceptRequest / interceptEvent│ │ │
|
||||||
|
│ │ - onWireMessageEnd: serialize │ │ │
|
||||||
|
│ │ FDs as JSON ProxyFDs inline │ │ │
|
||||||
|
│ │ - fastSync optimization │ │ │
|
||||||
|
│ └─────────────┬───────────────────────┘ │ │
|
||||||
|
│ │ outbound buffer │ │
|
||||||
|
│ ▼ │ │
|
||||||
|
│ ┌─────────────────────────────────────┐ │ │
|
||||||
|
│ │ ARQChannel (KCP-wrapped) │ │ │
|
||||||
|
│ │ MTU=64000, snd_wnd=128, rcv_wnd=1024│ │ │
|
||||||
|
│ │ KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data │
|
||||||
|
│ │ kcp.update() every 100ms │ │ │
|
||||||
|
│ │ - PROTOCOL (per wl_client) │ │ │
|
||||||
|
│ │ - FRAME (per wl_surface) │ │ │
|
||||||
|
│ │ - XWM (per XWayland) │ │ │
|
||||||
|
│ └─────────────┬───────────────────────┘ │ │
|
||||||
|
│ ┌─────────────┴───────────────────────┐ │ │
|
||||||
|
│ │ SimpleChannel (FEEDBACK, no KCP) │ │ │
|
||||||
|
│ │ 4 bytes: refreshIntervalMs + │ │ │
|
||||||
|
│ │ avgDecodeDurationMs │ │ │
|
||||||
|
│ └─────────────┬───────────────────────┘ │ │
|
||||||
|
│ ┌─────────────▼───────────────────────┐ │ │
|
||||||
|
│ │ NativeAppContext │ │ │
|
||||||
|
│ │ - signalingWebSocket (JSON msgs) │ │ │
|
||||||
|
│ │ - kill timers: SIGHUP@10min, │ │ │
|
||||||
|
│ │ SIGKILL@+10s │ │ │
|
||||||
|
│ └─────────────┬───────────────────────┘ │ │
|
||||||
|
│ ┌─────────────▼───────────────────────┐ │ │
|
||||||
|
│ │ SessionController │ │ │
|
||||||
|
│ │ ws.WebSocketServer (noServer) │ │ │
|
||||||
|
│ │ perMessageDeflate: false │ │ │
|
||||||
|
│ │ Endpoints: /signal /channel /mkfifo│ │ │
|
||||||
|
│ │ /mkstemp-mmap /read-fd /write-fd-as-stream│ │
|
||||||
|
│ │ /read-fd-as-stream /close-fd /request-keyframe│ │
|
||||||
|
│ └─────────────┬───────────────────────┘ │ │
|
||||||
|
└────────────────┼─────────────────────────────┘ │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────┐ │
|
||||||
|
│ │ main.ts (HTTP+WS hub) │ │
|
||||||
|
│ │ - socket.pause() + IPC │ │
|
||||||
|
│ │ send to session proc │ │
|
||||||
|
│ │ - noDelay:true (TCP) │ │
|
||||||
|
│ └────────────┬────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ◄──── 1 WebSocket per channel ────► │
|
||||||
|
│ ◄── 1 signaling WS per app ──► │
|
||||||
|
│ ◄── ad-hoc WebFD streams ─► │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────┐ │
|
||||||
|
│ │ ReconnectingWebSocket │ │
|
||||||
|
│ │ min 1-5s, factor 1.3, max 10s │ │
|
||||||
|
│ │ minUptime 5s, connTimeout 4s │ │
|
||||||
|
│ └────────────┬───────────────────────────┘ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────┐ │
|
||||||
|
│ │ ARQChannel / SimpleChannel (mirror image) │ │
|
||||||
|
│ │ snd_wnd=1024, rcv_wnd=128 │ │
|
||||||
|
│ │ kcp.input → peekSize/recv → onMessage │ │
|
||||||
|
│ └────────────┬───────────────────────────────┘ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────┐ │
|
||||||
|
│ │ RemoteAppLauncher.onChannel │ │
|
||||||
|
│ │ PROTOCOL → session.display.createClient │ │
|
||||||
|
│ │ FRAME → deliverContentToBufferStream │ │
|
||||||
|
│ │ XWM → XWindowManagerConnection │ │
|
||||||
|
│ │ FEEDBACK → ClientEncodersFeedback │ │
|
||||||
|
│ └────────────┬───────────────────────────────┘ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────┐ │
|
||||||
|
│ │ @gfld/compositor-protocol Client │ │
|
||||||
|
│ │ - parses Wayland wire format │ │
|
||||||
|
│ │ - dispatches to wl_*_resource impls │ │
|
||||||
|
│ │ - H264NALDecoder.worker.ts decodes frames │ │
|
||||||
|
│ └────────────┬───────────────────────────────┘ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────┐ │
|
||||||
|
│ │ Surface + Scene rendering │ │
|
||||||
|
│ │ - ImageBitmap / VideoFrame / Canvas │ │
|
||||||
|
│ │ - DOM input → wl_keyboard/pointer reqs │ │
|
||||||
|
│ └────────────┬───────────────────────────────┘ │
|
||||||
|
│ │ input events flow back via the │
|
||||||
|
│ │ PROTOCOL channel as ordinary │
|
||||||
|
│ │ Wayland requests (same framing) │
|
||||||
|
└──────────────────┘ │
|
||||||
|
Browser DOM ◄─┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key takeaways for building your own transport
|
||||||
|
|
||||||
|
1. **Separate control plane from data plane** — Greenfield uses `/signal`
|
||||||
|
(JSON) vs `/channel` (binary). The control plane handles app launch,
|
||||||
|
termination, kill timers; the data plane handles Wayland wire + encoded
|
||||||
|
frames.
|
||||||
|
2. **One WS per logical stream** — simpler, no HoL blocking, but many TCP
|
||||||
|
connections.
|
||||||
|
3. **KCP-over-WS** trades double-reliability overhead for fast (30 ms)
|
||||||
|
retransmits; reasonable for real-time video.
|
||||||
|
4. **Virtualize FDs as JSON tokens** with lazy separate-WS endpoints for
|
||||||
|
actual byte transfer.
|
||||||
|
5. **`bufferedAmount` check ≠ backpressure** — Greenfield's check is a drop
|
||||||
|
gate, not real flow control (acknowledged TODO).
|
||||||
|
6. **Distinguish transient vs permanent close** — codes 4000-4999 tear
|
||||||
|
down; lower codes auto-reconnect.
|
||||||
|
7. **Piggyback out-of-band opcodes** on the data channel via a leading u32
|
||||||
|
to avoid extra control WS.
|
||||||
|
8. **Asymmetric KCP windows** match each direction's bandwidth profile.
|
||||||
|
9. **Decode feedback drives frame pacing** — the proxy synthesizes
|
||||||
|
`wl_frame.done` callbacks at the browser's reported cadence.
|
||||||
|
10. **One Node child per browser session** for fault isolation and
|
||||||
|
per-session `wl_display` ownership.
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
### Native side
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor-proxy/src/Channel.ts` | `ARQChannel` + `SimpleChannel` |
|
||||||
|
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
|
||||||
|
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
|
||||||
|
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
|
||||||
|
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
|
||||||
|
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||||||
|
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
|
||||||
|
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
|
||||||
|
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
|
||||||
|
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
|
||||||
|
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
|
||||||
|
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||||||
|
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
|
||||||
|
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
|
||||||
|
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
|
||||||
|
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
|
||||||
|
|
||||||
|
### Browser side
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/remote/Channel.ts` | `ARQChannel` + `SimpleChannel` (mirror) |
|
||||||
|
| `packages/compositor/src/remote/kcp.ts` | KCP mirror |
|
||||||
|
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
|
||||||
|
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
|
||||||
|
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||||||
|
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
|
||||||
|
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||||||
|
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||||||
|
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||||||
|
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
|
||||||
|
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||||||
|
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||||||
|
| `packages/compositor/src/remote/api.ts` | Public exports |
|
||||||
|
|
||||||
|
### CLI
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
|
||||||
|
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
|
||||||
|
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
|
||||||
|
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
|
||||||
|
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
|
||||||
|
| `packages/compositor-proxy-cli/wait-until-ready.sh` | Startup readiness check |
|
||||||
@@ -0,0 +1,366 @@
|
|||||||
|
# 05 — Input Handling (Keyboard, Pointer, Touch, Focus)
|
||||||
|
|
||||||
|
This chapter covers Greenfield's input handling: keyboard (with xkbcommon in
|
||||||
|
browser), pointer, touch, focus management, and how input events travel from
|
||||||
|
the browser to native Linux apps via the proxy.
|
||||||
|
|
||||||
|
## Headline
|
||||||
|
|
||||||
|
Greenfield is a **browser-native Wayland compositor**. All seat logic —
|
||||||
|
keymap compilation, modifier state, pointer grabs, button/axis dispatch —
|
||||||
|
runs **in the browser**, against a **WASM build of libxkbcommon 1.5.0 +
|
||||||
|
pixman**. The `compositor-proxy` is *not* a compositor — it is a thin
|
||||||
|
libwayland-server bridge that relays Wayland wire messages between the
|
||||||
|
browser and native clients and provides file-descriptor shims.
|
||||||
|
|
||||||
|
There is no second xkbcommon on the proxy: the keymap is generated in the
|
||||||
|
browser from RMLVO and shipped to the proxy as a real native shm fd via a
|
||||||
|
custom `/mkstemp-mmap` WebSocket endpoint. `wl_touch` is advertised but
|
||||||
|
**never populated with events** — touch input is collapsed into Pointer
|
||||||
|
Events.
|
||||||
|
|
||||||
|
## The Seat abstraction
|
||||||
|
|
||||||
|
`Seat.ts` (in `packages/compositor/src/`) is the `wl_seat` global. It
|
||||||
|
aggregates keyboard, pointer, and touch, manages focus, and runs the input
|
||||||
|
queue. Each `Seat` has:
|
||||||
|
|
||||||
|
- `Keyboard` (always present)
|
||||||
|
- `Pointer` (always present)
|
||||||
|
- `Touch` (advertised only if `'ontouchstart' in document.documentElement`,
|
||||||
|
but inert — see §Touch below)
|
||||||
|
- `InputQueue` — coalesces events and dispatches them on the next RAF
|
||||||
|
- `InputOutput` — abstracts FD operations (browser pipes for in-browser
|
||||||
|
apps, WebSocket endpoints for native apps via proxy)
|
||||||
|
|
||||||
|
Each seat also tracks:
|
||||||
|
|
||||||
|
- The currently focused `Surface` for keyboard input
|
||||||
|
- The currently focused `Surface` for pointer input (transient)
|
||||||
|
- The active pointer grab (default, popup, drag, or resize)
|
||||||
|
- The active keyboard grab (default or popup)
|
||||||
|
|
||||||
|
## Keyboard — xkbcommon in the browser
|
||||||
|
|
||||||
|
### WASM xkbcommon build
|
||||||
|
|
||||||
|
`libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`,
|
||||||
|
applies a patch (removes tests, shims `strdup`), builds with emscripten
|
||||||
|
3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`,
|
||||||
|
exports the keymap/state C API as a single-file ES module.
|
||||||
|
|
||||||
|
The same build approach is used for pixman (for region math).
|
||||||
|
|
||||||
|
### Keymap compilation flow
|
||||||
|
|
||||||
|
The browser, not the proxy, is the source of truth for the keymap. The
|
||||||
|
flow is:
|
||||||
|
|
||||||
|
1. Browser receives a keymap config (either default or user-supplied via
|
||||||
|
`userShell.actions.setUserConfiguration({ keyboardLayoutName })`).
|
||||||
|
2. `Seat.notifyUpdateKeymap(nrmlvo)` is called.
|
||||||
|
3. `xkb_context_new` + `xkb_keymap_new_from_names` (WASM calls) compile the
|
||||||
|
RMLVO into a keymap.
|
||||||
|
4. `xkb_keymap_get_as_string` serializes the keymap as a long string.
|
||||||
|
5. `Keyboard.sendKeymap(resource)` is called for each `wl_keyboard`
|
||||||
|
resource:
|
||||||
|
- `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a
|
||||||
|
`/mkstemp-mmap` WebSocket, sends the blob, and receives a `ProxyFD {
|
||||||
|
handle: <native_fd>, type: 'shm', host: baseURL }`.
|
||||||
|
- `keyboardResource.keymap(xkbV1, ProxyFD, size)` is sent to the
|
||||||
|
native client.
|
||||||
|
6. The proxy deserializes the ProxyFD, looks up the native fd via
|
||||||
|
`proxyFDtoNativeFD` (identity when `host === baseURL`), ships it via
|
||||||
|
`sendEvents(wlClient, msg, fds)` from the N-API addon —
|
||||||
|
`libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain
|
||||||
|
socket.
|
||||||
|
7. The native client's libwayland receives the keymap fd, mmaps it, and
|
||||||
|
calls native `xkb_keymap_new_from_string` to compile its own copy.
|
||||||
|
|
||||||
|
So the keymap flows **browser → proxy → native**, not the reverse. This
|
||||||
|
is the opposite of what you might expect (since the proxy is on the Linux
|
||||||
|
host where xkbcommon natively lives), but it makes sense: the browser is
|
||||||
|
the actual compositor and is the one that needs the keymap to translate
|
||||||
|
DOM key events to Wayland keycodes.
|
||||||
|
|
||||||
|
### DOM → Wayland key event translation
|
||||||
|
|
||||||
|
Browser DOM `KeyboardEvent` has `event.code` (e.g. `"KeyA"`) and
|
||||||
|
`event.key` (the resolved glyph). Greenfield uses **only the physical
|
||||||
|
`.code`**, mirroring evdev:
|
||||||
|
|
||||||
|
1. `event.code` is looked up in `EvDevKeyCode` reverse-name-map → evdev
|
||||||
|
keycode (e.g. `30` for `KeyA`).
|
||||||
|
2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset).
|
||||||
|
3. `capsLock` / `numLock` are read from `event.getModifierState(...)`.
|
||||||
|
|
||||||
|
The DOM `.key` glyph is **ignored** — only physical `.code` is used, so
|
||||||
|
the same physical key always produces the same evdev keycode regardless
|
||||||
|
of layout. This is correct Wayland behavior.
|
||||||
|
|
||||||
|
### Modifier tracking
|
||||||
|
|
||||||
|
Modifier state is tracked entirely via the WASM libxkbcommon state
|
||||||
|
machine:
|
||||||
|
|
||||||
|
1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or
|
||||||
|
`xkb.keyUp(x11)`.
|
||||||
|
2. Internally, `_xkb_state_update_key` is invoked.
|
||||||
|
3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` /
|
||||||
|
`modsLocked` / `group` (each a `_xkb_state_serialize_mods` call).
|
||||||
|
4. High-level `KeyboardModifier` bits are computed by checking xkb
|
||||||
|
mod-indices resolved at keymap-compile time.
|
||||||
|
5. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)`
|
||||||
|
sends the `wl_keyboard.modifiers` event to the native client.
|
||||||
|
|
||||||
|
### Notable bug
|
||||||
|
|
||||||
|
`Seat.ts` lines 511-513 contain a copy/paste typo:
|
||||||
|
`modsLatched = modsLocked` (should be `= modsLatched`). The cached
|
||||||
|
`keyboard.modifiers` struct is unreliable but doesn't affect protocol
|
||||||
|
output because `notifyModifiers` reads from `xkb` directly each time.
|
||||||
|
|
||||||
|
## Pointer
|
||||||
|
|
||||||
|
### DOM → Wayland pointer events
|
||||||
|
|
||||||
|
Browser DOM `MouseEvent` is converted to Greenfield's internal
|
||||||
|
`ButtonEvent` / `AxisEvent`:
|
||||||
|
|
||||||
|
- `pointermove` → `createButtonEventFromMouseEvent` → `ButtonEvent`
|
||||||
|
(motion-only)
|
||||||
|
- `pointerdown` / `pointerup` → `ButtonEvent` (with button + state)
|
||||||
|
- `wheel` → `createAxisEventFromWheelEvent` → `AxisEvent`
|
||||||
|
|
||||||
|
### Pointer enter / leave between surfaces
|
||||||
|
|
||||||
|
`DefaultPointerGrab.focus()` runs on every motion event:
|
||||||
|
|
||||||
|
1. `Renderer.pickView` walks the view stack front-to-back.
|
||||||
|
2. Returns the first view whose `Surface.isWithinInputRegion(point)`
|
||||||
|
(pixman-WASM region) is true.
|
||||||
|
3. On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to
|
||||||
|
the old surface, then `wl_pointer.enter` + `frame` (+ modifiers if
|
||||||
|
keyboard is grabbed) to the new surface.
|
||||||
|
|
||||||
|
### Pointer capture / window leave
|
||||||
|
|
||||||
|
- `canvas.setPointerCapture` is called on `pointerdown`.
|
||||||
|
- `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus,
|
||||||
|
releases depressed keys, sets `needFocusInit=true` for re-sync),
|
||||||
|
resets `buttonCount=0`.
|
||||||
|
- DnD suppresses this handler mid-drag.
|
||||||
|
|
||||||
|
### Pointer acceleration / cursor images
|
||||||
|
|
||||||
|
**No acceleration** — verbatim `offsetX/Y`.
|
||||||
|
|
||||||
|
Cursor images are **CSS `cursor: url(data:...) x y, auto`**:
|
||||||
|
|
||||||
|
1. Native client calls `wl_pointer.set_cursor`.
|
||||||
|
2. The cursor surface commits a buffer.
|
||||||
|
3. `CursorRole` draws the bitmap to a hidden canvas.
|
||||||
|
4. `toDataURL()` produces a data URL.
|
||||||
|
5. Set on `document.body.style.cursor`.
|
||||||
|
|
||||||
|
OS composites the cursor. No rotation/scale transforms.
|
||||||
|
|
||||||
|
### Scroll wheel / touchpad
|
||||||
|
|
||||||
|
`WheelEvent.deltaMode`:
|
||||||
|
|
||||||
|
- `DOM_DELTA_LINE × 12` — discrete wheel clicks
|
||||||
|
- `DOM_DELTA_PAGE × surface size` — page-sized scrolls
|
||||||
|
- `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad
|
||||||
|
|
||||||
|
Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version
|
||||||
|
≥5 also gets `axis_source(wheel)` + `frame()` (groups events).
|
||||||
|
|
||||||
|
No `axis_discrete` / `value120` / `axis_stop` events are sent.
|
||||||
|
|
||||||
|
### Linux button codes
|
||||||
|
|
||||||
|
DOM `event.button` (0=left, 1=middle, 2=right, 3=back, 4=forward) is
|
||||||
|
mapped to Linux input protocol button codes via the `linuxInput` table
|
||||||
|
in `browser/input.ts`.
|
||||||
|
|
||||||
|
## Touch — declared but inert
|
||||||
|
|
||||||
|
`Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners.
|
||||||
|
`wl_touch` capability is advertised if `'ontouchstart' in
|
||||||
|
document.documentElement`, but no events are ever sent. Touch input flows
|
||||||
|
through Pointer Events as mouse input.
|
||||||
|
|
||||||
|
**For the user**: if you need real touch support (e.g. for tablets), this
|
||||||
|
is a feature you'll have to implement from scratch. The protocol stubs
|
||||||
|
exist (via generated `wl_touch` bindings), but no event source populates
|
||||||
|
them.
|
||||||
|
|
||||||
|
## Focus management
|
||||||
|
|
||||||
|
### DOM focus
|
||||||
|
|
||||||
|
The `<canvas>` owns keyboard focus permanently (`tabIndex=1`).
|
||||||
|
`Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between
|
||||||
|
native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface`
|
||||||
|
activation handlers.
|
||||||
|
|
||||||
|
### DOM → Wayland focus
|
||||||
|
|
||||||
|
- Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores
|
||||||
|
saved focus, re-syncs modifiers).
|
||||||
|
- Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves
|
||||||
|
focus, releases depressed keys, sets `needFocusInit=true`).
|
||||||
|
|
||||||
|
### Surface-level focus
|
||||||
|
|
||||||
|
- Click on a surface → `XdgToplevel` activation request →
|
||||||
|
`desktopSurface.activate()` → `gainFocus()` (refcounted via
|
||||||
|
`focusCount`, fires `surfaceActivationUpdated` event to the user shell
|
||||||
|
on 0↔1 transitions).
|
||||||
|
- Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`,
|
||||||
|
validating the serial against the keyboard/pointer `grabSerial`.
|
||||||
|
|
||||||
|
## Input queue
|
||||||
|
|
||||||
|
`InputQueue.ts` coalesces events and dispatches them on the next RAF
|
||||||
|
(`InputQueue.drainQueues`). This batches bursts of input events (e.g. fast
|
||||||
|
mouse motion) into a single dispatch, avoiding redundant work.
|
||||||
|
|
||||||
|
The queue is FIFO per event type (motion, axis, key, button). Each
|
||||||
|
dispatch calls the appropriate `seat.notify*` method.
|
||||||
|
|
||||||
|
## ASCII event-flow diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ BROWSER (DOM) │
|
||||||
|
│ │
|
||||||
|
│ <canvas tabIndex=1> ─┐ │
|
||||||
|
│ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │
|
||||||
|
│ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │
|
||||||
|
│ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │
|
||||||
|
│ │ • evdevKeyCode = EvDevKeyCode[event.code] │
|
||||||
|
│ │ • x11KeyCode = evdevKeyCode + 8 │
|
||||||
|
│ │ • capsLock/numLock = event.getModifierState(...) │
|
||||||
|
│ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │
|
||||||
|
│ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │
|
||||||
|
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||||||
|
│ session.inputQueue.queue{Motion,Axis,Key,Button}
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │
|
||||||
|
│ │
|
||||||
|
│ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame │
|
||||||
|
│ │ │
|
||||||
|
│ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │
|
||||||
|
│ │ └─ Keyboard.sendKey(event) │
|
||||||
|
│ │ └─ keyboardResource.key(serial,time,evdev,st) │
|
||||||
|
│ │ │
|
||||||
|
│ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │
|
||||||
|
│ │ └─ Pointer.sendButton │
|
||||||
|
│ │ └─ pointerResource.button(serial,time, │
|
||||||
|
│ │ linuxInput[btn], state) │
|
||||||
|
│ │ │
|
||||||
|
│ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │
|
||||||
|
│ │ └─ pointerResource.motion(time, sx_f, sy_f) │
|
||||||
|
│ │ │
|
||||||
|
│ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │
|
||||||
|
│ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │
|
||||||
|
│ │ └─ pointerResource.axis(time, axis, Fixed) │
|
||||||
|
│ │ + axisSource(wheel) + frame() if v>=5 │
|
||||||
|
│ │ │
|
||||||
|
│ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │
|
||||||
|
│ │ │
|
||||||
|
│ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │
|
||||||
|
│ └─ Keyboard.sendModifiers(modsDepressed,Latched, │
|
||||||
|
│ Locked,group) │
|
||||||
|
│ │
|
||||||
|
│ Keymap path (once per WlKeyboardResource): │
|
||||||
|
│ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │
|
||||||
|
│ → xkb_context_new + xkb_keymap_new_from_names (libxkbcommon WASM) │
|
||||||
|
│ → xkb_keymap_get_as_string ──► keymapString │
|
||||||
|
│ Keyboard.sendKeymap(resource): │
|
||||||
|
│ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │
|
||||||
|
│ keyboardResource.keymap(xkbV1, ProxyFD, size) │
|
||||||
|
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||||||
|
│ client.connection.onFlush(wireMessages)
|
||||||
|
│ ─ one big Uint32Array, FDs serialized as ProxyFD JSON
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ WIRE (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts) │
|
||||||
|
│ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ ws on proxy :SessionController │
|
||||||
|
│ Message layout (Uint32Array): │
|
||||||
|
│ [0] outOfBandOpcode (0 = in-band) │
|
||||||
|
│ [1] eventSerial │
|
||||||
|
│ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │
|
||||||
|
│ then concatenated Wayland wire messages │
|
||||||
|
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||||||
|
│ NativeWaylandClientSession.onMessage
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ PROXY (Node.js, compositor-proxy) │
|
||||||
|
│ │
|
||||||
|
│ SessionController wss paths: /mkfifo /mkstemp-mmap /read-fd /close-fd │
|
||||||
|
│ /read-fd-as-stream /write-fd-as-stream │
|
||||||
|
│ /signal /channel /request-keyframe │
|
||||||
|
│ │
|
||||||
|
│ mkstempMmap(ws): │
|
||||||
|
│ receives Blob → Buffer.concat → createMemoryMappedFile(buf) ← native addon │
|
||||||
|
│ returns ProxyFD { handle: <native fd>, type: 'shm', host: baseURL } │
|
||||||
|
│ │
|
||||||
|
│ onWireMessageEvents: │
|
||||||
|
│ for each ProxyFD in message: │
|
||||||
|
│ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) // just returns .handle │
|
||||||
|
│ └─ if proxyFD.host === baseURL → identity │
|
||||||
|
│ └─ if foreign host → spawn a bridging pipe + WebSocket │
|
||||||
|
│ messageInterceptor.interceptEvent(...) // empty stubs for kbd/ptr/touch │
|
||||||
|
│ sendEvents(wlClient, messageBuffer, fdsBuffer) ← libwayland-server addon │
|
||||||
|
│ flush(wlClient) │
|
||||||
|
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||||||
|
│ libwayland-server sends wire msg + SCM_RIGHTS FDs
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ NATIVE (Wayland client e.g. weston-terminal, Xwayland app) │
|
||||||
|
│ wl_display.fd readable → libwayland-client demarshals │
|
||||||
|
│ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │
|
||||||
|
│ wl_keyboard.key / enter / leave / modifiers → client app │
|
||||||
|
│ wl_pointer.enter / motion / button / axis / frame → client app │
|
||||||
|
└──────────────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key answers (quick reference)
|
||||||
|
|
||||||
|
| Question | Answer |
|
||||||
|
|---|---|
|
||||||
|
| DOM→Wayland key event translation | `event.code` (e.g. `"KeyA"`) is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30`). `x11KeyCode = evdevKeyCode + 8`. DOM `.key` glyph is **ignored** — only physical `.code` is used, mirroring evdev. |
|
||||||
|
| Modifier tracking | Entirely via WASM libxkbcommon state machine. `Seat.updateModifierState` calls `xkb.keyDown/Up(x11)` → `_xkb_state_update_key`, then `notifyModifiers` reads `xkb.modsDepressed/Latched/Locked/group` (each a `_xkb_state_serialize_mods` call). High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. |
|
||||||
|
| Focus: DOM ↔ native surfaces | The `<canvas>` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave/enter` between native surfaces. Activation is driven by `XdgToplevel`/`ShellSurface` activation handlers. |
|
||||||
|
| Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` calls `notifyKeyboardFocusOut` (saves kbd focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. DnD suppresses this handler mid-drag. |
|
||||||
|
| Touch event mapping | **Not implemented.** `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. |
|
||||||
|
| Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView` walks the view stack front-to-back, returns first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. On change, `Pointer.setFocus` emits `leave+frame` to old surface, `enter+frame` (+modifiers) to new. |
|
||||||
|
| Pointer acceleration / cursor images | **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: client calls `wl_pointer.set_cursor` → `CursorRole` → draw bitmap to hidden canvas → `toDataURL()` → set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. |
|
||||||
|
| Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: `DOM_DELTA_LINE × 12`, `DOM_DELTA_PAGE × surface size`, `DOM_DELTA_PIXEL ÷ 3`. Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. v≥5 also gets `axis_source(wheel)` + `frame()`. No `axis_discrete/value120/stop`. |
|
||||||
|
| WASM xkbcommon? | Yes. `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch, builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. Sibling build does the same for `pixman`. |
|
||||||
|
| Keymap FD workaround for WebSocket | Direction is **browser → proxy**, not the reverse. Browser calls `inputOutput.mkstempMmap(Blob)` → opens `/mkstemp-mmap` WebSocket → proxy creates real native shm fd via `createMemoryMappedFile` addon → returns `ProxyFD {handle: <native_fd>, type: 'shm', host: baseURL}`. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`; the ProxyFD is **serialized as JSON** in the wire message's `fds[]` array (WebSockets can't pass FDs natively). Proxy deserializes, maps `handle` → native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. |
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/Seat.ts` | `wl_seat` global, multi-seat support, focus management |
|
||||||
|
| `packages/compositor/src/Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events |
|
||||||
|
| `packages/compositor/src/Pointer.ts` | `wl_pointer` impl, motion, buttons, axis |
|
||||||
|
| `packages/compositor/src/Touch.ts` | `wl_touch` (46-line stub, inert) |
|
||||||
|
| `packages/compositor/src/Xkb.ts` | xkbcommon WASM wrapper |
|
||||||
|
| `packages/compositor/src/KeyEvent.ts` | Key event wrapper |
|
||||||
|
| `packages/compositor/src/ButtonEvent.ts` | Pointer button event wrapper |
|
||||||
|
| `packages/compositor/src/AxisEvent.ts` | Pointer axis (scroll) event wrapper |
|
||||||
|
| `packages/compositor/src/InputQueue.ts` | Event coalescing + RAF dispatch |
|
||||||
|
| `packages/compositor/src/InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) |
|
||||||
|
| `packages/compositor/src/browser/input.ts` | DOM event listeners → Greenfield events |
|
||||||
|
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer |
|
||||||
|
| `libs/compositor-wasm/` | pixman + xkbcommon WASM build |
|
||||||
|
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||||||
|
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||||||
|
| `packages/compositor-proxy/src/io/` | WebSocket server glue (mkstemp-mmap endpoint etc.) |
|
||||||
@@ -0,0 +1,314 @@
|
|||||||
|
# 06 — Data Device, Selection, DnD (`wl_data_device`)
|
||||||
|
|
||||||
|
This chapter covers Greenfield's data device / clipboard / drag-and-drop
|
||||||
|
implementation: how `wl_data_device_manager`, `wl_data_source`,
|
||||||
|
`wl_data_offer`, `wl_data_device` are implemented, how clipboard contents
|
||||||
|
are transferred between browser and native apps, and how DnD is coordinated.
|
||||||
|
|
||||||
|
## Faithful Wayland model
|
||||||
|
|
||||||
|
`DataDeviceManager` is a v3 global singleton. `wl_data_device` is per
|
||||||
|
seat-per-client and the implementation lives directly on `Seat` (which
|
||||||
|
implements `WlDataDeviceRequests`). `WaylandDataSource` / `DataOffer` are
|
||||||
|
the standard source/offer pair, ported almost verbatim from weston,
|
||||||
|
including the `chooseAction()` bit-trick for DnD action negotiation.
|
||||||
|
|
||||||
|
## The FD-over-WebSocket problem
|
||||||
|
|
||||||
|
Wayland's `wl_data_offer.receive(mimeType, fd)` /
|
||||||
|
`wl_data_source.send(mimeType, fd)` carry a real UNIX fd. Greenfield solves
|
||||||
|
this with a **`ProxyFD` JSON descriptor**:
|
||||||
|
|
||||||
|
```
|
||||||
|
ProxyFD = {
|
||||||
|
handle: number,
|
||||||
|
type: 'pipe-read' | 'pipe-write' | 'shm' | 'unknown',
|
||||||
|
host: string // proxy's HTTP base URL
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Three concrete `InputOutput` / `InputOutputFD` implementations:
|
||||||
|
|
||||||
|
### `webInputOutput` — for in-browser web apps
|
||||||
|
|
||||||
|
Pipes are `MessageChannel` ports, transferred as `ArrayBuffer`s. All
|
||||||
|
"fd" operations are in-browser zero-copy transfers.
|
||||||
|
|
||||||
|
### `RemoteInputOutput` — for native apps
|
||||||
|
|
||||||
|
All fd operations go over a secondary WebSocket to endpoints:
|
||||||
|
`/mkfifo`, `/mkstemp-mmap`, `/read-fd-as-stream`, `/write-fd-as-stream`,
|
||||||
|
`/read-fd`, `/close-fd`.
|
||||||
|
|
||||||
|
### `ProxyInputOutput` (proxy-side) — owns the real OS pipes and memfd files
|
||||||
|
|
||||||
|
`proxyFDtoNativeFD(proxyFD)` returns the raw int fd because every ProxyFD
|
||||||
|
has `host === proxy.baseURL`.
|
||||||
|
|
||||||
|
The proxy interceptors for `wl_data_source` / `wl_data_offer` /
|
||||||
|
`wl_data_device` / `wl_data_device_manager` are auto-generated no-ops that
|
||||||
|
only handle resource construction and lifetime — all actual semantics live
|
||||||
|
in the browser compositor.
|
||||||
|
|
||||||
|
### Crucial consequence
|
||||||
|
|
||||||
|
For native→native clipboard transfers (e.g. user copies in one Linux app,
|
||||||
|
pastes in another), the browser compositor never sees a single byte of the
|
||||||
|
data — it only relays the fd reference. The actual content stays inside
|
||||||
|
the proxy host's OS kernel pipe.
|
||||||
|
|
||||||
|
## MIME-type negotiation
|
||||||
|
|
||||||
|
At the Wayland level, mime types are just relayed strings. Mapping to the
|
||||||
|
browser's hard-coded allow-list `['text/plain', 'text/html', 'image/png']`
|
||||||
|
happens only at the DOM boundary:
|
||||||
|
|
||||||
|
### Native → browser
|
||||||
|
|
||||||
|
`selection.ts:handleWaylandDataSourceUpdate`: each source mime type is
|
||||||
|
substring-matched against the allow-list (`text/plain;charset=utf-8`
|
||||||
|
contains `text/plain`, so it maps), survivors are pulled as separate blobs
|
||||||
|
via `mkfifo + send + readBlob`, and finally
|
||||||
|
`navigator.clipboard.write([new ClipboardItem(...)])`.
|
||||||
|
|
||||||
|
### Browser → native
|
||||||
|
|
||||||
|
`BrowserDataSource.ts`: `ClipboardItem.types` become the source mime types,
|
||||||
|
plus a synthetic `text/plain;charset=utf-8` alias is appended (Linux apps
|
||||||
|
expect the charset form). On `send`, the alias is normalised back to
|
||||||
|
`text/plain` for the lookup.
|
||||||
|
|
||||||
|
### Caveat: substring matching is fragile
|
||||||
|
|
||||||
|
The substring match `dataSourceMimeType.indexOf(browserMimeType) !== -1`
|
||||||
|
can collide (`image/png` matching `image/png;base64` is fine, but
|
||||||
|
`text/plain;charset=iso-8859-1` silently becomes `text/plain`). Consider
|
||||||
|
an explicit prefix or exact-match policy in your implementation.
|
||||||
|
|
||||||
|
## Clipboard mirroring is pull-based
|
||||||
|
|
||||||
|
The browser has no `clipboardchange` event (Chromium issue 933608 is
|
||||||
|
cited in the code). Greenfield polls `navigator.clipboard.read()`:
|
||||||
|
|
||||||
|
- on init
|
||||||
|
- on every `window.focus` event
|
||||||
|
|
||||||
|
De-duplicates offers by mime-type set + total blob size, then calls
|
||||||
|
`seat.setSelectionInternal(createBrowserDataSource(offers), serial)`.
|
||||||
|
|
||||||
|
The feedback loop is broken because `handleWaylandDataSourceUpdate` returns
|
||||||
|
early when `dataSource instanceof BrowserDataSource`.
|
||||||
|
|
||||||
|
### Firefox fallback
|
||||||
|
|
||||||
|
`text-selection.ts` uses only `readText` / `writeText` and a
|
||||||
|
`BrowserTextDataSource` that serves `text/plain;charset=utf-8` — no images
|
||||||
|
or HTML.
|
||||||
|
|
||||||
|
## Drag-and-drop
|
||||||
|
|
||||||
|
### Wayland-side (Seat.startDrag)
|
||||||
|
|
||||||
|
1. Validates the implicit pointer grab.
|
||||||
|
2. Attaches `DragIconRole` to the optional icon surface.
|
||||||
|
3. Installs a `PointerDrag` grab on both pointer and keyboard.
|
||||||
|
|
||||||
|
`PointerDrag` tracks the focused data device, sends
|
||||||
|
`wl_data_device.enter` / `leave` / `motion` / `drop`, creates fresh
|
||||||
|
`wl_data_offer`s via `sendOffer`, and runs the v3 action negotiation.
|
||||||
|
|
||||||
|
### Browser-side (`BrowserDnD` class)
|
||||||
|
|
||||||
|
Creates a hidden full-canvas `dragHelper` div with `draggable=true` when
|
||||||
|
the Wayland drag-icon surface commits, so the browser shows the Wayland icon
|
||||||
|
via `event.dataTransfer.setDragImage(cursorCanvas, ...)`. While the user
|
||||||
|
drags, `dragover` events on the dragHelper are converted back to
|
||||||
|
`ButtonEvent`s and pushed through `session.inputQueue.queueMotion`, keeping
|
||||||
|
`PointerDrag` fed.
|
||||||
|
|
||||||
|
### Not implemented: dragging data FROM a Wayland app INTO a real DOM target
|
||||||
|
|
||||||
|
The FIXME in `dnd.ts` admits: dragging data FROM a Wayland app INTO a real
|
||||||
|
DOM drop target (file drag-out, etc.) is **not implemented**.
|
||||||
|
|
||||||
|
## XWayland bridge
|
||||||
|
|
||||||
|
`XWindowManager` listens on the X11 `CLIPBOARD` atom via `XFixes` and
|
||||||
|
builds `XDataSource` (clipboard) or `XDnDDataSource` (XDnD). When the
|
||||||
|
receiver pulls data:
|
||||||
|
|
||||||
|
1. `XDataSource.send` calls `XConvertSelection` to have the X server push
|
||||||
|
the bytes into a property on `selectionWindow`.
|
||||||
|
2. `writeProperty` copies them into the receiver's `InputOutputFD`.
|
||||||
|
3. INCR transfers are handled (`getIncrChunk`, `sendIncrChunk`).
|
||||||
|
|
||||||
|
## Primary selection is NOT implemented
|
||||||
|
|
||||||
|
A grep for `primary_selection`, `gtk_primary`, `zwp_primary_selection`
|
||||||
|
across the entire `packages/` tree returns zero hits. XWayland only
|
||||||
|
watches the `CLIPBOARD` atom, not `PRIMARY`. **Middle-click paste does
|
||||||
|
not work in Greenfield today.**
|
||||||
|
|
||||||
|
**For the user**: if your users expect Linux-style middle-click paste,
|
||||||
|
you'll need to implement `zwp_primary_selection_device_manager_v1` (or
|
||||||
|
`gtk_primary_selection`) on the compositor side and forward `PRIMARY`
|
||||||
|
notifications in `XWindowManager`.
|
||||||
|
|
||||||
|
## ASCII diagram — clipboard flow in both directions
|
||||||
|
|
||||||
|
```
|
||||||
|
CLIPBOARD / SELECTION FLOW
|
||||||
|
|
||||||
|
=================================================================
|
||||||
|
A) NATIVE APP COPIES ──► BROWSER DOM CLIPBOARD (Ctrl+C in Linux app)
|
||||||
|
=================================================================
|
||||||
|
|
||||||
|
┌──────────────┐ wl_data_device.set_selection(src,serial) ┌────────────────────────┐
|
||||||
|
│ Native app │ ─────────────────────────────────────────────► │ compositor │
|
||||||
|
│ (Wayland or │ │ (browser, in DOM) │
|
||||||
|
│ XWayland) │ ◄────── wl_data_offer.offer("text/plain") ─── │ │
|
||||||
|
└──────────────┘ (only if this app has keyboard focus)└───────────┬────────────┘
|
||||||
|
│ selectionListeners fire
|
||||||
|
▼
|
||||||
|
┌──────────────────────────┐
|
||||||
|
│ handleWaylandDataSource │
|
||||||
|
│ Update() (selection.ts)│
|
||||||
|
└──────────┬──────────────┘
|
||||||
|
mkfifo() ──────────────────────────► │ proxy /mkfifo
|
||||||
|
ProxyFD read+write (real OS pipe) ◄───── returns 2 ProxyFDs
|
||||||
|
▼
|
||||||
|
dataSource.send("text/plain", writeFD)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
wl_data_source.send("text/plain", writeFDProxyFD)
|
||||||
|
─────────────────────────────────────────► │ Native source
|
||||||
|
│ receives real write fd
|
||||||
|
│ (proxyFDtoNativeFD
|
||||||
|
│ is a passthrough)
|
||||||
|
▼
|
||||||
|
Native app writes bytes
|
||||||
|
──────────────► OS pipe ───┐
|
||||||
|
│
|
||||||
|
readFD.readBlob() ◄──────────── read from /read-fd-as-stream ◄─┘
|
||||||
|
▼ (proxy streams over WS)
|
||||||
|
Blob {type:"text/plain"}
|
||||||
|
▼
|
||||||
|
navigator.clipboard.write([ new ClipboardItem({"text/plain": blob}) ])
|
||||||
|
▼
|
||||||
|
┌────────────────────────────┐
|
||||||
|
│ DOM clipboard populated │
|
||||||
|
│ (Ctrl+V anywhere now works) │
|
||||||
|
└────────────────────────────┘
|
||||||
|
|
||||||
|
|
||||||
|
=================================================================
|
||||||
|
B) USER COPIES IN BROWSER ──► NATIVE APP GETS SELECTION (Ctrl+C in DOM)
|
||||||
|
=================================================================
|
||||||
|
|
||||||
|
┌─────────────────────────────┐
|
||||||
|
│ User copies in DOM │
|
||||||
|
│ (browser tab, textarea...) │
|
||||||
|
└──────────────┬──────────────┘
|
||||||
|
│ on window 'focus' event (or initial load)
|
||||||
|
▼
|
||||||
|
navigator.clipboard.read() ──► ClipboardItems[]
|
||||||
|
▼
|
||||||
|
handleBrowserDataSourceUpdate() — de-dup vs previous offers
|
||||||
|
▼
|
||||||
|
createBrowserDataSource(offers)
|
||||||
|
.mimeTypes = offer.types + ['text/plain;charset=utf-8'] alias
|
||||||
|
▼
|
||||||
|
seat.setSelectionInternal(browserDataSource, serial)
|
||||||
|
│
|
||||||
|
│ 1) selectionListeners fire — handleWaylandDataSourceUpdate
|
||||||
|
│ returns early (instanceof BrowserDataSource), breaking loop
|
||||||
|
│
|
||||||
|
│ 2) sendSelection(focusedClient)
|
||||||
|
▼
|
||||||
|
┌────────────────────────┐ wl_data_offer.offer("text/plain"), ┌──────────────┐
|
||||||
|
│ compositor │ ─ wl_data_device.selection(offer) ──► │ Native app │
|
||||||
|
│ (browser, in DOM) │ │ (focused) │
|
||||||
|
└───────────┬────────────┘ └──────┬───────┘
|
||||||
|
│ │
|
||||||
|
│ When the native app calls wl_data_offer.receive("text/plain", fd):
|
||||||
|
│ │
|
||||||
|
│ ◄──────── wl_data_offer.receive(mimeType, writeFDProxyFD) ◄── native pipe
|
||||||
|
│ (fd was created by the native app's libwayland; proxy
|
||||||
|
│ wrapped it as ProxyFD{host:proxy.baseURL, type:'unknown'}
|
||||||
|
│ then compositor's DataOffer.receive calls wrapFD → RemoteInputOutputFD)
|
||||||
|
▼
|
||||||
|
BrowserDataSource.send(mimeType, ioFD)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
matchingOffer.getType(mimeType) ─► Promise<Blob> (from ClipboardItem)
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
ioFD.write(blob) ─► POST /write-fd-as-stream?fd=<handle> ─► proxy
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
createWriteStream({fd, autoClose:true})
|
||||||
|
ws stream ──► native write fd
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Bytes flow through OS pipe into native app
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
ioFD.close() ─► /close-fd ─► EOF ─► native app's read returns 0
|
||||||
|
|
||||||
|
|
||||||
|
=================================================================
|
||||||
|
PROXY FD TRANSLATION (works the same way in B and A)
|
||||||
|
=================================================================
|
||||||
|
|
||||||
|
Native fd (in proxy process)
|
||||||
|
│
|
||||||
|
│ wrap into ProxyFD JSON: {handle, type, host: proxy.baseURL}
|
||||||
|
│ JSON-encode, length-prefix, send over WS protocol channel
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────┐
|
||||||
|
│ Browser compositor │
|
||||||
|
│ client.userData.inputOutput.wrapFD(fd, type) │
|
||||||
|
│ → RemoteInputOutputFD (RemoteInputOutput) │
|
||||||
|
│ → WebPipeInputOutputFD (webInputOutput, │
|
||||||
|
│ for in-browser apps)│
|
||||||
|
└─────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/DataDeviceManager.ts` | `wl_data_device_manager` global |
|
||||||
|
| `packages/compositor/src/DataSource.ts` | `wl_data_source` impl, the provider side |
|
||||||
|
| `packages/compositor/src/DataOffer.ts` | `wl_data_offer` impl, the receiver side |
|
||||||
|
| `packages/compositor/src/WaylandDataSource.ts` | Wrapper around native `wl_data_source` events |
|
||||||
|
| `packages/compositor/src/Seat.ts` | `wl_data_device` impl (Seat implements `WlDataDeviceRequests`) |
|
||||||
|
| `packages/compositor/src/browser/selection.ts` | Browser clipboard bridge (native → DOM) |
|
||||||
|
| `packages/compositor/src/browser/text-selection.ts` | Firefox fallback (text-only) |
|
||||||
|
| `packages/compositor/src/browser/dnd.ts` | Browser DnD bridge (Wayland → DOM) |
|
||||||
|
| `packages/compositor/src/InputOutput.ts` | FD abstraction interface |
|
||||||
|
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||||||
|
| `packages/compositor/src/remote/XWindowManager.ts` | X11 CLIPBOARD atom watcher |
|
||||||
|
| `packages/compositor/src/remote/XDataSource.ts` | X11 clipboard source (XWayland apps) |
|
||||||
|
| `packages/compositor/src/remote/XDnDDataSource.ts` | X11 DnD source |
|
||||||
|
|
||||||
|
## Next actions / suggested improvements
|
||||||
|
|
||||||
|
1. **Primary selection** — implement `zwp_primary_selection_device_manager`
|
||||||
|
v1 (or `gtk_primary_selection`) on the compositor side and forward
|
||||||
|
`PRIMARY` notifications in `XWindowManager`. This unlocks middle-click
|
||||||
|
paste from native apps.
|
||||||
|
2. **`clipboardchange` event** — when browsers ship it, replace the
|
||||||
|
`window.focus`-based polling in `selection.ts` with a proper event-driven
|
||||||
|
subscription.
|
||||||
|
3. **Native→DOM DnD** — implement the dataTransfer side of `BrowserDnD`
|
||||||
|
so a Wayland drag can deliver bytes to a real DOM drop target (the
|
||||||
|
`dnd.ts` FIXME).
|
||||||
|
4. **MIME allow-list discovery** — replace the hard-coded
|
||||||
|
`['text/plain', 'text/html', 'image/png']` with a runtime check once
|
||||||
|
browsers expose the supported types.
|
||||||
|
5. **MIME-type strict matching** — the substring match
|
||||||
|
`dataSourceMimeType.indexOf(browserMimeType) !== -1` can collide
|
||||||
|
(`image/png` matching `image/png;base64` is fine, but
|
||||||
|
`text/plain;charset=iso-8859-1` silently becomes `text/plain`).
|
||||||
|
Consider an explicit prefix or exact-match policy.
|
||||||
@@ -0,0 +1,404 @@
|
|||||||
|
# 07 — Native App Bridge (How Real Linux Apps Get Connected)
|
||||||
|
|
||||||
|
This chapter covers Greenfield's native app bridge: how real Linux
|
||||||
|
applications get launched, connected to the proxy, and routed to the
|
||||||
|
browser. This is the bridge that makes a Linux Wayland app appear inside
|
||||||
|
the browser. **This is the most important chapter for the user** — they
|
||||||
|
want to run real Linux apps in their Odoo 20 web app.
|
||||||
|
|
||||||
|
## Two app models — don't conflate them
|
||||||
|
|
||||||
|
Greenfield has **two app models**:
|
||||||
|
|
||||||
|
### Model 1: WASM web apps (browser-resident)
|
||||||
|
|
||||||
|
Built with the Greenfield SDK (Emscripten), run inside the browser via
|
||||||
|
`MessageChannel`. The `sdk/` directory serves only this model. Apps are
|
||||||
|
compiled to WASM, run in the same browser tab as the compositor, and
|
||||||
|
speak Wayland over a `MessagePort` instead of a UNIX socket.
|
||||||
|
|
||||||
|
### Model 2: Native Linux apps (the focus of this chapter)
|
||||||
|
|
||||||
|
Real ELF binaries (`gtk4-demo`, `kwrite`, `xterm`) running as child
|
||||||
|
processes of the Node.js proxy. Apps use the host's own system libraries
|
||||||
|
— **no sysroot, no chroot, no container**.
|
||||||
|
|
||||||
|
## Process model — three layers
|
||||||
|
|
||||||
|
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
|
||||||
|
|
||||||
|
A single Node.js process listens on `:8081`. Per browser session, it
|
||||||
|
`fork()`s one `SessionProcess` child — **one child per `compositorSessionId`**,
|
||||||
|
not per app. Multiple apps in the same browser tab share one child.
|
||||||
|
WebSocket upgrades are handed off to the matching child via
|
||||||
|
`childProcess.send(wsUpgrade, socket)`.
|
||||||
|
|
||||||
|
CLI args:
|
||||||
|
|
||||||
|
- `--bind-port=8081` (default)
|
||||||
|
- `--applications=<json>` (URL → executable map)
|
||||||
|
- `--encoder=x264|nvh264|vaapih264`
|
||||||
|
- `--render-device=/dev/dri/renderD128`
|
||||||
|
- `--base-url`, `--allow-origin`, `--bind-ip`, `--basic-auth`
|
||||||
|
|
||||||
|
### Layer 2: Session Process (`SessionProcess.ts`)
|
||||||
|
|
||||||
|
The forked child. Receives IPC messages:
|
||||||
|
|
||||||
|
- `start` — creates `Session` + `SessionController`
|
||||||
|
- `launchApp` — spawns the actual app binary
|
||||||
|
- `wsUpgrade` — hands WS to the controller
|
||||||
|
|
||||||
|
On `launchApp` it calls `launchApplication()` from `@gfld/compositor-proxy`,
|
||||||
|
which `spawn()`s the executable, captures `pid`, replies to main with
|
||||||
|
`{pid, key}`. On `session.close()` the process exits.
|
||||||
|
|
||||||
|
### Layer 3: Proxy Core (`compositor-proxy/src/`)
|
||||||
|
|
||||||
|
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
|
||||||
|
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
|
||||||
|
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
|
||||||
|
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
|
||||||
|
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
|
||||||
|
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
|
||||||
|
wl_display FD into Node's libuv loop via `startPoll()`.
|
||||||
|
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
|
||||||
|
Uses `MessageInterceptor` + auto-generated per-resource interceptor
|
||||||
|
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
|
||||||
|
`{native, browser, neverReplies?}` routing decision. Handles
|
||||||
|
`wl_display.sync` fast-path (skips browser round-trip when no reply
|
||||||
|
expected). Marshalls FDs as `ProxyFD` JSON.
|
||||||
|
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
|
||||||
|
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
|
||||||
|
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
|
||||||
|
(10 s) on force-quit, auto-destroy on app exit.
|
||||||
|
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
|
||||||
|
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
|
||||||
|
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
|
||||||
|
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
|
||||||
|
connect). Xwayland itself becomes a Wayland client of the proxy. The
|
||||||
|
`wm_fd` is bridged via xtsb to the browser, which implements the actual
|
||||||
|
XWM logic.
|
||||||
|
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
|
||||||
|
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
|
||||||
|
|
||||||
|
## App launch flow (definitive answer)
|
||||||
|
|
||||||
|
1. Browser sends `GET http://proxy:8081/<path>` with
|
||||||
|
`X-Compositor-Session-Id: <uuid>` header.
|
||||||
|
2. Main process forks a `SessionProcess` if none exists for that session
|
||||||
|
ID, sends `start` IPC.
|
||||||
|
3. Main sends `launchApp` IPC with `{name, executable, args, env, serial}`.
|
||||||
|
4. `SessionProcess` calls `launchApplication()`:
|
||||||
|
- `appEnv = {...process.env, ...env, WAYLAND_DISPLAY:
|
||||||
|
session.nativeWaylandCompositorSession.waylandDisplay}`
|
||||||
|
- `child_process.spawn(executable, args, {env: appEnv})` — **plain
|
||||||
|
`spawn`, no chroot/container**
|
||||||
|
5. On `spawn` event, captures `pid`, creates `NativeAppContext`, registers
|
||||||
|
`exit` listener.
|
||||||
|
6. Replies to main with `{pid, key}` → main returns HTTP 201 with
|
||||||
|
`{baseURL, signalURL, key, pid, name}`.
|
||||||
|
7. Browser opens `signalURL` as WebSocket → control channel established.
|
||||||
|
8. App connects to `$XDG_RUNTIME_DIR/wayland-N` → libwayland fires
|
||||||
|
`on_client_created` → proxy looks up PID via `getCredentials()`, walks
|
||||||
|
`/proc/<pid>/status` PPid chain to match a `NativeAppContext`.
|
||||||
|
9. For each new `wl_client`, creates a `PROTOCOL` ARQ channel and sends
|
||||||
|
`CONNECT_CHANNEL` over the signaling WS; browser opens a new WS at
|
||||||
|
`/channel?id=…`.
|
||||||
|
10. Wayland wire messages now flow: app → libwayland →
|
||||||
|
`setWireMessageCallback` → JS interceptor → routed to `{native,
|
||||||
|
browser}`.
|
||||||
|
|
||||||
|
## Native addon architecture
|
||||||
|
|
||||||
|
**Important**: The "native" code is **C, not C++** (one GLSL shader string +
|
||||||
|
GObject-style). Uses Node-API (N-API).
|
||||||
|
|
||||||
|
### `wayland-server-addon.node` (1231 lines C)
|
||||||
|
|
||||||
|
Wraps a **patched** `libwayland-server` (`libwayland-server.so.0` with
|
||||||
|
Greenfield additions in `native/wayland/changes.diff`):
|
||||||
|
|
||||||
|
- `wl_client_set_wire_message_cb`
|
||||||
|
- `wl_client_set_registry_created_cb`
|
||||||
|
- `wl_client_set_sync_done_cb`
|
||||||
|
- `wl_client_add_resource_created_listener`
|
||||||
|
- `wl_display_set_global_created_cb`
|
||||||
|
- `wl_resource_destroy_silently`
|
||||||
|
- `wl_get_server_object_ids_batch`
|
||||||
|
- `wl_connection_copy_fds_in`
|
||||||
|
|
||||||
|
Plus the `westfield` shared lib (EGL/GBM init, `wlr_linux_dmabuf_v1`,
|
||||||
|
`wlr_drm`, `westfield-xwayland`). **31 N-API functions** exposed. This is
|
||||||
|
what makes the proxy a real Wayland compositor.
|
||||||
|
|
||||||
|
### `proxy-poll-addon.node` (127 lines C)
|
||||||
|
|
||||||
|
Tiny wrapper around `uv_poll_t` (libuv, not raw epoll — though libuv uses
|
||||||
|
epoll under the hood). `startPoll(fd, cb)` integrates the wl_display
|
||||||
|
event-loop FD into Node's libuv loop so wayland events fire as Node
|
||||||
|
callbacks without blocking.
|
||||||
|
|
||||||
|
### `proxy-encoding-addon.node` (290 lines C) + `libproxy-encoding.so` (~1400 lines C)
|
||||||
|
|
||||||
|
GStreamer H.264/PNG encoder. Runs on a **dedicated GMainLoop thread** with
|
||||||
|
an `eventfd`-based message queue (`SyncSource` GSource). Two pipelines per
|
||||||
|
surface (opaque + alpha) for translucent windows. Supports `x264`
|
||||||
|
(software), `nvh264` (Nvidia NVENC), `vaapih264` (Intel, experimental).
|
||||||
|
Zero-copy DMA-BUF path via EGLImage → GStreamer GL. Uses
|
||||||
|
`napi_threadsafe_function` to marshal encoded samples back to Node.
|
||||||
|
|
||||||
|
## XWayland bridge
|
||||||
|
|
||||||
|
### Native side (`westfield-xwayland.c`, 466 lines C)
|
||||||
|
|
||||||
|
1. Creates `/tmp/.X11-unix/X<N>` listener socket
|
||||||
|
2. `setenv("DISPLAY", ":<N>", 1)`
|
||||||
|
3. Waits for X client connect
|
||||||
|
4. On connect: `socketpair(AF_UNIX)` × 3 (wayland / wm / display)
|
||||||
|
5. `wl_client_create(wl_display, sv[0])` — **Xwayland itself becomes a
|
||||||
|
Wayland client of the proxy!**
|
||||||
|
6. `fork()` + `execlp("Xwayland", "Xwayland", ":<N>", "-ac", "-rootless",
|
||||||
|
"-listen", "<unix_fd>", "-displayfd", "<display_fd>", "-wm", "<wm_fd>",
|
||||||
|
NULL)`
|
||||||
|
|
||||||
|
### xtsb (X11 TypeScript Bindings)
|
||||||
|
|
||||||
|
`libs/xtsb/`: pure-TS X11 client, no libxcb. Dual-environment (Node
|
||||||
|
`net.Socket` + browser WebSocket). Auto-generated from XCB XML protocol
|
||||||
|
files via `xcbgen/ts_client.py`. Handles MIT-MAGIC-COOKIE-1 auth from
|
||||||
|
`~/.Xauthority`. The browser-side xtsb implements the actual XWM logic
|
||||||
|
(translates X `CreateNotify` events → `xdg_toplevel` Wayland surfaces).
|
||||||
|
|
||||||
|
`XWaylandSession.ts` uses `nodeFDConnectionSetup(wmFd)` to wrap the wm_fd
|
||||||
|
as a `net.Socket`, performs X11 auth, and pipes the X11 wire protocol
|
||||||
|
through the XWM data channel to the browser.
|
||||||
|
|
||||||
|
## SDK misconception corrected
|
||||||
|
|
||||||
|
The `sdk/` directory is **NOT for native apps**. It's exclusively for
|
||||||
|
**WASM apps** that run inside the browser via Emscripten. The
|
||||||
|
`sysrootlibs/` cross-compiles wayland/gtk/cairo/etc. to wasm32. Native
|
||||||
|
Linux apps on the proxy use the host's own system libraries — the
|
||||||
|
Dockerfile installs them via `apt-get install libffi8 libgbm1
|
||||||
|
libgstreamer1.0-plugins-base … xwayland xauth`. The proxy's `pkg` config
|
||||||
|
only bundles three libs as assets (`libproxy-encoding.so`,
|
||||||
|
`libwayland-server.so.0`, `libwestfield.so`) — these are the proxy's OWN
|
||||||
|
addon dependencies, not the apps'.
|
||||||
|
|
||||||
|
## ASCII diagram — native app launch & connection
|
||||||
|
|
||||||
|
```
|
||||||
|
Browser (compositor-shell) Proxy host (Linux) App host (often same as proxy)
|
||||||
|
========================== ==================== ==============================
|
||||||
|
|
||||||
|
[1] User clicks app in shell $ compositor-proxy-cli \
|
||||||
|
│ └─── main.ts listens on :8081
|
||||||
|
│ sessionProcesses: Record<id, ChildProcess>
|
||||||
|
│
|
||||||
|
│ HTTP GET /gtk4-demo
|
||||||
|
│ Header: X-Compositor-Session-Id: <uuid>
|
||||||
|
├──────────────────────────────► │
|
||||||
|
│ │ if no child for <uuid>: fork(SessionProcess.ts)
|
||||||
|
│ │ └── IPC: {type:'start', payload:{config, sessionId}}
|
||||||
|
│ │ └── createSession() → NativeWaylandCompositorSession
|
||||||
|
│ │ ├── createDisplay()
|
||||||
|
│ │ │ └── wl_display_create() (libwayland-server)
|
||||||
|
│ │ ├── addSocketAuto()
|
||||||
|
│ │ │ └── creates $XDG_RUNTIME_DIR/wayland-N
|
||||||
|
│ │ │ returns "wayland-N" ◄── this becomes WAYLAND_DISPLAY
|
||||||
|
│ │ ├── initShm() + YUV420/NV12/YUV444/NV21
|
||||||
|
│ │ ├── initDrm("/dev/dri/renderD128")
|
||||||
|
│ │ │ └── westfield_egl_new() → EGLDisplay via GBM
|
||||||
|
│ │ │ wlr_linux_dmabuf_v1 global (zero-copy GPU buffers)
|
||||||
|
│ │ │ wlr_drm global (legacy dmabuf)
|
||||||
|
│ │ └── startPoll(wlDisplayFd, dispatchRequests)
|
||||||
|
│ │ └── uv_poll_t on EPOLLIN
|
||||||
|
│ │ (proxy-poll-addon.node)
|
||||||
|
│ │
|
||||||
|
│ │ IPC: {type:'launchApp', payload:{name, executable, args, env, serial}}
|
||||||
|
│ │ └── launchApplication() (NativeAppContext.ts)
|
||||||
|
│ │ ├── appEnv = {...process.env, ...env,
|
||||||
|
│ │ │ WAYLAND_DISPLAY: "wayland-N"}
|
||||||
|
│ │ ├── child_process.spawn("gtk4-demo", [], {env: appEnv})
|
||||||
|
│ │ │ ◄── posix_spawn/execvp, NO chroot, NO container
|
||||||
|
│ │ │ ──────────────────────────────► [2] gtk4-demo process starts
|
||||||
|
│ │ │ │
|
||||||
|
│ │ │ │ libwayland-client.so
|
||||||
|
│ │ │ │ connects to
|
||||||
|
│ │ │ │ $XDG_RUNTIME_DIR/wayland-N
|
||||||
|
│ │ │ │
|
||||||
|
│ │ │ ▼
|
||||||
|
│ │ │ ┌──────────────────────────┐
|
||||||
|
│ │ │ │ wl_display event loop │
|
||||||
|
│ │ │ │ (in proxy, libwayland- │
|
||||||
|
│ │ │ │ server.so.0) │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ │ │ ◄── uv_poll fires, │ on_client_created → │
|
||||||
|
│ │ │ dispatchRequests() │ getCredentials(pid) │
|
||||||
|
│ │ │ wl_event_loop_dispatch() │ findMatchingNativeApp │
|
||||||
|
│ │ │ │ Context(pid) by PPID │
|
||||||
|
│ │ │ │ walk │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ │ │ │ setWireMessageCallback │
|
||||||
|
│ │ │ │ setBufferCreated... │
|
||||||
|
│ │ │ │ setSyncDoneCallback │
|
||||||
|
│ │ │ │ setClientDestroyed... │
|
||||||
|
│ │ │ │ │
|
||||||
|
│ │ │ │ NativeWaylandClientSession
|
||||||
|
│ │ │ │ .messageInterceptor │
|
||||||
|
│ │ │ │ dispatches R<opcode> │
|
||||||
|
│ │ │ │ → {native, browser} │
|
||||||
|
│ │ │ └──────────────────────────┘
|
||||||
|
│ │ │
|
||||||
|
│ │ ├── child.on('spawn') → capture pid
|
||||||
|
│ │ │ createNativeAppContext(pid, name, external:false)
|
||||||
|
│ │ │ nativeAppContext.onDisconnect() // start 10-min SIGHUP timer
|
||||||
|
│ │ │
|
||||||
|
│ │ └── child.on('exit') → nativeAppContext.onExit({exitCode|signal})
|
||||||
|
│ │
|
||||||
|
│ │ IPC reply: {type:'launchAppSuccess', payload:{pid, key}}
|
||||||
|
│
|
||||||
|
│ HTTP 201 Created
|
||||||
|
│ Body: { baseURL, signalURL:"ws://…/signal?key=…", key, pid, name, internal:false }
|
||||||
|
◄────────────────────────────── │
|
||||||
|
│ │
|
||||||
|
│ Open WS: ws://…/signal?key=<16-hex>
|
||||||
|
├──────────────────────────────► │
|
||||||
|
│ │ SessionController demux /signal → nativeAppContext.onConnect(ws)
|
||||||
|
│ │ clears 10-min SIGHUP timer
|
||||||
|
│ │
|
||||||
|
│ │ For each new wl_client (one per wayland connection):
|
||||||
|
│ │ createProtocolChannel() → ARQChannel (KCP over WS)
|
||||||
|
│ │ sends CONNECT_CHANNEL {url:"ws://…/channel?id=…&key=…", desc}
|
||||||
|
│ ◄──── CONNECT_CHANNEL JSON ───┤
|
||||||
|
│ │
|
||||||
|
│ Open WS: ws://…/channel?id=…&key=…
|
||||||
|
├──────────────────────────────► │ binds WS to the ARQChannel
|
||||||
|
│ │
|
||||||
|
│ Wayland wire protocol flows over the ARQ channel (KCP reliable, 100ms interval):
|
||||||
|
│
|
||||||
|
│ App → wl_surface.commit ──── proxy intercepts R1 (attach) + R6 (commit) ────► browser compositor
|
||||||
|
│ App ← wl_surface.frame ──── proxy intercepts E1 (frame callback) ◄────── FrameFeedback paces callback
|
||||||
|
│ App → wl_display.sync ────── proxy fast-path sync (no roundtrip if no reply expected)
|
||||||
|
│ App ← wl_keyboard.enter ─── browser → proxy → sendEvents(wlClient, …) → flush(wlClient)
|
||||||
|
│
|
||||||
|
│ Buffer path (per wl_surface):
|
||||||
|
│ - App submits wl_buffer (dmabuf or shm)
|
||||||
|
│ - setBufferCreatedCallback → ProxyBuffer with creationSerial
|
||||||
|
│ - On commit: wl_surface_interceptor.R6 → Encoder.encodeFrame()
|
||||||
|
│ └── proxy-encoding-addon.encodeFrame(bufferId, contentSerial, creationSerial)
|
||||||
|
│ └── GStreamer pipeline (x264/nvh264/vaapih264) on dedicated GMainLoop thread
|
||||||
|
│ └── encoded H.264 sample → napi_threadsafe_function → Node main thread
|
||||||
|
│ └── sent over the surface's FRAME data channel (separate WS, also CONNECT_CHANNEL'd)
|
||||||
|
│
|
||||||
|
│ X11 path (xterm etc.):
|
||||||
|
│ - App connects to /tmp/.X11-unix/X1 (DISPLAY=:1, set by westfield_xwayland.c)
|
||||||
|
│ - First X connect → spawn_xserver() forks: execlp("Xwayland", …, "-rootless", "-wm", wm_fd)
|
||||||
|
│ - Xwayland itself becomes a Wayland client of the proxy (wl_client_create)
|
||||||
|
│ - When Xwayland ready (displayFd readable):
|
||||||
|
│ └── xtsb.nodeFDConnectionSetup(wmFd) → wraps as net.Socket
|
||||||
|
│ └── pipes through XWM data channel (4th channel type) to browser
|
||||||
|
│ └── browser-side xtsb speaks X11 → translates X windows → xdg_toplevel surfaces
|
||||||
|
│
|
||||||
|
│ Lifecycle:
|
||||||
|
│ - User closes browser tab → /signal WS closes with code 1001 → kill('SIGHUP')
|
||||||
|
│ - User clicks "force quit" in shell → /signal WS sends KILL_APP {signal:'SIGTERM'} → kill('SIGTERM')
|
||||||
|
│ └── if not exited in 10s → SIGKILL
|
||||||
|
│ - App crashes → child 'exit' event → nativeAppContext.onExit() → APP_TERMINATED msg → browser UI updates
|
||||||
|
│ - All clients gone for >1s → session.close() → process.exit() in SessionProcess.ts → main deletes entry
|
||||||
|
│
|
||||||
|
│ Native addon role summary:
|
||||||
|
│ - wayland-server-addon.node: full Wayland compositor (libwayland-server fork + westfield EGL/GBM/dmabuf/XWayland)
|
||||||
|
│ - proxy-poll-addon.node: libuv uv_poll_t hook so wayland FD events fire as Node callbacks
|
||||||
|
│ - proxy-encoding-addon.node: GStreamer H.264/PNG encoder on a dedicated GMainLoop thread, threadsafe_function back to Node
|
||||||
|
```
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
### CLI (`packages/compositor-proxy-cli/src/`)
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `main.ts` | HTTP + WS hub, forks per session |
|
||||||
|
| `main-controller.ts` | CLI arg parsing |
|
||||||
|
| `main-args.ts` | CLI arg definitions |
|
||||||
|
| `SessionProcess.ts` | Per-session child, IPC handler |
|
||||||
|
| `example-applications.json` | Pre-configured apps |
|
||||||
|
| `wait-until-ready.sh` | Startup readiness check |
|
||||||
|
|
||||||
|
### Proxy TS layer (`packages/compositor-proxy/src/`)
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `index.ts` | Exports `createSession()`, `launchApplication()` |
|
||||||
|
| `Session.ts` | Top-level proxy session |
|
||||||
|
| `SessionController.ts` | WS server with 9 endpoint paths |
|
||||||
|
| `NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
|
||||||
|
| `NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||||||
|
| `NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
|
||||||
|
| `Channel.ts` | ARQChannel + SimpleChannel |
|
||||||
|
| `kcp.ts` | ~1100-line KCP implementation |
|
||||||
|
| `FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
|
||||||
|
| `ProxyBuffer.ts` | Proxy-side buffer representation |
|
||||||
|
| `SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
|
||||||
|
| `XWaylandSession.ts` | XWayland spawn + bridge |
|
||||||
|
| `wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||||||
|
| `io/` | WebSocket server glue |
|
||||||
|
| `encoding/` | High-level encoding orchestration |
|
||||||
|
| `protocol/` | Generated interceptor modules |
|
||||||
|
| `addons/` | .node file shims (Linux-x64) |
|
||||||
|
| `config.ts` | Encoding-related configuration |
|
||||||
|
|
||||||
|
### Proxy C layer (`packages/compositor-proxy/native/`)
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|---|---|
|
||||||
|
| `CMakeLists.txt` | Build config |
|
||||||
|
| `wayland/src/addon.c` | Node-API addon: wayland-server-addon.node |
|
||||||
|
| `wayland/src/westfield-buffer.c` | Buffer helpers |
|
||||||
|
| `wayland/src/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM |
|
||||||
|
| `wayland/src/westfield-egl.c` | EGL zero-copy setup |
|
||||||
|
| `wayland/src/westfield-xwayland.c` | XWayland spawn + bridge |
|
||||||
|
| `wayland/src/wlr_drm.c` | wl_drm global impl |
|
||||||
|
| `wayland/src/wlr_linux_dmabuf_v1.c` | zwp_linux_dmabuf_v1 global impl |
|
||||||
|
| `wayland/src/drm-protocol.c` | DRM protocol helpers |
|
||||||
|
| `wayland/src/linux-dmabuf-unstable-v1-protocol.c` | linux-dmabuf-v1 protocol stubs |
|
||||||
|
| `wayland/src/wayland-server/` | Vendored + patched libwayland-server (with changes.diff) |
|
||||||
|
| `encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder |
|
||||||
|
| `encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
|
||||||
|
| `encoding/src/node_encoder.c` | Node-API addon: proxy-encoding-addon.node |
|
||||||
|
| `poll/src/poll.c` | libuv uv_poll_t wrapper |
|
||||||
|
| `node-api/` | Vendored Node-API headers |
|
||||||
|
|
||||||
|
### XWayland support (`libs/xtsb/`)
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `README.md` | Project intro |
|
||||||
|
| `package.json` | @gfld/xtsb package config |
|
||||||
|
| `protocol/xproto.xml` | X11 core protocol XML (5800+ lines) |
|
||||||
|
| `protocol/shape.xml` | Shape extension XML |
|
||||||
|
| `protocol/xfixes.xml` | XFIXES extension XML |
|
||||||
|
| `protocol/render.xml` | Render extension XML |
|
||||||
|
| `protocol/composite.xml` | Composite extension XML |
|
||||||
|
| `src/` | Generated TS bindings |
|
||||||
|
|
||||||
|
## Next actions for user (Odoo 20 integration)
|
||||||
|
|
||||||
|
1. Build the proxy on a Linux host (or use `udevbe/compositor-proxy-cli`
|
||||||
|
Docker image).
|
||||||
|
2. Write an `applications.json` mapping desired apps to HTTP paths.
|
||||||
|
3. Start proxy with `--allow-origin=https://your-odoo.example.com`.
|
||||||
|
4. Embed the Greenfield `compositor-shell` Preact SPA in Odoo (iframe or
|
||||||
|
QWeb template) — OR replace it with your Odoo 20 OWL shell (see
|
||||||
|
`10-porting-to-odoo-js-python.md`).
|
||||||
|
5. Trigger app launches from Odoo via
|
||||||
|
`fetch('http://<proxy>:8081/<path>', { headers: { 'X-Compositor-Session-Id': sessionId }})`
|
||||||
|
— Odoo frontend JS uses the returned `signalURL` to open the per-app
|
||||||
|
control WS.
|
||||||
|
6. For multi-tenant isolation: one `SessionProcess` per browser session is
|
||||||
|
automatic, but file-system isolation between tenants requires separate
|
||||||
|
proxy instances or containers per tenant.
|
||||||
|
7. For GPU acceleration: pass `--device /dev/dri:/dev/dri` to Docker; use
|
||||||
|
`--encoder=nvh264|vaapih264|x264` based on GPU.
|
||||||
@@ -0,0 +1,304 @@
|
|||||||
|
# 08 — App Lifecycle (XDG Shell, Toplevels, Popups, Subsurfaces)
|
||||||
|
|
||||||
|
This chapter covers Greenfield's application lifecycle layer: how xdg-shell
|
||||||
|
(toplevels, popups, positioners), wl_shell (legacy), and subsurface
|
||||||
|
compositing are implemented, and how window state transitions are handled.
|
||||||
|
EXCLUDE the actual UI shell (`packages/compositor-shell`) — focus on the
|
||||||
|
protocol-level lifecycle.
|
||||||
|
|
||||||
|
## Three role families, one shared abstraction
|
||||||
|
|
||||||
|
All surface roles implement `SurfaceRole` (`{ view, optional
|
||||||
|
desktopSurface, onCommit(surface) }`). `XdgToplevel`, `XdgPopup`, and
|
||||||
|
`ShellSurface` additionally implement `DesktopSurfaceRole`, which bridges
|
||||||
|
them to a `DesktopSurface` (`FloatingDesktopSurface` or
|
||||||
|
`AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`). The
|
||||||
|
`DesktopSurface` is where **every** `userShell.events.*` emission
|
||||||
|
originates — making it the true integration boundary.
|
||||||
|
|
||||||
|
## XdgToplevel lifecycle — the headline diagram
|
||||||
|
|
||||||
|
```
|
||||||
|
┌───────────────────────────────────────────────┐
|
||||||
|
│ wl_surface created (no role, no buffer yet) │
|
||||||
|
└─────────────────────┬─────────────────────────┘
|
||||||
|
│ client: xdg_wm_base.get_xdg_surface
|
||||||
|
│ (errors if surface already has buffer)
|
||||||
|
▼
|
||||||
|
┌───────────────────────────────────────────────┐
|
||||||
|
│ XdgSurface created │
|
||||||
|
│ • surface.hasKeyboard/Pointer/Touch = true │
|
||||||
|
│ • configured=false, configureList=[] │
|
||||||
|
└─────────────────────┬─────────────────────────┘
|
||||||
|
│ client: xdg_surface.get_toplevel
|
||||||
|
│ (errors if surface has another role)
|
||||||
|
▼
|
||||||
|
┌───────────────────────────────────────────────┐
|
||||||
|
│ XdgToplevel created │
|
||||||
|
│ • surface.role = xdgToplevel │
|
||||||
|
│ • View created │
|
||||||
|
│ • DesktopSurface created + init() │
|
||||||
|
│ • ackConfigure override installed │
|
||||||
|
└─────────────────────┬─────────────────────────┘
|
||||||
|
│ client: xdg_toplevel.set_title /
|
||||||
|
│ set_app_id / set_min/max_size /
|
||||||
|
│ set_parent (stored, propagated)
|
||||||
|
▼
|
||||||
|
┌───────────────────────────────────────────────┐
|
||||||
|
│ First wl_surface.commit (buffer === undefined) │
|
||||||
|
│ XdgToplevel.onCommit() → ensureAdded() │
|
||||||
|
│ • desktopSurface.add() │
|
||||||
|
│ → userShell.events.surfaceCreated(...) │
|
||||||
|
│ • scheduleConfigure(first) │
|
||||||
|
│ → microtask → sendConfigure() │
|
||||||
|
│ → resource.configure(w,h,states) │
|
||||||
|
│ • added=true │
|
||||||
|
└─────────────────────┬─────────────────────────┘
|
||||||
|
│ client receives configure
|
||||||
|
│ client: xdg_surface.ack_configure(serial)
|
||||||
|
▼
|
||||||
|
┌───────────────────────────────────────────────┐
|
||||||
|
│ ackConfigure(serial, configure) │
|
||||||
|
│ • handleAckConfigure trims older serials │
|
||||||
|
│ • configured=true │
|
||||||
|
│ • next.state = configure.state │
|
||||||
|
│ • next.size = configure.size │
|
||||||
|
└─────────────────────┬─────────────────────────┘
|
||||||
|
│ client: wl_surface.commit (with buffer)
|
||||||
|
▼
|
||||||
|
┌───────────────────────────────────────────────┐
|
||||||
|
│ ★ MAPPED (steady state) ★ │
|
||||||
|
│ onCommit(): │
|
||||||
|
│ • surface.commitPending() (state latch) │
|
||||||
|
│ • xdgSurface.commit() (geometry latch) │
|
||||||
|
│ • current.state ← next.state │
|
||||||
|
│ • current.min/max ← next.min/max │
|
||||||
|
│ • desktopSurface.commit() → render() │
|
||||||
|
└─────────────────────┬─────────────────────────┘
|
||||||
|
│
|
||||||
|
┌────────────────────────────────────────┼────────────────────────────┐
|
||||||
|
│ │ │
|
||||||
|
user/shell requests client requests destruction
|
||||||
|
│ │ │
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌──────────────────────────┐ ┌─────────────────────────────┐ ┌──────────────────────┐
|
||||||
|
│ configureMaximized(b) │ │ setMaximized / unsetMaximized│ │ wl_surface.destroy │
|
||||||
|
│ configureFullscreen(b) │ │ setFullscreen/unsetFullscreen│ │ or xdg_toplevel. │
|
||||||
|
│ configureActivated(b) │ │ setMinimized │ │ destroy │
|
||||||
|
│ configureResizing(b) │ │ setParent / setTitle / │ ├──────────────────────┤
|
||||||
|
│ configureSize(w,h) │ │ setAppId │ │ destroy listener: │
|
||||||
|
└────────────┬─────────────┘ │ move / resize / showWindowMenu│ │ • surface.unmap() │
|
||||||
|
│ └──────────────┬──────────────┘ │ • desktopSurface. │
|
||||||
|
│ each mutates │ │ removed() │
|
||||||
|
│ pending.state / pending.size │ │ → surfaceDestroyed │
|
||||||
|
▼ │ │ • view.destroy() │
|
||||||
|
┌──────────────────────────┐ │ └──────────────────────┘
|
||||||
|
│ scheduleConfigure( │ │
|
||||||
|
│ stateCompareEquals(), │ ┌─────────────────────────────┐
|
||||||
|
│ sendConfigure) │ │ setMinimized → │
|
||||||
|
│ • coalesce via µtask │ │ desktopSurface.minimize() │
|
||||||
|
│ • if pending==current → │ │ • view.mapped=false │
|
||||||
|
│ skip │ │ • seat drops keyboard focus │
|
||||||
|
│ • else: push serial, │ │ if this was the focused │
|
||||||
|
│ emit configure event │ │ surface │
|
||||||
|
│ │ └─────────────────────────────┘
|
||||||
|
└────────────┬─────────────┘
|
||||||
|
│
|
||||||
|
│ cycle repeats: ack_configure → commit → render
|
||||||
|
└────────────────────────────────────► back to ★ MAPPED ★
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configure / ack-configure round-trip
|
||||||
|
|
||||||
|
This is the **most important mechanism** to understand in xdg-shell. It's
|
||||||
|
an asynchronous protocol-level sync between the compositor and the
|
||||||
|
client.
|
||||||
|
|
||||||
|
### Why it's needed
|
||||||
|
|
||||||
|
The compositor wants to tell the client "please resize to (800, 600)
|
||||||
|
because the user maximized you". But the client may not be ready to
|
||||||
|
resize, or may want a different size (e.g. the app enforces a 16:9 aspect
|
||||||
|
ratio). The configure/ack round-trip lets the client acknowledge the
|
||||||
|
configure with a serial, then on the next commit, the compositor knows
|
||||||
|
which configure the client has applied.
|
||||||
|
|
||||||
|
### How Greenfield implements it
|
||||||
|
|
||||||
|
1. Compositor mutates `pending.{state,size}`.
|
||||||
|
2. Coalesces via `queueCancellableMicrotask` with a `stateCompareEquals`
|
||||||
|
short-circuit (if pending == current, skip).
|
||||||
|
3. Emits `resource.configure(w,h,states)` with a fresh serial.
|
||||||
|
4. Client acks: `xdg_surface.ack_configure(serial)`.
|
||||||
|
5. `handleAckConfigure` trims older serials, sets `configured=true`,
|
||||||
|
copies `configure.{state,size}` into `next`.
|
||||||
|
6. Next commit latches `next → current`.
|
||||||
|
|
||||||
|
The `configured` flag gates buffer attachment: an `unconfiguredBuffer`
|
||||||
|
error is raised if the client tries to attach a buffer before acking.
|
||||||
|
|
||||||
|
## XdgPositioner math
|
||||||
|
|
||||||
|
`anchor` selects a point on the anchor rect (in parent window-geometry
|
||||||
|
space); `gravity` selects which corner/edge of the popup is placed at
|
||||||
|
that point; `offset` is applied last.
|
||||||
|
|
||||||
|
### Constraint adjustment cascade
|
||||||
|
|
||||||
|
Runs as X-then-Y cascade:
|
||||||
|
|
||||||
|
1. Try **flip** — swap anchor+gravity via `inverseX` / `inverseY` tables,
|
||||||
|
revert if still violating.
|
||||||
|
2. Try **slide** — shift offset by violation delta.
|
||||||
|
3. Try **resize** — shrink `sizeRect`.
|
||||||
|
|
||||||
|
Slides/flips are skipped if popup size on that axis exceeds viewport.
|
||||||
|
|
||||||
|
## Popup grabs
|
||||||
|
|
||||||
|
`XdgPopup.grab` rejects if:
|
||||||
|
|
||||||
|
- Surface is already mapped, OR
|
||||||
|
- Parent isn't either:
|
||||||
|
- a toplevel (root of new chain), OR
|
||||||
|
- the topmost popup of an existing grab (`notTheTopmostPopup` error)
|
||||||
|
|
||||||
|
`seat.popupGrabStart` validates the serial against the keyboard/pointer
|
||||||
|
`grabSerial`, then starts a `PopupGrab` on both devices. `popupGrabEnd`
|
||||||
|
walks the chain in reverse calling `requestClose()` on each (sends
|
||||||
|
`popup_done` / `xdg_popup.popup_done`).
|
||||||
|
|
||||||
|
## Subsurfaces sync/desync
|
||||||
|
|
||||||
|
Three-layer mechanism:
|
||||||
|
|
||||||
|
1. **`cachedState`** — full pending-state shadow on the subsurface role.
|
||||||
|
2. **`commitCache`** — flush on parent commit.
|
||||||
|
3. **`effectiveSync`** getter — cascades: if any ancestor is sync, this
|
||||||
|
subsurface is effectively sync.
|
||||||
|
|
||||||
|
### Decision matrix
|
||||||
|
|
||||||
|
| Sync mode | Dirty | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| Sync | yes | Cache to `cachedState`, flush when parent commits |
|
||||||
|
| Sync | no | No-op (already cached) |
|
||||||
|
| Desync | yes | Flush immediately to parent's pending state |
|
||||||
|
| Desync | no | Acts like a normal surface (commit directly to its own pendingState) |
|
||||||
|
|
||||||
|
`setDesync` immediately calls `onCommit` to flush any pending cache.
|
||||||
|
|
||||||
|
## wl_shell (legacy path)
|
||||||
|
|
||||||
|
Legacy path. Reduced feature set:
|
||||||
|
|
||||||
|
- `configureActivated` / `configureResizing` are explicit no-ops.
|
||||||
|
- State machine via `SurfaceStates` enum (MAXIMIZED / FULLSCREEN / POPUP /
|
||||||
|
TRANSIENT / TOP_LEVEL) and `changeState()`.
|
||||||
|
- Per-surface ping (separate from `XdgWmBase`'s per-client ping).
|
||||||
|
|
||||||
|
## The UserShellApi seam — critical for the Odoo 20 OWL shell
|
||||||
|
|
||||||
|
Located in `UserShellApi.ts`, exposed as `session.userShell: UserShellApi =
|
||||||
|
{ events, actions }`. The default `createUserShellApi(session)` returns
|
||||||
|
`events: {}` (empty) and a populated `actions` object. **The shell
|
||||||
|
assigns the event callbacks; the compositor always invokes them with
|
||||||
|
`?.()` so unhandled events are silent no-ops.** This is the single
|
||||||
|
integration seam — the protocol layer above, and the shell UI below.
|
||||||
|
|
||||||
|
### Events (compositor → shell) — all optional
|
||||||
|
|
||||||
|
| Event | Payload | Fired from | When |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `clientCreated` | `CompositorClient` | `WebConnectionHandler`, `RemoteAppLauncher` | A new app client connects. |
|
||||||
|
| `clientDestroyed` | `CompositorClient` | same | A client disconnects. |
|
||||||
|
| `clientUnresponsiveUpdated` | `(client, unresponsive: boolean)` | `XdgWmBase`, `ShellSurface` | Ping (2s timeout, every 5s) — `true` = hung, `false` = recovered. |
|
||||||
|
| `surfaceCreated` | `CompositorSurface` | `FloatingDesktopSurface.add()`, `AlwaysFullScreenDesktopSurface` | Toplevel first added to desktop. |
|
||||||
|
| `surfaceDestroyed` | `CompositorSurface` | `.removed()` | Window destroyed. |
|
||||||
|
| `surfaceTitleUpdated` | `(surface, title)` | `.setTitle()` | `xdg_toplevel.set_title` / `wl_shell_surface.set_title`. |
|
||||||
|
| `surfaceAppIdUpdated` | `(surface, appId)` | `.setAppId()` | `xdg_toplevel.set_app_id` / `wl_shell_surface.set_class`. |
|
||||||
|
| `surfaceActivationUpdated` | `(surface, active: boolean)` | `gainFocus()` / `loseFocus()` | Refcounted via `focusCount` — only fires on 0↔1 transitions. |
|
||||||
|
| `notify` | `(variant, message)` | (declared, no direct emissions) | Free-form notification channel. |
|
||||||
|
| `sceneRefreshed` | `sceneId: string` | `Scene:75` | Render finished. |
|
||||||
|
|
||||||
|
### Actions (shell → compositor) — provided by default
|
||||||
|
|
||||||
|
| Action | Effect |
|
||||||
|
|---|---|
|
||||||
|
| `initScene(canvasCreator)` | Bootstraps a WebGL canvas + input/output. One per output. |
|
||||||
|
| `refreshScene()` | `session.renderer.render()` — manual redraw. |
|
||||||
|
| `destroyScene(sceneId)` | Destroys a specific scene. |
|
||||||
|
| `setUserConfiguration(config)` | Updates `scrollFactor`, `keyboardLayoutName`. |
|
||||||
|
| `closeClient({ id })` | Closes a client connection. |
|
||||||
|
| `activateSurface(compositorSurface)` | **Click-to-focus entry point from the shell's taskbar.** Reverse-lookups the surface via `session.display.clients[clientId].connection.wlObjects[surfaceId]` and calls `desktopSurface.activate()`. |
|
||||||
|
|
||||||
|
### Opaque handle types
|
||||||
|
|
||||||
|
```
|
||||||
|
interface CompositorSurface { id: number; client: CompositorClient }
|
||||||
|
interface CompositorClient { id: string }
|
||||||
|
```
|
||||||
|
|
||||||
|
`toCompositorSurface(desktopSurface)` adapts `{ surface.resource.id,
|
||||||
|
surface.resource.client.id }`; `lookupSurface(session, compositorSurface)`
|
||||||
|
is the reverse walk.
|
||||||
|
|
||||||
|
## Notable gaps for the Odoo OWL shell
|
||||||
|
|
||||||
|
1. **No maximize/fullscreen/resize/minimize events** in
|
||||||
|
`UserShellApiEvents`. The shell can't directly observe window-state
|
||||||
|
transitions through this API — only focus (`surfaceActivationUpdated`)
|
||||||
|
and metadata (title/appId). For a taskbar this is fine; for
|
||||||
|
window-decorations rendering it is not. Options: (a) extend
|
||||||
|
`UserShellApiEvents` with `surfaceStateChanged`; (b) the OWL shell
|
||||||
|
imports `DesktopSurface` directly and introspects; (c) add a
|
||||||
|
`querySurfaceState` action.
|
||||||
|
2. **No geometry/position event** — the shell can't render a frame that
|
||||||
|
hugs surface bounds from the shell side without taking over rendering.
|
||||||
|
3. **`notify` declared but unused** in compositor core — dead interface
|
||||||
|
slot.
|
||||||
|
4. **`surfaceCreated` fires once on first commit; no separate map/unmap
|
||||||
|
signal** — minimization is internal (`view.mapped=false`) and not
|
||||||
|
propagated.
|
||||||
|
|
||||||
|
## Risks flagged
|
||||||
|
|
||||||
|
- `XdgWmBase.doPing` uses serial 0 (`// FIXME use a proper serial`) — works
|
||||||
|
only because there's a single outstanding ping per client.
|
||||||
|
- `XdgToplevel.onCommit` has commented-out buffer-size-vs-configured-state
|
||||||
|
validation (lines 140–158) — the compositor trusts clients to honor
|
||||||
|
configured sizes.
|
||||||
|
- `XdgSurface.scheduleConfigure`'s `pendingSame` argument inverts meaning
|
||||||
|
depending on whether a microtask is already queued — subtle and easy to
|
||||||
|
break in maintenance.
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/Shell.ts` | `wl_shell` global (legacy) |
|
||||||
|
| `packages/compositor/src/ShellSurface.ts` | `wl_shell_surface` role |
|
||||||
|
| `packages/compositor/src/Subcompositor.ts` | `wl_subcompositor` global |
|
||||||
|
| `packages/compositor/src/Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
|
||||||
|
| `packages/compositor/src/SurfaceRole.ts` | SurfaceRole interface |
|
||||||
|
| `packages/compositor/src/SurfaceChild.ts` | Surface tree child abstraction |
|
||||||
|
| `packages/compositor/src/UserShellApi.ts` | The integration seam — events + actions |
|
||||||
|
| `packages/compositor/src/XdgWmBase.ts` | `xdg_wm_base` global, ping-based unresponsiveness detection |
|
||||||
|
| `packages/compositor/src/XdgSurface.ts` | `xdg_surface` role, configure/ack round-trip |
|
||||||
|
| `packages/compositor/src/XdgToplevel.ts` | `xdg_toplevel` role, window state |
|
||||||
|
| `packages/compositor/src/XdgPopup.ts` | `xdg_popup` role, transient windows, grabs |
|
||||||
|
| `packages/compositor/src/XdgPositioner.ts` | `xdg_positioner`, popup positioning math |
|
||||||
|
| `packages/compositor/src/desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
|
||||||
|
|
||||||
|
## Next actions (for Odoo 20 OWL shell)
|
||||||
|
|
||||||
|
1. Design `createOdooUserShellApi(session)` returning the default
|
||||||
|
`actions` and OWL-signal-bound callbacks for the 8 events.
|
||||||
|
2. Decide how to surface maximize/fullscreen/resize/minimize state (extend
|
||||||
|
events vs. introspect `DesktopSurface`).
|
||||||
|
3. Wire `clientCreated` / `clientDestroyed` to an Odoo `AppContext`-style
|
||||||
|
registry for the app launcher UI.
|
||||||
|
4. Feed `surfaceActivationUpdated` into an OWL reactive store as the
|
||||||
|
single source of truth for active-window focus.
|
||||||
|
5. `initScene(canvasCreator)` — OWL shell owns the `HTMLCanvasElement`
|
||||||
|
lifecycle, one per output.
|
||||||
@@ -0,0 +1,423 @@
|
|||||||
|
# 09 — Hardware Acceleration, Encoding, WASM, GPU Textures
|
||||||
|
|
||||||
|
This chapter covers Greenfield's hardware acceleration / encoding / WASM
|
||||||
|
pipeline: how native buffers get encoded on the proxy side, transported,
|
||||||
|
decoded in the browser, and turned into WebGL textures or canvas images.
|
||||||
|
|
||||||
|
## What Greenfield actually ships
|
||||||
|
|
||||||
|
### Two Wasm libraries under `libs/`
|
||||||
|
|
||||||
|
**`compositor-wasm`** builds **pixman 0.42.2 + xkbcommon 1.5.0**
|
||||||
|
(Emscripten 3.1.46, `-msimd128 -O3 -flto`). Used by the browser for
|
||||||
|
**region math** (damage tracking) and **keyboard layout** — *not* for
|
||||||
|
pixel format conversion or H.264 decode.
|
||||||
|
|
||||||
|
**`compositor-ffmpeg-h264`** is a **stripped-down FFmpeg n5.1.2**
|
||||||
|
(`--disable-everything --enable-decoder=h264`) compiled to a 1.7 MB
|
||||||
|
single-file Wasm worker. A 30-line `decoder.c` wrapper exposes
|
||||||
|
`create_codec_context` / `decode` / `close_frame` / `destroy_codec_context`.
|
||||||
|
Decodes to **I420 (YUV 4:2:0 planar)** and returns plane pointers; planes
|
||||||
|
are `postMessage`'d back to the main thread as **transferable
|
||||||
|
`ArrayBuffer`s**. No threads (single-threaded, SIMD only).
|
||||||
|
|
||||||
|
### Native encoder addon (`packages/compositor-proxy/native/encoding/`)
|
||||||
|
|
||||||
|
A **GStreamer wrapper** exposed via Node-API. The TS side (`Encoder.ts`)
|
||||||
|
calls `appEndpointNative.encodeFrame(encoder, bufferId, contentSerial,
|
||||||
|
creationSerial)`. The C side (`node_encoder.c`) sniffs the wl_resource:
|
||||||
|
|
||||||
|
- `zwp_linux_dmabuf_v1` → DMA path
|
||||||
|
- Legacy `wl_drm` → DMA path
|
||||||
|
- `wl_shm_buffer` → SHM path
|
||||||
|
|
||||||
|
For DMA-BUFs the proxy wraps the dma-buf as an `EGLImage` and then as a
|
||||||
|
`GstGLMemoryEGL`, so pixels **stay on GPU** through the color-convert
|
||||||
|
stages.
|
||||||
|
|
||||||
|
### Threading
|
||||||
|
|
||||||
|
Encoding runs on a dedicated GLib main-loop thread fed by an
|
||||||
|
`eventfd`-signalled `GAsyncQueue`. Output returns to Node via a
|
||||||
|
`napi_threadsafe_function` with **zero-copy** external Buffers (finalised
|
||||||
|
back to C on GC).
|
||||||
|
|
||||||
|
## Encoder selection (per-frame, deterministic)
|
||||||
|
|
||||||
|
`gst_frame_encoder.c::frame_encoder_descriptions[]` declares **four**
|
||||||
|
pipelines:
|
||||||
|
|
||||||
|
| name | element | when used | split_alpha | coded-size multiple |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| `x264` | `x264enc` (software, default) | >256×256 + ARGB/XRGB/dmabuf | true (parallel alpha H.264 stream) | 128 |
|
||||||
|
| `nvh264` | `nvh264enc` (Nvidia NVENC) | same, `--encoder=nvh264` | true | 128 |
|
||||||
|
| `vaapih264` | `vaapih264enc` (Intel/AMD VA-API, "experimental") | same, `--encoder=vaapih264` | true | 128 |
|
||||||
|
| `png` | `pngenc` (fallback) | ≤256×256 + ARGB/XRGB/dmabuf | false (PNG has native alpha) | 16 |
|
||||||
|
|
||||||
|
**Rule**: pixel_count ≤ 256×256 → PNG; otherwise the configured H.264
|
||||||
|
encoder. YUV420/NV12/NV21/Y444 SHM formats are *not* supported on the
|
||||||
|
H.264 path and only by PNG at small sizes — effectively unsupported for
|
||||||
|
video-class content.
|
||||||
|
|
||||||
|
### Canonical H.264 pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
appsrc → glupload → glcolorconvert → glshader(scale→coded_size) →
|
||||||
|
glcolorconvert → video/x-raw(memory:GLMemory),format=NV12 → gldownload →
|
||||||
|
queue → x264enc tune=zerolatency bitrate=12000 vbv-buf-capacity=160
|
||||||
|
bframes=0 rc-lookahead=0 sliced-threads byte-stream=true →
|
||||||
|
video/x-h264,profile=high,stream-format=byte-stream,alignment=au → appsink
|
||||||
|
```
|
||||||
|
|
||||||
|
The `glshader` element does two jobs:
|
||||||
|
|
||||||
|
1. Pad up to the encoder's macroblock multiple (128 for H.264, 16 for PNG)
|
||||||
|
using a `u_transformation` matrix.
|
||||||
|
2. For the **alpha pipeline**, run `vec4(pix.a, pix.a, pix.a, 0.)` so
|
||||||
|
alpha is encoded as a grayscale H.264 stream — H.264 has no alpha, so
|
||||||
|
Greenfield ships two parallel AU streams.
|
||||||
|
|
||||||
|
Bitrates: opaque 12000 / alpha 1200 (x264), 12000 / 1200 (nvh264).
|
||||||
|
|
||||||
|
## Wire format
|
||||||
|
|
||||||
|
Single contiguous blob, little-endian:
|
||||||
|
|
||||||
|
```
|
||||||
|
struct packed props {
|
||||||
|
u32 buffer_id, creation_serial, content_serial,
|
||||||
|
u32 frame_encoding_type (0=h264, 1=png),
|
||||||
|
u32 width, height, coded_width, coded_height
|
||||||
|
}
|
||||||
|
u32 opaque_length
|
||||||
|
u8 opaque[opaque_length] // H.264 AU or PNG blob
|
||||||
|
u32 alpha_length // 0 if no alpha split
|
||||||
|
u8 alpha[alpha_length] // H.264 AU of alpha-only stream
|
||||||
|
```
|
||||||
|
|
||||||
|
Parsed by `remote/EncodedFrame.ts::createEncodedFrame`.
|
||||||
|
|
||||||
|
## Transport (`Channel.ts`)
|
||||||
|
|
||||||
|
Four logical channels per client multiplexed over WebSockets:
|
||||||
|
|
||||||
|
| Channel | Type | Purpose |
|
||||||
|
|---|---|---|
|
||||||
|
| PROTOCOL | ARQ (KCP) | Wayland wire protocol + out-of-band ops |
|
||||||
|
| FRAME | ARQ (KCP) | Encoded surface frames |
|
||||||
|
| XWM | ARQ (KCP) | XWayland WM messages |
|
||||||
|
| FEEDBACK | SIMPLE (raw WS) | 4-byte feedback messages |
|
||||||
|
|
||||||
|
ARQ = a JS KCP implementation layered over WebSocket. Tuning:
|
||||||
|
`MTU=64000`, `MAX_BUFFERED_AMOUNT=2949120` (~2.8 MB — the back-pressure
|
||||||
|
ceiling), `SND_WINDOW=128`, `RCV_WINDOW=1024`,
|
||||||
|
`setNoDelay(1, 100, 0, 1)`.
|
||||||
|
|
||||||
|
## Browser decode (`packages/compositor/src/remote/`)
|
||||||
|
|
||||||
|
At session startup `Session.create()` probes
|
||||||
|
`VideoDecoder.isConfigSupported({codec:'avc1.64001f', optimizeForLatency:true,
|
||||||
|
hardwareAcceleration:'prefer-software'})`:
|
||||||
|
|
||||||
|
### WebCodecs preferred
|
||||||
|
|
||||||
|
Software-only; the `prefer-hardware` path is implemented but commented out
|
||||||
|
with `// FIXME hardware decoding frame offset is wrong`.
|
||||||
|
|
||||||
|
### FFmpeg Wasm fallback
|
||||||
|
|
||||||
|
If WebCodecs unavailable: two persistent Workers (`h264NALDecoderOpaque`,
|
||||||
|
`h264NALDecoderAlpha`), one `AVCodecContext` per surface per worker,
|
||||||
|
keyed by `surfaceH264DecodeId`. Encoded NALs are transferred zero-copy
|
||||||
|
into the worker; decoded I420 planes are transferred back.
|
||||||
|
|
||||||
|
Per-surface decoder context is lazily created on first H.264 frame.
|
||||||
|
|
||||||
|
### PNG path (both decoders)
|
||||||
|
|
||||||
|
Identical in both branches: `new Blob([opaque], {type:'image/png'})` →
|
||||||
|
`createImageBitmap(blob, sx, sy, w, h)` (the `sx/sy` crops the encoder
|
||||||
|
padding back to logical size).
|
||||||
|
|
||||||
|
## WebGL upload (`render/`)
|
||||||
|
|
||||||
|
WebGL 1 with `desynchronized:true` for low-latency presentation. Per Scene
|
||||||
|
(canvas/output), three GL helpers:
|
||||||
|
|
||||||
|
### `YUVA2RGBA`
|
||||||
|
|
||||||
|
FBO pass: uploads Y/U/V/alphaY as **4× LUMINANCE textures**, renders into
|
||||||
|
the surface's RGBA texture via `YUVA2RGBAShader` (BT.601 coefficients,
|
||||||
|
alpha recovered by running the alpha-Y plane through the same YUV→RGB
|
||||||
|
matrix and taking R).
|
||||||
|
|
||||||
|
### `RGBXandA2RGBA`
|
||||||
|
|
||||||
|
FBO pass for the `DualPlaneRGBA*` WebCodecs outputs: opaque + alpha planes
|
||||||
|
composited via `RGBandA2RGBAShader`.
|
||||||
|
|
||||||
|
### `SceneShader`
|
||||||
|
|
||||||
|
Final pass: draws every view's RGBA texture to the canvas with
|
||||||
|
`gl.blendFunc(ONE, ONE_MINUS_SRC_ALPHA)` (note: expects premultiplied
|
||||||
|
alpha but the YUVA shader outputs straight alpha — a minor inconsistency).
|
||||||
|
|
||||||
|
PNG and web-bitmapbuf paths skip the FBO and call
|
||||||
|
`texImage2D(ImageBitmap)` directly.
|
||||||
|
|
||||||
|
### Alpha pre-multiplication
|
||||||
|
|
||||||
|
Not done explicitly anywhere. The proxy extracts alpha as a Y plane; the
|
||||||
|
browser shader recombines; the canvas blend mode assumes premultiplied but
|
||||||
|
receives straight alpha — works acceptably because Wayland surfaces rarely
|
||||||
|
use partial alpha.
|
||||||
|
|
||||||
|
## End-to-end pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────── NATIVE CLIENT PROCESS ────────────────────┐
|
||||||
|
│ App renders via GTK/Qt/raw EGL into │
|
||||||
|
│ ┌────────────────────────────────────┐ │
|
||||||
|
│ │ wl_shm buffer (ARGB8888/XRGB8888)│ ─── or ─────────┐ │
|
||||||
|
│ │ zwp_linux_dmabuf_v1 (DMA-BUF) │ │ │
|
||||||
|
│ │ wl_drm create_prime_buffer │ │ │
|
||||||
|
│ └────────────────────────────────────┘ │ │
|
||||||
|
└────────────────────────┬──────────────────────────────────┼────┘
|
||||||
|
│ Wayland socket (UNIX domain) │
|
||||||
|
▼ │
|
||||||
|
┌────────────────────── COMPOSITOR PROXY (Node + N-API addons)─┐
|
||||||
|
│ wayland-server-addon.node │
|
||||||
|
│ └─ wl_display → wl_client → wl_resource (buffer) │
|
||||||
|
│ └─ initShm(), initDrm("/dev/dri/renderD128") │
|
||||||
|
│ └─ westfield-egl.c: EGLDevice + EGLDisplay + EGLContext │
|
||||||
|
│ │
|
||||||
|
│ proxy-encoding-addon.node (per-surface Encoder) │
|
||||||
|
│ ┌─ node_encoder.c → frame_buffer {type: SHM|DMA, w, h, fmt} │
|
||||||
|
│ │ │
|
||||||
|
│ │ gst_main_loop.c (own GLib thread, eventfd work queue) │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ ▼ │
|
||||||
|
│ │ gst_frame_encoder.c │
|
||||||
|
│ │ ┌─ Encoder selection (per-frame): │
|
||||||
|
│ │ │ pixel_count ≤ 256×256 ─► PNG pipeline │
|
||||||
|
│ │ │ > 256×256 + ARGB/XRGB/dmabuf ─► configured h264 │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ │ H.264 pipelines (parallel): │
|
||||||
|
│ │ │ opaque: appsrc → glupload → glcolorconvert → │
|
||||||
|
│ │ │ glshader(scale→coded_size) → │
|
||||||
|
│ │ │ glcolorconvert(NV12) → gldownload → │
|
||||||
|
│ │ │ queue → x264enc|nvh264enc|vaapih264enc → │
|
||||||
|
│ │ │ byte-stream AU → appsink │
|
||||||
|
│ │ │ alpha: same source, fragment shader │
|
||||||
|
│ │ │ `vec4(pix.a,pix.a,pix.a,0.)` → │
|
||||||
|
│ │ │ second H.264 stream │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ │ PNG pipeline (single): │
|
||||||
|
│ │ │ appsrc → glupload → glcolorconvert → glshader → │
|
||||||
|
│ │ │ RGBA → gldownload → pngenc → appsink │
|
||||||
|
│ │ │ │
|
||||||
|
│ │ └─ frame_encoding_result_to_encoded_frame(): │
|
||||||
|
│ │ packed struct {buffer_id, creation_serial, │
|
||||||
|
│ │ content_serial, type(0=h264,1=png), w, h, │
|
||||||
|
│ │ coded_w, coded_h} + opaque_len + opaque + │
|
||||||
|
│ │ alpha_len + alpha → Node Buffer (zero-copy ext.) │
|
||||||
|
│ │ │
|
||||||
|
│ └─ SurfaceBufferEncoding.ts │
|
||||||
|
│ └─ frameDataChannel.send(buffer) │
|
||||||
|
│ │
|
||||||
|
│ Channel.ts: │
|
||||||
|
│ ARQChannel (KCP-reliable over WebSocket, MTU 64 KB) │
|
||||||
|
│ SimpleChannel (raw WebSocket, for FEEDBACK only) │
|
||||||
|
│ ▲ │
|
||||||
|
│ │ FrameFeedback.ts: 16-bit rAF interval + 16-bit avg │
|
||||||
|
│ │ decode duration → used only to pace wl_frame.done │
|
||||||
|
└───────┼───────────────────────────────────────────────────────┘
|
||||||
|
│ 4 logical channels per client: PROTOCOL / FRAME / XWM / FEEDBACK
|
||||||
|
▼
|
||||||
|
┌────────────────────── BROWSER (compositor) ────────────────────┐
|
||||||
|
│ WebSocket → ReconnectingWebSocket │
|
||||||
|
│ ↓ │
|
||||||
|
│ Channel.ts: ARQChannel.input(kcp) → onMessage │
|
||||||
|
│ ↓ │
|
||||||
|
│ RemoteAppLauncher.setupFrameDataChannel → │
|
||||||
|
│ BufferStream.deliverContentToBufferStream (parse serials) │
|
||||||
|
│ ↓ │
|
||||||
|
│ StreamingBuffer.getContents(surface, serial) │
|
||||||
|
│ ↓ │
|
||||||
|
│ Session.frameDecoder.decode(surface, encodedFrame) │
|
||||||
|
│ │ │
|
||||||
|
│ ├─ PNG path (both decoders): │
|
||||||
|
│ │ new Blob([opaque], {type:'image/png'}) → │
|
||||||
|
│ │ createImageBitmap(blob, sx, sy, w, h) // crop padding │
|
||||||
|
│ │ → SinglePlane { bitmap: ImageBitmap } │
|
||||||
|
│ │ │
|
||||||
|
│ └─ H.264 path: │
|
||||||
|
│ (preferred) WebCodecs VideoDecoder │
|
||||||
|
│ config: avc1.64001f, prefer-software, optimizeForLatency│
|
||||||
|
│ opaqueDecoder + alphaDecoder per surface │
|
||||||
|
│ EncodedVideoChunk {type:key|delta, data:nal} │
|
||||||
|
│ → VideoFrame.output → │
|
||||||
|
│ if I420: copyTo → DualPlaneYUVAArrayBuffer │
|
||||||
|
│ else: keep VideoFrame → DualPlaneRGBAVideoFrame │
|
||||||
|
│ (fallback) FFmpeg Wasm in 2 Web Workers │
|
||||||
|
│ worker.postMessage(nal, [nal.buffer]) // transferable │
|
||||||
|
│ libavH264._decode → I420 planes │
|
||||||
|
│ postMessage back with [y.buffer, u.buffer, v.buffer] │
|
||||||
|
│ → DualPlaneYUVASplitBuffer │
|
||||||
|
│ │
|
||||||
|
│ Renderer.render() (rAF-driven, queueCancellableMicrotask) │
|
||||||
|
│ ↓ │
|
||||||
|
│ Scene.render(viewStack) │
|
||||||
|
│ ├─ Scene['image/png'] → texImage2D(ImageBitmap) │
|
||||||
|
│ ├─ Scene['image/bitmap'] (web-bitmapbuf path) → texImage2D │
|
||||||
|
│ └─ Scene['video/h264']: │
|
||||||
|
│ YUVA2RGBA.convertYUVASplitBufferInto / │
|
||||||
|
│ convertYUVAArrayBufferInto: │
|
||||||
|
│ Y/U/V/alphaY → 4× LUMINANCE textures │
|
||||||
|
│ bind FBO → renderState.texture (RGBA) as color attach │
|
||||||
|
│ YUVA2RGBAShader (BT.601) → drawArrays → RGBA in tex │
|
||||||
|
│ or RGBXandA2RGBA.convertInto (WebCodecs RGBA): │
|
||||||
|
│ VideoFrame/ImageBitmap → texImage2D directly │
|
||||||
|
│ ↓ │
|
||||||
|
│ SceneShader.draw(): drawArrays with surface's RGBA texture, │
|
||||||
|
│ blend gl.ONE / gl.ONE_MINUS_SRC_ALPHA, desynchronized canvas │
|
||||||
|
│ ↓ │
|
||||||
|
│ Browser composites canvas into the page │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
|
||||||
|
|
||||||
|
Same-frame web app fast path (web-bitmapbuf):
|
||||||
|
┌───────────────┐ MessagePort.postMessage([view, ...fds], ┌──────────────┐
|
||||||
|
│ Web app │ ──► [imageBitmap]) ──────────────────────────► │ Compositor │
|
||||||
|
│ (iframe) │ web_bitmapbuf_factory.create_buffer(id, fd) │ WebConnHandl │
|
||||||
|
└───────────────┘ → wl_buffer → ImageBitmapBuffer └──────────────┘
|
||||||
|
(no encoding, no WebSocket, no GStreamer)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key answers
|
||||||
|
|
||||||
|
### Why H.264 over PNG?
|
||||||
|
|
||||||
|
H.264 at 12 Mbps × 1/60s ≈ 25 KB/frame for 1080p, vs PNG 100 KB – 3 MB.
|
||||||
|
PNG wins for ≤256×256 because H.264 I-frame overhead and alpha-split
|
||||||
|
doubling dominate at small sizes.
|
||||||
|
|
||||||
|
### WebCodecs?
|
||||||
|
|
||||||
|
Yes — preferred path, software-only (`prefer-hardware` is wired but
|
||||||
|
disabled with FIXME "frame offset is wrong"). FFmpeg Wasm is the fallback.
|
||||||
|
|
||||||
|
### Trade-offs
|
||||||
|
|
||||||
|
- H.264 = small wire / decode cost
|
||||||
|
- PNG = medium wire / browser-native fast decode (`createImageBitmap`)
|
||||||
|
- raw SHM = **not used** between proxy and browser — every surface commit
|
||||||
|
is encoded.
|
||||||
|
|
||||||
|
### Per-surface policy?
|
||||||
|
|
||||||
|
No — per-frame deterministic. Only configurable knob is
|
||||||
|
`--encoder=x264|nvh264|vaapih264`. PNG is always automatic fallback.
|
||||||
|
|
||||||
|
### Cursors?
|
||||||
|
|
||||||
|
Always PNG (32×32 ≤ 256×256). Decoded to ImageBitmap and set as the CSS
|
||||||
|
cursor via `setBrowserCursor` — never enters the WebGL compositor.
|
||||||
|
|
||||||
|
### Alpha pre-multiplication?
|
||||||
|
|
||||||
|
Implicit in shaders. Proxy extracts alpha as a Y plane (no premult);
|
||||||
|
browser recombines via BT.601 YUVA→RGBA shader outputting straight alpha;
|
||||||
|
canvas blend is `ONE, ONE_MINUS_SRC_ALPHA` (expects premult) — minor
|
||||||
|
inconsistency, not visually problematic for typical Wayland content.
|
||||||
|
|
||||||
|
### OffscreenCanvas / Web Workers?
|
||||||
|
|
||||||
|
Workers yes (two for Wasm decoder, shared across surfaces). OffscreenCanvas
|
||||||
|
**no** — renderer is main-thread WebGL with `desynchronized:true`.
|
||||||
|
WebGPU/WebGL2 not used.
|
||||||
|
|
||||||
|
## Notable design trade-offs for your compositor
|
||||||
|
|
||||||
|
1. Always-encode policy (no raw SHM escape hatch even on LAN).
|
||||||
|
2. One encoder per surface, EOS-and-recreate on resize/format change.
|
||||||
|
3. Alpha as a separate H.264 stream — doubles cost for transparent windows.
|
||||||
|
4. Per-surface decoder context, no decoder pooling.
|
||||||
|
5. Only aggregate feedback (1 Hz rAF interval + decode duration); no
|
||||||
|
per-frame ACKs, no bitrate adaptation; `requestKeyUnit` plumbing exists
|
||||||
|
end-to-end but the browser never calls it.
|
||||||
|
6. KCP over WebSocket (not WebRTC datachannels) — simpler infra, no
|
||||||
|
STUN/TURN, but no real congestion control.
|
||||||
|
7. Emscripten SIMD (not threads) — avoids COOP/COEP requirements.
|
||||||
|
8. Two parallel GL pipelines (opaque + alpha) for H.264.
|
||||||
|
9. PNG fallback boundary empirically at 256×256 pixels.
|
||||||
|
10. WebGL 1, not 2/WebGPU — maximum browser compat at the cost of manual
|
||||||
|
LUMINANCE-texture YUV juggling and a hidden FBO intermediate.
|
||||||
|
|
||||||
|
## File reference
|
||||||
|
|
||||||
|
### Wasm libraries
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|---|---|
|
||||||
|
| `libs/compositor-wasm/build_wasm.sh` | Builds pixman 0.42.2 + xkbcommon 1.5.0 to WASM (emsdk 3.1.46) |
|
||||||
|
| `libs/compositor-wasm/emscripten-build.ini` | Emscripten build config |
|
||||||
|
| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function |
|
||||||
|
| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon build (clones 1.5.0, embeds xkeyboard-config DB) |
|
||||||
|
| `libs/compositor-wasm/pixman/build.sh` | pixman build |
|
||||||
|
| `libs/compositor-ffmpeg-h264/build_wasm.sh` | Builds FFmpeg n5.1.2 to WASM (single-file, decoder-only) |
|
||||||
|
| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API |
|
||||||
|
| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module |
|
||||||
|
| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point |
|
||||||
|
| `libs/compositor-ffmpeg-h264/src/index.ts` | Exports |
|
||||||
|
|
||||||
|
### Native encoder addon
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon (290 lines) |
|
||||||
|
| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder (~1400 lines) |
|
||||||
|
| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
|
||||||
|
| `packages/compositor-proxy/native/CMakeLists.txt` | Build config |
|
||||||
|
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | TS-side encoding orchestration |
|
||||||
|
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
|
||||||
|
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback |
|
||||||
|
| `packages/compositor-proxy/src/encoding/` | High-level encoding files |
|
||||||
|
|
||||||
|
### Browser decode
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||||||
|
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs VideoDecoder path |
|
||||||
|
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||||||
|
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||||||
|
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
|
||||||
|
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||||||
|
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||||||
|
|
||||||
|
### WebGL rendering
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor/src/render/Renderer.ts` | 4-phase render pipeline |
|
||||||
|
| `packages/compositor/src/render/Scene.ts` | One per output, manages view stack + textures |
|
||||||
|
| `packages/compositor/src/render/SceneShader.ts` | Final pass: textured quads to canvas |
|
||||||
|
| `packages/compositor/src/render/Texture.ts` | WebGL texture wrapper |
|
||||||
|
| `packages/compositor/src/render/RenderState.ts` | Per-view-per-scene texture + transform |
|
||||||
|
| `packages/compositor/src/render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
|
||||||
|
| `packages/compositor/src/render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
|
||||||
|
| `packages/compositor/src/render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
|
||||||
|
| `packages/compositor/src/render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
|
||||||
|
| `packages/compositor/src/render/RGBandA2RGBAShader.ts` | Shader for above |
|
||||||
|
| `packages/compositor/src/render/Program.ts` | WebGL program wrapper |
|
||||||
|
| `packages/compositor/src/render/ShaderCompiler.ts` | Shader compile/link helper |
|
||||||
|
| `packages/compositor/src/render/ShaderSources.ts` | GLSL source strings |
|
||||||
|
|
||||||
|
### Protocol extensions
|
||||||
|
|
||||||
|
| Path | Role |
|
||||||
|
|---|---|
|
||||||
|
| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom Wayland extension for browser JS apps |
|
||||||
|
| `protocol/wayland-drm.xml` | Standard wl_drm for DMA-BUF (proxy-only) |
|
||||||
|
| `packages/compositor/src/web/WebBuffersFactory.ts` | Browser-side impl of web_bitmapbuf_factory |
|
||||||
@@ -0,0 +1,644 @@
|
|||||||
|
# 10 — Porting Blueprint (JS Browser + Python Host + Odoo 20 OWL Shell)
|
||||||
|
|
||||||
|
This is the standalone porting chapter. It maps every Greenfield layer
|
||||||
|
analyzed in chapters 02-09 to concrete guidance for building a
|
||||||
|
Wayland-to-HTML compositor that integrates with an Odoo 20 addon shell and
|
||||||
|
a JS + Python backend.
|
||||||
|
|
||||||
|
## Target architecture
|
||||||
|
|
||||||
|
The recommended architecture:
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ BROWSER (Odoo 20 web client) │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Odoo 20 OWL Shell addon (your code) │ │
|
||||||
|
│ │ - Application launcher (lists apps from /api/compositor/apps) │ │
|
||||||
|
│ │ - Window management UI (taskbar, decorations) │ │
|
||||||
|
│ │ - Keyboard layout picker (RMLVO) │ │
|
||||||
|
│ │ - Per-output <canvas> element │ │
|
||||||
|
│ │ - Triggers app launches via Odoo /api/compositor/launch │ │
|
||||||
|
│ │ - Implements UserShellApi.events.* callbacks │ │
|
||||||
|
│ └──────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ userShell.events + userShell.actions │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ @gfld/compositor (browser-side compositor library) │ │
|
||||||
|
│ │ - Use as-is from npm: @gfld/compositor │ │
|
||||||
|
│ │ - Replaces compositor-shell with createOdooUserShellApi(session) │ │
|
||||||
|
│ │ - Renders surfaces to <canvas> via WebGL │ │
|
||||||
|
│ │ - Manages keyboard/pointer/touch │ │
|
||||||
|
│ └──────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
└──────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
│ WebSocket (TCP, KCP-over-WS)
|
||||||
|
│ 9 endpoint paths per session
|
||||||
|
▼
|
||||||
|
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ PYTHON HOST (Linux server) │
|
||||||
|
│ │
|
||||||
|
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Odoo 20 backend addon (your code) │ │
|
||||||
|
│ │ - /api/compositor/apps → list of available apps │ │
|
||||||
|
│ │ - /api/compositor/launch → triggers proxy to spawn app │ │
|
||||||
|
│ │ - Session management (one Odoo session ↔ one proxy session) │ │
|
||||||
|
│ │ - Per-tenant isolation, ACL, audit log │ │
|
||||||
|
│ └──────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ spawn / IPC │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Python compositor-proxy (your code — replaces @gfld/compositor-proxy) │ │
|
||||||
|
│ │ - aiohttp/FastAPI WebSocket server (replaces ws + ws-server) │ │
|
||||||
|
│ │ - pywayland (cffi bindings to libwayland-server) │ │
|
||||||
|
│ │ - python-xkbcommon for keymap compilation │ │
|
||||||
|
│ │ - GStreamer Python bindings (gi.repository.Gst) for H.264 encoding │ │
|
||||||
|
│ │ - asyncio event loop integrating libwayland FD polling │ │
|
||||||
|
│ │ - Per-app child_process spawning (subprocess.Popen with env) │ │
|
||||||
|
│ │ - Sandboxing via bubblewrap + seccomp (NEW — Greenfield has none) │ │
|
||||||
|
│ └──────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ WAYLAND_DISPLAY=wayland-N │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Native Linux apps (gtk4-demo, xterm, kwrite, ...) │ │
|
||||||
|
│ │ - Use host system libraries (no sysroot) │ │
|
||||||
|
│ │ - Sandboxed via bubblewrap + seccomp │ │
|
||||||
|
│ └──────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
└──────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Per-layer porting guidance
|
||||||
|
|
||||||
|
For each Greenfield layer analyzed in chapters 02-09:
|
||||||
|
|
||||||
|
### Layer 1: Wayland Protocol XML
|
||||||
|
|
||||||
|
**What to copy as-is**: All four protocol XML files (`wayland.xml`,
|
||||||
|
`xdg-shell.xml`, `wayland-drm.xml`, `web-bitmapbuf-unstable-v1.xml`).
|
||||||
|
They are upstream standard + Greenfield's custom extension. No need to
|
||||||
|
rewrite.
|
||||||
|
|
||||||
|
**What to replace**: Nothing. The XML is the source of truth.
|
||||||
|
|
||||||
|
**Where Python fits**: Python doesn't need to parse the XML directly —
|
||||||
|
use `pywayland` which already ships with parsed bindings for the standard
|
||||||
|
protocols. For Greenfield's custom `web-bitmapbuf-unstable-v1`, you can
|
||||||
|
either skip it (native apps don't use it) or generate Python bindings via
|
||||||
|
`pywayland-protocol-scanner`.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: Not applicable. This layer is
|
||||||
|
below the shell.
|
||||||
|
|
||||||
|
### Layer 2: Protocol Generators
|
||||||
|
|
||||||
|
**What to copy as-is**: The TypeScript generators for the browser side
|
||||||
|
(`client-generator`, `compositor-generator`). They produce stubs for the
|
||||||
|
browser compositor and browser-resident JS apps.
|
||||||
|
|
||||||
|
**What to replace**: The `compositor-proxy-generator`. Greenfield's proxy
|
||||||
|
generates CommonJS `*_interface.js` + `*_interceptor.js` files for the
|
||||||
|
Node N-API addon. For Python, you don't need a generator — use
|
||||||
|
`pywayland`'s built-in protocol bindings.
|
||||||
|
|
||||||
|
**Where Python fits**: `pywayland` provides `wl_interface` structs and
|
||||||
|
request/event dispatch in pure Python via cffi. You write Python classes
|
||||||
|
that implement `WlCompositorRequests`, `WlSeatRequests`, etc.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: Not applicable.
|
||||||
|
|
||||||
|
### Layer 3: Wayland Runtime
|
||||||
|
|
||||||
|
**What to copy as-is**: `@gfld/common` (`Connection.ts`) and
|
||||||
|
`@gfld/compositor-protocol` for the browser side. These implement the
|
||||||
|
standard Wayland wire format and object lifecycle, and they're MIT-licensed.
|
||||||
|
|
||||||
|
**What to replace**: The proxy's `wayland-server.ts` (which wraps the
|
||||||
|
`wayland-server-addon.node` N-API addon). Replace with `pywayland`'s
|
||||||
|
`Display` and `Client` classes.
|
||||||
|
|
||||||
|
**Where Python fits**: `pywayland` provides:
|
||||||
|
- `pywayland.server.Display` — wraps `wl_display`
|
||||||
|
- `pywayland.server.Client` — wraps `wl_client`
|
||||||
|
- `pywayland.server.Resource` — wraps `wl_resource`
|
||||||
|
- `pywayland.server.Global` — wraps `wl_global`
|
||||||
|
- Per-resource event/request dispatch via Python method overrides
|
||||||
|
|
||||||
|
You write Python classes that subclass `WlCompositor`, `WlSeat`, etc. —
|
||||||
|
similar to how Greenfield's TypeScript classes implement `*Requests`
|
||||||
|
interfaces.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: Not applicable. The runtime
|
||||||
|
layer is below the shell.
|
||||||
|
|
||||||
|
### Layer 4: Transport
|
||||||
|
|
||||||
|
**What to copy as-is**: The browser side of `Channel.ts` (`ARQChannel`
|
||||||
|
+ `SimpleChannel` + `ReconnectingWebSocket`) and `kcp.ts`. The TypeScript
|
||||||
|
KCP implementation is solid and battle-tested.
|
||||||
|
|
||||||
|
**What to replace**: The native side of `Channel.ts`. Replace Node's `ws`
|
||||||
|
library with `aiohttp` WebSocket or `FastAPI` WebSocket.
|
||||||
|
|
||||||
|
**Where Python fits**:
|
||||||
|
|
||||||
|
- Use `aiohttp.web.WebSocketResponse` or `starlette.websockets.WebSocket`
|
||||||
|
for the WebSocket server.
|
||||||
|
- Port the KCP implementation from TypeScript to Python (or wrap a C
|
||||||
|
implementation via cffi — `pykcp` exists).
|
||||||
|
- Use `asyncio` event loop integration: `pywayland` exposes the
|
||||||
|
libwayland event-loop FD, which you can integrate via
|
||||||
|
`asyncio.loop.add_reader()`.
|
||||||
|
- For multiplexing 9 endpoints, use `aiohttp.web.Application` with 9
|
||||||
|
routes, or `FastAPI` with 9 WebSocket endpoints.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: The shell connects to the
|
||||||
|
proxy via the same WebSocket URLs. No change needed.
|
||||||
|
|
||||||
|
### Layer 5: Rendering Pipeline (browser side)
|
||||||
|
|
||||||
|
**What to copy as-is**: The entire `packages/compositor/src/` rendering
|
||||||
|
pipeline. This is the heart of Greenfield and it's where most of the
|
||||||
|
engineering effort went. Reuse it wholesale:
|
||||||
|
- `Surface.ts`, `View.ts`, `Output.ts`, `Loop.ts`, `Callback.ts`,
|
||||||
|
`Region.ts`, `Shm.ts`, `BufferContents.ts`, `BufferImplementation.ts`,
|
||||||
|
`ImageBitmapBuffer.ts`
|
||||||
|
- `render/` directory (Renderer, Scene, SceneShader, Texture, RenderState,
|
||||||
|
shaders)
|
||||||
|
- `remote/` directory (StreamingBuffer, BufferStream, EncodedFrame,
|
||||||
|
EncoderFeedback, decoders)
|
||||||
|
|
||||||
|
**What to replace**: Nothing. This layer is pure browser TypeScript and
|
||||||
|
works as-is.
|
||||||
|
|
||||||
|
**Where Python fits**: Not applicable. The browser does all rendering.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: The shell owns the
|
||||||
|
`<canvas>` element and passes it to
|
||||||
|
`userShell.actions.initScene(canvasCreator)`. The compositor creates a
|
||||||
|
`Scene` per canvas.
|
||||||
|
|
||||||
|
### Layer 6: Surface Lifecycle / Roles
|
||||||
|
|
||||||
|
**What to copy as-is**: All of `XdgWmBase.ts`, `XdgSurface.ts`,
|
||||||
|
`XdgToplevel.ts`, `XdgPopup.ts`, `XdgPositioner.ts`, `Shell.ts`,
|
||||||
|
`ShellSurface.ts`, `Subcompositor.ts`, `Subsurface.ts`, `SurfaceRole.ts`,
|
||||||
|
`SurfaceChild.ts`. These implement Wayland's role state machines correctly.
|
||||||
|
|
||||||
|
**What to replace**: Nothing.
|
||||||
|
|
||||||
|
**Where Python fits**: Not applicable (this is browser-side).
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: The shell implements
|
||||||
|
`UserShellApi.events` callbacks (see `08-app-lifecycle.md`). Specifically:
|
||||||
|
- `surfaceCreated` → add a window to the OWL taskbar
|
||||||
|
- `surfaceDestroyed` → remove from taskbar
|
||||||
|
- `surfaceTitleUpdated` → update window title
|
||||||
|
- `surfaceActivationUpdated` → mark window as active
|
||||||
|
- `clientCreated` / `clientDestroyed` → update app registry
|
||||||
|
- `clientUnresponsiveUpdated` → show "not responding" indicator
|
||||||
|
|
||||||
|
### Layer 7: Desktop / Layout Policy
|
||||||
|
|
||||||
|
**What to copy as-is**: `desktop/Desktop.ts` provides Floating and
|
||||||
|
AlwaysFullScreen modes. Floating is what most users want (windows can be
|
||||||
|
moved/resized freely).
|
||||||
|
|
||||||
|
**What to replace**: If your Odoo shell needs custom layout policies
|
||||||
|
(e.g. tiling, tabbed), you'll need to extend `DesktopSurface` with new
|
||||||
|
modes. But you can start with Floating.
|
||||||
|
|
||||||
|
**Where Python fits**: Not applicable.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: The shell calls
|
||||||
|
`userShell.actions.activateSurface(compositorSurface)` when the user
|
||||||
|
clicks a taskbar entry.
|
||||||
|
|
||||||
|
### Layer 8: User Shell (the integration seam)
|
||||||
|
|
||||||
|
**What to copy as-is**: `UserShellApi.ts`. The interface is clean and
|
||||||
|
shell-agnostic.
|
||||||
|
|
||||||
|
**What to replace**: Replace `compositor-shell` (Preact) with your Odoo
|
||||||
|
20 OWL addon.
|
||||||
|
|
||||||
|
**Where Python fits**: Not applicable. The shell is browser-side.
|
||||||
|
|
||||||
|
**Where the Odoo 20 OWL shell hooks in**: This IS the integration point.
|
||||||
|
Create an OWL component that:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// Pseudocode for Odoo 20 OWL shell
|
||||||
|
import { Component, useState, onMounted } from "@odoo/owl";
|
||||||
|
import { createCompositorSession } from "@gfld/compositor";
|
||||||
|
|
||||||
|
export class CompositorShell extends Component {
|
||||||
|
setup() {
|
||||||
|
this.state = useState({
|
||||||
|
surfaces: new Map(), // id → { title, appId, active }
|
||||||
|
clients: new Map(),
|
||||||
|
});
|
||||||
|
|
||||||
|
onMounted(async () => {
|
||||||
|
const session = createCompositorSession();
|
||||||
|
this.session = session;
|
||||||
|
|
||||||
|
// Wire UserShellApi events to OWL reactive state
|
||||||
|
session.userShell.events.surfaceCreated = (surface) => {
|
||||||
|
this.state.surfaces.set(surface.id, {
|
||||||
|
title: "", appId: "", active: false
|
||||||
|
});
|
||||||
|
};
|
||||||
|
session.userShell.events.surfaceDestroyed = (surface) => {
|
||||||
|
this.state.surfaces.delete(surface.id);
|
||||||
|
};
|
||||||
|
session.userShell.events.surfaceTitleUpdated = (surface, title) => {
|
||||||
|
const s = this.state.surfaces.get(surface.id);
|
||||||
|
if (s) s.title = title;
|
||||||
|
};
|
||||||
|
session.userShell.events.surfaceActivationUpdated = (surface, active) => {
|
||||||
|
const s = this.state.surfaces.get(surface.id);
|
||||||
|
if (s) s.active = active;
|
||||||
|
};
|
||||||
|
// ... etc for clientCreated, clientDestroyed, etc.
|
||||||
|
|
||||||
|
// Initialize the canvas
|
||||||
|
const canvas = this.canvasRef.el;
|
||||||
|
session.userShell.actions.initScene(() => ({
|
||||||
|
canvas,
|
||||||
|
width: canvas.clientWidth,
|
||||||
|
height: canvas.clientHeight,
|
||||||
|
}));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// User clicks app in launcher
|
||||||
|
async launchApp(appId) {
|
||||||
|
const response = await fetch("/web/compositor/launch", {
|
||||||
|
method: "POST",
|
||||||
|
headers: { "Content-Type": "application/json" },
|
||||||
|
body: JSON.stringify({ app_id: appId }),
|
||||||
|
});
|
||||||
|
const { signal_url } = await response.json();
|
||||||
|
// Browser JS opens signal_url WebSocket → compositor handles the rest
|
||||||
|
}
|
||||||
|
|
||||||
|
// User clicks taskbar entry
|
||||||
|
activateSurface(surfaceId) {
|
||||||
|
this.session.userShell.actions.activateSurface({ id: surfaceId });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Odoo 20 backend addon
|
||||||
|
|
||||||
|
The Odoo 20 backend addon acts as a thin proxy between the OWL shell and
|
||||||
|
the Python compositor-proxy. It handles:
|
||||||
|
|
||||||
|
- App launcher config (per-user, per-tenant allowed apps list)
|
||||||
|
- Auth (Odoo session → compositor-proxy session mapping)
|
||||||
|
- Audit log (which user launched which app when)
|
||||||
|
- Multi-tenant isolation (separate Python proxy processes per tenant,
|
||||||
|
containerized)
|
||||||
|
- Per-tenant resource limits (max concurrent apps, max memory)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Pseudocode for Odoo 20 backend controller
|
||||||
|
from odoo import http
|
||||||
|
from odoo.http import request
|
||||||
|
|
||||||
|
class CompositorController(http.Controller):
|
||||||
|
|
||||||
|
@http.route("/web/compositor/apps", type="json", auth="user")
|
||||||
|
def list_apps(self):
|
||||||
|
# Return user-allowed apps from applications.json
|
||||||
|
user = request.env.user
|
||||||
|
return request.env["compositor.app"].search_read(
|
||||||
|
[("allowed_group_ids", "in", user.groups_id.ids)],
|
||||||
|
["name", "icon", "app_id"],
|
||||||
|
)
|
||||||
|
|
||||||
|
@http.route("/web/compositor/launch", type="json", auth="user")
|
||||||
|
def launch_app(self, app_id):
|
||||||
|
user = request.env.user
|
||||||
|
# Look up the user's compositor-proxy host (per-tenant routing)
|
||||||
|
proxy_host = user.compositor_proxy_host
|
||||||
|
proxy_port = user.compositor_proxy_port
|
||||||
|
|
||||||
|
# Tell the Python proxy to spawn the app
|
||||||
|
# The proxy returns a signal_url + auth key
|
||||||
|
import aiohttp
|
||||||
|
async with aiohttp.ClientSession() as sess:
|
||||||
|
async with sess.post(
|
||||||
|
f"http://{proxy_host}:{proxy_port}/launch",
|
||||||
|
json={
|
||||||
|
"app_id": app_id,
|
||||||
|
"odoo_session_id": request.session.sid,
|
||||||
|
"user_id": user.id,
|
||||||
|
},
|
||||||
|
) as resp:
|
||||||
|
result = await resp.json()
|
||||||
|
|
||||||
|
# Audit log
|
||||||
|
request.env["compositor.audit.log"].create({
|
||||||
|
"user_id": user.id,
|
||||||
|
"app_id": app_id,
|
||||||
|
"timestamp": fields.Datetime.now(),
|
||||||
|
})
|
||||||
|
|
||||||
|
return {"signal_url": result["signal_url"]}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python compositor-proxy (the big new code)
|
||||||
|
|
||||||
|
This is the bulk of the new code. Replace Node's `compositor-proxy` +
|
||||||
|
`compositor-proxy-cli` with a Python equivalent. The key components:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Pseudocode for Python compositor-proxy
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
|
from aiohttp import web, WSMsgType
|
||||||
|
from pywayland.server import Display
|
||||||
|
from xkbcommon import xkb
|
||||||
|
import gi
|
||||||
|
gi.require_version("Gst", "1.0")
|
||||||
|
gi.require_version("GstGL", "1.0")
|
||||||
|
from gi.repository import Gst, GstGL, GLib
|
||||||
|
|
||||||
|
class CompositorProxy:
|
||||||
|
def __init__(self, bind_ip="0.0.0.0", bind_port=8081, render_device="/dev/dri/renderD128"):
|
||||||
|
self.bind_ip = bind_ip
|
||||||
|
self.bind_port = bind_port
|
||||||
|
self.render_device = render_device
|
||||||
|
self.display = Display()
|
||||||
|
self.sessions = {} # session_id → SessionProcess
|
||||||
|
|
||||||
|
async def start(self):
|
||||||
|
# 1. Create wl_display + add socket
|
||||||
|
self.display.create()
|
||||||
|
socket_name = self.display.add_socket_auto()
|
||||||
|
os.environ["WAYLAND_DISPLAY"] = socket_name
|
||||||
|
|
||||||
|
# 2. Init SHM formats (YUV420, NV12, YUV444, NV21)
|
||||||
|
self.init_shm()
|
||||||
|
|
||||||
|
# 3. Init DRM (open /dev/dri/renderD128, register dmabuf globals)
|
||||||
|
self.init_drm()
|
||||||
|
|
||||||
|
# 4. Integrate wl_display FD into asyncio loop
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
loop.add_reader(self.display.get_fd(), self.dispatch_requests)
|
||||||
|
|
||||||
|
# 5. Start GStreamer
|
||||||
|
Gst.init(None)
|
||||||
|
self.gst_loop = GLib.MainLoop()
|
||||||
|
# Run GLib loop in a separate thread (GStreamer requires it)
|
||||||
|
import threading
|
||||||
|
threading.Thread(target=self.gst_loop.run, daemon=True).start()
|
||||||
|
|
||||||
|
# 6. Start aiohttp server
|
||||||
|
app = web.Application()
|
||||||
|
app.add_routes([
|
||||||
|
web.get("/signal", self.handle_signal),
|
||||||
|
web.get("/channel", self.handle_channel),
|
||||||
|
web.post("/launch", self.handle_launch),
|
||||||
|
# ... 6 more endpoints
|
||||||
|
])
|
||||||
|
runner = web.AppRunner(app)
|
||||||
|
await runner.setup()
|
||||||
|
site = web.TCPSite(runner, self.bind_ip, self.bind_port)
|
||||||
|
await site.start()
|
||||||
|
|
||||||
|
# Run forever
|
||||||
|
await asyncio.Event().wait()
|
||||||
|
|
||||||
|
def dispatch_requests(self):
|
||||||
|
"""Called by asyncio when wl_display FD is readable."""
|
||||||
|
self.display.dispatch_events_pending()
|
||||||
|
|
||||||
|
async def handle_launch(self, request):
|
||||||
|
body = await request.json()
|
||||||
|
app_id = body["app_id"]
|
||||||
|
|
||||||
|
# Spawn the app, set WAYLAND_DISPLAY
|
||||||
|
env = {**os.environ, "WAYLAND_DISPLAY": os.environ["WAYLAND_DISPLAY"]}
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
self.applications[app_id]["executable"],
|
||||||
|
env=env,
|
||||||
|
# NEW vs Greenfield: sandbox with bubblewrap
|
||||||
|
preexec_fn=lambda: self.setup_sandbox(),
|
||||||
|
)
|
||||||
|
|
||||||
|
return web.json_response({
|
||||||
|
"signal_url": f"ws://{self.bind_ip}:{self.bind_port}/signal?app_id={app_id}",
|
||||||
|
"pid": proc.pid,
|
||||||
|
})
|
||||||
|
|
||||||
|
def setup_sandbox(self):
|
||||||
|
"""NEW: bubblewrap + seccomp sandbox (Greenfield has none)."""
|
||||||
|
import ctypes
|
||||||
|
libc = ctypes.CDLL("libc.so.6")
|
||||||
|
# ... bwrap / unshare / seccomp_filter setup
|
||||||
|
```
|
||||||
|
|
||||||
|
### Native app bridge (Python equivalent)
|
||||||
|
|
||||||
|
For each Wayland client that connects (one per app), spawn a Python
|
||||||
|
`NativeWaylandClientSession` equivalent that:
|
||||||
|
|
||||||
|
1. Sets up the wire-message interceptor (using pywayland's resource
|
||||||
|
dispatch).
|
||||||
|
2. Creates a `PROTOCOL` ARQ channel (port `kcp.ts` to Python or use
|
||||||
|
`pykcp`).
|
||||||
|
3. Creates a `FRAME` ARQ channel for the surface's encoded content.
|
||||||
|
4. Manages the app's lifecycle (10-min SIGHUP grace, SIGTERM escalation).
|
||||||
|
|
||||||
|
For XWayland, replace `westfield-xwayland.c` with a Python equivalent
|
||||||
|
that:
|
||||||
|
|
||||||
|
1. Creates `/tmp/.X11-unix/X<N>` listener socket.
|
||||||
|
2. Sets `DISPLAY=:N` environment variable.
|
||||||
|
3. Spawns `Xwayland :N -ac -rootless -listen <fd> -displayfd <fd> -wm <fd>`.
|
||||||
|
4. Wraps the `wm_fd` as a Python file-like object.
|
||||||
|
5. Uses `xtsb` (the existing TypeScript X11 bindings) on the browser side
|
||||||
|
to handle XWM logic — the browser side stays the same.
|
||||||
|
|
||||||
|
### Hardware acceleration (Python equivalent)
|
||||||
|
|
||||||
|
For the encoding pipeline:
|
||||||
|
|
||||||
|
- Use GStreamer Python bindings (`gi.repository.Gst`):
|
||||||
|
- `appsrc` for input buffers
|
||||||
|
- `glupload`, `glcolorconvert`, `glshader` for GPU processing
|
||||||
|
- `x264enc` / `nvh264enc` / `vaapih264enc` for H.264 encoding
|
||||||
|
- `appsink` for output samples
|
||||||
|
- Same encoder selection logic as Greenfield (PNG ≤ 256×256, H.264 above).
|
||||||
|
- Run the GLib main loop in a separate thread, use `asyncio.run_in_executor`
|
||||||
|
for cross-thread encoding requests.
|
||||||
|
- For zero-copy dmabuf, use `GstGLMemoryEGL` (same as Greenfield).
|
||||||
|
|
||||||
|
For the browser decode side: **no change**. The browser uses WebCodecs
|
||||||
|
preferred + FFmpeg Wasm fallback, same as Greenfield.
|
||||||
|
|
||||||
|
## Key trade-offs vs staying on Node
|
||||||
|
|
||||||
|
### Staying on Node (use Greenfield as-is, just replace compositor-shell with Odoo)
|
||||||
|
|
||||||
|
**Pros**:
|
||||||
|
- ~90% of Greenfield's code reused as-is (browser compositor + native proxy)
|
||||||
|
- Less code to write/maintain
|
||||||
|
- Faster time-to-market
|
||||||
|
- Battle-tested in Greenfield's rc1 cycle
|
||||||
|
|
||||||
|
**Cons**:
|
||||||
|
- Native Node addons (`.node` files) are Linux-x64 only, ABI-locked to
|
||||||
|
Node major
|
||||||
|
- GStreamer + libwayland integration is via C N-API, harder to extend
|
||||||
|
- Odoo team may have more Python expertise than Node expertise
|
||||||
|
- AGPL-3.0 license on `compositor-proxy` — viral to your modifications
|
||||||
|
|
||||||
|
### Switching to Python
|
||||||
|
|
||||||
|
**Pros**:
|
||||||
|
- Aligns with Odoo's Python backend stack
|
||||||
|
- `pywayland` (cffi) is easier to debug than N-API addons
|
||||||
|
- Python GStreamer bindings are well-documented
|
||||||
|
- Easier to integrate with Odoo's auth/session model
|
||||||
|
- Can avoid AGPL by reimplementing from scratch (or use only MIT libs)
|
||||||
|
|
||||||
|
**Cons**:
|
||||||
|
- Significant new code (rewrite the proxy + native addons in Python)
|
||||||
|
- Python GIL may bottleneck encoding throughput (use multiprocessing or
|
||||||
|
run GStreamer in separate thread)
|
||||||
|
- pywayland is less mature than libwayland's native bindings
|
||||||
|
- You take on the maintenance of a forked Greenfield-equivalent
|
||||||
|
|
||||||
|
### Recommendation: hybrid approach
|
||||||
|
|
||||||
|
1. **Browser side**: Use Greenfield's `@gfld/compositor` library as-is
|
||||||
|
(MIT-licensed via `@gfld/compositor-protocol` + `@gfld/common`, AGPL only
|
||||||
|
on `@gfld/compositor` itself — but the shell is yours so the AGPL
|
||||||
|
virality is contained to the compositor library).
|
||||||
|
2. **Native side**: Start with Greenfield's `compositor-proxy` (AGPL) to
|
||||||
|
get to MVP fast. Then progressively port to Python as needs evolve
|
||||||
|
(sandboxing, multi-tenant, custom auth, Odoo session integration).
|
||||||
|
3. **Shell**: Build the Odoo 20 OWL addon from scratch (clean room,
|
||||||
|
MIT/Apache license).
|
||||||
|
|
||||||
|
## Implementation phases
|
||||||
|
|
||||||
|
### Phase 1 (1-2 weeks): MVP — display a single Linux app
|
||||||
|
|
||||||
|
1. Run Greenfield's compositor-proxy-cli as a Docker container.
|
||||||
|
2. Build minimal Odoo 20 OWL addon that creates a `<canvas>` and
|
||||||
|
`userShell.events` handlers.
|
||||||
|
3. Embed the OWL addon in an Odoo action.
|
||||||
|
4. Trigger app launch via HTTP fetch to the proxy.
|
||||||
|
5. Verify the Linux app displays in the canvas.
|
||||||
|
|
||||||
|
### Phase 2 (2-4 weeks): Multi-app, basic window management
|
||||||
|
|
||||||
|
1. Implement `surfaceCreated` / `surfaceDestroyed` handlers.
|
||||||
|
2. Build a taskbar OWL component listing active surfaces.
|
||||||
|
3. Implement `activateSurface` action on taskbar click.
|
||||||
|
4. Test with 2-3 concurrent apps.
|
||||||
|
|
||||||
|
### Phase 3 (4-8 weeks): Python port of proxy
|
||||||
|
|
||||||
|
1. Replace Node `compositor-proxy-cli` with a Python equivalent.
|
||||||
|
2. Implement WebSocket endpoints in `aiohttp` or `FastAPI`.
|
||||||
|
3. Port the KCP TypeScript implementation to Python (or use `pykcp`).
|
||||||
|
4. Replace `wayland-server-addon.node` with `pywayland`.
|
||||||
|
5. Replace `proxy-encoding-addon.node` with GStreamer Python bindings.
|
||||||
|
6. Add bubblewrap sandboxing (NEW — Greenfield has none).
|
||||||
|
|
||||||
|
### Phase 4 (8-16 weeks): Polish & scale
|
||||||
|
|
||||||
|
1. Multi-tenant isolation (separate Python proxy processes per tenant).
|
||||||
|
2. Per-tenant resource limits (cgroups, max memory, max CPU).
|
||||||
|
3. Audit log integration with Odoo.
|
||||||
|
4. Clipboard sync (native → DOM and reverse).
|
||||||
|
5. Touch input implementation (Greenfield has stub only).
|
||||||
|
6. Primary selection (middle-click paste).
|
||||||
|
7. Adaptive bitrate (consume `EncoderFeedback` data, Greenfield doesn't).
|
||||||
|
|
||||||
|
## Specific replacements for each Greenfield native component
|
||||||
|
|
||||||
|
| Greenfield native component | Python replacement | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `wayland-server-addon.node` (1231 lines C) | `pywayland` (cffi bindings) | Same libwayland-server, Python wrapper |
|
||||||
|
| `proxy-poll-addon.node` (127 lines C) | `asyncio.loop.add_reader()` | Built-in, no addon needed |
|
||||||
|
| `proxy-encoding-addon.node` + `libproxy-encoding.so` (~1700 lines C) | `gi.repository.Gst` (GStreamer Python bindings) | Same GStreamer, Python wrapper |
|
||||||
|
| Patched `libwayland-server` (vendored at `native/wayland/src/wayland-server/`) | Use upstream libwayland-server + Greenfield's patches applied | Apply `changes.diff` to a fork of libwayland, install via system package |
|
||||||
|
| `westfield-*.c` (EGL, dmabuf, xwayland helpers) | Reimplement in Python via `pyEGL` or `python-xkbcommon` + `pygobject`'s `GstGL` | Less mature than Greenfield's C — be prepared to debug |
|
||||||
|
| `xtsb` (X11 TypeScript bindings) | Use as-is on browser side, replace Python side with `python-xlib` | Browser-side xtsb is MIT-licensed |
|
||||||
|
| Node `ws` library | `aiohttp.web.WebSocketResponse` or `starlette.websockets.WebSocket` | Mature, well-documented |
|
||||||
|
| Node `pkg` (single-binary packaging) | PyInstaller or `shiv` | Same end result |
|
||||||
|
| Node `fork()` per session | `multiprocessing.Process` or `os.fork()` | Direct equivalent |
|
||||||
|
| `compositor-proxy-cli` (CLI args) | `argparse` | Standard Python |
|
||||||
|
| `ajv` (JSON schema validation) | `jsonschema` | Standard Python |
|
||||||
|
|
||||||
|
## Final architecture diagram (target state)
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ BROWSER (Odoo 20 web client) │
|
||||||
|
│ │
|
||||||
|
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Odoo 20 OWL Shell addon (your code, MIT/Apache) │ │
|
||||||
|
│ │ - Application launcher, window manager, keyboard picker │ │
|
||||||
|
│ │ - Per-output <canvas> │ │
|
||||||
|
│ │ - Implements UserShellApi.events.* callbacks │ │
|
||||||
|
│ └────────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ userShell.events + actions │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ @gfld/compositor (browser-side, MIT libs + AGPL compositor) │ │
|
||||||
|
│ │ - Surface, View, Output, Loop, Seat, Keyboard, Pointer, Touch │ │
|
||||||
|
│ │ - render/ (WebGL shaders) │ │
|
||||||
|
│ │ - remote/ (Channel, KCP, decoders) │ │
|
||||||
|
│ └────────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
└──────────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
│ WebSocket + KCP-over-WS
|
||||||
|
│ 9 endpoint paths per session
|
||||||
|
▼
|
||||||
|
┌──────────────────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ PYTHON HOST (Linux server) │
|
||||||
|
│ │
|
||||||
|
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Odoo 20 backend addon (your code, MPL-2.0/LGPL) │ │
|
||||||
|
│ │ - /web/compositor/apps, /web/compositor/launch │ │
|
||||||
|
│ │ - Per-user, per-tenant allowed apps list │ │
|
||||||
|
│ │ - Audit log │ │
|
||||||
|
│ └────────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ subprocess / IPC │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Python compositor-proxy (your code, MIT/Apache) │ │
|
||||||
|
│ │ - aiohttp/FastAPI WebSocket server (9 endpoints) │ │
|
||||||
|
│ │ - pywayland (cffi bindings to libwayland-server) │ │
|
||||||
|
│ │ - python-xkbcommon for keymap compilation │ │
|
||||||
|
│ │ - GStreamer Python bindings for H.264 encoding │ │
|
||||||
|
│ │ - asyncio event loop integrating libwayland FD polling │ │
|
||||||
|
│ │ - Per-app subprocess.Popen with bubblewrap sandbox │ │
|
||||||
|
│ │ - KCP port (pykcp or hand-rolled) │ │
|
||||||
|
│ └────────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ │ WAYLAND_DISPLAY=wayland-N │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌────────────────────────────────────────────────────────────────────────────┐ │
|
||||||
|
│ │ Native Linux apps (gtk4-demo, xterm, kwrite, ...) │ │
|
||||||
|
│ │ - Use host system libraries │ │
|
||||||
|
│ │ - Sandboxed via bubblewrap + seccomp │ │
|
||||||
|
│ └────────────────────────────────────────────────────────────────────────────┘ │
|
||||||
|
└──────────────────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
+532
@@ -0,0 +1,532 @@
|
|||||||
|
# 11 — Pitfalls, Anti-patterns, and What to Avoid
|
||||||
|
|
||||||
|
This is the prominent pitfalls chapter. Greenfield's known weaknesses,
|
||||||
|
anti-patterns, and design pitfalls are catalogued here so the user can
|
||||||
|
avoid making the same mistakes in their own Wayland-to-HTML compositor
|
||||||
|
(Odoo 20 + JS/Python). Each pitfall comes with: **what Greenfield does**
|
||||||
|
(with file/directory reference), **why it's a problem** (technical
|
||||||
|
reason), and **what the user should do instead** (concrete recommendation
|
||||||
|
for their JS/Python + Odoo 20 stack).
|
||||||
|
|
||||||
|
## Headline numbers
|
||||||
|
|
||||||
|
- **299** `TODO|FIXME|HACK|XXX|@deprecated|workaround` hits across non-vendored source (after excluding `examples/sdk/*` upstreams and `xtsb/protocol/xproto.xml`).
|
||||||
|
- **13 packages** all pinned to `"version": "1.0.0-rc1"` — no `CHANGELOG.md`, no `SECURITY.md`, no `CONTRIBUTING.md`, no issue/PR templates.
|
||||||
|
- **1 commit** visible on `master` (`6c578f4`, "Fix typo in code snippet (#161)", 2025-10-15) — implying the public repo is a release snapshot, not the live dev branch.
|
||||||
|
- **3 native Node `.node` addons**, all Linux-x64 only.
|
||||||
|
- **0** uses of `OffscreenCanvas` or `transferControlToOffscreen` anywhere in the compositor.
|
||||||
|
- **2** Web Workers used — and only for the WASM H.264 *fallback* decoder path. The primary WebCodecs path uses none.
|
||||||
|
|
||||||
|
## Catalogued pitfalls (A–U)
|
||||||
|
|
||||||
|
### Critical (will bite you on day 1)
|
||||||
|
|
||||||
|
#### A. Native Node addons
|
||||||
|
|
||||||
|
**What Greenfield does**: Three Node-API addons:
|
||||||
|
- `wayland-server-addon.node` (1231 lines C, wraps patched libwayland-server)
|
||||||
|
- `proxy-encoding-addon.node` (290 lines C, wraps GStreamer)
|
||||||
|
- `proxy-poll-addon.node` (127 lines C, wraps libuv `uv_poll_t`)
|
||||||
|
|
||||||
|
All Linux-x64 only, built with CMake + Ninja. ABI-locked to Node major
|
||||||
|
versions.
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Linux-x64 only — no ARM64 support
|
||||||
|
- ABI breaks on Node major upgrades
|
||||||
|
- Hard to ship — must rebuild for each target platform
|
||||||
|
- Hard to debug — C N-API debugging requires core dumps
|
||||||
|
- GStreamer integration is via C, not Python
|
||||||
|
- No Python story
|
||||||
|
|
||||||
|
**What you should do instead**: Use `pywayland` (cffi bindings to
|
||||||
|
libwayland-server), `python-xkbcommon`, and GStreamer's Python bindings
|
||||||
|
(`gi.repository.Gst`). One Python process can replace the entire
|
||||||
|
`compositor-proxy-cli`. See `10-porting-to-odoo-js-python.md` for details.
|
||||||
|
|
||||||
|
#### B. rc1 with no policy
|
||||||
|
|
||||||
|
**What Greenfield does**: All 13 packages at `1.0.0-rc1`. No CHANGELOG,
|
||||||
|
no SECURITY.md, no CONTRIBUTING.md.
|
||||||
|
|
||||||
|
**Why it's a problem**: Breaking changes can land silently. The
|
||||||
|
maintainer's roadmap is unclear.
|
||||||
|
|
||||||
|
**What you should do instead**: Pin to a specific commit hash if you fork
|
||||||
|
it. Track upstream commits manually. Plan for the eventuality that
|
||||||
|
1.0.0 final may break APIs.
|
||||||
|
|
||||||
|
#### C. Encoding rigidity
|
||||||
|
|
||||||
|
**What Greenfield does**: Three H.264 GStreamer pipelines (x264/nvh264/
|
||||||
|
vaapih264) with hand-tuned magic constants:
|
||||||
|
- `bitrate=12000 vbv-buf-capacity=160 bframes=0 rc-lookahead=0`
|
||||||
|
- VA-API is "experimental"
|
||||||
|
- AMD (`vah264enc`) absent
|
||||||
|
- PNG fallback at hardcoded ≤256×256 threshold
|
||||||
|
- Browser-side hardware decode is explicitly disabled in `Session.ts:93`
|
||||||
|
with a FIXME "hardware decoding frame offset is wrong"
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- No adaptive bitrate — video plays at 12 Mbps regardless of network
|
||||||
|
conditions
|
||||||
|
- No AMD support — limits hardware encoder options
|
||||||
|
- Browser decode is software-only — wastes GPU
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Build an adaptive multi-codec encoding router (PNG for ≤64×64 or static,
|
||||||
|
VP8/H.264 for mid-size, H.264/AV1 for video)
|
||||||
|
- Drive the router by the `EncoderFeedback` measurement Greenfield already
|
||||||
|
produces but doesn't consume
|
||||||
|
- Use `pykcp` or similar for congestion-aware bitrate adaptation
|
||||||
|
- Enable browser hardware decode (`prefer-hardware`) — fix the "frame
|
||||||
|
offset is wrong" bug
|
||||||
|
|
||||||
|
#### D. Wire protocol has no schema version field
|
||||||
|
|
||||||
|
**What Greenfield does**: No version prefix on any binary message:
|
||||||
|
- `EncodedFrame` (FRAME channel)
|
||||||
|
- `RemoteOutOfBandChannel` (PROTOCOL channel OOB)
|
||||||
|
- The FD-channel in `RemoteAppLauncher.ts:612`
|
||||||
|
- Undocumented magic close codes (`4200`, `4201`, `4403`, `4001`, `1005`)
|
||||||
|
|
||||||
|
All break-lockstep — if you change the wire format, both sides must
|
||||||
|
upgrade simultaneously or messages will be misinterpreted.
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Cannot evolve the protocol without breaking all clients
|
||||||
|
- Cannot detect version mismatches
|
||||||
|
- Hard to debug — close codes are magic numbers
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Prefix every binary message with `u16 protocolVersion`
|
||||||
|
- Document every close code in a comment
|
||||||
|
- Use JSON `{"v":1, "type":"..."}` for the control plane
|
||||||
|
- Build a feature-flag system so old clients can negotiate capabilities
|
||||||
|
|
||||||
|
#### E. Main-thread decode + render
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `VideoDecoder` runs on main thread
|
||||||
|
- WebGL context bound to `HTMLCanvasElement`
|
||||||
|
- No `OffscreenCanvas`
|
||||||
|
- No `Worker` in render path
|
||||||
|
- 0 uses of `OffscreenCanvas` or `transferControlToOffscreen` in the
|
||||||
|
compositor
|
||||||
|
- Only 2 Web Workers used (FFmpeg Wasm fallback), shared across surfaces
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Jank on 1080p surfaces — main thread blocks on decode
|
||||||
|
- No way to recover if main thread stalls (input also blocked)
|
||||||
|
- Can't use `requestAnimationFrame` for compositing if main thread is busy
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Render in a Web Worker with `OffscreenCanvas` from day one
|
||||||
|
- Ship `EncodedVideoChunk` / `ArrayBuffer` transfers (zero-copy) to the
|
||||||
|
worker
|
||||||
|
- Use `VideoFrame` directly as a WebGL texture source — no `copyTo`
|
||||||
|
- Use multiple workers (one per output) for parallel rendering
|
||||||
|
|
||||||
|
### High severity
|
||||||
|
|
||||||
|
#### F. xkbcommon Wasm is multi-MB
|
||||||
|
|
||||||
|
**What Greenfield does**: `libs/compositor-wasm/xkbcommon/build.sh`
|
||||||
|
clones libxkbcommon 1.5.0, embeds the entire xkeyboard-config database
|
||||||
|
via `--embed-file`. Result: multi-MB WASM module loaded on every page.
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Slow initial load
|
||||||
|
- Layout list parsed via fragile string matching
|
||||||
|
- Hard to update keyboard layouts without rebuilding WASM
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Use the browser's `navigator.keyboard.getLayoutMap()` API for basic
|
||||||
|
layout detection
|
||||||
|
- For full xkbcommon support, lazy-load the WASM only when needed
|
||||||
|
- Consider server-side keymap compilation (Python `python-xkbcommon`)
|
||||||
|
and ship only the result string to the browser
|
||||||
|
|
||||||
|
#### G. Clipboard edge cases
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- Hardcoded MIME list `['text/plain', 'text/html', 'image/png']`
|
||||||
|
- No primary selection (`wp_primary_selection_v1` not implemented)
|
||||||
|
- Change detection by blob size (false negatives on same-size different
|
||||||
|
content)
|
||||||
|
- Focus check at `DataOffer.ts:73` is a FIXME
|
||||||
|
- Browser-side clipboard polling (no `clipboardchange` event)
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Middle-click paste doesn't work (Linux users expect it)
|
||||||
|
- Large clipboard content (e.g. copied image) is poorly handled
|
||||||
|
- MIME negotiation races
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Implement `zwp_primary_selection_device_manager_v1` from day one
|
||||||
|
- Use exact MIME matching, not substring
|
||||||
|
- For large clipboard content, use chunked transfer
|
||||||
|
- Watch for the `clipboardchange` event in modern browsers (currently
|
||||||
|
behind flags in Chrome)
|
||||||
|
|
||||||
|
#### H. XWayland/xtsb complexity
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- A full XCB reimplementation in TypeScript with its own 5800-line
|
||||||
|
`xproto.xml`
|
||||||
|
- The XWM is a hand-port of wlroots' `xwm.c` with FIXMEs pointing at
|
||||||
|
specific wlroots commits
|
||||||
|
- `XWindowManager.ts:746` literally says "this makes xterm close somehow"
|
||||||
|
- Title bars painted pixel-by-pixel in `XWindowFrame.ts`
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- 5800 lines of X11 protocol XML to maintain
|
||||||
|
- Hand-ported XWM logic is fragile and out of sync with upstream wlroots
|
||||||
|
- Magic workarounds in code
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Use `python-xlib` on the Python proxy side instead of porting xtsb
|
||||||
|
- Consider running XWayland with a minimal window manager (e.g. mutter
|
||||||
|
in kiosk mode) instead of implementing XWM yourself
|
||||||
|
- For XWayland apps only (no full X11 desktop needed), the XWM can be
|
||||||
|
much simpler than wlroots'
|
||||||
|
|
||||||
|
#### I. Adaptive bitrate feedback loop is wired but not consumed
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `EncoderFeedback.ts` measures client decode duration at 1 Hz
|
||||||
|
- 4-byte feedback messages sent from browser to proxy
|
||||||
|
- Proxy receives them but does NOT use them to adapt bitrate
|
||||||
|
- `Channel.ts:181` has a TODO confirming the gap
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Video plays at fixed 12 Mbps regardless of network conditions
|
||||||
|
- No way to recover from congestion
|
||||||
|
- Wasted bandwidth on slow networks
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Wire `EncoderFeedback` to a bitrate adaptation controller
|
||||||
|
- Use `GstElement.set_property("bitrate", ...)` to adjust x264enc/nvh264enc
|
||||||
|
- Implement `requestKeyUnit` plumbing (Greenfield has it end-to-end but
|
||||||
|
the browser never calls it)
|
||||||
|
|
||||||
|
#### J. Single-seat, single-session model
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- One seat per browser session
|
||||||
|
- One Node process per browser session (per `compositorSessionId`)
|
||||||
|
- No multi-seat support (e.g. multiple keyboards/mice for collaborative
|
||||||
|
editing)
|
||||||
|
- No multi-region isolation
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Can't support multi-user scenarios (e.g. conference room with multiple
|
||||||
|
keyboards)
|
||||||
|
- No way to isolate apps between tenants
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- For multi-tenant isolation: separate Python proxy processes per tenant,
|
||||||
|
containerized (Docker + bubblewrap)
|
||||||
|
- For multi-seat: implement `wl_seat` global per seat, route input
|
||||||
|
events by seat
|
||||||
|
|
||||||
|
#### K. Custom web-bitmapbuf protocol — apps must be Greenfield-aware
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `protocol/web-bitmapbuf-unstable-v1.xml` is Greenfield's custom
|
||||||
|
extension
|
||||||
|
- Lets browser JS apps hand a GPU-backed `ImageBitmap` directly to the
|
||||||
|
compositor
|
||||||
|
- Standard Wayland apps (even browser-resident ones) fall to slow H.264
|
||||||
|
path
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Apps must be rebuilt against `@gfld/client-protocol` to use the fast
|
||||||
|
path
|
||||||
|
- Standard Wayland apps can't take advantage of zero-copy GPU textures
|
||||||
|
- Browser-resident WASM apps are slow unless rebuilt
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Document clearly that the fast path requires Greenfield-aware apps
|
||||||
|
- For native Linux apps (your main use case), this isn't an issue — they
|
||||||
|
go through the H.264 path regardless
|
||||||
|
- For browser-resident WASM apps, encourage users to build against
|
||||||
|
`@gfld/client-protocol`
|
||||||
|
|
||||||
|
#### L. Connection loss handling
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- 10-minute zombie timeout on disconnect (`NativeAppContext.ts:152-160`)
|
||||||
|
- No state resumption on reconnect
|
||||||
|
- `eagerBufferContents` cache in `BufferStream.ts:31` is unbounded (TODO
|
||||||
|
at `:87` confirms the leak)
|
||||||
|
- App kill timers: SIGHUP@10min, SIGKILL@+10s
|
||||||
|
- Close code 1001 → immediate SIGHUP; 4001 → immediate SIGTERM
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- User loses all running apps on browser refresh (unless they reconnect
|
||||||
|
within 10 min)
|
||||||
|
- Memory leak from unbounded cache
|
||||||
|
- No way to survive proxy restart
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Bound the `eagerBufferContents` cache (e.g. max 100 frames)
|
||||||
|
- Persist app state on browser side (localStorage) so users can resume
|
||||||
|
- For proxy restart, checkpoint app state to disk (json file)
|
||||||
|
- Implement graceful reconnection with state sync (the KCP plumbing
|
||||||
|
exists)
|
||||||
|
|
||||||
|
#### M. No sandboxing
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `spawn(executable, args, { env })` with no bubblewrap/seccomp/namespaces
|
||||||
|
- Basic Auth is a single replayable credential compared with `!==`
|
||||||
|
- `compositorSessionId` is client-generated
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Native apps have full access to the host filesystem
|
||||||
|
- One compromised app can read all other apps' data
|
||||||
|
- Basic Auth credentials can be replayed
|
||||||
|
- Client-generated session IDs can be spoofed
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Sandbox every spawned app with bubblewrap + seccomp + per-session Linux
|
||||||
|
user namespace
|
||||||
|
- Replace Basic Auth with short-lived signed JWT bound to Odoo's session
|
||||||
|
model
|
||||||
|
- Server-generated session IDs (use `secrets.token_urlsafe(32)`)
|
||||||
|
|
||||||
|
#### N. Heavy toolchain
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- Yarn 4 + CMake + Ninja + Meson + pkg-config
|
||||||
|
- Two different Emscripten versions: 3.1.46 for FFmpeg, 3.1.60 for SDK
|
||||||
|
- SDK builds 30+ C libraries from source
|
||||||
|
- gtk4 WASM is currently broken
|
||||||
|
- Native build needs `autoconf libtool automake meson ninja-build bison
|
||||||
|
cmake build-essential gperf`
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Complex build environment setup
|
||||||
|
- Long build times
|
||||||
|
- Emscripten version skew between libs and SDK
|
||||||
|
- gtk4 WASM broken means no GTK4 browser apps
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Stick with a single Emscripten version
|
||||||
|
- Pre-build WASM modules and ship them as artifacts (don't rebuild every
|
||||||
|
time)
|
||||||
|
- For Python, use `pip install` for all dependencies (simpler than CMake
|
||||||
|
+ Ninja)
|
||||||
|
- Skip the WASM app SDK entirely if you only care about native Linux apps
|
||||||
|
|
||||||
|
### Medium / low (but worth fixing)
|
||||||
|
|
||||||
|
#### O. WebCodecs usage
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- WebCodecs IS used (correcting the brief in some agents' notes)
|
||||||
|
- BUT HW-decode is disabled
|
||||||
|
- `VideoEncoder` is unused
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Wastes GPU on software decode
|
||||||
|
- Could encode in browser (for upload to other peers) but doesn't
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Enable HW-decode by fixing the "frame offset is wrong" FIXME
|
||||||
|
- Consider WebCodecs `VideoEncoder` for any peer-to-peer video paths
|
||||||
|
|
||||||
|
#### P. WebCodecs hardware decode disabled
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `Session.ts:93` has `// FIXME hardware decoding frame offset is wrong`
|
||||||
|
- `hardwareAcceleration: 'prefer-software'` is hardcoded
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Software decode at 1080p60 is ~30% CPU on a modern laptop
|
||||||
|
- Battery life impact on mobile
|
||||||
|
- Wastes GPU
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Investigate and fix the frame offset bug
|
||||||
|
- Enable `prefer-hardware` for capable browsers
|
||||||
|
- Provide a runtime toggle
|
||||||
|
|
||||||
|
#### Q. Unbounded `eagerBufferContents` cache
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `BufferStream.ts:31` has unbounded cache
|
||||||
|
- TODO at `:87` confirms the leak
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Memory leak on long-running sessions
|
||||||
|
- Can OOM the browser tab
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Bound the cache to a maximum number of frames (e.g. 100)
|
||||||
|
- Use LRU eviction
|
||||||
|
- Periodic cleanup of orphaned entries
|
||||||
|
|
||||||
|
#### R. `H264NALDecoder.worker.ts` is minimal
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- 22 lines of code
|
||||||
|
- `@ts-ignore`'d
|
||||||
|
- No error boundary
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Worker crashes silently
|
||||||
|
- No retry on transient errors
|
||||||
|
- No metrics on decode failures
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Add error boundary with retry logic
|
||||||
|
- Log decode failures to the proxy (for diagnostics)
|
||||||
|
- Pool workers for resilience
|
||||||
|
|
||||||
|
#### S. Magic undocumented WebSocket close codes
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- Close codes `4200`, `4201`, `4403`, `4001`, `1005` are used without
|
||||||
|
documentation
|
||||||
|
- Only codes 4000-4999 (application-level) tear down KCP
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Hard to debug connection issues
|
||||||
|
- Close codes are magic numbers
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Document every close code in code comments
|
||||||
|
- Use named constants (e.g. `CLOSE_APP_TERMINATED = 4001`)
|
||||||
|
- Build a close-code → human-readable mapping
|
||||||
|
|
||||||
|
#### T. Generated protocol files committed
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `packages/compositor-proxy/src/protocol/*.js` is generated but
|
||||||
|
committed to the repo
|
||||||
|
- Can drift from XML source
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Easy to forget to regenerate
|
||||||
|
- PRs may include stale generated code
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Don't commit generated files
|
||||||
|
- Use `prebuild` script to generate them
|
||||||
|
- Add a CI check that regenerated files match committed ones
|
||||||
|
|
||||||
|
#### U. Confirmed copy-paste bug in `webcodec-buffer-decoder.ts`
|
||||||
|
|
||||||
|
**What Greenfield does**:
|
||||||
|
- `webcodec-buffer-decoder.ts:142-144, 156-157` has:
|
||||||
|
`this.opaqueDecoder = undefined` twice
|
||||||
|
- Should be `alphaDecoder` the second time
|
||||||
|
- Leaks the alpha decoder after errors
|
||||||
|
|
||||||
|
**Why it's a problem**:
|
||||||
|
- Memory leak
|
||||||
|
- Wasm decoder context not cleaned up properly
|
||||||
|
- Can cause decoder pool exhaustion
|
||||||
|
|
||||||
|
**What you should do instead**:
|
||||||
|
- Audit all `cleanup` paths in decoder code
|
||||||
|
- Add unit tests for error scenarios
|
||||||
|
|
||||||
|
## Design wins to copy
|
||||||
|
|
||||||
|
Not all of Greenfield's choices are bad. Here are 12 design wins worth
|
||||||
|
copying verbatim:
|
||||||
|
|
||||||
|
1. **Generator-driven protocol stubs** (`libs/compositor-generator/`,
|
||||||
|
`client-generator/`, `compositor-proxy-generator/`) — single XML source
|
||||||
|
for both sides. Eliminates wire-format drift.
|
||||||
|
2. **Dual-buffer commit semantics** (`Surface.ts`: `state` vs
|
||||||
|
`pendingState`, atomic merge with pixman damage regions). Classic
|
||||||
|
Wayland correctness.
|
||||||
|
3. **UserShellApi decoupling** — shell sees only `{events, actions}`,
|
||||||
|
never Wayland resources. This is THE integration seam you'll use.
|
||||||
|
4. **Asynchronous browser compositor** — per-client channels, never blocks
|
||||||
|
on a slow client. Essential for multi-app scenarios.
|
||||||
|
5. **KCP-over-WebSocket** for UDP-like latency over TCP (`kcp.ts`, 1170
|
||||||
|
lines). Better than plain TCP for real-time video.
|
||||||
|
6. **WebCodecs primary + WASM fallback + capability probe** (`Session.ts`).
|
||||||
|
Modern browsers get the fast path, older ones still work.
|
||||||
|
7. **Zero-copy dmabuf → encoder** via standard `linux-dmabuf-unstable-v1`
|
||||||
|
(`westfield-dmabuf.c`). Pixels stay on GPU.
|
||||||
|
8. **Per-surface encoder feedback channel** (`EncoderFeedback.ts`)
|
||||||
|
measuring client decode duration — even though adaptation isn't wired
|
||||||
|
up yet, the telemetry is right.
|
||||||
|
9. **`ReconnectingWebSocket`** with exponential backoff + message queue.
|
||||||
|
Survives transient network drops.
|
||||||
|
10. **Minimal shell** — 562 lines of TypeScript total in `compositor-shell`;
|
||||||
|
kiosk variant is 40 lines. Demonstrates that the shell can be tiny.
|
||||||
|
11. **Honest docs** — `docs/pages/design/` openly admits WebSocket/TCP is
|
||||||
|
wrong for real-time and waits on WebTransport. Useful as a reference
|
||||||
|
for what trade-offs were considered.
|
||||||
|
12. **AGPL-3.0** — deliberate copyleft, but be aware before integrating.
|
||||||
|
If you want to keep your modifications proprietary, fork the MIT
|
||||||
|
libs and reimplement the AGPL parts.
|
||||||
|
|
||||||
|
## Top-5 recommendations for the user's Odoo 20 + JS/Python compositor
|
||||||
|
|
||||||
|
1. **Skip native Node addons entirely.** Use `pywayland` (cffi),
|
||||||
|
`python-xkbcommon`, and GStreamer's `gi.repository.Gst` Python
|
||||||
|
bindings. One Python process can replace the entire
|
||||||
|
`compositor-proxy-cli`. See `10-porting-to-odoo-js-python.md` for
|
||||||
|
details.
|
||||||
|
|
||||||
|
2. **Sandbox every spawned app** with bubblewrap + seccomp + per-session
|
||||||
|
Linux user namespace; replace Basic Auth with short-lived signed JWT
|
||||||
|
bound to Odoo's session model.
|
||||||
|
|
||||||
|
3. **Render in a Web Worker with `OffscreenCanvas`** from day one; ship
|
||||||
|
`EncodedVideoChunk` / `ArrayBuffer` transfers (zero-copy) to the
|
||||||
|
worker; use `VideoFrame` directly as a WebGL texture source — no
|
||||||
|
`copyTo`.
|
||||||
|
|
||||||
|
4. **Build an adaptive multi-codec encoding router** (PNG for ≤64×64 or
|
||||||
|
static, VP8/H.264 for mid-size, H.264/AV1 for video), driven by the
|
||||||
|
`EncoderFeedback` measurement Greenfield already produces but doesn't
|
||||||
|
consume.
|
||||||
|
|
||||||
|
5. **Version your wire protocol from commit #1** — prefix every binary
|
||||||
|
message with `u16 protocolVersion`; document every close code; use
|
||||||
|
JSON `{"v":1, "type":"..."}` for the control plane.
|
||||||
|
|
||||||
|
## License implications of AGPL-3.0
|
||||||
|
|
||||||
|
The four `packages/*` (compositor, compositor-proxy, compositor-proxy-cli,
|
||||||
|
compositor-shell) are **AGPL-3.0-or-later**. Implications:
|
||||||
|
|
||||||
|
- If you host Greenfield's `compositor-proxy` as a network service and
|
||||||
|
modify it, you MUST release your modifications under AGPL-3.0+.
|
||||||
|
- The browser-side `@gfld/compositor` is also AGPL. If you modify it
|
||||||
|
(e.g. to integrate with Odoo), your modifications must be AGPL.
|
||||||
|
- The lower-level `libs/*` (common, client-protocol, compositor-protocol,
|
||||||
|
the generators, compositor-wasm, compositor-ffmpeg-h264, xtsb) are MIT
|
||||||
|
or Apache-2.0 — these are safe to use without copyleft obligations.
|
||||||
|
- **Strategy to avoid AGPL**: use the MIT libs as-is, reimplement the
|
||||||
|
AGPL parts from scratch (browser compositor + native proxy). The
|
||||||
|
Python port described in `10-porting-to-odoo-js-python.md` is a clean
|
||||||
|
way to do this — if you write your Python proxy from scratch, you can
|
||||||
|
license it however you want.
|
||||||
|
|
||||||
|
## What Greenfield's codebase reveals about its engineering process
|
||||||
|
|
||||||
|
The codebase reveals several engineering process insights:
|
||||||
|
|
||||||
|
1. **One-person show** — most commits are by Erik De Rijcke. The codebase
|
||||||
|
has the strengths (consistent vision, tight integration) and weaknesses
|
||||||
|
(single point of failure, no code review diversity) of a one-person
|
||||||
|
project.
|
||||||
|
2. **Long-running project** — code references back to 2017-era commits
|
||||||
|
(WebRTC was a previous transport; the `kcp.ts` MTU comment says
|
||||||
|
"webrtc datachannel MTU"). The architecture has evolved through several
|
||||||
|
major pivots.
|
||||||
|
3. **rc1 is meaningful** — the maintainer has been working toward 1.0 for
|
||||||
|
years and rc1 represents a deliberate freeze. Don't expect 1.0 final
|
||||||
|
any time soon, but also don't expect major architecture changes.
|
||||||
|
4. **Honest about limitations** — docs/pages/design/ explicitly admits
|
||||||
|
what's wrong and what's planned. Use this to inform your own roadmap.
|
||||||
+400
@@ -0,0 +1,400 @@
|
|||||||
|
# 12 — Appendix: Glossary, File Index, References
|
||||||
|
|
||||||
|
## Glossary of Greenfield terms
|
||||||
|
|
||||||
|
### Wayland concepts
|
||||||
|
|
||||||
|
| Term | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `wl_display` | The Wayland display connection — top-level object for both clients and the compositor. |
|
||||||
|
| `wl_registry` | Per-client registry of available globals (compositor, seat, shm, output, etc.). |
|
||||||
|
| `wl_compositor` | Global that creates `wl_surface` and `wl_region` objects. |
|
||||||
|
| `wl_surface` | A 2D rectangular pixel buffer with state stack — the basic drawable. |
|
||||||
|
| `wl_seat` | Represents a set of input devices (keyboard, pointer, touch). |
|
||||||
|
| `wl_keyboard` | Per-seat keyboard interface — sends keymap, key events, modifiers. |
|
||||||
|
| `wl_pointer` | Per-seat pointer interface — sends enter/leave/motion/button/axis events. |
|
||||||
|
| `wl_touch` | Per-seat touch interface (declared but inert in Greenfield). |
|
||||||
|
| `wl_output` | A monitor/screen — surfaces are positioned relative to outputs. |
|
||||||
|
| `wl_shm` | Shared memory buffer mechanism — client provides raw pixel buffer. |
|
||||||
|
| `wl_shm_pool` | A pool of shared memory that can be sliced into multiple `wl_buffer`s. |
|
||||||
|
| `wl_buffer` | A pixel buffer that can be attached to a `wl_surface`. |
|
||||||
|
| `wl_callback` | One-shot frame callback — fired when the compositor presents a frame. |
|
||||||
|
| `wl_data_device_manager` | Global for clipboard / DnD. |
|
||||||
|
| `wl_data_device` | Per-seat-per-client data device (selection, DnD). |
|
||||||
|
| `wl_data_source` | The provider side of a selection (the app that has data to offer). |
|
||||||
|
| `wl_data_offer` | The receiver side of a selection (the app that wants to receive data). |
|
||||||
|
| `wl_drm` | Legacy protocol for DMA-BUF buffers (used by Mesa, XWayland). |
|
||||||
|
| `wl_subcompositor` | Global for subsurfaces (nested surfaces). |
|
||||||
|
| `wl_subsurface` | A surface nested inside another surface (sync or desync). |
|
||||||
|
| `wl_region` | A region (set of rectangles) used for input/opaque/damage. |
|
||||||
|
|
||||||
|
### xdg-shell concepts
|
||||||
|
|
||||||
|
| Term | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `xdg_wm_base` | The xdg-shell global — clients use it to register as desktop apps. |
|
||||||
|
| `xdg_surface` | A wrapper around `wl_surface` for xdg-shell semantics. |
|
||||||
|
| `xdg_toplevel` | A top-level window (with title, app_id, state, parent). |
|
||||||
|
| `xdg_popup` | A transient popup window (menus, tooltips, dropdowns). |
|
||||||
|
| `xdg_positioner` | A specification for how to position a popup relative to its parent. |
|
||||||
|
| `configure` | Compositor → client event telling the client what state/size to apply. |
|
||||||
|
| `ack_configure` | Client → compositor request acknowledging a configure with a serial. |
|
||||||
|
|
||||||
|
### Greenfield-specific terms
|
||||||
|
|
||||||
|
| Term | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `Session` | Top-level browser-side compositor session. Created by `createCompositorSession()`. |
|
||||||
|
| `Channel` | Transport abstraction. `ARQChannel` (KCP-wrapped) for reliable channels, `SimpleChannel` for raw WebSocket. |
|
||||||
|
| `ARQChannel` | Automatic Repeat reQuest channel — KCP-over-WebSocket, reliable, low-latency. |
|
||||||
|
| `SimpleChannel` | Raw WebSocket channel, no KCP. Used for FEEDBACK (4-byte messages). |
|
||||||
|
| `Globals` | Registry of Wayland globals on the compositor side (`Globals.ts`). |
|
||||||
|
| `Seat` | Browser-side `wl_seat` implementation. Aggregates keyboard/pointer/touch + focus management. |
|
||||||
|
| `Surface` | Browser-side `wl_surface` implementation. Double-buffered state (`state` vs `pendingState`). |
|
||||||
|
| `View` | A surface's position on an output, used for compositing. One view per surface per output. |
|
||||||
|
| `Output` | Browser-side `wl_output` implementation. Wraps an HTML `<canvas>` with WebGL context. |
|
||||||
|
| `RenderState` | Per-view-per-scene texture + transform state. |
|
||||||
|
| `Scene` | One per output/canvas. Manages the view stack and WebGL textures. |
|
||||||
|
| `BufferContents` | Abstraction over a buffer's content (mimeType + payload). |
|
||||||
|
| `BufferImplementation` | Interface for buffer types: SHM, ImageBitmap, StreamingBuffer. |
|
||||||
|
| `StreamingBuffer` | Per-surface buffer that receives encoded frames over the FRAME channel. |
|
||||||
|
| `BufferStream` | Manages serial → frame promise resolution. |
|
||||||
|
| `EncodedFrame` | Wire format parser for encoded frames (H.264 / PNG). |
|
||||||
|
| `EncoderFeedback` | 4-byte feedback messages sent from browser to proxy at 1 Hz (refreshInterval + decode duration). |
|
||||||
|
| `FrameFeedback` | Proxy-side receiver of `EncoderFeedback` messages. Used to pace `wl_frame.done` callbacks. |
|
||||||
|
| `ProxyFD` | JSON descriptor for an FD over WebSocket: `{handle, type, host}`. |
|
||||||
|
| `webFS` | Browser-side resolver for ProxyFD references. |
|
||||||
|
| `InputOutput` | FD abstraction (browser pipes for in-browser apps, WebSocket endpoints for native apps). |
|
||||||
|
| `RemoteInputOutput` | Native-app FD operations via WebSocket endpoints. |
|
||||||
|
| `webInputOutput` | In-browser-app FD operations via `MessageChannel` ports. |
|
||||||
|
| `UserShellApi` | The integration seam — `{events, actions}` interface between compositor and shell. |
|
||||||
|
| `CompositorClient` | Opaque handle for a Wayland client (browser-side). |
|
||||||
|
| `CompositorSurface` | Opaque handle for a Wayland surface (browser-side). |
|
||||||
|
| `DesktopSurface` | Floating or AlwaysFullScreen mode wrapper. Emits `userShell.events.*`. |
|
||||||
|
| `FloatingDesktopSurface` | Default floating window mode. |
|
||||||
|
| `AlwaysFullScreenDesktopSurface` | Kiosk mode — one window per output. |
|
||||||
|
| `XdgWmBase` | `xdg_wm_base` global, with ping-based unresponsiveness detection. |
|
||||||
|
| `XdgSurface` | `xdg_surface` role wrapper. |
|
||||||
|
| `XdgToplevel` | `xdg_toplevel` role — top-level windows. |
|
||||||
|
| `XdgPopup` | `xdg_popup` role — transient windows. |
|
||||||
|
| `XdgPositioner` | `xdg_positioner` — popup positioning math. |
|
||||||
|
| `web-bitmapbuf-unstable-v1` | Greenfield's custom Wayland protocol extension for browser JS apps. |
|
||||||
|
| `wayland-server-addon` | C N-API addon wrapping patched libwayland-server. |
|
||||||
|
| `proxy-encoding-addon` | C N-API addon wrapping GStreamer H.264/PNG encoder. |
|
||||||
|
| `proxy-poll-addon` | C N-API addon wrapping libuv `uv_poll_t` for wl_display FD polling. |
|
||||||
|
| `xtsb` | X11 TypeScript Bindings — pure-TS X11 client (no libxcb dependency). |
|
||||||
|
| `westfield-*` | Greenfield's C helpers: `westfield-buffer.c`, `westfield-dmabuf.c`, `westfield-egl.c`, `westfield-xwayland.c`. |
|
||||||
|
| `wlr_drm` | `wl_drm` global implementation (in `wlr_drm.c`). |
|
||||||
|
| `wlr_linux_dmabuf_v1` | `zwp_linux_dmabuf_v1` global implementation. |
|
||||||
|
|
||||||
|
### Encoding / rendering terms
|
||||||
|
|
||||||
|
| Term | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `dmabuf` | Linux DMA-BUF — a buffer that can be zero-copied between processes and GPUs. |
|
||||||
|
| `EGLImage` | EGL wrapper around a dmabuf, used by GStreamer GL. |
|
||||||
|
| `GstGLMemoryEGL` | GStreamer GL memory backed by an EGLImage — zero-copy GPU buffer. |
|
||||||
|
| `I420` | YUV 4:2:0 planar format (Y, U, V as separate planes). |
|
||||||
|
| `NV12` | YUV 4:2:0 semi-planar format (Y plane + interleaved UV plane). |
|
||||||
|
| `BT.601` | Color space conversion standard used by Greenfield's YUV→RGB shaders. |
|
||||||
|
| `NAL` | H.264 Network Abstraction Layer unit — a single encoded video frame or part thereof. |
|
||||||
|
| `AU` | H.264 Access Unit — a complete frame. |
|
||||||
|
| `keyframe` | H.264 I-frame — independently decodable, used to start a stream. |
|
||||||
|
| `WebCodecs` | Browser API for native video decode/encode (`VideoDecoder`, `VideoEncoder`). |
|
||||||
|
| `WebTransport` | Future browser API for UDP-like transport over QUIC. Greenfield plans to use it. |
|
||||||
|
| `KCP` | Reliable UDP protocol implemented in user-space. Greenfield runs it over WebSocket. |
|
||||||
|
|
||||||
|
## File-to-feature index
|
||||||
|
|
||||||
|
### Browser side (`packages/compositor/src/`)
|
||||||
|
|
||||||
|
| File | Implements / Role |
|
||||||
|
|---|---|
|
||||||
|
| `index.ts` | Public API: `createCompositorSession()`, `createAppLauncher()` |
|
||||||
|
| `Session.ts` | Top-level browser session, decoder setup |
|
||||||
|
| `Compositor.ts` | `wl_compositor` global |
|
||||||
|
| `Globals.ts` | Wayland globals registry |
|
||||||
|
| `Surface.ts` | `wl_surface` impl, double-buffered state, commit semantics |
|
||||||
|
| `SurfaceChild.ts` | Surface tree child abstraction |
|
||||||
|
| `SurfaceRole.ts` | SurfaceRole interface |
|
||||||
|
| `Subsurface.ts` | `wl_subsurface` role, sync/desync mode |
|
||||||
|
| `Subcompositor.ts` | `wl_subcompositor` global |
|
||||||
|
| `View.ts` | Positioned surface for compositing |
|
||||||
|
| `Output.ts` | `wl_output`, WebGL canvas binding |
|
||||||
|
| `Shm.ts` | `wl_shm` / shm pool, BMP-header hack |
|
||||||
|
| `BufferContents.ts` | Buffer abstraction (with mimeType) |
|
||||||
|
| `BufferImplementation.ts` | Buffer implementation interface |
|
||||||
|
| `ImageBitmapBuffer.ts` | ImageBitmap-backed buffer |
|
||||||
|
| `Region.ts` | `wl_region`, pixman-WASM damage regions |
|
||||||
|
| `Callback.ts` | Frame callback (`wl_callback`) |
|
||||||
|
| `Loop.ts` | Compositor main loop, RAF integration |
|
||||||
|
| `Seat.ts` | `wl_seat` global, multi-seat support, focus management |
|
||||||
|
| `Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events |
|
||||||
|
| `Pointer.ts` | `wl_pointer` impl, motion, buttons, axis |
|
||||||
|
| `Touch.ts` | `wl_touch` (46-line stub, inert) |
|
||||||
|
| `Xkb.ts` | xkbcommon WASM wrapper |
|
||||||
|
| `KeyEvent.ts` | Key event wrapper |
|
||||||
|
| `ButtonEvent.ts` | Pointer button event wrapper |
|
||||||
|
| `AxisEvent.ts` | Pointer axis (scroll) event wrapper |
|
||||||
|
| `InputQueue.ts` | Event coalescing + RAF dispatch |
|
||||||
|
| `InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) |
|
||||||
|
| `DataDeviceManager.ts` | `wl_data_device_manager` global |
|
||||||
|
| `DataSource.ts` | `wl_data_source` impl, the provider side |
|
||||||
|
| `DataOffer.ts` | `wl_data_offer` impl, the receiver side |
|
||||||
|
| `WaylandDataSource.ts` | Wrapper around native `wl_data_source` events |
|
||||||
|
| `Shell.ts` | `wl_shell` global (legacy) |
|
||||||
|
| `ShellSurface.ts` | `wl_shell_surface` role |
|
||||||
|
| `XdgWmBase.ts` | `xdg_wm_base` global, ping-based unresponsiveness detection |
|
||||||
|
| `XdgSurface.ts` | `xdg_surface` role, configure/ack round-trip |
|
||||||
|
| `XdgToplevel.ts` | `xdg_toplevel` role, window state |
|
||||||
|
| `XdgPopup.ts` | `xdg_popup` role, transient windows, grabs |
|
||||||
|
| `XdgPositioner.ts` | `xdg_positioner`, popup positioning math |
|
||||||
|
| `UserShellApi.ts` | The integration seam — events + actions |
|
||||||
|
| `render/Renderer.ts` | 4-phase render pipeline |
|
||||||
|
| `render/Scene.ts` | One per output, manages view stack + textures |
|
||||||
|
| `render/SceneShader.ts` | Final pass: textured quads to canvas |
|
||||||
|
| `render/Texture.ts` | WebGL texture wrapper |
|
||||||
|
| `render/RenderState.ts` | Per-view-per-scene texture + transform |
|
||||||
|
| `render/Program.ts` | WebGL program wrapper |
|
||||||
|
| `render/ShaderCompiler.ts` | Shader compile/link helper |
|
||||||
|
| `render/ShaderSources.ts` | GLSL source strings |
|
||||||
|
| `render/YUVA2RGBA.ts` | YUVA→RGBA FBO pass |
|
||||||
|
| `render/YUVA2RGBAShader.ts` | BT.601 YUVA→RGBA shader |
|
||||||
|
| `render/YUV2RGBShader.ts` | YUV→RGB shader (no alpha) |
|
||||||
|
| `render/RGBXandA2RGBA.ts` | RGB+Alpha → RGBA FBO pass (WebCodecs output) |
|
||||||
|
| `render/RGBandA2RGBAShader.ts` | Shader for above |
|
||||||
|
| `remote/Channel.ts` | `ARQChannel` + `SimpleChannel` |
|
||||||
|
| `remote/kcp.ts` | ~1100-line KCP implementation |
|
||||||
|
| `remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
|
||||||
|
| `remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
|
||||||
|
| `remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||||||
|
| `remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
|
||||||
|
| `remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||||||
|
| `remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||||||
|
| `remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||||||
|
| `remote/buffer-decoder.ts` | Decoder interface |
|
||||||
|
| `remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
|
||||||
|
| `remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||||||
|
| `remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||||||
|
| `remote/XWindowManager.ts` | X11 CLIPBOARD atom watcher (XWayland) |
|
||||||
|
| `remote/XDataSource.ts` | X11 clipboard source (XWayland apps) |
|
||||||
|
| `remote/XDnDDataSource.ts` | X11 DnD source |
|
||||||
|
| `remote/api.ts` | Public exports |
|
||||||
|
| `browser/input.ts` | DOM keyboard → wl_keyboard events |
|
||||||
|
| `browser/pointer.ts` | DOM pointer → wl_pointer events |
|
||||||
|
| `browser/dnd.ts` | DnD bridge |
|
||||||
|
| `browser/selection.ts` | Clipboard bridge |
|
||||||
|
| `browser/text-selection.ts` | Firefox fallback (text-only) |
|
||||||
|
| `browser/capabilities.ts` | Browser feature detection |
|
||||||
|
| `desktop/Desktop.ts` | Floating vs AlwaysFullScreen modes |
|
||||||
|
| `math/Mat4.ts` | 4×4 matrix math |
|
||||||
|
| `math/Rect.ts` | Rectangle math |
|
||||||
|
| `web/WebBuffersFactory.ts` | Browser-side impl of `web_bitmapbuf_factory` |
|
||||||
|
| `web/WebInputOutput.ts` | In-browser-app FD operations |
|
||||||
|
| `web/WebConnectionHandler.ts` | In-browser-app connection handling |
|
||||||
|
| `polyfill/createImageBitmap.js` | ImageBitmap polyfill |
|
||||||
|
|
||||||
|
### Native side
|
||||||
|
|
||||||
|
| File | Implements / Role |
|
||||||
|
|---|---|
|
||||||
|
| `packages/compositor-proxy/src/index.ts` | Exports `createSession()`, `launchApplication()` |
|
||||||
|
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
|
||||||
|
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
|
||||||
|
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
|
||||||
|
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||||||
|
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
|
||||||
|
| `packages/compositor-proxy/src/Channel.ts` | ARQChannel + SimpleChannel |
|
||||||
|
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
|
||||||
|
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
|
||||||
|
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
|
||||||
|
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
|
||||||
|
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
|
||||||
|
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||||||
|
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
|
||||||
|
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
|
||||||
|
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
|
||||||
|
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
|
||||||
|
| `packages/compositor-proxy/src/config.ts` | Encoding-related configuration |
|
||||||
|
| `packages/compositor-proxy/native/CMakeLists.txt` | Build config |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/addon.c` | Node-API addon: wayland-server-addon.node |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/westfield-buffer.c` | Buffer helpers |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/westfield-dmabuf.c` | DMA-BUF import via EGL/GBM |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/westfield-egl.c` | EGL zero-copy setup |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/westfield-xwayland.c` | XWayland spawn + bridge |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/wlr_drm.c` | `wl_drm` global impl |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/wlr_linux_dmabuf_v1.c` | `zwp_linux_dmabuf_v1` global impl |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/drm-protocol.c` | DRM protocol helpers |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/linux-dmabuf-unstable-v1-protocol.c` | linux-dmabuf-v1 protocol stubs |
|
||||||
|
| `packages/compositor-proxy/native/wayland/src/wayland-server/` | Vendored + patched libwayland-server |
|
||||||
|
| `packages/compositor-proxy/native/encoding/src/gst_frame_encoder.c` | GStreamer H.264/PNG encoder |
|
||||||
|
| `packages/compositor-proxy/native/encoding/src/gst_main_loop.c` | Dedicated GLib main loop for encoding |
|
||||||
|
| `packages/compositor-proxy/native/encoding/src/node_encoder.c` | Node-API addon: proxy-encoding-addon.node |
|
||||||
|
| `packages/compositor-proxy/native/poll/src/poll.c` | libuv uv_poll_t wrapper |
|
||||||
|
| `packages/compositor-proxy/native/node-api/` | Vendored Node-API headers |
|
||||||
|
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
|
||||||
|
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
|
||||||
|
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
|
||||||
|
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
|
||||||
|
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
|
||||||
|
|
||||||
|
### Libs
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `libs/common/src/Connection.ts` | Shared wire format codec |
|
||||||
|
| `libs/common/src/index.ts` | Public exports |
|
||||||
|
| `libs/client-generator/{ProtocolParser,ProtocolArguments,index}.mjs` | Browser client bindings generator |
|
||||||
|
| `libs/compositor-generator/{ProtocolParser,ProtocolArguments,index}.mjs` | Browser compositor bindings generator |
|
||||||
|
| `libs/compositor-proxy-generator/src/{EndpointProtocolParser,EndpointProtocolArguments,index}.mjs` | Proxy interceptor generator |
|
||||||
|
| `libs/client-protocol/src/westfield-runtime-client.ts` | Client-side runtime |
|
||||||
|
| `libs/client-protocol/src/protocol/` | Generated client bindings |
|
||||||
|
| `libs/compositor-protocol/src/westfield-runtime-server.ts` | Server-side runtime |
|
||||||
|
| `libs/compositor-protocol/src/protocol/` | Generated server bindings |
|
||||||
|
| `libs/compositor-wasm/build_wasm.sh` | pixman + xkbcommon WASM build |
|
||||||
|
| `libs/compositor-wasm/xkbcommon/build.sh` | xkbcommon 1.5.0 build |
|
||||||
|
| `libs/compositor-wasm/pixman/build.sh` | pixman 0.42.2 build |
|
||||||
|
| `libs/compositor-wasm/src/index.ts` | TS wrapper, `init` function |
|
||||||
|
| `libs/compositor-ffmpeg-h264/build_wasm.sh` | FFmpeg n5.1.2 → WASM build |
|
||||||
|
| `libs/compositor-ffmpeg-h264/native/decoder.c` | 30-line wrapper exposing decode API |
|
||||||
|
| `libs/compositor-ffmpeg-h264/src/H264Decoder.ts` | TS wrapper around the Wasm module |
|
||||||
|
| `libs/compositor-ffmpeg-h264/src/H264Worker.ts` | Worker entry point |
|
||||||
|
| `libs/xtsb/protocol/xproto.xml` | X11 core protocol XML (5800+ lines) |
|
||||||
|
| `libs/xtsb/protocol/shape.xml` | Shape extension XML |
|
||||||
|
| `libs/xtsb/protocol/xfixes.xml` | XFIXES extension XML |
|
||||||
|
| `libs/xtsb/protocol/render.xml` | Render extension XML |
|
||||||
|
| `libs/xtsb/protocol/composite.xml` | Composite extension XML |
|
||||||
|
| `libs/xtsb/src/` | Generated TS X11 bindings |
|
||||||
|
|
||||||
|
### Protocol XML
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `protocol/wayland.xml` | Standard upstream Wayland core protocol |
|
||||||
|
| `protocol/xdg-shell.xml` | Stable xdg-shell protocol |
|
||||||
|
| `protocol/wayland-drm.xml` | Legacy wl_drm for DMA-BUF (proxy-only) |
|
||||||
|
| `protocol/web-bitmapbuf-unstable-v1.xml` | Greenfield's custom extension for browser JS apps |
|
||||||
|
|
||||||
|
## External references
|
||||||
|
|
||||||
|
### Wayland protocol
|
||||||
|
|
||||||
|
- **Wayland documentation**: https://wayland.freedesktop.org/docs/html/
|
||||||
|
- **Wayland protocol XML**: https://gitlab.freedesktop.org/wayland/wayland/-/blob/main/protocol/wayland.xml
|
||||||
|
- **xdg-shell protocol**: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/stable/xdg-shell/xdg-shell.xml
|
||||||
|
- **wayland-drm protocol**: https://gitlab.freedesktop.org/wayland/wayland/-/blob/main/protocol/wayland.xml (wayland-drm is included)
|
||||||
|
- **linux-dmabuf-unstable-v1**: https://gitlab.freedesktop.org/wayland/wayland-protocols/-/blob/main/unstable/linux-dmabuf/linux-dmabuf-unstable-v1.xml
|
||||||
|
|
||||||
|
### libwayland
|
||||||
|
|
||||||
|
- **libwayland source**: https://gitlab.freedesktop.org/wayland/wayland
|
||||||
|
- **Wayland book**: https://wayland-book.com/
|
||||||
|
- **pywayland (Python bindings)**: https://pywayland.readthedocs.io/
|
||||||
|
|
||||||
|
### xkbcommon
|
||||||
|
|
||||||
|
- **xkbcommon homepage**: https://xkbcommon.org/
|
||||||
|
- **xkbcommon source**: https://github.com/xkbcommon/libxkbcommon
|
||||||
|
- **python-xkbcommon**: https://github.com/sde1000/python-xkbcommon
|
||||||
|
|
||||||
|
### GStreamer
|
||||||
|
|
||||||
|
- **GStreamer homepage**: https://gstreamer.freedesktop.org/
|
||||||
|
- **GStreamer Python bindings**: https://gstreamer.freedesktop.org/documentation/tutorials/basic/hello-world.html
|
||||||
|
- **GStreamer GL plugins**: https://gstreamer.freedesktop.org/documentation/opengl/index.html
|
||||||
|
- **x264enc docs**: https://gstreamer.freedesktop.org/documentation/x264/index.html
|
||||||
|
- **nvh264enc docs**: https://gstreamer.freedesktop.org/documentation/nvcodec/index.html
|
||||||
|
|
||||||
|
### FFmpeg Wasm
|
||||||
|
|
||||||
|
- **FFmpeg homepage**: https://ffmpeg.org/
|
||||||
|
- **FFmpeg Wasm build**: https://github.com/ffmpegwasm/ffmpeg.wasm
|
||||||
|
- **Emscripten**: https://emscripten.org/
|
||||||
|
|
||||||
|
### WebCodecs
|
||||||
|
|
||||||
|
- **WebCodecs API**: https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API
|
||||||
|
- **VideoDecoder**: https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder
|
||||||
|
- **WebCodecs samples**: https://w3c.github.io/webcodecs/samples/
|
||||||
|
|
||||||
|
### WebTransport
|
||||||
|
|
||||||
|
- **WebTransport**: https://developer.mozilla.org/en-US/docs/Web/API/WebTransport
|
||||||
|
- **WebTransport over HTTP/3**: https://datatracker.ietf.org/doc/html/draft-ietf-webtrans-http3
|
||||||
|
|
||||||
|
### KCP
|
||||||
|
|
||||||
|
- **KCP protocol**: https://github.com/skywind3000/kcp
|
||||||
|
- **KCP documentation (Chinese)**: https://skywind3000.github.io/kcp/
|
||||||
|
- **pykcp (Python bindings)**: https://github.com/Zero3K/pykcp
|
||||||
|
|
||||||
|
### Emscripten
|
||||||
|
|
||||||
|
- **Emscripten homepage**: https://emscripten.org/
|
||||||
|
- **Emscripten porting guide**: https://emscripten.org/docs/porting/index.html
|
||||||
|
|
||||||
|
### X11 / XCB
|
||||||
|
|
||||||
|
- **XCB documentation**: https://xcb.freedesktop.org/
|
||||||
|
- **X protocol reference**: https://www.x.org/releases/X11R7.7/doc/xproto/x11protocol.html
|
||||||
|
- **python-xlib**: https://github.com/python-xlib/python-xlib
|
||||||
|
|
||||||
|
### XWayland
|
||||||
|
|
||||||
|
- **XWayland docs**: https://wayland.freedesktop.org/xserver.html
|
||||||
|
- **wlroots XWM source**: https://gitlab.freedesktop.org/wlroots/wlroots/-/tree/master/xcage
|
||||||
|
|
||||||
|
### Greenfield project
|
||||||
|
|
||||||
|
- **Greenfield homepage**: https://www.greenfield.app
|
||||||
|
- **Greenfield documentation**: http://udev.be/greenfield
|
||||||
|
- **Greenfield source**: https://github.com/udevbe/greenfield
|
||||||
|
- **Greenfield design docs**: https://github.com/udevbe/greenfield/blob/master/docs/pages/design/index.md
|
||||||
|
- **Greenfield future plans**: https://github.com/udevbe/greenfield/blob/master/docs/pages/future_plans/index.md
|
||||||
|
- **Author's blog**: https://wayouttheresoftware.blogspot.com/2023/07/some-history-about-greenfield.html
|
||||||
|
|
||||||
|
### Odoo 20
|
||||||
|
|
||||||
|
- **Odoo homepage**: https://www.odoo.com/
|
||||||
|
- **Odoo documentation**: https://www.odoo.com/documentation/20.0/
|
||||||
|
- **Odoo OWL framework**: https://github.com/odoo/owl
|
||||||
|
- **Odoo development tutorial**: https://www.odoo.com/documentation/20.0/developer/howtos.html
|
||||||
|
|
||||||
|
## Pointers to the user's next steps
|
||||||
|
|
||||||
|
1. **Read chapters in order**: `00` → `01` → `02` → `07` → `08` → `10` → `11`.
|
||||||
|
This gives you the architecture map, the protocol layer, the native app
|
||||||
|
bridge (your primary use case), the UserShellApi integration seam, the
|
||||||
|
porting blueprint, and the pitfalls to avoid.
|
||||||
|
|
||||||
|
2. **Start with Phase 1 of the porting blueprint** (`10-porting-to-odoo-js-python.md`):
|
||||||
|
Run Greenfield's compositor-proxy-cli as a Docker container, build a
|
||||||
|
minimal Odoo 20 OWL addon that creates a `<canvas>`, and verify a single
|
||||||
|
Linux app displays.
|
||||||
|
|
||||||
|
3. **Decide on the AGPL question early**: Will you accept AGPL on the
|
||||||
|
browser compositor + native proxy (and release your modifications), or
|
||||||
|
will you reimplement them from scratch (and license them however you
|
||||||
|
want)?
|
||||||
|
|
||||||
|
4. **Choose your Python stack**: `aiohttp` vs `FastAPI` for the WebSocket
|
||||||
|
server. `pywayland` for the Wayland bindings. `python-xkbcommon` for
|
||||||
|
keymaps. `gi.repository.Gst` for GStreamer encoding. `bubblewrap` +
|
||||||
|
`seccomp` for sandboxing.
|
||||||
|
|
||||||
|
5. **Plan for sandboxing from day one**: Greenfield has none. If you skip
|
||||||
|
this, your Odoo users can run arbitrary Linux code on your server.
|
||||||
|
That's a security hole you can't ship.
|
||||||
|
|
||||||
|
6. **Build the OWL shell with `UserShellApi.events` callbacks**: This is
|
||||||
|
the clean integration seam. Don't try to import Greenfield internals
|
||||||
|
into your Odoo shell — use the public `UserShellApi` interface.
|
||||||
|
|
||||||
|
7. **Budget for the Python port**: ~4-8 weeks for a working Python proxy.
|
||||||
|
~8-16 weeks for production quality (sandboxing, multi-tenant, adaptive
|
||||||
|
bitrate, clipboard sync, touch input, primary selection).
|
||||||
|
|
||||||
|
8. **Track upstream Greenfield commits**: Pin to a specific commit hash
|
||||||
|
if you fork it. Watch for 1.0 final (may break APIs).
|
||||||
|
|
||||||
|
Good luck with your Wayland-to-HTML compositor build!
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
# Greenfield Architecture Analysis — for an Odoo 20 + JS/Python Wayland-to-HTML Compositor
|
||||||
|
|
||||||
|
> **Purpose of this bundle.** A deep, source-level architectural analysis of the
|
||||||
|
> Greenfield repository (https://github.com/udevbe/greenfield, commit `6c578f4`,
|
||||||
|
> `1.0.0-rc1`) written specifically for an engineering team that is about to build
|
||||||
|
> its own Wayland-to-HTML compositor. The target stack is **JavaScript (browser) +
|
||||||
|
> Python (host) + Odoo 20 addon shell**. Greenfield is treated purely as a
|
||||||
|
> reference implementation; the actual Greenfield UI shell (`packages/compositor-shell`)
|
||||||
|
> is **explicitly out of scope** — the user will supply their own Odoo 20 OWL shell.
|
||||||
|
|
||||||
|
## How to read this bundle
|
||||||
|
|
||||||
|
The bundle is split into 13 Markdown files. They are designed to be read in
|
||||||
|
order, but each file is self-contained and can be read alone if you only need
|
||||||
|
one layer. Files are organized from the highest-level view down to the
|
||||||
|
implementation-level details, then the porting blueprint, then pitfalls.
|
||||||
|
|
||||||
|
| # | File | What you'll find inside |
|
||||||
|
|---|------|-------------------------|
|
||||||
|
| 0 | `00-overview.md` | What Greenfield is, repo layout, package inventory, tech stack table, ASCII architecture diagram. |
|
||||||
|
| 1 | `01-architecture.md` | Layer-by-layer map from Wayland XML → generators → runtime → transport → rendering. |
|
||||||
|
| 2 | `02-wayland-protocol.md` | Protocol layer: 3 generators, wire format, object lifecycle, FD transport, custom `web-bitmapbuf` extension. |
|
||||||
|
| 3 | `03-html5-rendering.md` | Surface → canvas compositing pipeline: Surface state, damage, FrameLoop, WebGL shaders, multi-output. |
|
||||||
|
| 4 | `04-transport.md` | WebSocket + KCP-over-WebSocket, multi-channel design, message formats, FD virtualization, reconnection. |
|
||||||
|
| 5 | `05-input-handling.md` | Keyboard / pointer / touch / focus, xkbcommon WASM, keymap FD workaround, DOM → Wayland event mapping. |
|
||||||
|
| 6 | `06-data-clipboard.md` | `wl_data_device`, selection, drag-and-drop, MIME negotiation, XWayland bridge, primary-selection gap. |
|
||||||
|
| 7 | `07-native-app-bridge.md` | How real Linux apps get launched, the 3 native Node addons, XWayland bridge, app lifecycle & kill timers. |
|
||||||
|
| 8 | `08-app-lifecycle.md` | xdg-shell state machine, configure/ack round-trips, popup positioner math, subsurface sync, `UserShellApi`. |
|
||||||
|
| 9 | `09-hardware-accel.md` | GStreamer H.264 encoders, WASM FFmpeg fallback, WebCodecs preferred path, WebGL texture upload, alpha split. |
|
||||||
|
| 10 | `10-porting-to-odoo-js-python.md` | Concrete blueprint for porting each Greenfield layer to **JS browser compositor + Python host proxy + Odoo 20 OWL shell**. |
|
||||||
|
| 11 | `11-pitfalls.md` | 21 catalogued pitfalls + 12 design wins, with file references and what-to-do-instead for each. |
|
||||||
|
| 12 | `12-appendix.md` | Glossary, file-to-feature index, external references. |
|
||||||
|
|
||||||
|
## Recommended reading order
|
||||||
|
|
||||||
|
- **Architects / tech leads** — read `00` → `01` → `11` → `10`. You'll get the
|
||||||
|
lay of the land, what to copy, what to avoid, and the porting blueprint.
|
||||||
|
- **Browser-compositor engineers** — read `02` → `03` → `04` → `05` → `08` →
|
||||||
|
`09`. The core engine chapters, in dependency order.
|
||||||
|
- **Host-proxy engineers (Python)** — read `07` → `04` → `09` → `02` (proxy
|
||||||
|
generator section) → `10`. Most of the porting work is on your side.
|
||||||
|
- **Odoo 20 OWL shell engineers** — read `08` (especially the `UserShellApi`
|
||||||
|
section) → `01` → `10` (Odoo integration section). The shell is the
|
||||||
|
integration seam where Greenfield stops and your work begins.
|
||||||
|
- **Skip-it sections** — `06` (clipboard) is only needed if you care about
|
||||||
|
clipboard sync between browser and native apps.
|
||||||
|
|
||||||
|
## Scope & exclusions
|
||||||
|
|
||||||
|
**In scope** — every Greenfield package, lib, generator, native addon,
|
||||||
|
protocol file, transport channel, rendering path, encoding pipeline, and
|
||||||
|
the `UserShellApi` integration seam.
|
||||||
|
|
||||||
|
**Out of scope** — the actual `packages/compositor-shell` Preact+Tailwind
|
||||||
|
SPA. The user is building their own shell as an Odoo 20 addon. We only
|
||||||
|
describe the protocol-level integration point (`UserShellApi`) so the
|
||||||
|
Odoo shell knows what to consume.
|
||||||
|
|
||||||
|
**Not covered in depth** — the `sdk/` (Emscripten-based WASM app SDK for
|
||||||
|
browser-resident Wayland apps). Mentioned in `00-overview.md` and
|
||||||
|
`10-porting-to-odoo-js-python.md` but not analyzed layer-by-layer,
|
||||||
|
because the user's goal is running **real Linux apps** in the browser,
|
||||||
|
not browser-resident WASM apps.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **File paths** are always relative to the repo root, e.g.
|
||||||
|
`packages/compositor/src/Surface.ts`.
|
||||||
|
- **ASCII diagrams** are used throughout — they render correctly in any
|
||||||
|
Markdown viewer, survive copy-paste, and work in code reviews.
|
||||||
|
- **No code excerpts** by user request. We describe algorithms in prose
|
||||||
|
and reference file paths so you can read the source yourself.
|
||||||
|
- **Cross-references** look like `(see §3 of `03-html5-rendering.md`)`.
|
||||||
|
|
||||||
|
## How this analysis was produced
|
||||||
|
|
||||||
|
This bundle was produced by 10 parallel research agents, each assigned to a
|
||||||
|
single layer of the Greenfield codebase:
|
||||||
|
|
||||||
|
- **Agent A** — top-level architecture
|
||||||
|
- **Agent B** — Wayland protocol layer (generators + runtime)
|
||||||
|
- **Agent C** — HTML5 rendering pipeline
|
||||||
|
- **Agent D** — transport layer (WebSocket + KCP)
|
||||||
|
- **Agent E** — input handling (keyboard / pointer / touch / xkbcommon)
|
||||||
|
- **Agent F** — data device & clipboard
|
||||||
|
- **Agent G** — native app bridge (process model + native addons)
|
||||||
|
- **Agent H** — xdg-shell / app lifecycle / `UserShellApi`
|
||||||
|
- **Agent I** — hardware acceleration / encoding / WASM
|
||||||
|
- **Agent J** — pitfalls, anti-patterns, design wins
|
||||||
|
|
||||||
|
Each agent read the relevant source files end-to-end, produced a stage
|
||||||
|
summary, and appended its findings to `/home/z/my-project/worklog.md`.
|
||||||
|
This bundle is the synthesis of those findings into a single readable
|
||||||
|
document set.
|
||||||
|
|
||||||
|
## License of the source
|
||||||
|
|
||||||
|
Greenfield's runtime packages (`compositor`, `compositor-proxy`,
|
||||||
|
`compositor-proxy-cli`, `compositor-shell`) are **AGPL-3.0-or-later**.
|
||||||
|
The lower-level libs (generators, `common`, `compositor-wasm`,
|
||||||
|
`compositor-ffmpeg-h264`, `xtsb`) are MIT or Apache-2.0. Anyone forking
|
||||||
|
or building on the runtime packages must respect AGPL — see
|
||||||
|
`11-pitfalls.md` §License for implications.
|
||||||
Reference in new issue
Block a user