1b-agent/one-os-dev-wayland-html5-compositor-greenfield-analysis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents)

Browse Source
This commit is contained in:
1b-agent committed 2026-10-08 08:48:25 +00:00
commit c4b45cf218
15 files changed
+5444
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

01-architecture.md
+443
View File
@@ -0,0 +1,443 @@
# 01 — Layered Architecture Map
This chapter walks Greenfield's architecture layer by layer, from the lowest
abstraction (Wayland XML) up to the highest (the user-facing shell). Each layer
is described in terms of: **responsibility**, **key abstractions**,
**dependencies**, and **files of interest**. The chapter ends with an ASCII
diagram showing how the layers stack and where data flows between them.
## The eight layers
```
┌──────────────────────────────────────────────────────────────────────────┐
│ Layer 8: USER SHELL (Odoo 20 OWL addon, supplied by user) │
│ Hooks: UserShellApi (events + actions) │
│ ─ NOT covered in this chapter (see 08-app-lifecycle.md) │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 7: DESKTOP / LAYOUT POLICY │
│ Floating vs AlwaysFullScreen, view stacking, focus activation refcount │
│ packages/compositor/src/desktop/Desktop.ts │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 6: SURFACE LIFECYCLE / ROLES │
│ XdgWmBase, XdgSurface, XdgToplevel, XdgPopup, XdgPositioner, Shell, │
│ ShellSurface, Subcompositor, Subsurface, SurfaceRole │
│ packages/compositor/src/{Xdg*, Shell*, Sub*, SurfaceRole, SurfaceChild} │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 5: RENDERING PIPELINE │
│ Surface (state stack), View, Output (WebGL canvas), Renderer, Scene, │
│ Shaders (YUV→RGBA, YUVA→RGBA, RGB+A→RGBA), Loop (RAF), Callback │
│ packages/compositor/src/{Surface, View, Output, Loop, Callback, Region} │
│ packages/compositor/src/render/ │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 4: TRANSPORT (browser side) │
│ Channel (ARQ + Simple), ReconnectingWebSocket, RemoteAppLauncher, │
│ StreamingBuffer, BufferStream, EncodedFrame, EncoderFeedback, decoders │
│ (WebCodecs + WASM fallback) │
│ packages/compositor/src/remote/ │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 4: TRANSPORT (native side) │
│ Channel (ARQ + Simple), SessionController, kcp.ts, NativeAppContext, │
│ FrameFeedback, SurfaceBufferEncoding, wayland-server.ts │
│ packages/compositor-proxy/src/{Channel, kcp, Session*, Native*, ...} │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 3: WAYLAND RUNTIME (browser side) │
│ Connection (wire framing), Client, WlObject registry, marshallers │
│ libs/common/src/Connection.ts │
│ libs/compositor-protocol/src/{westfield-runtime-server.ts, generated/*} │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 3: WAYLAND RUNTIME (native side) │
│ wayland-server.ts + wayland-server-addon.node (N-API → libwayland-server)│
│ MessageInterceptor (per-resource request/event routing) │
│ packages/compositor-proxy/src/{wayland-server.ts, protocol/*_interceptor}│
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 2: PROTOCOL GENERATORS │
│ client-generator, compositor-generator, compositor-proxy-generator │
│ libs/{client,compositor,compositor-proxy}-generator/ │
├──────────────────────────────────────────────────────────────────────────┤
│ Layer 1: WAYLAND PROTOCOL XML │
│ protocol/{wayland, xdg-shell, wayland-drm, web-bitmapbuf-unstable-v1}.xml│
└──────────────────────────────────────────────────────────────────────────┘
```
## Layer 1 — Wayland Protocol XML
**Responsibility**: declare the protocol surface — what interfaces, requests,
events, enums, and bitfield types exist on the wire.
**Files of interest**:
- `protocol/wayland.xml` — standard upstream Wayland core. 22 interfaces,
`wl_compositor` v4, `wl_seat` v6, `wl_surface` v4 (roughly Wayland
1.20-ish). Kristian Høgsberg copyright. Used by all 3 generators.
- `protocol/xdg-shell.xml` — stable `xdg_shell` v1, 5 interfaces (xdg_wm_base,
xdg_surface, xdg_toplevel, xdg_popup, xdg_positioner). Used by all 3
generators.
- `protocol/wayland-drm.xml` — legacy `wl_drm` v2, used by native apps (Mesa,
XWayland) to advertise dmabuf buffers. **Only the proxy generator
processes this file** — so the proxy can intercept `create_buffer` /
`create_planar_buffer` / `create_prime_buffer` and turn dmabuf fds into
ProxyFDs for the browser's GStreamer encoding pipeline. Browser-side libs
do not generate `wl_drm` stubs.
- `protocol/web-bitmapbuf-unstable-v1.xml` — **Greenfield's custom
extension**. Single interface `web_bitmapbuf_factory` v1, single request
`create_buffer(id: new_id<wl_buffer>, bitmap: fd)`. Lets browser JS apps
hand a GPU-backed `ImageBitmap` directly to the compositor as a `wl_buffer`
— bypassing SHM copy, encoding, and dmabuf import. The `fd` arg is
marshalled via the same `fileDescriptor(...)` path as
`wl_shm.create_pool`'s fd, but holds an `ImageBitmap` (becomes a
transferable on the flush path). **Not processed by the proxy generator**
— native apps never use it.
**Dependencies**: none. This is the source of truth.
## Layer 2 — Protocol Generators
**Responsibility**: parse the XML with `xml2js`, walk the interface tree,
emit typed TypeScript bindings for each side.
**Key abstractions**: `ProtocolParser.mjs` (shared XML walker) +
`ProtocolArguments.mjs` (per-side argument-type dispatch table).
**Files of interest**:
- `libs/client-generator/{ProtocolParser, ProtocolArguments, index}.mjs` —
emits TS `*Proxy` classes for browser apps that want to be Wayland clients
of the browser compositor (via `MessagePort`).
- `libs/compositor-generator/{ProtocolParser, ProtocolArguments, index}.mjs`
— emits TS `*Resource` + `*Requests` classes for the browser-resident
compositor itself (receives requests, dispatches to `*Requests` impls,
emits events). Skips `wl_display`, `wl_registry` (hand-written in runtime).
- `libs/compositor-proxy-generator/{EndpointProtocolParser,
EndpointProtocolArguments, index}.mjs` — emits CommonJS
`*_interface.js` (wl_interface struct definitions for the N-API addon)
+ `*_interceptor.js` (per-opcode interceptors that decide routing) for
the Node.js native proxy.
**Why three generators?** Because each side has a fundamentally different job:
1. **client-generator** → browser apps that want to be Wayland clients of the
browser compositor (via `MessagePort`).
2. **compositor-generator** → the browser-resident compositor itself (receives
requests, dispatches to `*Requests` impls, emits events).
3. **compositor-proxy-generator** → the Node.js native proxy bridging real
libwayland-client apps to the browser. Emits `wl_interface` struct
definitions for the N-API addon + per-opcode interceptors that decide
routing.
**Dependencies**: `xml2js`, `camelcase`, `uppercamelcase`, `meow`.
## Layer 3 — Wayland Runtime
**Responsibility**: marshal and unmarshal the actual wire bytes. Implement
object lifecycle (creation, dispatch, destruction), new-id tracking, listener
dispatch, file descriptor handling.
**Browser side** (`libs/common/src/Connection.ts`) — the shared runtime for
both `@gfld/client-protocol` and `@gfld/compositor-protocol`:
- Wire format is **standard upstream Wayland**, byte-for-byte. 8-byte
header: `u32 object_id | u16 opcode | u16 size-incl-header`, then per-arg
bytes. `u/i/f/n/o` are 4 bytes; `s` and `a` are length-prefixed u32 +
payload padded to 4-byte alignment; `h` (FD) takes zero bytes — FDs travel
out-of-band in `wireMsg.fds[]`.
- Reading: `wlObject[opcode](message)` — the generated classes use
computed-property-key methods `[0](message) { ... }`, `[1](message) { ... }`
so opcode is the array index.
- Client side: `Proxy.marshallConstructor` calls `display.generateNextId()`
(1-indexed, recycles ids from `wl_display.delete_id`), constructs the new
Proxy (self-registers in constructor), patches the `new_id` arg's `value`,
then flushes.
- Server side: `Client.marshallConstructor` allocates from
`SERVER_OBJECT_ID_BASE = 0xff000000` (or recycles locally). When a
client-allocated id is destroyed, the server sends `wl_display.delete_id`
so the client can recycle.
- Connection registry: `wlObjects: { [id]: WlObject }`; `registerWlObject`
throws on collision.
- `Connection.message` dispatches by literal `wlObject[opcode](message)`.
Each handler returns void or a Promise; `Connection` awaits the promise
before reading the next message — this preserves wire-order while still
allowing async implementations. Per-client Connections mean a slow client
doesn't block others.
**Native side** (`packages/compositor-proxy/src/wayland-server.ts`) — a TS
wrapper around `wayland-server-addon.node` (the C N-API addon that wraps a
**patched** `libwayland-server`):
- `MessageInterceptor` — per-resource request/event dispatch. Each
`wl_*_interceptor.js` file declares `R<opcode>(...args)` and
`E<opcode>(...args)` functions that decide whether a message is routed
`{native: true}` (handle in the proxy), `{browser: true}` (forward to
browser compositor), or both.
- Special cases: `wl_display.sync` → both native and browser (must complete
before `done` event), `wl_display.get_registry` → both, `wl_registry.bind`
→ native if id is in `nativeGlobalNames[]` (e.g. `wl_drm`, `wl_shm`),
else browser. Fast-sync optimization skips browser round trip when no
prior browser-bound request occurred.
**File descriptor handling — the Greenfield innovation**:
`type FD = unknown` — it's intentionally opaque. Browser-side FDs are
**transferable JS objects**:
- `ArrayBufferView` for `wl_shm.create_pool`'s `fd` (SHM backing memory lives
in browser, zero-copy transferred via `MessagePort.postMessage(...,
transferList)`).
- `ImageBitmap` for `web_bitmapbuf_factory.create_buffer`'s `bitmap` arg
(GPU-backed texture, zero-copy).
- `MessagePort` for `wl_data_offer.receive`'s `fd` (in-browser pipe).
Native-side (proxy) FDs are real POSIX ints.
`NativeWaylandClientSession.onWireMessageEnd` converts each to a JSON
`ProxyFD = {handle, type, host}` (host = proxy's HTTP base URL),
length-prefixed + 4-byte aligned, prepended to the wire message before
sending over WebSocket/KCP. Browser's `webFS` resolves them by fetching
data over HTTP.
**Dependencies**: Layer 2's generated bindings + `@gfld/common`.
## Layer 4 — Transport
**Responsibility**: carry wire bytes between the browser compositor and the
native proxy. Multiplex logical channels. Handle reconnection, backpressure,
and latency.
**Browser side** (`packages/compositor/src/remote/`):
- `Channel.ts` — defines `ARQChannel` (KCP-wrapped, reliable, low-latency)
and `SimpleChannel` (raw WebSocket).
- `ReconnectingWebSocket` — vendored from `pladaria/reconnecting-websocket`
with `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`, `minUptime=5s`,
`connTimeout=4s`, `retries=∞`.
- `RemoteAppLauncher.ts` — sets up the 4 logical channels per app client
(PROTOCOL, FRAME, XWM, FEEDBACK) and routes incoming messages.
- `RemoteInputOutput.ts` — implements the `InputOutput` interface for native
apps, with `mkfifo`, `mkstempMmap`, `readFd`, `writeFdAsStream`, `closeFd`
over separate WebSocket endpoints.
- `StreamingBuffer.ts`, `BufferStream.ts`, `EncodedFrame.ts`,
`EncoderFeedback.ts` — decode pipeline for incoming H.264 / PNG frames.
- `webcodec-buffer-decoder.ts`, `wasm-buffer-decoder.ts`,
`H264NALDecoder.worker.ts` — the actual decoders.
**Native side** (`packages/compositor-proxy/src/`):
- `Channel.ts` — mirror image of the browser side. `ARQChannel` and
`SimpleChannel`.
- `kcp.ts` — ~1100-line TypeScript port of the canonical C `ikcp.c`.
- `SessionController.ts` — WS server with 9 endpoint paths:
`/signal`, `/channel`, `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
- `Session.ts`, `SessionProcess.ts`, `SessionController.ts` — process model:
one main process, one forked child per browser session.
- `NativeAppContext.ts` — per-app state, signaling WS, lifecycle (10-min
SIGHUP grace on disconnect, SIGTERM→SIGKILL escalation).
- `FrameFeedback.ts` — receives 4-byte feedback messages from the browser,
uses them to pace `wl_frame.done` callbacks.
- `SurfaceBufferEncoding.ts`, `ProxyBuffer.ts` — encoding pipeline.
- `io/` subdirectory — WebSocket server glue.
- `encoding/` subdirectory — high-level encoding orchestration.
**Wire format**: see `04-transport.md` for the full byte-level breakdown of
PROTOCOL, FRAME, XWM, and FEEDBACK channels.
**Dependencies**: Layer 3 + `ws` (Node) + browser-native WebSocket + `ReconnectingWebSocket`.
## Layer 5 — Rendering Pipeline
**Responsibility**: take incoming buffer frames (SHM or encoded), turn them
into pixels on a browser canvas/WebGL context. Manage surface state, damage
tracking, frame callbacks, and the compositing loop.
**Key abstractions**:
- `Surface.ts` — the `wl_surface` implementation. Double-buffered state
(`state` vs `pendingState`), atomic merge in `commitPending()`. Subsurface
sync mode cascades through the ancestor chain.
- `SurfaceChild.ts`, `SurfaceRole.ts`, `Subsurface.ts`, `Subcompositor.ts` —
surface tree / role system.
- `View.ts` — represents a surface's position on an output, used for
compositing.
- `Output.ts` — the `wl_output` implementation; how output is rendered to a
WebGL canvas.
- `Shm.ts` — `wl_shm` / shm pool handling. How shared memory buffers arrive
in the browser.
- `BufferContents.ts`, `BufferImplementation.ts`, `ImageBitmapBuffer.ts` —
buffer abstraction hierarchy.
- `Region.ts` — `wl_region`, damage regions (pixman-WASM backed).
- `Callback.ts` — frame callback mechanism.
- `Loop.ts` — the compositor main loop / `requestAnimationFrame` integration.
- `Compositor.ts` — the `wl_compositor` global; how it creates surfaces and
regions.
- `Session.ts` — top-level browser-side session.
- `Globals.ts` — how Wayland globals are registered.
- `render/` subdirectory — the WebGL renderer: `Renderer.ts`, `Scene.ts`,
`SceneShader.ts`, `Texture.ts`, `RenderState.ts`, `Program.ts`,
`ShaderCompiler.ts`, `ShaderSources.ts`, `YUVA2RGBA.ts`, `YUVA2RGBAShader.ts`,
`YUV2RGBShader.ts`, `RGBXandAToRGBA.ts`, `RGBandA2RGBAShader.ts`.
- `browser/` subdirectory — browser integration glue (canvas, RAF,
ResizeObserver, dnd, selection, input).
- `math/` subdirectory — matrix/region math (`Mat4.ts`, `Rect.ts`).
**Pipeline**: see `03-html5-rendering.md` for the full
commit → damage → render → present pipeline.
**Dependencies**: Layer 4 + WebGL + WebCodecs + `@gfld/compositor-wasm` (pixman + xkbcommon).
## Layer 6 — Surface Lifecycle / Roles
**Responsibility**: implement the role-specific surface state machines
(xdg-toplevel, xdg-popup, xdg-positioner, wl_shell_surface, subsurface).
Manage configure/ack-configure round-trips, popup grabs, window state.
**Files of interest**:
- `XdgWmBase.ts` — the `xdg_wm_base` global. How clients advertise they're
XDG clients. Includes ping-based unresponsiveness detection.
- `XdgSurface.ts` — `xdg_surface`. The role wrapper around a `wl_surface`.
- `XdgToplevel.ts` — `xdg_toplevel`. Top-level windows. State (maximized,
fullscreen, minimized, activated, resizing). Configure/ack-configure
round-trip.
- `XdgPopup.ts` — `xdg_popup`. Transient windows, menus, tooltips. Grab
semantics.
- `XdgPositioner.ts` — `xdg_positioner`. How popups are positioned relative
to parent.
- `Shell.ts`, `ShellSurface.ts` — the legacy `wl_shell` (deprecated but still
implemented for old clients).
- `Subcompositor.ts`, `Subsurface.ts`, `SurfaceChild.ts`, `SurfaceRole.ts`
— subsurface compositing, the surface tree.
**State machines**: see `08-app-lifecycle.md` for the full
XdgToplevel lifecycle, configure/ack round-trip semantics, popup
positioner math, and subsurface sync/desync mode.
**Dependencies**: Layer 5 + Layer 3 (Wayland runtime).
## Layer 7 — Desktop / Layout Policy
**Responsibility**: arrange surfaces on the output. Decide stacking order,
focus policy, and the high-level window management mode (Floating vs
AlwaysFullScreen).
**Files of interest**:
- `desktop/Desktop.ts` — implements both `FloatingDesktopSurface` and
`AlwaysFullScreenDesktopSurface`, chosen by `session.config.mode`.
Floating lets the user move/resize windows freely; AlwaysFullScreen
enforces one window per output (kiosk mode).
**Dependencies**: Layer 6 + Layer 5 (for View rendering).
## Layer 8 — User Shell
**Responsibility**: the actual user-facing UI. The taskbar, window
decorations, app launcher, keyboard layout picker, etc.
**Out of scope for the user** — they will build their own shell as an Odoo 20
OWL addon. The integration point is `UserShellApi.ts` (see
`08-app-lifecycle.md`).
**Greenfield's reference shell** (`packages/compositor-shell`) is a Preact +
Tailwind SPA, 562 lines of TypeScript total. Mentioned here only for
orientation; not analyzed in this bundle.
## How data flows across the layers
```
NATIVE APP (e.g. gtk4-demo, X11 via XWayland)
│
│ libwayland-client → UNIX socket + SCM_RIGHTS FDs
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ NATIVE PROXY (Node.js + C N-API addons) │
│ │
│ Layer 3 (native): wayland-server.ts + wayland-server-addon.node │
│ ↳ patched libwayland-server parses wire bytes │
│ ↳ MessageInterceptor routes each request to {native, browser} │
│ ↳ FDs wrapped as ProxyFD JSON │
│ │
│ Layer 4 (native): Channel.ts + kcp.ts + SessionController │
│ ↳ 4 logical channels: PROTOCOL, FRAME, XWM, FEEDBACK │
│ ↳ KCP-over-WebSocket for ARQ channels │
│ ↳ SimpleChannel for FEEDBACK │
│ │
│ Layer 5 (encoding): SurfaceBufferEncoding + proxy-encoding-addon │
│ ↳ GStreamer H.264 (x264 / nvh264 / vaapih264) on dedicated thread │
│ ↳ PNG fallback for ≤256×256 surfaces │
│ ↳ Zero-copy dmabuf via EGLImage → GstGLMemoryEGL │
│ │
│ Layer 1 (XML) → Layer 2 (proxy-generator) feeds Layer 3 interceptors │
└─────────────────────────────────────────────────────────────────────────┘
│
│ WebSocket (TCP) + KCP-over-WebSocket
│ 9 endpoint paths per session
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ BROWSER COMPOSITOR (TypeScript + WebGL + WebCodecs + WASM) │
│ │
│ Layer 4 (browser): Channel.ts + RemoteAppLauncher + decoders │
│ ↳ ReconnectingWebSocket with exponential backoff │
│ ↳ ARQChannel + SimpleChannel mirror the native side │
│ ↳ WebCodecs VideoDecoder preferred, FFmpeg WASM fallback │
│ ↳ H264NALDecoder.worker.ts (2 workers: opaque + alpha) │
│ │
│ Layer 3 (browser): @gfld/common Connection + compositor-protocol │
│ ↳ Standard Wayland wire format, byte-for-byte │
│ ↳ WlObject registry, opcode dispatch via wlObject[opcode](msg) │
│ ↳ Per-client Connections (slow client never blocks another) │
│ │
│ Layer 5: Surface.ts + View.ts + Output.ts + render/ │
│ ↳ Double-buffered state (state vs pendingState) │
│ ↳ Damage tracking via pixman-WASM regions │
│ ↳ WebGL1 with FBOs for YUV→RGBA, RGB+Alpha→RGBA conversion │
│ ↳ desynchronized:true canvas for low-latency presentation │
│ ↳ Frame callbacks fired on requestAnimationFrame vsync │
│ │
│ Layer 6: XdgWmBase + XdgSurface + XdgToplevel + XdgPopup + ... │
│ ↳ Configure/ack-configure round-trips │
│ ↳ Popup positioner math (anchor + gravity + constraints) │
│ ↳ Subsurface sync/desync mode cascading │
│ │
│ Layer 7: desktop/Desktop.ts │
│ ↳ Floating vs AlwaysFullScreen mode │
│ ↳ View stacking, focus refcount │
│ │
│ Layer 8: UserShellApi (events + actions) │
│ ↳ Events: clientCreated, surfaceCreated, surfaceTitleUpdated, │
│ surfaceActivationUpdated, ... │
│ ↳ Actions: initScene, refreshScene, destroyScene, │
│ closeClient, activateSurface, setUserConfiguration │
│ ↳ THE INTEGRATION SEAM — Odoo 20 OWL shell hooks here │
└─────────────────────────────────────────────────────────────────────────┘
│
│ UserShellApi.events.* fired by compositor
│ UserShellApi.actions.* called by shell
▼
┌─────────────────────────────────────────────────────────────────────────┐
│ USER SHELL (Odoo 20 OWL addon — supplied by user) │
│ │
│ - Application launcher (lists apps from --applications JSON) │
│ - Window management UI (taskbar, decorations, minimize/maximize) │
│ - Keyboard layout picker (RMLVO) │
│ - Per-output canvas (HTMLCanvasElement, one per Output) │
│ - Trigger app launches via fetch('http://proxy:8081/<path>') │
│ - Open signalURL WebSocket to receive CONNECT_CHANNEL messages │
│ │
│ Not analyzed in this bundle (out of scope). │
└─────────────────────────────────────────────────────────────────────────┘
```
## Reading guide for the rest of the bundle
Now that you've seen the layer map, the remaining chapters drill into each
layer in detail:
- `02-wayland-protocol.md` — Layers 1, 2, 3 (XML → generators → runtime)
- `03-html5-rendering.md` — Layer 5 (rendering pipeline)
- `04-transport.md` — Layer 4 (transport, both sides)
- `05-input-handling.md` — Layer 5 input subsystem (cross-cuts Layers 3, 4, 5)
- `06-data-clipboard.md` — Layer 5 data device (cross-cuts Layers 3, 4, 5)
- `07-native-app-bridge.md` — Layer 4 native + Layer 3 native + Layer 1 XML native (proxy-only `wl_drm`)
- `08-app-lifecycle.md` — Layer 6 + Layer 8 (UserShellApi seam)
- `09-hardware-accel.md` — Layer 5 encoding + Layer 4 frame channel + Layer 3 WASM
- `10-porting-to-odoo-js-python.md` — how to rebuild all 8 layers in your stack
- `11-pitfalls.md` — what Greenfield does wrong, layer by layer
- `12-appendix.md` — glossary, file index, external references