Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents)

This commit is contained in:
1b-agent committed 2026-10-08 08:48:25 +00:00
commit c4b45cf218
15 files changed
+5444

No files matched your search

+397
View File
@@ -0,0 +1,397 @@
# 04 — Transport Layer (WebSocket / KCP / Channel Layer)
This chapter covers Greenfield's transport: how Wayland wire traffic flows
between the Linux-side proxy and the browser-side compositor over WebSocket,
the binary framing, message ordering, latency/backpressure, KCP usage,
multiplexing, and connection lifecycle.
## Architecture at a glance
Greenfield uses a **multi-WebSocket, KCP-wrapped, asymmetric flow-control**
design:
- **Two-tier process model**: A `main.ts` HTTP hub forks one `SessionProcess`
per browser session (per `compositorSessionId`). Raw `net.Socket`s are
handed to children via Node IPC `sendHandle`. Each session child owns its
own `wl_display` (via `wayland-server-addon.node`).
- **9 WebSocket endpoints** per session: `/signal` (JSON control),
`/channel` (binary data channels), `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
All with `perMessageDeflate: false`, TCP `noDelay: true`.
## Channel types
There are **4 logical channel types** per app client, each implemented as a
separate WebSocket:
| Type | Class | KCP? | Purpose |
|------|-------|------|---------|
| PROTOCOL | ARQChannel | Yes | Wayland wire protocol, per `wl_client` |
| FRAME | ARQChannel | Yes | H.264/PNG encoded surface content, per `wl_surface` |
| XWM | ARQChannel | Yes | XWayland window manager protocol |
| FEEDBACK | SimpleChannel | No | 4-byte telemetry (refreshInterval + decode avg), 1 Hz |
**There is no multiplexing** of multiple Wayland clients onto one WS. Each
logical channel is its own WS, eliminating head-of-line blocking between
video, input, and control at the cost of many connections per app (10+ for
complex apps).
## KCP-over-WebSocket
`kcp.ts` is a ~1100-line TypeScript port of the canonical C `ikcp.c`.
Greenfield runs it **on top of WebSocket** (not real UDP) for:
- 30 ms min RTO vs TCP's 200 ms.
- 100 ms update tick (`setInterval(kcp.update, 100)`).
- Per-channel congestion domains independent of TCP.
- `setNoDelay(1, 100, 0, 1)` — nodelay enabled, congestion control disabled
(`nc=1`).
- 24-byte segment header: `conv|cmd|frg|wnd|ts|sn|una|len`.
- MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC
design).
### Asymmetric windows (deliberate)
- Proxy side: `snd_wnd=128, rcv_wnd=1024`
- Browser side: `snd_wnd=1024, rcv_wnd=128`
So proxy→browser (heavy video) has a smaller in-flight cap to bound latency;
browser→proxy (bursty input) has a large window to avoid stalls.
### Why KCP-over-WebSocket?
Because real-time video over plain TCP suffers from head-of-line blocking
and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in
user-space, while still riding on WebSocket (the only bidirectional binary
transport universally available in browsers today).
The long-term plan, documented in `docs/pages/design/index.md`, is to
switch to **WebTransport** (real UDP) when browser support is widespread,
and pair it with KCP for reliability. The KCP code is already in place;
only the WebSocket transport layer needs to be swapped out.
## Binary message format
All multi-byte fields are **little-endian**.
### PROTOCOL channel
```
u32 outOfBandOpcode (0 = in-band events, else 1=ResourceDestroyed,
2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial)
u32 eventSerial
u32 fdsCount
for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned)
concatenated Wayland wire messages (object_id + opcode/size + args)
```
### FRAME channel (per H.264 frame)
```
u32 bufferId
u32 bufferCreationSerial
u32 bufferContentSerial
encodedFrame:
u32 contentSerial, u32 encodingType (0=h264/1=png),
u32 width, u32 height, u32 encodedWidth, u32 encodedHeight,
u32 opaqueLength, u8[] opaqueData,
u32 alphaLength, u8[] alphaData ← alpha encoded as a SEPARATE H.264 stream
```
### FEEDBACK channel
Just 4 bytes: `u16 refreshIntervalMs + u16 avgDecodeDurationMs`. Sent at
1 Hz from browser to proxy. Used by the proxy to pace `wl_frame.done`
callbacks so the native app sees a virtual monitor refresh rate matching
the browser's actual RAF cadence.
## File descriptor handling
Can't send FDs over WebSocket — Greenfield virtualizes them:
- **ProxyFD** = `{handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown',
host:string}`.
- Serialized as JSON inline in the message stream with a u32 length prefix.
- Browser opens **separate ad-hoc WebSocket** to `/read-fd`,
`/write-fd-as-stream`, etc., only when it actually needs the bytes.
- Most `wl_buffer` FDs never need to be read because the proxy intercepts
`wl_surface.commit`, reads the buffer locally via DRM/SHM, encodes to
H.264, and ships only the encoded bytes.
- **Foreign ProxyFD** (`proxyFD.host !== local baseURL`) triggers
proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the
foreign proxy's `/write-fd-as-stream`, and pipes the data through. This
is how copy-paste between two proxy instances avoids a browser round-trip.
## Reconnection & lifecycle
### Browser side
Vendored `ReconnectingWebSocket` (pladaria) with:
- `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`
- `minUptime=5s`, `connTimeout=4s`
- `retries=∞`
Channel state preservation: only close codes 4000–4999 (application-level)
tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps
ticking and the WS reconnects underneath.
### Proxy side
No auto-reconnect (it's the server). The Channel object lives on, waiting
for a new WS upgrade with matching `id` + `key`. Outbound messages queue
in `outboundMessages` and flush on `onOpen`.
### App kill timers
When the browser disconnects from the `/signal` WebSocket:
- **10-minute SIGHUP grace period** — gives the user time to reconnect
(e.g. if they refresh the browser tab).
- After 10 minutes of no reconnect, escalating to **SIGKILL after another
10s**.
- Close code 1001 → immediate SIGHUP (user closed tab intentionally).
- Close code 4001 → immediate SIGTERM (user clicked "force quit" in the
shell).
### Session close
When all `wl_clients` disconnect, 1-second grace then `session.close()` →
`process.exit()` in the session child. Main process deletes the entry
from its `sessionProcesses` map.
## Process model — three layers
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
A single Node.js process listens on `:8081`. Per browser session, it
`fork()`s one `SessionProcess` child — one child per `compositorSessionId`,
**not** per app. Multiple apps in the same browser tab share one child.
WebSocket upgrades are handed off to the matching child via
`childProcess.send(wsUpgrade, socket)`.
### Layer 2: Session Process (`SessionProcess.ts`)
The forked child. Receives `start` (creates `Session` +
`SessionController`), `launchApp` (spawns the actual app binary), and
`wsUpgrade` (hands WS to the controller) IPC messages. On `launchApp` it
calls `launchApplication()` from `@gfld/compositor-proxy`, which `spawn()`s
the executable, captures `pid`, replies to main with `{pid, key}`. On
`session.close()` the process exits.
### Layer 3: Proxy Core (`compositor-proxy/src/`)
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
wl_display FD into Node's libuv loop via `startPoll()`.
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
Uses `MessageInterceptor` + auto-generated per-resource interceptor
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
`{native, browser, neverReplies?}` routing decision. Handles
`wl_display.sync` fast-path (skips browser round-trip when no reply
expected). Marshalls FDs as `ProxyFD` JSON.
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
(10 s) on force-quit, auto-destroy on app exit.
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
connect). Xwayland itself becomes a Wayland client of the proxy. The
`wm_fd` is bridged via xtsb to the browser, which implements the actual
XWM logic.
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
## ASCII diagram of the full transport stack
```
LINUX SIDE | BROWSER SIDE
|
┌─────────────────────────┐ | ┌──────────────────────────────┐
│ gtk4-demo (process) │ | │ Browser tab │
│ libwayland-client.so │ | │ (Greenfield compositor) │
└───────────┬─────────────┘ | └──────────────┬───────────────┘
│ WAYLAND_DISPLAY=wayland-0 │
│ Unix socket + SCM_RIGHTS FD passing │ DOM events → input
▼ │ WebSocket API
┌──────────────────────────────────────────┐ │
│ SessionProcess (fork'd Node child) │ │
│ ┌────────────────────────────────────┐ │ │
│ │ wayland-server-addon.node │ │ │
│ │ (libwayland-server N-API) │ │ │
│ │ - wl_display + wl_client │ │ │
│ │ - epoll via proxy-poll-addon │ │ │
│ └─────────────┬──────────────────────┘ │ │
│ │ wire msg + FDs │ │
│ ▼ │ │
│ ┌─────────────────────────────────────┐ │ │
│ │ NativeWaylandClientSession │ │ │
│ │ - interceptRequest / interceptEvent│ │ │
│ │ - onWireMessageEnd: serialize │ │ │
│ │ FDs as JSON ProxyFDs inline │ │ │
│ │ - fastSync optimization │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ │ outbound buffer │ │
│ ▼ │ │
│ ┌─────────────────────────────────────┐ │ │
│ │ ARQChannel (KCP-wrapped) │ │ │
│ │ MTU=64000, snd_wnd=128, rcv_wnd=1024│ │ │
│ │ KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data │
│ │ kcp.update() every 100ms │ │ │
│ │ - PROTOCOL (per wl_client) │ │ │
│ │ - FRAME (per wl_surface) │ │ │
│ │ - XWM (per XWayland) │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ ┌─────────────┴───────────────────────┐ │ │
│ │ SimpleChannel (FEEDBACK, no KCP) │ │ │
│ │ 4 bytes: refreshIntervalMs + │ │ │
│ │ avgDecodeDurationMs │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ ┌─────────────▼───────────────────────┐ │ │
│ │ NativeAppContext │ │ │
│ │ - signalingWebSocket (JSON msgs) │ │ │
│ │ - kill timers: SIGHUP@10min, │ │ │
│ │ SIGKILL@+10s │ │ │
│ └─────────────┬───────────────────────┘ │ │
│ ┌─────────────▼───────────────────────┐ │ │
│ │ SessionController │ │ │
│ │ ws.WebSocketServer (noServer) │ │ │
│ │ perMessageDeflate: false │ │ │
│ │ Endpoints: /signal /channel /mkfifo│ │ │
│ │ /mkstemp-mmap /read-fd /write-fd-as-stream│ │
│ │ /read-fd-as-stream /close-fd /request-keyframe│ │
│ └─────────────┬───────────────────────┘ │ │
└────────────────┼─────────────────────────────┘ │
│ │
│ ┌──────────────────────────┐ │
│ │ main.ts (HTTP+WS hub) │ │
│ │ - socket.pause() + IPC │ │
│ │ send to session proc │ │
│ │ - noDelay:true (TCP) │ │
│ └────────────┬────────────┘ │
│ │ │
│ ◄──── 1 WebSocket per channel ────► │
│ ◄── 1 signaling WS per app ──► │
│ ◄── ad-hoc WebFD streams ─► │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────┐ │
│ │ ReconnectingWebSocket │ │
│ │ min 1-5s, factor 1.3, max 10s │ │
│ │ minUptime 5s, connTimeout 4s │ │
│ └────────────┬───────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ ARQChannel / SimpleChannel (mirror image) │ │
│ │ snd_wnd=1024, rcv_wnd=128 │ │
│ │ kcp.input → peekSize/recv → onMessage │ │
│ └────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ RemoteAppLauncher.onChannel │ │
│ │ PROTOCOL → session.display.createClient │ │
│ │ FRAME → deliverContentToBufferStream │ │
│ │ XWM → XWindowManagerConnection │ │
│ │ FEEDBACK → ClientEncodersFeedback │ │
│ └────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ @gfld/compositor-protocol Client │ │
│ │ - parses Wayland wire format │ │
│ │ - dispatches to wl_*_resource impls │ │
│ │ - H264NALDecoder.worker.ts decodes frames │ │
│ └────────────┬───────────────────────────────┘ │
│ ▼ │
│ ┌────────────────────────────────────────────┐ │
│ │ Surface + Scene rendering │ │
│ │ - ImageBitmap / VideoFrame / Canvas │ │
│ │ - DOM input → wl_keyboard/pointer reqs │ │
│ └────────────┬───────────────────────────────┘ │
│ │ input events flow back via the │
│ │ PROTOCOL channel as ordinary │
│ │ Wayland requests (same framing) │
└──────────────────┘ │
Browser DOM ◄─┘
```
## Key takeaways for building your own transport
1. **Separate control plane from data plane** — Greenfield uses `/signal`
(JSON) vs `/channel` (binary). The control plane handles app launch,
termination, kill timers; the data plane handles Wayland wire + encoded
frames.
2. **One WS per logical stream** — simpler, no HoL blocking, but many TCP
connections.
3. **KCP-over-WS** trades double-reliability overhead for fast (30 ms)
retransmits; reasonable for real-time video.
4. **Virtualize FDs as JSON tokens** with lazy separate-WS endpoints for
actual byte transfer.
5. **`bufferedAmount` check ≠ backpressure** — Greenfield's check is a drop
gate, not real flow control (acknowledged TODO).
6. **Distinguish transient vs permanent close** — codes 4000-4999 tear
down; lower codes auto-reconnect.
7. **Piggyback out-of-band opcodes** on the data channel via a leading u32
to avoid extra control WS.
8. **Asymmetric KCP windows** match each direction's bandwidth profile.
9. **Decode feedback drives frame pacing** — the proxy synthesizes
`wl_frame.done` callbacks at the browser's reported cadence.
10. **One Node child per browser session** for fault isolation and
per-session `wl_display` ownership.
## File reference
### Native side
| File | Role |
|---|---|
| `packages/compositor-proxy/src/Channel.ts` | `ARQChannel` + `SimpleChannel` |
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
### Browser side
| File | Role |
|---|---|
| `packages/compositor/src/remote/Channel.ts` | `ARQChannel` + `SimpleChannel` (mirror) |
| `packages/compositor/src/remote/kcp.ts` | KCP mirror |
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
| `packages/compositor/src/remote/api.ts` | Public exports |
### CLI
| File | Role |
|---|---|
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
| `packages/compositor-proxy-cli/wait-until-ready.sh` | Startup readiness check |