Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents)
This commit is contained in:
commit
c4b45cf218
15 files changed
+5444
No files matched your search
+397
@@ -0,0 +1,397 @@
|
||||
# 04 — Transport Layer (WebSocket / KCP / Channel Layer)
|
||||
|
||||
This chapter covers Greenfield's transport: how Wayland wire traffic flows
|
||||
between the Linux-side proxy and the browser-side compositor over WebSocket,
|
||||
the binary framing, message ordering, latency/backpressure, KCP usage,
|
||||
multiplexing, and connection lifecycle.
|
||||
|
||||
## Architecture at a glance
|
||||
|
||||
Greenfield uses a **multi-WebSocket, KCP-wrapped, asymmetric flow-control**
|
||||
design:
|
||||
|
||||
- **Two-tier process model**: A `main.ts` HTTP hub forks one `SessionProcess`
|
||||
per browser session (per `compositorSessionId`). Raw `net.Socket`s are
|
||||
handed to children via Node IPC `sendHandle`. Each session child owns its
|
||||
own `wl_display` (via `wayland-server-addon.node`).
|
||||
- **9 WebSocket endpoints** per session: `/signal` (JSON control),
|
||||
`/channel` (binary data channels), `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
|
||||
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
|
||||
All with `perMessageDeflate: false`, TCP `noDelay: true`.
|
||||
|
||||
## Channel types
|
||||
|
||||
There are **4 logical channel types** per app client, each implemented as a
|
||||
separate WebSocket:
|
||||
|
||||
| Type | Class | KCP? | Purpose |
|
||||
|------|-------|------|---------|
|
||||
| PROTOCOL | ARQChannel | Yes | Wayland wire protocol, per `wl_client` |
|
||||
| FRAME | ARQChannel | Yes | H.264/PNG encoded surface content, per `wl_surface` |
|
||||
| XWM | ARQChannel | Yes | XWayland window manager protocol |
|
||||
| FEEDBACK | SimpleChannel | No | 4-byte telemetry (refreshInterval + decode avg), 1 Hz |
|
||||
|
||||
**There is no multiplexing** of multiple Wayland clients onto one WS. Each
|
||||
logical channel is its own WS, eliminating head-of-line blocking between
|
||||
video, input, and control at the cost of many connections per app (10+ for
|
||||
complex apps).
|
||||
|
||||
## KCP-over-WebSocket
|
||||
|
||||
`kcp.ts` is a ~1100-line TypeScript port of the canonical C `ikcp.c`.
|
||||
Greenfield runs it **on top of WebSocket** (not real UDP) for:
|
||||
|
||||
- 30 ms min RTO vs TCP's 200 ms.
|
||||
- 100 ms update tick (`setInterval(kcp.update, 100)`).
|
||||
- Per-channel congestion domains independent of TCP.
|
||||
- `setNoDelay(1, 100, 0, 1)` — nodelay enabled, congestion control disabled
|
||||
(`nc=1`).
|
||||
- 24-byte segment header: `conv|cmd|frg|wnd|ts|sn|una|len`.
|
||||
- MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC
|
||||
design).
|
||||
|
||||
### Asymmetric windows (deliberate)
|
||||
|
||||
- Proxy side: `snd_wnd=128, rcv_wnd=1024`
|
||||
- Browser side: `snd_wnd=1024, rcv_wnd=128`
|
||||
|
||||
So proxy→browser (heavy video) has a smaller in-flight cap to bound latency;
|
||||
browser→proxy (bursty input) has a large window to avoid stalls.
|
||||
|
||||
### Why KCP-over-WebSocket?
|
||||
|
||||
Because real-time video over plain TCP suffers from head-of-line blocking
|
||||
and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in
|
||||
user-space, while still riding on WebSocket (the only bidirectional binary
|
||||
transport universally available in browsers today).
|
||||
|
||||
The long-term plan, documented in `docs/pages/design/index.md`, is to
|
||||
switch to **WebTransport** (real UDP) when browser support is widespread,
|
||||
and pair it with KCP for reliability. The KCP code is already in place;
|
||||
only the WebSocket transport layer needs to be swapped out.
|
||||
|
||||
## Binary message format
|
||||
|
||||
All multi-byte fields are **little-endian**.
|
||||
|
||||
### PROTOCOL channel
|
||||
|
||||
```
|
||||
u32 outOfBandOpcode (0 = in-band events, else 1=ResourceDestroyed,
|
||||
2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial)
|
||||
u32 eventSerial
|
||||
u32 fdsCount
|
||||
for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned)
|
||||
concatenated Wayland wire messages (object_id + opcode/size + args)
|
||||
```
|
||||
|
||||
### FRAME channel (per H.264 frame)
|
||||
|
||||
```
|
||||
u32 bufferId
|
||||
u32 bufferCreationSerial
|
||||
u32 bufferContentSerial
|
||||
encodedFrame:
|
||||
u32 contentSerial, u32 encodingType (0=h264/1=png),
|
||||
u32 width, u32 height, u32 encodedWidth, u32 encodedHeight,
|
||||
u32 opaqueLength, u8[] opaqueData,
|
||||
u32 alphaLength, u8[] alphaData ← alpha encoded as a SEPARATE H.264 stream
|
||||
```
|
||||
|
||||
### FEEDBACK channel
|
||||
|
||||
Just 4 bytes: `u16 refreshIntervalMs + u16 avgDecodeDurationMs`. Sent at
|
||||
1 Hz from browser to proxy. Used by the proxy to pace `wl_frame.done`
|
||||
callbacks so the native app sees a virtual monitor refresh rate matching
|
||||
the browser's actual RAF cadence.
|
||||
|
||||
## File descriptor handling
|
||||
|
||||
Can't send FDs over WebSocket — Greenfield virtualizes them:
|
||||
|
||||
- **ProxyFD** = `{handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown',
|
||||
host:string}`.
|
||||
- Serialized as JSON inline in the message stream with a u32 length prefix.
|
||||
- Browser opens **separate ad-hoc WebSocket** to `/read-fd`,
|
||||
`/write-fd-as-stream`, etc., only when it actually needs the bytes.
|
||||
- Most `wl_buffer` FDs never need to be read because the proxy intercepts
|
||||
`wl_surface.commit`, reads the buffer locally via DRM/SHM, encodes to
|
||||
H.264, and ships only the encoded bytes.
|
||||
- **Foreign ProxyFD** (`proxyFD.host !== local baseURL`) triggers
|
||||
proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the
|
||||
foreign proxy's `/write-fd-as-stream`, and pipes the data through. This
|
||||
is how copy-paste between two proxy instances avoids a browser round-trip.
|
||||
|
||||
## Reconnection & lifecycle
|
||||
|
||||
### Browser side
|
||||
|
||||
Vendored `ReconnectingWebSocket` (pladaria) with:
|
||||
|
||||
- `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`
|
||||
- `minUptime=5s`, `connTimeout=4s`
|
||||
- `retries=∞`
|
||||
|
||||
Channel state preservation: only close codes 4000–4999 (application-level)
|
||||
tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps
|
||||
ticking and the WS reconnects underneath.
|
||||
|
||||
### Proxy side
|
||||
|
||||
No auto-reconnect (it's the server). The Channel object lives on, waiting
|
||||
for a new WS upgrade with matching `id` + `key`. Outbound messages queue
|
||||
in `outboundMessages` and flush on `onOpen`.
|
||||
|
||||
### App kill timers
|
||||
|
||||
When the browser disconnects from the `/signal` WebSocket:
|
||||
|
||||
- **10-minute SIGHUP grace period** — gives the user time to reconnect
|
||||
(e.g. if they refresh the browser tab).
|
||||
- After 10 minutes of no reconnect, escalating to **SIGKILL after another
|
||||
10s**.
|
||||
- Close code 1001 → immediate SIGHUP (user closed tab intentionally).
|
||||
- Close code 4001 → immediate SIGTERM (user clicked "force quit" in the
|
||||
shell).
|
||||
|
||||
### Session close
|
||||
|
||||
When all `wl_clients` disconnect, 1-second grace then `session.close()` →
|
||||
`process.exit()` in the session child. Main process deletes the entry
|
||||
from its `sessionProcesses` map.
|
||||
|
||||
## Process model — three layers
|
||||
|
||||
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
|
||||
|
||||
A single Node.js process listens on `:8081`. Per browser session, it
|
||||
`fork()`s one `SessionProcess` child — one child per `compositorSessionId`,
|
||||
**not** per app. Multiple apps in the same browser tab share one child.
|
||||
WebSocket upgrades are handed off to the matching child via
|
||||
`childProcess.send(wsUpgrade, socket)`.
|
||||
|
||||
### Layer 2: Session Process (`SessionProcess.ts`)
|
||||
|
||||
The forked child. Receives `start` (creates `Session` +
|
||||
`SessionController`), `launchApp` (spawns the actual app binary), and
|
||||
`wsUpgrade` (hands WS to the controller) IPC messages. On `launchApp` it
|
||||
calls `launchApplication()` from `@gfld/compositor-proxy`, which `spawn()`s
|
||||
the executable, captures `pid`, replies to main with `{pid, key}`. On
|
||||
`session.close()` the process exits.
|
||||
|
||||
### Layer 3: Proxy Core (`compositor-proxy/src/`)
|
||||
|
||||
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
|
||||
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
|
||||
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
|
||||
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
|
||||
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
|
||||
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
|
||||
wl_display FD into Node's libuv loop via `startPoll()`.
|
||||
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
|
||||
Uses `MessageInterceptor` + auto-generated per-resource interceptor
|
||||
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
|
||||
`{native, browser, neverReplies?}` routing decision. Handles
|
||||
`wl_display.sync` fast-path (skips browser round-trip when no reply
|
||||
expected). Marshalls FDs as `ProxyFD` JSON.
|
||||
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
|
||||
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
|
||||
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
|
||||
(10 s) on force-quit, auto-destroy on app exit.
|
||||
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
|
||||
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
|
||||
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
|
||||
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
|
||||
connect). Xwayland itself becomes a Wayland client of the proxy. The
|
||||
`wm_fd` is bridged via xtsb to the browser, which implements the actual
|
||||
XWM logic.
|
||||
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
|
||||
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
|
||||
|
||||
## ASCII diagram of the full transport stack
|
||||
|
||||
```
|
||||
LINUX SIDE | BROWSER SIDE
|
||||
|
|
||||
┌─────────────────────────┐ | ┌──────────────────────────────┐
|
||||
│ gtk4-demo (process) │ | │ Browser tab │
|
||||
│ libwayland-client.so │ | │ (Greenfield compositor) │
|
||||
└───────────┬─────────────┘ | └──────────────┬───────────────┘
|
||||
│ WAYLAND_DISPLAY=wayland-0 │
|
||||
│ Unix socket + SCM_RIGHTS FD passing │ DOM events → input
|
||||
▼ │ WebSocket API
|
||||
┌──────────────────────────────────────────┐ │
|
||||
│ SessionProcess (fork'd Node child) │ │
|
||||
│ ┌────────────────────────────────────┐ │ │
|
||||
│ │ wayland-server-addon.node │ │ │
|
||||
│ │ (libwayland-server N-API) │ │ │
|
||||
│ │ - wl_display + wl_client │ │ │
|
||||
│ │ - epoll via proxy-poll-addon │ │ │
|
||||
│ └─────────────┬──────────────────────┘ │ │
|
||||
│ │ wire msg + FDs │ │
|
||||
│ ▼ │ │
|
||||
│ ┌─────────────────────────────────────┐ │ │
|
||||
│ │ NativeWaylandClientSession │ │ │
|
||||
│ │ - interceptRequest / interceptEvent│ │ │
|
||||
│ │ - onWireMessageEnd: serialize │ │ │
|
||||
│ │ FDs as JSON ProxyFDs inline │ │ │
|
||||
│ │ - fastSync optimization │ │ │
|
||||
│ └─────────────┬───────────────────────┘ │ │
|
||||
│ │ outbound buffer │ │
|
||||
│ ▼ │ │
|
||||
│ ┌─────────────────────────────────────┐ │ │
|
||||
│ │ ARQChannel (KCP-wrapped) │ │ │
|
||||
│ │ MTU=64000, snd_wnd=128, rcv_wnd=1024│ │ │
|
||||
│ │ KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data │
|
||||
│ │ kcp.update() every 100ms │ │ │
|
||||
│ │ - PROTOCOL (per wl_client) │ │ │
|
||||
│ │ - FRAME (per wl_surface) │ │ │
|
||||
│ │ - XWM (per XWayland) │ │ │
|
||||
│ └─────────────┬───────────────────────┘ │ │
|
||||
│ ┌─────────────┴───────────────────────┐ │ │
|
||||
│ │ SimpleChannel (FEEDBACK, no KCP) │ │ │
|
||||
│ │ 4 bytes: refreshIntervalMs + │ │ │
|
||||
│ │ avgDecodeDurationMs │ │ │
|
||||
│ └─────────────┬───────────────────────┘ │ │
|
||||
│ ┌─────────────▼───────────────────────┐ │ │
|
||||
│ │ NativeAppContext │ │ │
|
||||
│ │ - signalingWebSocket (JSON msgs) │ │ │
|
||||
│ │ - kill timers: SIGHUP@10min, │ │ │
|
||||
│ │ SIGKILL@+10s │ │ │
|
||||
│ └─────────────┬───────────────────────┘ │ │
|
||||
│ ┌─────────────▼───────────────────────┐ │ │
|
||||
│ │ SessionController │ │ │
|
||||
│ │ ws.WebSocketServer (noServer) │ │ │
|
||||
│ │ perMessageDeflate: false │ │ │
|
||||
│ │ Endpoints: /signal /channel /mkfifo│ │ │
|
||||
│ │ /mkstemp-mmap /read-fd /write-fd-as-stream│ │
|
||||
│ │ /read-fd-as-stream /close-fd /request-keyframe│ │
|
||||
│ └─────────────┬───────────────────────┘ │ │
|
||||
└────────────────┼─────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────┐ │
|
||||
│ │ main.ts (HTTP+WS hub) │ │
|
||||
│ │ - socket.pause() + IPC │ │
|
||||
│ │ send to session proc │ │
|
||||
│ │ - noDelay:true (TCP) │ │
|
||||
│ └────────────┬────────────┘ │
|
||||
│ │ │
|
||||
│ ◄──── 1 WebSocket per channel ────► │
|
||||
│ ◄── 1 signaling WS per app ──► │
|
||||
│ ◄── ad-hoc WebFD streams ─► │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────────────┐ │
|
||||
│ │ ReconnectingWebSocket │ │
|
||||
│ │ min 1-5s, factor 1.3, max 10s │ │
|
||||
│ │ minUptime 5s, connTimeout 4s │ │
|
||||
│ └────────────┬───────────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────────────────┐ │
|
||||
│ │ ARQChannel / SimpleChannel (mirror image) │ │
|
||||
│ │ snd_wnd=1024, rcv_wnd=128 │ │
|
||||
│ │ kcp.input → peekSize/recv → onMessage │ │
|
||||
│ └────────────┬───────────────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────────────────┐ │
|
||||
│ │ RemoteAppLauncher.onChannel │ │
|
||||
│ │ PROTOCOL → session.display.createClient │ │
|
||||
│ │ FRAME → deliverContentToBufferStream │ │
|
||||
│ │ XWM → XWindowManagerConnection │ │
|
||||
│ │ FEEDBACK → ClientEncodersFeedback │ │
|
||||
│ └────────────┬───────────────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────────────────┐ │
|
||||
│ │ @gfld/compositor-protocol Client │ │
|
||||
│ │ - parses Wayland wire format │ │
|
||||
│ │ - dispatches to wl_*_resource impls │ │
|
||||
│ │ - H264NALDecoder.worker.ts decodes frames │ │
|
||||
│ └────────────┬───────────────────────────────┘ │
|
||||
│ ▼ │
|
||||
│ ┌────────────────────────────────────────────┐ │
|
||||
│ │ Surface + Scene rendering │ │
|
||||
│ │ - ImageBitmap / VideoFrame / Canvas │ │
|
||||
│ │ - DOM input → wl_keyboard/pointer reqs │ │
|
||||
│ └────────────┬───────────────────────────────┘ │
|
||||
│ │ input events flow back via the │
|
||||
│ │ PROTOCOL channel as ordinary │
|
||||
│ │ Wayland requests (same framing) │
|
||||
└──────────────────┘ │
|
||||
Browser DOM ◄─┘
|
||||
```
|
||||
|
||||
## Key takeaways for building your own transport
|
||||
|
||||
1. **Separate control plane from data plane** — Greenfield uses `/signal`
|
||||
(JSON) vs `/channel` (binary). The control plane handles app launch,
|
||||
termination, kill timers; the data plane handles Wayland wire + encoded
|
||||
frames.
|
||||
2. **One WS per logical stream** — simpler, no HoL blocking, but many TCP
|
||||
connections.
|
||||
3. **KCP-over-WS** trades double-reliability overhead for fast (30 ms)
|
||||
retransmits; reasonable for real-time video.
|
||||
4. **Virtualize FDs as JSON tokens** with lazy separate-WS endpoints for
|
||||
actual byte transfer.
|
||||
5. **`bufferedAmount` check ≠ backpressure** — Greenfield's check is a drop
|
||||
gate, not real flow control (acknowledged TODO).
|
||||
6. **Distinguish transient vs permanent close** — codes 4000-4999 tear
|
||||
down; lower codes auto-reconnect.
|
||||
7. **Piggyback out-of-band opcodes** on the data channel via a leading u32
|
||||
to avoid extra control WS.
|
||||
8. **Asymmetric KCP windows** match each direction's bandwidth profile.
|
||||
9. **Decode feedback drives frame pacing** — the proxy synthesizes
|
||||
`wl_frame.done` callbacks at the browser's reported cadence.
|
||||
10. **One Node child per browser session** for fault isolation and
|
||||
per-session `wl_display` ownership.
|
||||
|
||||
## File reference
|
||||
|
||||
### Native side
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `packages/compositor-proxy/src/Channel.ts` | `ARQChannel` + `SimpleChannel` |
|
||||
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
|
||||
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
|
||||
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
|
||||
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
|
||||
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||||
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
|
||||
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
|
||||
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
|
||||
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
|
||||
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
|
||||
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||||
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
|
||||
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
|
||||
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
|
||||
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
|
||||
|
||||
### Browser side
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `packages/compositor/src/remote/Channel.ts` | `ARQChannel` + `SimpleChannel` (mirror) |
|
||||
| `packages/compositor/src/remote/kcp.ts` | KCP mirror |
|
||||
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
|
||||
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
|
||||
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||||
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
|
||||
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||||
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||||
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||||
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
|
||||
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||||
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||||
| `packages/compositor/src/remote/api.ts` | Public exports |
|
||||
|
||||
### CLI
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
|
||||
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
|
||||
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
|
||||
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
|
||||
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
|
||||
| `packages/compositor-proxy-cli/wait-until-ready.sh` | Startup readiness check |
|
||||
Reference in new issue
Block a user