398 lines
24 KiB
Markdown
398 lines
24 KiB
Markdown
# 04 — Transport Layer (WebSocket / KCP / Channel Layer)
|
||
|
||
This chapter covers Greenfield's transport: how Wayland wire traffic flows
|
||
between the Linux-side proxy and the browser-side compositor over WebSocket,
|
||
the binary framing, message ordering, latency/backpressure, KCP usage,
|
||
multiplexing, and connection lifecycle.
|
||
|
||
## Architecture at a glance
|
||
|
||
Greenfield uses a **multi-WebSocket, KCP-wrapped, asymmetric flow-control**
|
||
design:
|
||
|
||
- **Two-tier process model**: A `main.ts` HTTP hub forks one `SessionProcess`
|
||
per browser session (per `compositorSessionId`). Raw `net.Socket`s are
|
||
handed to children via Node IPC `sendHandle`. Each session child owns its
|
||
own `wl_display` (via `wayland-server-addon.node`).
|
||
- **9 WebSocket endpoints** per session: `/signal` (JSON control),
|
||
`/channel` (binary data channels), `/mkfifo`, `/mkstemp-mmap`, `/read-fd`,
|
||
`/read-fd-as-stream`, `/write-fd-as-stream`, `/close-fd`, `/request-keyframe`.
|
||
All with `perMessageDeflate: false`, TCP `noDelay: true`.
|
||
|
||
## Channel types
|
||
|
||
There are **4 logical channel types** per app client, each implemented as a
|
||
separate WebSocket:
|
||
|
||
| Type | Class | KCP? | Purpose |
|
||
|------|-------|------|---------|
|
||
| PROTOCOL | ARQChannel | Yes | Wayland wire protocol, per `wl_client` |
|
||
| FRAME | ARQChannel | Yes | H.264/PNG encoded surface content, per `wl_surface` |
|
||
| XWM | ARQChannel | Yes | XWayland window manager protocol |
|
||
| FEEDBACK | SimpleChannel | No | 4-byte telemetry (refreshInterval + decode avg), 1 Hz |
|
||
|
||
**There is no multiplexing** of multiple Wayland clients onto one WS. Each
|
||
logical channel is its own WS, eliminating head-of-line blocking between
|
||
video, input, and control at the cost of many connections per app (10+ for
|
||
complex apps).
|
||
|
||
## KCP-over-WebSocket
|
||
|
||
`kcp.ts` is a ~1100-line TypeScript port of the canonical C `ikcp.c`.
|
||
Greenfield runs it **on top of WebSocket** (not real UDP) for:
|
||
|
||
- 30 ms min RTO vs TCP's 200 ms.
|
||
- 100 ms update tick (`setInterval(kcp.update, 100)`).
|
||
- Per-channel congestion domains independent of TCP.
|
||
- `setNoDelay(1, 100, 0, 1)` — nodelay enabled, congestion control disabled
|
||
(`nc=1`).
|
||
- 24-byte segment header: `conv|cmd|frg|wnd|ts|sn|una|len`.
|
||
- MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC
|
||
design).
|
||
|
||
### Asymmetric windows (deliberate)
|
||
|
||
- Proxy side: `snd_wnd=128, rcv_wnd=1024`
|
||
- Browser side: `snd_wnd=1024, rcv_wnd=128`
|
||
|
||
So proxy→browser (heavy video) has a smaller in-flight cap to bound latency;
|
||
browser→proxy (bursty input) has a large window to avoid stalls.
|
||
|
||
### Why KCP-over-WebSocket?
|
||
|
||
Because real-time video over plain TCP suffers from head-of-line blocking
|
||
and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in
|
||
user-space, while still riding on WebSocket (the only bidirectional binary
|
||
transport universally available in browsers today).
|
||
|
||
The long-term plan, documented in `docs/pages/design/index.md`, is to
|
||
switch to **WebTransport** (real UDP) when browser support is widespread,
|
||
and pair it with KCP for reliability. The KCP code is already in place;
|
||
only the WebSocket transport layer needs to be swapped out.
|
||
|
||
## Binary message format
|
||
|
||
All multi-byte fields are **little-endian**.
|
||
|
||
### PROTOCOL channel
|
||
|
||
```
|
||
u32 outOfBandOpcode (0 = in-band events, else 1=ResourceDestroyed,
|
||
2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial)
|
||
u32 eventSerial
|
||
u32 fdsCount
|
||
for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned)
|
||
concatenated Wayland wire messages (object_id + opcode/size + args)
|
||
```
|
||
|
||
### FRAME channel (per H.264 frame)
|
||
|
||
```
|
||
u32 bufferId
|
||
u32 bufferCreationSerial
|
||
u32 bufferContentSerial
|
||
encodedFrame:
|
||
u32 contentSerial, u32 encodingType (0=h264/1=png),
|
||
u32 width, u32 height, u32 encodedWidth, u32 encodedHeight,
|
||
u32 opaqueLength, u8[] opaqueData,
|
||
u32 alphaLength, u8[] alphaData ← alpha encoded as a SEPARATE H.264 stream
|
||
```
|
||
|
||
### FEEDBACK channel
|
||
|
||
Just 4 bytes: `u16 refreshIntervalMs + u16 avgDecodeDurationMs`. Sent at
|
||
1 Hz from browser to proxy. Used by the proxy to pace `wl_frame.done`
|
||
callbacks so the native app sees a virtual monitor refresh rate matching
|
||
the browser's actual RAF cadence.
|
||
|
||
## File descriptor handling
|
||
|
||
Can't send FDs over WebSocket — Greenfield virtualizes them:
|
||
|
||
- **ProxyFD** = `{handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown',
|
||
host:string}`.
|
||
- Serialized as JSON inline in the message stream with a u32 length prefix.
|
||
- Browser opens **separate ad-hoc WebSocket** to `/read-fd`,
|
||
`/write-fd-as-stream`, etc., only when it actually needs the bytes.
|
||
- Most `wl_buffer` FDs never need to be read because the proxy intercepts
|
||
`wl_surface.commit`, reads the buffer locally via DRM/SHM, encodes to
|
||
H.264, and ships only the encoded bytes.
|
||
- **Foreign ProxyFD** (`proxyFD.host !== local baseURL`) triggers
|
||
proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the
|
||
foreign proxy's `/write-fd-as-stream`, and pipes the data through. This
|
||
is how copy-paste between two proxy instances avoids a browser round-trip.
|
||
|
||
## Reconnection & lifecycle
|
||
|
||
### Browser side
|
||
|
||
Vendored `ReconnectingWebSocket` (pladaria) with:
|
||
|
||
- `minDelay=1-5s`, `factor=1.3`, `maxDelay=10s`
|
||
- `minUptime=5s`, `connTimeout=4s`
|
||
- `retries=∞`
|
||
|
||
Channel state preservation: only close codes 4000–4999 (application-level)
|
||
tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps
|
||
ticking and the WS reconnects underneath.
|
||
|
||
### Proxy side
|
||
|
||
No auto-reconnect (it's the server). The Channel object lives on, waiting
|
||
for a new WS upgrade with matching `id` + `key`. Outbound messages queue
|
||
in `outboundMessages` and flush on `onOpen`.
|
||
|
||
### App kill timers
|
||
|
||
When the browser disconnects from the `/signal` WebSocket:
|
||
|
||
- **10-minute SIGHUP grace period** — gives the user time to reconnect
|
||
(e.g. if they refresh the browser tab).
|
||
- After 10 minutes of no reconnect, escalating to **SIGKILL after another
|
||
10s**.
|
||
- Close code 1001 → immediate SIGHUP (user closed tab intentionally).
|
||
- Close code 4001 → immediate SIGTERM (user clicked "force quit" in the
|
||
shell).
|
||
|
||
### Session close
|
||
|
||
When all `wl_clients` disconnect, 1-second grace then `session.close()` →
|
||
`process.exit()` in the session child. Main process deletes the entry
|
||
from its `sessionProcesses` map.
|
||
|
||
## Process model — three layers
|
||
|
||
### Layer 1: CLI Front Door (`compositor-proxy-cli/src/main.ts`)
|
||
|
||
A single Node.js process listens on `:8081`. Per browser session, it
|
||
`fork()`s one `SessionProcess` child — one child per `compositorSessionId`,
|
||
**not** per app. Multiple apps in the same browser tab share one child.
|
||
WebSocket upgrades are handed off to the matching child via
|
||
`childProcess.send(wsUpgrade, socket)`.
|
||
|
||
### Layer 2: Session Process (`SessionProcess.ts`)
|
||
|
||
The forked child. Receives `start` (creates `Session` +
|
||
`SessionController`), `launchApp` (spawns the actual app binary), and
|
||
`wsUpgrade` (hands WS to the controller) IPC messages. On `launchApp` it
|
||
calls `launchApplication()` from `@gfld/compositor-proxy`, which `spawn()`s
|
||
the executable, captures `pid`, replies to main with `{pid, key}`. On
|
||
`session.close()` the process exits.
|
||
|
||
### Layer 3: Proxy Core (`compositor-proxy/src/`)
|
||
|
||
- **`NativeWaylandCompositorSession`** — creates a real `wl_display` via
|
||
`createDisplay()`, calls `addSocketAuto()` which creates the UNIX socket
|
||
at `$XDG_RUNTIME_DIR/wayland-N` and returns the name `wayland-N` (this
|
||
becomes `WAYLAND_DISPLAY`). Calls `initShm()` (adds YUV420/NV12/YUV444/
|
||
NV21 formats) and `initDrm()` (opens `/dev/dri/renderD128` via EGL/GBM,
|
||
registers `zwp_linux_dmabuf_v1` + `wl_drm` globals). Integrates the
|
||
wl_display FD into Node's libuv loop via `startPoll()`.
|
||
- **`NativeWaylandClientSession`** — per-client wire-message interceptor.
|
||
Uses `MessageInterceptor` + auto-generated per-resource interceptor
|
||
modules (`src/protocol/wl_*_interceptor.js`). Each request returns
|
||
`{native, browser, neverReplies?}` routing decision. Handles
|
||
`wl_display.sync` fast-path (skips browser round-trip when no reply
|
||
expected). Marshalls FDs as `ProxyFD` JSON.
|
||
- **`NativeAppContext`** — per-app state: `pid`, `key` (16-hex random
|
||
signaling auth), signaling WS, client sessions, data channels. Lifecycle:
|
||
10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation
|
||
(10 s) on force-quit, auto-destroy on app exit.
|
||
- **`SessionController`** — WS demultiplexer by pathname: `/signal`
|
||
(per-app control), `/channel` (multiplexed data channels), `/mkfifo`,
|
||
`/mkstemp-mmap`, `/read-fd`, `/write-fd-as-stream`, `/request-keyframe`.
|
||
- **`XWaylandSession`** — lazy Xwayland fork (only on first X client
|
||
connect). Xwayland itself becomes a Wayland client of the proxy. The
|
||
`wm_fd` is bridged via xtsb to the browser, which implements the actual
|
||
XWM logic.
|
||
- **`Channel`** — `ARQChannel` (KCP-over-WebSocket, reliable, low-latency)
|
||
for PROTOCOL/FRAME/XWM; `SimpleChannel` for FEEDBACK.
|
||
|
||
## ASCII diagram of the full transport stack
|
||
|
||
```
|
||
LINUX SIDE | BROWSER SIDE
|
||
|
|
||
┌─────────────────────────┐ | ┌──────────────────────────────┐
|
||
│ gtk4-demo (process) │ | │ Browser tab │
|
||
│ libwayland-client.so │ | │ (Greenfield compositor) │
|
||
└───────────┬─────────────┘ | └──────────────┬───────────────┘
|
||
│ WAYLAND_DISPLAY=wayland-0 │
|
||
│ Unix socket + SCM_RIGHTS FD passing │ DOM events → input
|
||
▼ │ WebSocket API
|
||
┌──────────────────────────────────────────┐ │
|
||
│ SessionProcess (fork'd Node child) │ │
|
||
│ ┌────────────────────────────────────┐ │ │
|
||
│ │ wayland-server-addon.node │ │ │
|
||
│ │ (libwayland-server N-API) │ │ │
|
||
│ │ - wl_display + wl_client │ │ │
|
||
│ │ - epoll via proxy-poll-addon │ │ │
|
||
│ └─────────────┬──────────────────────┘ │ │
|
||
│ │ wire msg + FDs │ │
|
||
│ ▼ │ │
|
||
│ ┌─────────────────────────────────────┐ │ │
|
||
│ │ NativeWaylandClientSession │ │ │
|
||
│ │ - interceptRequest / interceptEvent│ │ │
|
||
│ │ - onWireMessageEnd: serialize │ │ │
|
||
│ │ FDs as JSON ProxyFDs inline │ │ │
|
||
│ │ - fastSync optimization │ │ │
|
||
│ └─────────────┬───────────────────────┘ │ │
|
||
│ │ outbound buffer │ │
|
||
│ ▼ │ │
|
||
│ ┌─────────────────────────────────────┐ │ │
|
||
│ │ ARQChannel (KCP-wrapped) │ │ │
|
||
│ │ MTU=64000, snd_wnd=128, rcv_wnd=1024│ │ │
|
||
│ │ KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data │
|
||
│ │ kcp.update() every 100ms │ │ │
|
||
│ │ - PROTOCOL (per wl_client) │ │ │
|
||
│ │ - FRAME (per wl_surface) │ │ │
|
||
│ │ - XWM (per XWayland) │ │ │
|
||
│ └─────────────┬───────────────────────┘ │ │
|
||
│ ┌─────────────┴───────────────────────┐ │ │
|
||
│ │ SimpleChannel (FEEDBACK, no KCP) │ │ │
|
||
│ │ 4 bytes: refreshIntervalMs + │ │ │
|
||
│ │ avgDecodeDurationMs │ │ │
|
||
│ └─────────────┬───────────────────────┘ │ │
|
||
│ ┌─────────────▼───────────────────────┐ │ │
|
||
│ │ NativeAppContext │ │ │
|
||
│ │ - signalingWebSocket (JSON msgs) │ │ │
|
||
│ │ - kill timers: SIGHUP@10min, │ │ │
|
||
│ │ SIGKILL@+10s │ │ │
|
||
│ └─────────────┬───────────────────────┘ │ │
|
||
│ ┌─────────────▼───────────────────────┐ │ │
|
||
│ │ SessionController │ │ │
|
||
│ │ ws.WebSocketServer (noServer) │ │ │
|
||
│ │ perMessageDeflate: false │ │ │
|
||
│ │ Endpoints: /signal /channel /mkfifo│ │ │
|
||
│ │ /mkstemp-mmap /read-fd /write-fd-as-stream│ │
|
||
│ │ /read-fd-as-stream /close-fd /request-keyframe│ │
|
||
│ └─────────────┬───────────────────────┘ │ │
|
||
└────────────────┼─────────────────────────────┘ │
|
||
│ │
|
||
│ ┌──────────────────────────┐ │
|
||
│ │ main.ts (HTTP+WS hub) │ │
|
||
│ │ - socket.pause() + IPC │ │
|
||
│ │ send to session proc │ │
|
||
│ │ - noDelay:true (TCP) │ │
|
||
│ └────────────┬────────────┘ │
|
||
│ │ │
|
||
│ ◄──── 1 WebSocket per channel ────► │
|
||
│ ◄── 1 signaling WS per app ──► │
|
||
│ ◄── ad-hoc WebFD streams ─► │
|
||
│ │ │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────┐ │
|
||
│ │ ReconnectingWebSocket │ │
|
||
│ │ min 1-5s, factor 1.3, max 10s │ │
|
||
│ │ minUptime 5s, connTimeout 4s │ │
|
||
│ └────────────┬───────────────────────────┘ │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────────┐ │
|
||
│ │ ARQChannel / SimpleChannel (mirror image) │ │
|
||
│ │ snd_wnd=1024, rcv_wnd=128 │ │
|
||
│ │ kcp.input → peekSize/recv → onMessage │ │
|
||
│ └────────────┬───────────────────────────────┘ │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────────┐ │
|
||
│ │ RemoteAppLauncher.onChannel │ │
|
||
│ │ PROTOCOL → session.display.createClient │ │
|
||
│ │ FRAME → deliverContentToBufferStream │ │
|
||
│ │ XWM → XWindowManagerConnection │ │
|
||
│ │ FEEDBACK → ClientEncodersFeedback │ │
|
||
│ └────────────┬───────────────────────────────┘ │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────────┐ │
|
||
│ │ @gfld/compositor-protocol Client │ │
|
||
│ │ - parses Wayland wire format │ │
|
||
│ │ - dispatches to wl_*_resource impls │ │
|
||
│ │ - H264NALDecoder.worker.ts decodes frames │ │
|
||
│ └────────────┬───────────────────────────────┘ │
|
||
│ ▼ │
|
||
│ ┌────────────────────────────────────────────┐ │
|
||
│ │ Surface + Scene rendering │ │
|
||
│ │ - ImageBitmap / VideoFrame / Canvas │ │
|
||
│ │ - DOM input → wl_keyboard/pointer reqs │ │
|
||
│ └────────────┬───────────────────────────────┘ │
|
||
│ │ input events flow back via the │
|
||
│ │ PROTOCOL channel as ordinary │
|
||
│ │ Wayland requests (same framing) │
|
||
└──────────────────┘ │
|
||
Browser DOM ◄─┘
|
||
```
|
||
|
||
## Key takeaways for building your own transport
|
||
|
||
1. **Separate control plane from data plane** — Greenfield uses `/signal`
|
||
(JSON) vs `/channel` (binary). The control plane handles app launch,
|
||
termination, kill timers; the data plane handles Wayland wire + encoded
|
||
frames.
|
||
2. **One WS per logical stream** — simpler, no HoL blocking, but many TCP
|
||
connections.
|
||
3. **KCP-over-WS** trades double-reliability overhead for fast (30 ms)
|
||
retransmits; reasonable for real-time video.
|
||
4. **Virtualize FDs as JSON tokens** with lazy separate-WS endpoints for
|
||
actual byte transfer.
|
||
5. **`bufferedAmount` check ≠ backpressure** — Greenfield's check is a drop
|
||
gate, not real flow control (acknowledged TODO).
|
||
6. **Distinguish transient vs permanent close** — codes 4000-4999 tear
|
||
down; lower codes auto-reconnect.
|
||
7. **Piggyback out-of-band opcodes** on the data channel via a leading u32
|
||
to avoid extra control WS.
|
||
8. **Asymmetric KCP windows** match each direction's bandwidth profile.
|
||
9. **Decode feedback drives frame pacing** — the proxy synthesizes
|
||
`wl_frame.done` callbacks at the browser's reported cadence.
|
||
10. **One Node child per browser session** for fault isolation and
|
||
per-session `wl_display` ownership.
|
||
|
||
## File reference
|
||
|
||
### Native side
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `packages/compositor-proxy/src/Channel.ts` | `ARQChannel` + `SimpleChannel` |
|
||
| `packages/compositor-proxy/src/kcp.ts` | ~1100-line KCP implementation |
|
||
| `packages/compositor-proxy/src/Session.ts` | Top-level proxy session |
|
||
| `packages/compositor-proxy/src/SessionController.ts` | WS server with 9 endpoint paths |
|
||
| `packages/compositor-proxy/src/NativeAppContext.ts` | Per-app state, signaling WS, kill timers |
|
||
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||
| `packages/compositor-proxy/src/NativeWaylandCompositorSession.ts` | wl_display creation, initShm, initDrm |
|
||
| `packages/compositor-proxy/src/FrameFeedback.ts` | Receives 4-byte decode feedback, paces wl_frame.done |
|
||
| `packages/compositor-proxy/src/ProxyBuffer.ts` | Proxy-side buffer representation |
|
||
| `packages/compositor-proxy/src/SurfaceBufferEncoding.ts` | Encoding pipeline orchestration |
|
||
| `packages/compositor-proxy/src/XWaylandSession.ts` | XWayland spawn + bridge |
|
||
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||
| `packages/compositor-proxy/src/io/` | WebSocket server glue |
|
||
| `packages/compositor-proxy/src/encoding/` | High-level encoding orchestration |
|
||
| `packages/compositor-proxy/src/protocol/` | Generated interceptor modules |
|
||
| `packages/compositor-proxy/src/addons/` | .node file shims (Linux-x64) |
|
||
|
||
### Browser side
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `packages/compositor/src/remote/Channel.ts` | `ARQChannel` + `SimpleChannel` (mirror) |
|
||
| `packages/compositor/src/remote/kcp.ts` | KCP mirror |
|
||
| `packages/compositor/src/remote/RemoteAppLauncher.ts` | Channel setup per app, frame delivery |
|
||
| `packages/compositor/src/remote/RemoteOutOfBandChannel.ts` | OOB opcode dispatch |
|
||
| `packages/compositor/src/remote/RemoteInputOutput.ts` | FD-over-WebSocket for native apps |
|
||
| `packages/compositor/src/remote/EncoderFeedback.ts` | Sends 4-byte decode feedback at 1 Hz |
|
||
| `packages/compositor/src/remote/StreamingBuffer.ts` | Per-surface encoded-frame buffer |
|
||
| `packages/compositor/src/remote/BufferStream.ts` | Manages serial → frame promise resolution |
|
||
| `packages/compositor/src/remote/EncodedFrame.ts` | Wire format parser for encoded frames |
|
||
| `packages/compositor/src/remote/webcodec-buffer-decoder.ts` | WebCodecs `VideoDecoder` path |
|
||
| `packages/compositor/src/remote/wasm-buffer-decoder.ts` | FFmpeg Wasm fallback path |
|
||
| `packages/compositor/src/remote/H264NALDecoder.worker.ts` | Worker entry, runs FFmpeg Wasm decoder |
|
||
| `packages/compositor/src/remote/api.ts` | Public exports |
|
||
|
||
### CLI
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `packages/compositor-proxy-cli/src/main.ts` | HTTP + WS hub, forks per session |
|
||
| `packages/compositor-proxy-cli/src/main-controller.ts` | CLI arg parsing |
|
||
| `packages/compositor-proxy-cli/src/main-args.ts` | CLI arg definitions |
|
||
| `packages/compositor-proxy-cli/src/SessionProcess.ts` | Per-session child, IPC handler |
|
||
| `packages/compositor-proxy-cli/example-applications.json` | Pre-configured apps |
|
||
| `packages/compositor-proxy-cli/wait-until-ready.sh` | Startup readiness check |
|