1b-agent/one-os-dev-wayland-html5-compositor-greenfield-analysis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
one-os-dev-wayland-html5-co…/04-transport.md
T

24 KiB
Raw Permalink Blame History

04 — Transport Layer (WebSocket / KCP / Channel Layer)

This chapter covers Greenfield's transport: how Wayland wire traffic flows between the Linux-side proxy and the browser-side compositor over WebSocket, the binary framing, message ordering, latency/backpressure, KCP usage, multiplexing, and connection lifecycle.

Architecture at a glance

Greenfield uses a multi-WebSocket, KCP-wrapped, asymmetric flow-control design:

  • Two-tier process model: A main.ts HTTP hub forks one SessionProcess per browser session (per compositorSessionId). Raw net.Sockets are handed to children via Node IPC sendHandle. Each session child owns its own wl_display (via wayland-server-addon.node).
  • 9 WebSocket endpoints per session: /signal (JSON control), /channel (binary data channels), /mkfifo, /mkstemp-mmap, /read-fd, /read-fd-as-stream, /write-fd-as-stream, /close-fd, /request-keyframe. All with perMessageDeflate: false, TCP noDelay: true.

Channel types

There are 4 logical channel types per app client, each implemented as a separate WebSocket:

Type Class KCP? Purpose
PROTOCOL ARQChannel Yes Wayland wire protocol, per wl_client
FRAME ARQChannel Yes H.264/PNG encoded surface content, per wl_surface
XWM ARQChannel Yes XWayland window manager protocol
FEEDBACK SimpleChannel No 4-byte telemetry (refreshInterval + decode avg), 1 Hz

There is no multiplexing of multiple Wayland clients onto one WS. Each logical channel is its own WS, eliminating head-of-line blocking between video, input, and control at the cost of many connections per app (10+ for complex apps).

KCP-over-WebSocket

kcp.ts is a ~1100-line TypeScript port of the canonical C ikcp.c. Greenfield runs it on top of WebSocket (not real UDP) for:

  • 30 ms min RTO vs TCP's 200 ms.
  • 100 ms update tick (setInterval(kcp.update, 100)).
  • Per-channel congestion domains independent of TCP.
  • setNoDelay(1, 100, 0, 1) — nodelay enabled, congestion control disabled (nc=1).
  • 24-byte segment header: conv|cmd|frg|wnd|ts|sn|una|len.
  • MTU=64000 (commented "webrtc datachannel MTU" — legacy of a prior WebRTC design).

Asymmetric windows (deliberate)

  • Proxy side: snd_wnd=128, rcv_wnd=1024
  • Browser side: snd_wnd=1024, rcv_wnd=128

So proxy→browser (heavy video) has a smaller in-flight cap to bound latency; browser→proxy (bursty input) has a large window to avoid stalls.

Why KCP-over-WebSocket?

Because real-time video over plain TCP suffers from head-of-line blocking and slow retransmit. KCP gives Greenfield a UDP-like congestion behavior in user-space, while still riding on WebSocket (the only bidirectional binary transport universally available in browsers today).

The long-term plan, documented in docs/pages/design/index.md, is to switch to WebTransport (real UDP) when browser support is widespread, and pair it with KCP for reliability. The KCP code is already in place; only the WebSocket transport layer needs to be swapped out.

Binary message format

All multi-byte fields are little-endian.

PROTOCOL channel

u32 outOfBandOpcode  (0 = in-band events, else 1=ResourceDestroyed,
                       2=BufferCreation, 6=RecycledResourceIds, 7=CommitSerial)
u32 eventSerial
u32 fdsCount
  for each fd: u32 jsonLen + u8[] JSON{handle,type,host} (4-byte aligned)
concatenated Wayland wire messages (object_id + opcode/size + args)

FRAME channel (per H.264 frame)

u32 bufferId
u32 bufferCreationSerial
u32 bufferContentSerial
encodedFrame:
  u32 contentSerial, u32 encodingType (0=h264/1=png),
  u32 width, u32 height, u32 encodedWidth, u32 encodedHeight,
  u32 opaqueLength, u8[] opaqueData,
  u32 alphaLength, u8[] alphaData   ← alpha encoded as a SEPARATE H.264 stream

FEEDBACK channel

Just 4 bytes: u16 refreshIntervalMs + u16 avgDecodeDurationMs. Sent at 1 Hz from browser to proxy. Used by the proxy to pace wl_frame.done callbacks so the native app sees a virtual monitor refresh rate matching the browser's actual RAF cadence.

File descriptor handling

Can't send FDs over WebSocket — Greenfield virtualizes them:

  • ProxyFD = {handle:number, type:'pipe-read'|'pipe-write'|'shm'|'unknown', host:string}.
  • Serialized as JSON inline in the message stream with a u32 length prefix.
  • Browser opens separate ad-hoc WebSocket to /read-fd, /write-fd-as-stream, etc., only when it actually needs the bytes.
  • Most wl_buffer FDs never need to be read because the proxy intercepts wl_surface.commit, reads the buffer locally via DRM/SHM, encodes to H.264, and ships only the encoded bytes.
  • Foreign ProxyFD (proxyFD.host !== local baseURL) triggers proxy-to-proxy relay: local proxy creates a native pipe, opens a WS to the foreign proxy's /write-fd-as-stream, and pipes the data through. This is how copy-paste between two proxy instances avoids a browser round-trip.

Reconnection & lifecycle

Browser side

Vendored ReconnectingWebSocket (pladaria) with:

  • minDelay=1-5s, factor=1.3, maxDelay=10s
  • minUptime=5s, connTimeout=4s
  • retries=∞

Channel state preservation: only close codes 4000–4999 (application-level) tear down KCP. Transient codes (1006, 1011) are absorbed — KCP keeps ticking and the WS reconnects underneath.

Proxy side

No auto-reconnect (it's the server). The Channel object lives on, waiting for a new WS upgrade with matching id + key. Outbound messages queue in outboundMessages and flush on onOpen.

App kill timers

When the browser disconnects from the /signal WebSocket:

  • 10-minute SIGHUP grace period — gives the user time to reconnect (e.g. if they refresh the browser tab).
  • After 10 minutes of no reconnect, escalating to SIGKILL after another 10s.
  • Close code 1001 → immediate SIGHUP (user closed tab intentionally).
  • Close code 4001 → immediate SIGTERM (user clicked "force quit" in the shell).

Session close

When all wl_clients disconnect, 1-second grace then session.close() → process.exit() in the session child. Main process deletes the entry from its sessionProcesses map.

Process model — three layers

Layer 1: CLI Front Door (compositor-proxy-cli/src/main.ts)

A single Node.js process listens on :8081. Per browser session, it fork()s one SessionProcess child — one child per compositorSessionId, not per app. Multiple apps in the same browser tab share one child. WebSocket upgrades are handed off to the matching child via childProcess.send(wsUpgrade, socket).

Layer 2: Session Process (SessionProcess.ts)

The forked child. Receives start (creates Session + SessionController), launchApp (spawns the actual app binary), and wsUpgrade (hands WS to the controller) IPC messages. On launchApp it calls launchApplication() from @gfld/compositor-proxy, which spawn()s the executable, captures pid, replies to main with {pid, key}. On session.close() the process exits.

Layer 3: Proxy Core (compositor-proxy/src/)

  • NativeWaylandCompositorSession — creates a real wl_display via createDisplay(), calls addSocketAuto() which creates the UNIX socket at $XDG_RUNTIME_DIR/wayland-N and returns the name wayland-N (this becomes WAYLAND_DISPLAY). Calls initShm() (adds YUV420/NV12/YUV444/ NV21 formats) and initDrm() (opens /dev/dri/renderD128 via EGL/GBM, registers zwp_linux_dmabuf_v1 + wl_drm globals). Integrates the wl_display FD into Node's libuv loop via startPoll().
  • NativeWaylandClientSession — per-client wire-message interceptor. Uses MessageInterceptor + auto-generated per-resource interceptor modules (src/protocol/wl_*_interceptor.js). Each request returns {native, browser, neverReplies?} routing decision. Handles wl_display.sync fast-path (skips browser round-trip when no reply expected). Marshalls FDs as ProxyFD JSON.
  • NativeAppContext — per-app state: pid, key (16-hex random signaling auth), signaling WS, client sessions, data channels. Lifecycle: 10-min SIGHUP grace on browser disconnect, SIGTERM→SIGKILL escalation (10 s) on force-quit, auto-destroy on app exit.
  • SessionController — WS demultiplexer by pathname: /signal (per-app control), /channel (multiplexed data channels), /mkfifo, /mkstemp-mmap, /read-fd, /write-fd-as-stream, /request-keyframe.
  • XWaylandSession — lazy Xwayland fork (only on first X client connect). Xwayland itself becomes a Wayland client of the proxy. The wm_fd is bridged via xtsb to the browser, which implements the actual XWM logic.
  • Channel — ARQChannel (KCP-over-WebSocket, reliable, low-latency) for PROTOCOL/FRAME/XWM; SimpleChannel for FEEDBACK.

ASCII diagram of the full transport stack

       LINUX SIDE                                   |       BROWSER SIDE
                                                   |
┌─────────────────────────┐                        |   ┌──────────────────────────────┐
│  gtk4-demo (process)    │                        |   │   Browser tab                │
│  libwayland-client.so   │                        |   │   (Greenfield compositor)    │
└───────────┬─────────────┘                        |   └──────────────┬───────────────┘
            │ WAYLAND_DISPLAY=wayland-0                             │
            │ Unix socket + SCM_RIGHTS FD passing                   │ DOM events → input
            ▼                                                      │ WebSocket API
┌──────────────────────────────────────────┐                       │
│  SessionProcess (fork'd Node child)      │                       │
│  ┌────────────────────────────────────┐   │                       │
│  │ wayland-server-addon.node          │   │                       │
│  │  (libwayland-server N-API)         │   │                       │
│  │  - wl_display + wl_client          │   │                       │
│  │  - epoll via proxy-poll-addon      │   │                       │
│  └─────────────┬──────────────────────┘   │                       │
│                │ wire msg + FDs            │                       │
│                ▼                           │                       │
│  ┌─────────────────────────────────────┐  │                       │
│  │ NativeWaylandClientSession          │  │                       │
│  │  - interceptRequest / interceptEvent│  │                       │
│  │  - onWireMessageEnd: serialize      │  │                       │
│  │    FDs as JSON ProxyFDs inline      │  │                       │
│  │  - fastSync optimization            │  │                       │
│  └─────────────┬───────────────────────┘  │                       │
│                │ outbound buffer            │                       │
│                ▼                            │                       │
│  ┌─────────────────────────────────────┐  │                       │
│  │ ARQChannel (KCP-wrapped)            │  │                       │
│  │  MTU=64000, snd_wnd=128, rcv_wnd=1024│ │                       │
│  │  KCP seg: conv|cmd|frg|wnd|ts|sn|una|len (24B) + data          │
│  │  kcp.update() every 100ms           │  │                       │
│  │  - PROTOCOL (per wl_client)         │  │                       │
│  │  - FRAME     (per wl_surface)       │  │                       │
│  │  - XWM       (per XWayland)        │  │                       │
│  └─────────────┬───────────────────────┘  │                       │
│  ┌─────────────┴───────────────────────┐  │                       │
│  │ SimpleChannel (FEEDBACK, no KCP)    │  │                       │
│  │  4 bytes: refreshIntervalMs +       │  │                       │
│  │          avgDecodeDurationMs        │  │                       │
│  └─────────────┬───────────────────────┘  │                       │
│  ┌─────────────▼───────────────────────┐  │                       │
│  │ NativeAppContext                    │  │                       │
│  │  - signalingWebSocket (JSON msgs)   │  │                       │
│  │  - kill timers: SIGHUP@10min,       │  │                       │
│  │    SIGKILL@+10s                     │  │                       │
│  └─────────────┬───────────────────────┘  │                       │
│  ┌─────────────▼───────────────────────┐  │                       │
│  │ SessionController                   │  │                       │
│  │  ws.WebSocketServer (noServer)      │  │                       │
│  │  perMessageDeflate: false           │  │                       │
│  │  Endpoints: /signal /channel /mkfifo│  │                       │
│  │  /mkstemp-mmap /read-fd /write-fd-as-stream│                  │
│  │  /read-fd-as-stream /close-fd /request-keyframe│              │
│  └─────────────┬───────────────────────┘  │                       │
└────────────────┼─────────────────────────────┘                       │
                 │                                                       │
                 │         ┌──────────────────────────┐                 │
                 │         │  main.ts (HTTP+WS hub)  │                 │
                 │         │  - socket.pause() + IPC │                 │
                 │         │    send to session proc │                 │
                 │         │  - noDelay:true (TCP)   │                 │
                 │         └────────────┬────────────┘                 │
                 │                      │                                │
                 │  ◄──── 1 WebSocket per channel ────►                  │
                 │     ◄── 1 signaling WS per app ──►                   │
                 │     ◄── ad-hoc WebFD streams    ─►                   │
                 │                      │                                │
                 │                      ▼                                │
                 │     ┌────────────────────────────────────────┐       │
                 │     │  ReconnectingWebSocket                 │       │
                 │     │   min 1-5s, factor 1.3, max 10s        │       │
                 │     │   minUptime 5s, connTimeout 4s         │       │
                 │     └────────────┬───────────────────────────┘       │
                 │                  ▼                                    │
                 │     ┌────────────────────────────────────────────┐   │
                 │     │ ARQChannel / SimpleChannel (mirror image) │   │
                 │     │   snd_wnd=1024, rcv_wnd=128                │   │
                 │     │   kcp.input → peekSize/recv → onMessage    │   │
                 │     └────────────┬───────────────────────────────┘   │
                 │                  ▼                                    │
                 │     ┌────────────────────────────────────────────┐   │
                 │     │ RemoteAppLauncher.onChannel                │   │
                 │     │   PROTOCOL  → session.display.createClient │   │
                 │     │   FRAME     → deliverContentToBufferStream │   │
                 │     │   XWM       → XWindowManagerConnection     │   │
                 │     │   FEEDBACK  → ClientEncodersFeedback       │   │
                 │     └────────────┬───────────────────────────────┘   │
                 │                  ▼                                    │
                 │     ┌────────────────────────────────────────────┐   │
                 │     │ @gfld/compositor-protocol Client           │   │
                 │     │  - parses Wayland wire format              │   │
                 │     │  - dispatches to wl_*_resource impls       │   │
                 │     │  - H264NALDecoder.worker.ts decodes frames │   │
                 │     └────────────┬───────────────────────────────┘   │
                 │                  ▼                                    │
                 │     ┌────────────────────────────────────────────┐   │
                 │     │ Surface + Scene rendering                  │   │
                 │     │  - ImageBitmap / VideoFrame / Canvas       │   │
                 │     │  - DOM input → wl_keyboard/pointer reqs    │   │
                 │     └────────────┬───────────────────────────────┘   │
                 │                  │ input events flow back via the      │
                 │                  │ PROTOCOL channel as ordinary        │
                 │                  │ Wayland requests (same framing)     │
                 └──────────────────┘                                    │
                                                          Browser DOM ◄─┘

Key takeaways for building your own transport

  1. Separate control plane from data plane — Greenfield uses /signal (JSON) vs /channel (binary). The control plane handles app launch, termination, kill timers; the data plane handles Wayland wire + encoded frames.
  2. One WS per logical stream — simpler, no HoL blocking, but many TCP connections.
  3. KCP-over-WS trades double-reliability overhead for fast (30 ms) retransmits; reasonable for real-time video.
  4. Virtualize FDs as JSON tokens with lazy separate-WS endpoints for actual byte transfer.
  5. bufferedAmount check ≠ backpressure — Greenfield's check is a drop gate, not real flow control (acknowledged TODO).
  6. Distinguish transient vs permanent close — codes 4000-4999 tear down; lower codes auto-reconnect.
  7. Piggyback out-of-band opcodes on the data channel via a leading u32 to avoid extra control WS.
  8. Asymmetric KCP windows match each direction's bandwidth profile.
  9. Decode feedback drives frame pacing — the proxy synthesizes wl_frame.done callbacks at the browser's reported cadence.
  10. One Node child per browser session for fault isolation and per-session wl_display ownership.

File reference

Native side

File Role
packages/compositor-proxy/src/Channel.ts ARQChannel + SimpleChannel
packages/compositor-proxy/src/kcp.ts ~1100-line KCP implementation
packages/compositor-proxy/src/Session.ts Top-level proxy session
packages/compositor-proxy/src/SessionController.ts WS server with 9 endpoint paths
packages/compositor-proxy/src/NativeAppContext.ts Per-app state, signaling WS, kill timers
packages/compositor-proxy/src/NativeWaylandClientSession.ts Per-client message interceptor
packages/compositor-proxy/src/NativeWaylandCompositorSession.ts wl_display creation, initShm, initDrm
packages/compositor-proxy/src/FrameFeedback.ts Receives 4-byte decode feedback, paces wl_frame.done
packages/compositor-proxy/src/ProxyBuffer.ts Proxy-side buffer representation
packages/compositor-proxy/src/SurfaceBufferEncoding.ts Encoding pipeline orchestration
packages/compositor-proxy/src/XWaylandSession.ts XWayland spawn + bridge
packages/compositor-proxy/src/wayland-server.ts TS wrapper around wayland-server-addon.node
packages/compositor-proxy/src/io/ WebSocket server glue
packages/compositor-proxy/src/encoding/ High-level encoding orchestration
packages/compositor-proxy/src/protocol/ Generated interceptor modules
packages/compositor-proxy/src/addons/ .node file shims (Linux-x64)

Browser side

File Role
packages/compositor/src/remote/Channel.ts ARQChannel + SimpleChannel (mirror)
packages/compositor/src/remote/kcp.ts KCP mirror
packages/compositor/src/remote/RemoteAppLauncher.ts Channel setup per app, frame delivery
packages/compositor/src/remote/RemoteOutOfBandChannel.ts OOB opcode dispatch
packages/compositor/src/remote/RemoteInputOutput.ts FD-over-WebSocket for native apps
packages/compositor/src/remote/EncoderFeedback.ts Sends 4-byte decode feedback at 1 Hz
packages/compositor/src/remote/StreamingBuffer.ts Per-surface encoded-frame buffer
packages/compositor/src/remote/BufferStream.ts Manages serial → frame promise resolution
packages/compositor/src/remote/EncodedFrame.ts Wire format parser for encoded frames
packages/compositor/src/remote/webcodec-buffer-decoder.ts WebCodecs VideoDecoder path
packages/compositor/src/remote/wasm-buffer-decoder.ts FFmpeg Wasm fallback path
packages/compositor/src/remote/H264NALDecoder.worker.ts Worker entry, runs FFmpeg Wasm decoder
packages/compositor/src/remote/api.ts Public exports

CLI

File Role
packages/compositor-proxy-cli/src/main.ts HTTP + WS hub, forks per session
packages/compositor-proxy-cli/src/main-controller.ts CLI arg parsing
packages/compositor-proxy-cli/src/main-args.ts CLI arg definitions
packages/compositor-proxy-cli/src/SessionProcess.ts Per-session child, IPC handler
packages/compositor-proxy-cli/example-applications.json Pre-configured apps
packages/compositor-proxy-cli/wait-until-ready.sh Startup readiness check
Reference in new issue View Git Blame Copy Permalink