Initial commit: Greenfield architectural analysis (14 files, ~80 pages, 10 parallel research agents)

This commit is contained in:
1b-agent committed 2026-10-08 08:48:25 +00:00
commit c4b45cf218
15 files changed
+5444

No files matched your search

+366
View File
@@ -0,0 +1,366 @@
# 05 — Input Handling (Keyboard, Pointer, Touch, Focus)
This chapter covers Greenfield's input handling: keyboard (with xkbcommon in
browser), pointer, touch, focus management, and how input events travel from
the browser to native Linux apps via the proxy.
## Headline
Greenfield is a **browser-native Wayland compositor**. All seat logic —
keymap compilation, modifier state, pointer grabs, button/axis dispatch —
runs **in the browser**, against a **WASM build of libxkbcommon 1.5.0 +
pixman**. The `compositor-proxy` is *not* a compositor — it is a thin
libwayland-server bridge that relays Wayland wire messages between the
browser and native clients and provides file-descriptor shims.
There is no second xkbcommon on the proxy: the keymap is generated in the
browser from RMLVO and shipped to the proxy as a real native shm fd via a
custom `/mkstemp-mmap` WebSocket endpoint. `wl_touch` is advertised but
**never populated with events** — touch input is collapsed into Pointer
Events.
## The Seat abstraction
`Seat.ts` (in `packages/compositor/src/`) is the `wl_seat` global. It
aggregates keyboard, pointer, and touch, manages focus, and runs the input
queue. Each `Seat` has:
- `Keyboard` (always present)
- `Pointer` (always present)
- `Touch` (advertised only if `'ontouchstart' in document.documentElement`,
but inert — see §Touch below)
- `InputQueue` — coalesces events and dispatches them on the next RAF
- `InputOutput` — abstracts FD operations (browser pipes for in-browser
apps, WebSocket endpoints for native apps via proxy)
Each seat also tracks:
- The currently focused `Surface` for keyboard input
- The currently focused `Surface` for pointer input (transient)
- The active pointer grab (default, popup, drag, or resize)
- The active keyboard grab (default or popup)
## Keyboard — xkbcommon in the browser
### WASM xkbcommon build
`libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`,
applies a patch (removes tests, shims `strdup`), builds with emscripten
3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`,
exports the keymap/state C API as a single-file ES module.
The same build approach is used for pixman (for region math).
### Keymap compilation flow
The browser, not the proxy, is the source of truth for the keymap. The
flow is:
1. Browser receives a keymap config (either default or user-supplied via
`userShell.actions.setUserConfiguration({ keyboardLayoutName })`).
2. `Seat.notifyUpdateKeymap(nrmlvo)` is called.
3. `xkb_context_new` + `xkb_keymap_new_from_names` (WASM calls) compile the
RMLVO into a keymap.
4. `xkb_keymap_get_as_string` serializes the keymap as a long string.
5. `Keyboard.sendKeymap(resource)` is called for each `wl_keyboard`
resource:
- `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a
`/mkstemp-mmap` WebSocket, sends the blob, and receives a `ProxyFD {
handle: <native_fd>, type: 'shm', host: baseURL }`.
- `keyboardResource.keymap(xkbV1, ProxyFD, size)` is sent to the
native client.
6. The proxy deserializes the ProxyFD, looks up the native fd via
`proxyFDtoNativeFD` (identity when `host === baseURL`), ships it via
`sendEvents(wlClient, msg, fds)` from the N-API addon —
`libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain
socket.
7. The native client's libwayland receives the keymap fd, mmaps it, and
calls native `xkb_keymap_new_from_string` to compile its own copy.
So the keymap flows **browser → proxy → native**, not the reverse. This
is the opposite of what you might expect (since the proxy is on the Linux
host where xkbcommon natively lives), but it makes sense: the browser is
the actual compositor and is the one that needs the keymap to translate
DOM key events to Wayland keycodes.
### DOM → Wayland key event translation
Browser DOM `KeyboardEvent` has `event.code` (e.g. `"KeyA"`) and
`event.key` (the resolved glyph). Greenfield uses **only the physical
`.code`**, mirroring evdev:
1. `event.code` is looked up in `EvDevKeyCode` reverse-name-map → evdev
keycode (e.g. `30` for `KeyA`).
2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset).
3. `capsLock` / `numLock` are read from `event.getModifierState(...)`.
The DOM `.key` glyph is **ignored** — only physical `.code` is used, so
the same physical key always produces the same evdev keycode regardless
of layout. This is correct Wayland behavior.
### Modifier tracking
Modifier state is tracked entirely via the WASM libxkbcommon state
machine:
1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or
`xkb.keyUp(x11)`.
2. Internally, `_xkb_state_update_key` is invoked.
3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` /
`modsLocked` / `group` (each a `_xkb_state_serialize_mods` call).
4. High-level `KeyboardModifier` bits are computed by checking xkb
mod-indices resolved at keymap-compile time.
5. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)`
sends the `wl_keyboard.modifiers` event to the native client.
### Notable bug
`Seat.ts` lines 511-513 contain a copy/paste typo:
`modsLatched = modsLocked` (should be `= modsLatched`). The cached
`keyboard.modifiers` struct is unreliable but doesn't affect protocol
output because `notifyModifiers` reads from `xkb` directly each time.
## Pointer
### DOM → Wayland pointer events
Browser DOM `MouseEvent` is converted to Greenfield's internal
`ButtonEvent` / `AxisEvent`:
- `pointermove` → `createButtonEventFromMouseEvent` → `ButtonEvent`
(motion-only)
- `pointerdown` / `pointerup` → `ButtonEvent` (with button + state)
- `wheel` → `createAxisEventFromWheelEvent` → `AxisEvent`
### Pointer enter / leave between surfaces
`DefaultPointerGrab.focus()` runs on every motion event:
1. `Renderer.pickView` walks the view stack front-to-back.
2. Returns the first view whose `Surface.isWithinInputRegion(point)`
(pixman-WASM region) is true.
3. On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to
the old surface, then `wl_pointer.enter` + `frame` (+ modifiers if
keyboard is grabbed) to the new surface.
### Pointer capture / window leave
- `canvas.setPointerCapture` is called on `pointerdown`.
- `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus,
releases depressed keys, sets `needFocusInit=true` for re-sync),
resets `buttonCount=0`.
- DnD suppresses this handler mid-drag.
### Pointer acceleration / cursor images
**No acceleration** — verbatim `offsetX/Y`.
Cursor images are **CSS `cursor: url(data:...) x y, auto`**:
1. Native client calls `wl_pointer.set_cursor`.
2. The cursor surface commits a buffer.
3. `CursorRole` draws the bitmap to a hidden canvas.
4. `toDataURL()` produces a data URL.
5. Set on `document.body.style.cursor`.
OS composites the cursor. No rotation/scale transforms.
### Scroll wheel / touchpad
`WheelEvent.deltaMode`:
- `DOM_DELTA_LINE × 12` — discrete wheel clicks
- `DOM_DELTA_PAGE × surface size` — page-sized scrolls
- `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad
Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version
≥5 also gets `axis_source(wheel)` + `frame()` (groups events).
No `axis_discrete` / `value120` / `axis_stop` events are sent.
### Linux button codes
DOM `event.button` (0=left, 1=middle, 2=right, 3=back, 4=forward) is
mapped to Linux input protocol button codes via the `linuxInput` table
in `browser/input.ts`.
## Touch — declared but inert
`Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners.
`wl_touch` capability is advertised if `'ontouchstart' in
document.documentElement`, but no events are ever sent. Touch input flows
through Pointer Events as mouse input.
**For the user**: if you need real touch support (e.g. for tablets), this
is a feature you'll have to implement from scratch. The protocol stubs
exist (via generated `wl_touch` bindings), but no event source populates
them.
## Focus management
### DOM focus
The `<canvas>` owns keyboard focus permanently (`tabIndex=1`).
`Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between
native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface`
activation handlers.
### DOM → Wayland focus
- Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores
saved focus, re-syncs modifiers).
- Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves
focus, releases depressed keys, sets `needFocusInit=true`).
### Surface-level focus
- Click on a surface → `XdgToplevel` activation request →
`desktopSurface.activate()` → `gainFocus()` (refcounted via
`focusCount`, fires `surfaceActivationUpdated` event to the user shell
on 0↔1 transitions).
- Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`,
validating the serial against the keyboard/pointer `grabSerial`.
## Input queue
`InputQueue.ts` coalesces events and dispatches them on the next RAF
(`InputQueue.drainQueues`). This batches bursts of input events (e.g. fast
mouse motion) into a single dispatch, avoiding redundant work.
The queue is FIFO per event type (motion, axis, key, button). Each
dispatch calls the appropriate `seat.notify*` method.
## ASCII event-flow diagram
```
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER (DOM) │
│ │
│ <canvas tabIndex=1> ─┐ │
│ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │
│ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │
│ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │
│ │ • evdevKeyCode = EvDevKeyCode[event.code] │
│ │ • x11KeyCode = evdevKeyCode + 8 │
│ │ • capsLock/numLock = event.getModifierState(...) │
│ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │
│ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ session.inputQueue.queue{Motion,Axis,Key,Button}
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │
│ │
│ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame │
│ │ │
│ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │
│ │ └─ Keyboard.sendKey(event) │
│ │ └─ keyboardResource.key(serial,time,evdev,st) │
│ │ │
│ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │
│ │ └─ Pointer.sendButton │
│ │ └─ pointerResource.button(serial,time, │
│ │ linuxInput[btn], state) │
│ │ │
│ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │
│ │ └─ pointerResource.motion(time, sx_f, sy_f) │
│ │ │
│ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │
│ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │
│ │ └─ pointerResource.axis(time, axis, Fixed) │
│ │ + axisSource(wheel) + frame() if v>=5 │
│ │ │
│ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │
│ │ │
│ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │
│ └─ Keyboard.sendModifiers(modsDepressed,Latched, │
│ Locked,group) │
│ │
│ Keymap path (once per WlKeyboardResource): │
│ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │
│ → xkb_context_new + xkb_keymap_new_from_names (libxkbcommon WASM) │
│ → xkb_keymap_get_as_string ──► keymapString │
│ Keyboard.sendKeymap(resource): │
│ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │
│ keyboardResource.keymap(xkbV1, ProxyFD, size) │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ client.connection.onFlush(wireMessages)
│ ─ one big Uint32Array, FDs serialized as ProxyFD JSON
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ WIRE (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts) │
│ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ ws on proxy :SessionController │
│ Message layout (Uint32Array): │
│ [0] outOfBandOpcode (0 = in-band) │
│ [1] eventSerial │
│ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │
│ then concatenated Wayland wire messages │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ NativeWaylandClientSession.onMessage
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ PROXY (Node.js, compositor-proxy) │
│ │
│ SessionController wss paths: /mkfifo /mkstemp-mmap /read-fd /close-fd │
│ /read-fd-as-stream /write-fd-as-stream │
│ /signal /channel /request-keyframe │
│ │
│ mkstempMmap(ws): │
│ receives Blob → Buffer.concat → createMemoryMappedFile(buf) ← native addon │
│ returns ProxyFD { handle: <native fd>, type: 'shm', host: baseURL } │
│ │
│ onWireMessageEvents: │
│ for each ProxyFD in message: │
│ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) // just returns .handle │
│ └─ if proxyFD.host === baseURL → identity │
│ └─ if foreign host → spawn a bridging pipe + WebSocket │
│ messageInterceptor.interceptEvent(...) // empty stubs for kbd/ptr/touch │
│ sendEvents(wlClient, messageBuffer, fdsBuffer) ← libwayland-server addon │
│ flush(wlClient) │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
│ libwayland-server sends wire msg + SCM_RIGHTS FDs
▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ NATIVE (Wayland client e.g. weston-terminal, Xwayland app) │
│ wl_display.fd readable → libwayland-client demarshals │
│ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │
│ wl_keyboard.key / enter / leave / modifiers → client app │
│ wl_pointer.enter / motion / button / axis / frame → client app │
└──────────────────────────────────────────────────────────────────────────────────────┘
```
## Key answers (quick reference)
| Question | Answer |
|---|---|
| DOM→Wayland key event translation | `event.code` (e.g. `"KeyA"`) is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30`). `x11KeyCode = evdevKeyCode + 8`. DOM `.key` glyph is **ignored** — only physical `.code` is used, mirroring evdev. |
| Modifier tracking | Entirely via WASM libxkbcommon state machine. `Seat.updateModifierState` calls `xkb.keyDown/Up(x11)` → `_xkb_state_update_key`, then `notifyModifiers` reads `xkb.modsDepressed/Latched/Locked/group` (each a `_xkb_state_serialize_mods` call). High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. |
| Focus: DOM ↔ native surfaces | The `<canvas>` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave/enter` between native surfaces. Activation is driven by `XdgToplevel`/`ShellSurface` activation handlers. |
| Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` calls `notifyKeyboardFocusOut` (saves kbd focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. DnD suppresses this handler mid-drag. |
| Touch event mapping | **Not implemented.** `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. |
| Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView` walks the view stack front-to-back, returns first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. On change, `Pointer.setFocus` emits `leave+frame` to old surface, `enter+frame` (+modifiers) to new. |
| Pointer acceleration / cursor images | **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: client calls `wl_pointer.set_cursor` → `CursorRole` → draw bitmap to hidden canvas → `toDataURL()` → set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. |
| Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: `DOM_DELTA_LINE × 12`, `DOM_DELTA_PAGE × surface size`, `DOM_DELTA_PIXEL ÷ 3`. Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. v≥5 also gets `axis_source(wheel)` + `frame()`. No `axis_discrete/value120/stop`. |
| WASM xkbcommon? | Yes. `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch, builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. Sibling build does the same for `pixman`. |
| Keymap FD workaround for WebSocket | Direction is **browser → proxy**, not the reverse. Browser calls `inputOutput.mkstempMmap(Blob)` → opens `/mkstemp-mmap` WebSocket → proxy creates real native shm fd via `createMemoryMappedFile` addon → returns `ProxyFD {handle: <native_fd>, type: 'shm', host: baseURL}`. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`; the ProxyFD is **serialized as JSON** in the wire message's `fds[]` array (WebSockets can't pass FDs natively). Proxy deserializes, maps `handle` → native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. |
## File reference
| File | Role |
|---|---|
| `packages/compositor/src/Seat.ts` | `wl_seat` global, multi-seat support, focus management |
| `packages/compositor/src/Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events |
| `packages/compositor/src/Pointer.ts` | `wl_pointer` impl, motion, buttons, axis |
| `packages/compositor/src/Touch.ts` | `wl_touch` (46-line stub, inert) |
| `packages/compositor/src/Xkb.ts` | xkbcommon WASM wrapper |
| `packages/compositor/src/KeyEvent.ts` | Key event wrapper |
| `packages/compositor/src/ButtonEvent.ts` | Pointer button event wrapper |
| `packages/compositor/src/AxisEvent.ts` | Pointer axis (scroll) event wrapper |
| `packages/compositor/src/InputQueue.ts` | Event coalescing + RAF dispatch |
| `packages/compositor/src/InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) |
| `packages/compositor/src/browser/input.ts` | DOM event listeners → Greenfield events |
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer |
| `libs/compositor-wasm/` | pixman + xkbcommon WASM build |
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
| `packages/compositor-proxy/src/io/` | WebSocket server glue (mkstemp-mmap endpoint etc.) |