367 lines
24 KiB
Markdown
367 lines
24 KiB
Markdown
# 05 — Input Handling (Keyboard, Pointer, Touch, Focus)
|
||
|
||
This chapter covers Greenfield's input handling: keyboard (with xkbcommon in
|
||
browser), pointer, touch, focus management, and how input events travel from
|
||
the browser to native Linux apps via the proxy.
|
||
|
||
## Headline
|
||
|
||
Greenfield is a **browser-native Wayland compositor**. All seat logic —
|
||
keymap compilation, modifier state, pointer grabs, button/axis dispatch —
|
||
runs **in the browser**, against a **WASM build of libxkbcommon 1.5.0 +
|
||
pixman**. The `compositor-proxy` is *not* a compositor — it is a thin
|
||
libwayland-server bridge that relays Wayland wire messages between the
|
||
browser and native clients and provides file-descriptor shims.
|
||
|
||
There is no second xkbcommon on the proxy: the keymap is generated in the
|
||
browser from RMLVO and shipped to the proxy as a real native shm fd via a
|
||
custom `/mkstemp-mmap` WebSocket endpoint. `wl_touch` is advertised but
|
||
**never populated with events** — touch input is collapsed into Pointer
|
||
Events.
|
||
|
||
## The Seat abstraction
|
||
|
||
`Seat.ts` (in `packages/compositor/src/`) is the `wl_seat` global. It
|
||
aggregates keyboard, pointer, and touch, manages focus, and runs the input
|
||
queue. Each `Seat` has:
|
||
|
||
- `Keyboard` (always present)
|
||
- `Pointer` (always present)
|
||
- `Touch` (advertised only if `'ontouchstart' in document.documentElement`,
|
||
but inert — see §Touch below)
|
||
- `InputQueue` — coalesces events and dispatches them on the next RAF
|
||
- `InputOutput` — abstracts FD operations (browser pipes for in-browser
|
||
apps, WebSocket endpoints for native apps via proxy)
|
||
|
||
Each seat also tracks:
|
||
|
||
- The currently focused `Surface` for keyboard input
|
||
- The currently focused `Surface` for pointer input (transient)
|
||
- The active pointer grab (default, popup, drag, or resize)
|
||
- The active keyboard grab (default or popup)
|
||
|
||
## Keyboard — xkbcommon in the browser
|
||
|
||
### WASM xkbcommon build
|
||
|
||
`libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`,
|
||
applies a patch (removes tests, shims `strdup`), builds with emscripten
|
||
3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`,
|
||
exports the keymap/state C API as a single-file ES module.
|
||
|
||
The same build approach is used for pixman (for region math).
|
||
|
||
### Keymap compilation flow
|
||
|
||
The browser, not the proxy, is the source of truth for the keymap. The
|
||
flow is:
|
||
|
||
1. Browser receives a keymap config (either default or user-supplied via
|
||
`userShell.actions.setUserConfiguration({ keyboardLayoutName })`).
|
||
2. `Seat.notifyUpdateKeymap(nrmlvo)` is called.
|
||
3. `xkb_context_new` + `xkb_keymap_new_from_names` (WASM calls) compile the
|
||
RMLVO into a keymap.
|
||
4. `xkb_keymap_get_as_string` serializes the keymap as a long string.
|
||
5. `Keyboard.sendKeymap(resource)` is called for each `wl_keyboard`
|
||
resource:
|
||
- `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a
|
||
`/mkstemp-mmap` WebSocket, sends the blob, and receives a `ProxyFD {
|
||
handle: <native_fd>, type: 'shm', host: baseURL }`.
|
||
- `keyboardResource.keymap(xkbV1, ProxyFD, size)` is sent to the
|
||
native client.
|
||
6. The proxy deserializes the ProxyFD, looks up the native fd via
|
||
`proxyFDtoNativeFD` (identity when `host === baseURL`), ships it via
|
||
`sendEvents(wlClient, msg, fds)` from the N-API addon —
|
||
`libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain
|
||
socket.
|
||
7. The native client's libwayland receives the keymap fd, mmaps it, and
|
||
calls native `xkb_keymap_new_from_string` to compile its own copy.
|
||
|
||
So the keymap flows **browser → proxy → native**, not the reverse. This
|
||
is the opposite of what you might expect (since the proxy is on the Linux
|
||
host where xkbcommon natively lives), but it makes sense: the browser is
|
||
the actual compositor and is the one that needs the keymap to translate
|
||
DOM key events to Wayland keycodes.
|
||
|
||
### DOM → Wayland key event translation
|
||
|
||
Browser DOM `KeyboardEvent` has `event.code` (e.g. `"KeyA"`) and
|
||
`event.key` (the resolved glyph). Greenfield uses **only the physical
|
||
`.code`**, mirroring evdev:
|
||
|
||
1. `event.code` is looked up in `EvDevKeyCode` reverse-name-map → evdev
|
||
keycode (e.g. `30` for `KeyA`).
|
||
2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset).
|
||
3. `capsLock` / `numLock` are read from `event.getModifierState(...)`.
|
||
|
||
The DOM `.key` glyph is **ignored** — only physical `.code` is used, so
|
||
the same physical key always produces the same evdev keycode regardless
|
||
of layout. This is correct Wayland behavior.
|
||
|
||
### Modifier tracking
|
||
|
||
Modifier state is tracked entirely via the WASM libxkbcommon state
|
||
machine:
|
||
|
||
1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or
|
||
`xkb.keyUp(x11)`.
|
||
2. Internally, `_xkb_state_update_key` is invoked.
|
||
3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` /
|
||
`modsLocked` / `group` (each a `_xkb_state_serialize_mods` call).
|
||
4. High-level `KeyboardModifier` bits are computed by checking xkb
|
||
mod-indices resolved at keymap-compile time.
|
||
5. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)`
|
||
sends the `wl_keyboard.modifiers` event to the native client.
|
||
|
||
### Notable bug
|
||
|
||
`Seat.ts` lines 511-513 contain a copy/paste typo:
|
||
`modsLatched = modsLocked` (should be `= modsLatched`). The cached
|
||
`keyboard.modifiers` struct is unreliable but doesn't affect protocol
|
||
output because `notifyModifiers` reads from `xkb` directly each time.
|
||
|
||
## Pointer
|
||
|
||
### DOM → Wayland pointer events
|
||
|
||
Browser DOM `MouseEvent` is converted to Greenfield's internal
|
||
`ButtonEvent` / `AxisEvent`:
|
||
|
||
- `pointermove` → `createButtonEventFromMouseEvent` → `ButtonEvent`
|
||
(motion-only)
|
||
- `pointerdown` / `pointerup` → `ButtonEvent` (with button + state)
|
||
- `wheel` → `createAxisEventFromWheelEvent` → `AxisEvent`
|
||
|
||
### Pointer enter / leave between surfaces
|
||
|
||
`DefaultPointerGrab.focus()` runs on every motion event:
|
||
|
||
1. `Renderer.pickView` walks the view stack front-to-back.
|
||
2. Returns the first view whose `Surface.isWithinInputRegion(point)`
|
||
(pixman-WASM region) is true.
|
||
3. On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to
|
||
the old surface, then `wl_pointer.enter` + `frame` (+ modifiers if
|
||
keyboard is grabbed) to the new surface.
|
||
|
||
### Pointer capture / window leave
|
||
|
||
- `canvas.setPointerCapture` is called on `pointerdown`.
|
||
- `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus,
|
||
releases depressed keys, sets `needFocusInit=true` for re-sync),
|
||
resets `buttonCount=0`.
|
||
- DnD suppresses this handler mid-drag.
|
||
|
||
### Pointer acceleration / cursor images
|
||
|
||
**No acceleration** — verbatim `offsetX/Y`.
|
||
|
||
Cursor images are **CSS `cursor: url(data:...) x y, auto`**:
|
||
|
||
1. Native client calls `wl_pointer.set_cursor`.
|
||
2. The cursor surface commits a buffer.
|
||
3. `CursorRole` draws the bitmap to a hidden canvas.
|
||
4. `toDataURL()` produces a data URL.
|
||
5. Set on `document.body.style.cursor`.
|
||
|
||
OS composites the cursor. No rotation/scale transforms.
|
||
|
||
### Scroll wheel / touchpad
|
||
|
||
`WheelEvent.deltaMode`:
|
||
|
||
- `DOM_DELTA_LINE × 12` — discrete wheel clicks
|
||
- `DOM_DELTA_PAGE × surface size` — page-sized scrolls
|
||
- `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad
|
||
|
||
Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version
|
||
≥5 also gets `axis_source(wheel)` + `frame()` (groups events).
|
||
|
||
No `axis_discrete` / `value120` / `axis_stop` events are sent.
|
||
|
||
### Linux button codes
|
||
|
||
DOM `event.button` (0=left, 1=middle, 2=right, 3=back, 4=forward) is
|
||
mapped to Linux input protocol button codes via the `linuxInput` table
|
||
in `browser/input.ts`.
|
||
|
||
## Touch — declared but inert
|
||
|
||
`Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners.
|
||
`wl_touch` capability is advertised if `'ontouchstart' in
|
||
document.documentElement`, but no events are ever sent. Touch input flows
|
||
through Pointer Events as mouse input.
|
||
|
||
**For the user**: if you need real touch support (e.g. for tablets), this
|
||
is a feature you'll have to implement from scratch. The protocol stubs
|
||
exist (via generated `wl_touch` bindings), but no event source populates
|
||
them.
|
||
|
||
## Focus management
|
||
|
||
### DOM focus
|
||
|
||
The `<canvas>` owns keyboard focus permanently (`tabIndex=1`).
|
||
`Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between
|
||
native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface`
|
||
activation handlers.
|
||
|
||
### DOM → Wayland focus
|
||
|
||
- Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores
|
||
saved focus, re-syncs modifiers).
|
||
- Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves
|
||
focus, releases depressed keys, sets `needFocusInit=true`).
|
||
|
||
### Surface-level focus
|
||
|
||
- Click on a surface → `XdgToplevel` activation request →
|
||
`desktopSurface.activate()` → `gainFocus()` (refcounted via
|
||
`focusCount`, fires `surfaceActivationUpdated` event to the user shell
|
||
on 0↔1 transitions).
|
||
- Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`,
|
||
validating the serial against the keyboard/pointer `grabSerial`.
|
||
|
||
## Input queue
|
||
|
||
`InputQueue.ts` coalesces events and dispatches them on the next RAF
|
||
(`InputQueue.drainQueues`). This batches bursts of input events (e.g. fast
|
||
mouse motion) into a single dispatch, avoiding redundant work.
|
||
|
||
The queue is FIFO per event type (motion, axis, key, button). Each
|
||
dispatch calls the appropriate `seat.notify*` method.
|
||
|
||
## ASCII event-flow diagram
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ BROWSER (DOM) │
|
||
│ │
|
||
│ <canvas tabIndex=1> ─┐ │
|
||
│ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │
|
||
│ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │
|
||
│ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │
|
||
│ │ • evdevKeyCode = EvDevKeyCode[event.code] │
|
||
│ │ • x11KeyCode = evdevKeyCode + 8 │
|
||
│ │ • capsLock/numLock = event.getModifierState(...) │
|
||
│ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │
|
||
│ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ session.inputQueue.queue{Motion,Axis,Key,Button}
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │
|
||
│ │
|
||
│ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame │
|
||
│ │ │
|
||
│ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │
|
||
│ │ └─ Keyboard.sendKey(event) │
|
||
│ │ └─ keyboardResource.key(serial,time,evdev,st) │
|
||
│ │ │
|
||
│ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │
|
||
│ │ └─ Pointer.sendButton │
|
||
│ │ └─ pointerResource.button(serial,time, │
|
||
│ │ linuxInput[btn], state) │
|
||
│ │ │
|
||
│ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │
|
||
│ │ └─ pointerResource.motion(time, sx_f, sy_f) │
|
||
│ │ │
|
||
│ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │
|
||
│ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │
|
||
│ │ └─ pointerResource.axis(time, axis, Fixed) │
|
||
│ │ + axisSource(wheel) + frame() if v>=5 │
|
||
│ │ │
|
||
│ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │
|
||
│ │ │
|
||
│ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │
|
||
│ └─ Keyboard.sendModifiers(modsDepressed,Latched, │
|
||
│ Locked,group) │
|
||
│ │
|
||
│ Keymap path (once per WlKeyboardResource): │
|
||
│ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │
|
||
│ → xkb_context_new + xkb_keymap_new_from_names (libxkbcommon WASM) │
|
||
│ → xkb_keymap_get_as_string ──► keymapString │
|
||
│ Keyboard.sendKeymap(resource): │
|
||
│ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │
|
||
│ keyboardResource.keymap(xkbV1, ProxyFD, size) │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ client.connection.onFlush(wireMessages)
|
||
│ ─ one big Uint32Array, FDs serialized as ProxyFD JSON
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ WIRE (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts) │
|
||
│ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ ws on proxy :SessionController │
|
||
│ Message layout (Uint32Array): │
|
||
│ [0] outOfBandOpcode (0 = in-band) │
|
||
│ [1] eventSerial │
|
||
│ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │
|
||
│ then concatenated Wayland wire messages │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ NativeWaylandClientSession.onMessage
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ PROXY (Node.js, compositor-proxy) │
|
||
│ │
|
||
│ SessionController wss paths: /mkfifo /mkstemp-mmap /read-fd /close-fd │
|
||
│ /read-fd-as-stream /write-fd-as-stream │
|
||
│ /signal /channel /request-keyframe │
|
||
│ │
|
||
│ mkstempMmap(ws): │
|
||
│ receives Blob → Buffer.concat → createMemoryMappedFile(buf) ← native addon │
|
||
│ returns ProxyFD { handle: <native fd>, type: 'shm', host: baseURL } │
|
||
│ │
|
||
│ onWireMessageEvents: │
|
||
│ for each ProxyFD in message: │
|
||
│ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) // just returns .handle │
|
||
│ └─ if proxyFD.host === baseURL → identity │
|
||
│ └─ if foreign host → spawn a bridging pipe + WebSocket │
|
||
│ messageInterceptor.interceptEvent(...) // empty stubs for kbd/ptr/touch │
|
||
│ sendEvents(wlClient, messageBuffer, fdsBuffer) ← libwayland-server addon │
|
||
│ flush(wlClient) │
|
||
└──────────────────────────────┬──────────────────────────────────────────────────────┘
|
||
│ libwayland-server sends wire msg + SCM_RIGHTS FDs
|
||
▼
|
||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
||
│ NATIVE (Wayland client e.g. weston-terminal, Xwayland app) │
|
||
│ wl_display.fd readable → libwayland-client demarshals │
|
||
│ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │
|
||
│ wl_keyboard.key / enter / leave / modifiers → client app │
|
||
│ wl_pointer.enter / motion / button / axis / frame → client app │
|
||
└──────────────────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
## Key answers (quick reference)
|
||
|
||
| Question | Answer |
|
||
|---|---|
|
||
| DOM→Wayland key event translation | `event.code` (e.g. `"KeyA"`) is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30`). `x11KeyCode = evdevKeyCode + 8`. DOM `.key` glyph is **ignored** — only physical `.code` is used, mirroring evdev. |
|
||
| Modifier tracking | Entirely via WASM libxkbcommon state machine. `Seat.updateModifierState` calls `xkb.keyDown/Up(x11)` → `_xkb_state_update_key`, then `notifyModifiers` reads `xkb.modsDepressed/Latched/Locked/group` (each a `_xkb_state_serialize_mods` call). High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. |
|
||
| Focus: DOM ↔ native surfaces | The `<canvas>` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave/enter` between native surfaces. Activation is driven by `XdgToplevel`/`ShellSurface` activation handlers. |
|
||
| Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` calls `notifyKeyboardFocusOut` (saves kbd focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. DnD suppresses this handler mid-drag. |
|
||
| Touch event mapping | **Not implemented.** `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. |
|
||
| Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView` walks the view stack front-to-back, returns first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. On change, `Pointer.setFocus` emits `leave+frame` to old surface, `enter+frame` (+modifiers) to new. |
|
||
| Pointer acceleration / cursor images | **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: client calls `wl_pointer.set_cursor` → `CursorRole` → draw bitmap to hidden canvas → `toDataURL()` → set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. |
|
||
| Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: `DOM_DELTA_LINE × 12`, `DOM_DELTA_PAGE × surface size`, `DOM_DELTA_PIXEL ÷ 3`. Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. v≥5 also gets `axis_source(wheel)` + `frame()`. No `axis_discrete/value120/stop`. |
|
||
| WASM xkbcommon? | Yes. `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch, builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. Sibling build does the same for `pixman`. |
|
||
| Keymap FD workaround for WebSocket | Direction is **browser → proxy**, not the reverse. Browser calls `inputOutput.mkstempMmap(Blob)` → opens `/mkstemp-mmap` WebSocket → proxy creates real native shm fd via `createMemoryMappedFile` addon → returns `ProxyFD {handle: <native_fd>, type: 'shm', host: baseURL}`. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`; the ProxyFD is **serialized as JSON** in the wire message's `fds[]` array (WebSockets can't pass FDs natively). Proxy deserializes, maps `handle` → native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. |
|
||
|
||
## File reference
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `packages/compositor/src/Seat.ts` | `wl_seat` global, multi-seat support, focus management |
|
||
| `packages/compositor/src/Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events |
|
||
| `packages/compositor/src/Pointer.ts` | `wl_pointer` impl, motion, buttons, axis |
|
||
| `packages/compositor/src/Touch.ts` | `wl_touch` (46-line stub, inert) |
|
||
| `packages/compositor/src/Xkb.ts` | xkbcommon WASM wrapper |
|
||
| `packages/compositor/src/KeyEvent.ts` | Key event wrapper |
|
||
| `packages/compositor/src/ButtonEvent.ts` | Pointer button event wrapper |
|
||
| `packages/compositor/src/AxisEvent.ts` | Pointer axis (scroll) event wrapper |
|
||
| `packages/compositor/src/InputQueue.ts` | Event coalescing + RAF dispatch |
|
||
| `packages/compositor/src/InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) |
|
||
| `packages/compositor/src/browser/input.ts` | DOM event listeners → Greenfield events |
|
||
| `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer |
|
||
| `libs/compositor-wasm/` | pixman + xkbcommon WASM build |
|
||
| `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node |
|
||
| `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor |
|
||
| `packages/compositor-proxy/src/io/` | WebSocket server glue (mkstemp-mmap endpoint etc.) |
|