# 05 — Input Handling (Keyboard, Pointer, Touch, Focus) This chapter covers Greenfield's input handling: keyboard (with xkbcommon in browser), pointer, touch, focus management, and how input events travel from the browser to native Linux apps via the proxy. ## Headline Greenfield is a **browser-native Wayland compositor**. All seat logic — keymap compilation, modifier state, pointer grabs, button/axis dispatch — runs **in the browser**, against a **WASM build of libxkbcommon 1.5.0 + pixman**. The `compositor-proxy` is *not* a compositor — it is a thin libwayland-server bridge that relays Wayland wire messages between the browser and native clients and provides file-descriptor shims. There is no second xkbcommon on the proxy: the keymap is generated in the browser from RMLVO and shipped to the proxy as a real native shm fd via a custom `/mkstemp-mmap` WebSocket endpoint. `wl_touch` is advertised but **never populated with events** — touch input is collapsed into Pointer Events. ## The Seat abstraction `Seat.ts` (in `packages/compositor/src/`) is the `wl_seat` global. It aggregates keyboard, pointer, and touch, manages focus, and runs the input queue. Each `Seat` has: - `Keyboard` (always present) - `Pointer` (always present) - `Touch` (advertised only if `'ontouchstart' in document.documentElement`, but inert — see §Touch below) - `InputQueue` — coalesces events and dispatches them on the next RAF - `InputOutput` — abstracts FD operations (browser pipes for in-browser apps, WebSocket endpoints for native apps via proxy) Each seat also tracks: - The currently focused `Surface` for keyboard input - The currently focused `Surface` for pointer input (transient) - The active pointer grab (default, popup, drag, or resize) - The active keyboard grab (default or popup) ## Keyboard — xkbcommon in the browser ### WASM xkbcommon build `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch (removes tests, shims `strdup`), builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. The same build approach is used for pixman (for region math). ### Keymap compilation flow The browser, not the proxy, is the source of truth for the keymap. The flow is: 1. Browser receives a keymap config (either default or user-supplied via `userShell.actions.setUserConfiguration({ keyboardLayoutName })`). 2. `Seat.notifyUpdateKeymap(nrmlvo)` is called. 3. `xkb_context_new` + `xkb_keymap_new_from_names` (WASM calls) compile the RMLVO into a keymap. 4. `xkb_keymap_get_as_string` serializes the keymap as a long string. 5. `Keyboard.sendKeymap(resource)` is called for each `wl_keyboard` resource: - `inputOutput.mkstempMmap(new Blob([utf8(keymapString)]))` opens a `/mkstemp-mmap` WebSocket, sends the blob, and receives a `ProxyFD { handle: , type: 'shm', host: baseURL }`. - `keyboardResource.keymap(xkbV1, ProxyFD, size)` is sent to the native client. 6. The proxy deserializes the ProxyFD, looks up the native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships it via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. 7. The native client's libwayland receives the keymap fd, mmaps it, and calls native `xkb_keymap_new_from_string` to compile its own copy. So the keymap flows **browser → proxy → native**, not the reverse. This is the opposite of what you might expect (since the proxy is on the Linux host where xkbcommon natively lives), but it makes sense: the browser is the actual compositor and is the one that needs the keymap to translate DOM key events to Wayland keycodes. ### DOM → Wayland key event translation Browser DOM `KeyboardEvent` has `event.code` (e.g. `"KeyA"`) and `event.key` (the resolved glyph). Greenfield uses **only the physical `.code`**, mirroring evdev: 1. `event.code` is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30` for `KeyA`). 2. `x11KeyCode = evdevKeyCode + 8` (Linux input protocol offset). 3. `capsLock` / `numLock` are read from `event.getModifierState(...)`. The DOM `.key` glyph is **ignored** — only physical `.code` is used, so the same physical key always produces the same evdev keycode regardless of layout. This is correct Wayland behavior. ### Modifier tracking Modifier state is tracked entirely via the WASM libxkbcommon state machine: 1. `Seat.updateModifierState` calls `xkb.keyDown(x11)` or `xkb.keyUp(x11)`. 2. Internally, `_xkb_state_update_key` is invoked. 3. `notifyModifiers` reads `xkb.modsDepressed` / `modsLatched` / `modsLocked` / `group` (each a `_xkb_state_serialize_mods` call). 4. High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. 5. `Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group)` sends the `wl_keyboard.modifiers` event to the native client. ### Notable bug `Seat.ts` lines 511-513 contain a copy/paste typo: `modsLatched = modsLocked` (should be `= modsLatched`). The cached `keyboard.modifiers` struct is unreliable but doesn't affect protocol output because `notifyModifiers` reads from `xkb` directly each time. ## Pointer ### DOM → Wayland pointer events Browser DOM `MouseEvent` is converted to Greenfield's internal `ButtonEvent` / `AxisEvent`: - `pointermove` → `createButtonEventFromMouseEvent` → `ButtonEvent` (motion-only) - `pointerdown` / `pointerup` → `ButtonEvent` (with button + state) - `wheel` → `createAxisEventFromWheelEvent` → `AxisEvent` ### Pointer enter / leave between surfaces `DefaultPointerGrab.focus()` runs on every motion event: 1. `Renderer.pickView` walks the view stack front-to-back. 2. Returns the first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. 3. On change, `Pointer.setFocus` emits `wl_pointer.leave` + `frame` to the old surface, then `wl_pointer.enter` + `frame` (+ modifiers if keyboard is grabbed) to the new surface. ### Pointer capture / window leave - `canvas.setPointerCapture` is called on `pointerdown`. - `pointerleave` calls `notifyKeyboardFocusOut` (saves keyboard focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. - DnD suppresses this handler mid-drag. ### Pointer acceleration / cursor images **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: 1. Native client calls `wl_pointer.set_cursor`. 2. The cursor surface commits a buffer. 3. `CursorRole` draws the bitmap to a hidden canvas. 4. `toDataURL()` produces a data URL. 5. Set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. ### Scroll wheel / touchpad `WheelEvent.deltaMode`: - `DOM_DELTA_LINE × 12` — discrete wheel clicks - `DOM_DELTA_PAGE × surface size` — page-sized scrolls - `DOM_DELTA_PIXEL ÷ 3` — fine-grained touchpad Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. Wayland version ≥5 also gets `axis_source(wheel)` + `frame()` (groups events). No `axis_discrete` / `value120` / `axis_stop` events are sent. ### Linux button codes DOM `event.button` (0=left, 1=middle, 2=right, 3=back, 4=forward) is mapped to Linux input protocol button codes via the `linuxInput` table in `browser/input.ts`. ## Touch — declared but inert `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. **For the user**: if you need real touch support (e.g. for tablets), this is a feature you'll have to implement from scratch. The protocol stubs exist (via generated `wl_touch` bindings), but no event source populates them. ## Focus management ### DOM focus The `` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave` / `enter` between native surfaces. Activation is driven by `XdgToplevel` / `ShellSurface` activation handlers. ### DOM → Wayland focus - Browser DOM `focus` event → `seat.notifyKeyboardFocusIn()` (restores saved focus, re-syncs modifiers). - Browser DOM `blur` event → `seat.notifyKeyboardFocusOut()` (saves focus, releases depressed keys, sets `needFocusInit=true`). ### Surface-level focus - Click on a surface → `XdgToplevel` activation request → `desktopSurface.activate()` → `gainFocus()` (refcounted via `focusCount`, fires `surfaceActivationUpdated` event to the user shell on 0↔1 transitions). - Popup grabs take keyboard + pointer focus via `seat.popupGrabStart`, validating the serial against the keyboard/pointer `grabSerial`. ## Input queue `InputQueue.ts` coalesces events and dispatches them on the next RAF (`InputQueue.drainQueues`). This batches bursts of input events (e.g. fast mouse motion) into a single dispatch, avoiding redundant work. The queue is FIFO per event type (motion, axis, key, button). Each dispatch calls the appropriate `seat.notify*` method. ## ASCII event-flow diagram ``` ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ BROWSER (DOM) │ │ │ │ ─┐ │ │ pointermove/dn/up ───┤── createButtonEventFromMouseEvent() ── ButtonEvent │ │ wheel ────┤── createAxisEventFromWheelEvent() ── AxisEvent │ │ keydown / keyup ────┤── createKeyEventFromKeyboardEvent() ── KeyEvent │ │ │ • evdevKeyCode = EvDevKeyCode[event.code] │ │ │ • x11KeyCode = evdevKeyCode + 8 │ │ │ • capsLock/numLock = event.getModifierState(...) │ │ pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods) │ │ focus ────┴── seat.notifyKeyboardFocusIn() (restores saved focus) │ └──────────────────────────────┬──────────────────────────────────────────────────────┘ │ session.inputQueue.queue{Motion,Axis,Key,Button} ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ BROWSER COMPOSITOR (InputQueue.drainQueues, scheduled on next animation frame) │ │ │ │ seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame │ │ │ │ │ ├─► Seat.notifyKey ─► Keyboard.grab.key(event) │ │ │ └─ Keyboard.sendKey(event) │ │ │ └─ keyboardResource.key(serial,time,evdev,st) │ │ │ │ │ ├─► Seat.notifyButton ─► runButtonBindings + grab.button(event) │ │ │ └─ Pointer.sendButton │ │ │ └─ pointerResource.button(serial,time, │ │ │ linuxInput[btn], state) │ │ │ │ │ ├─► Seat.notifyMotion ─► grab.motion ─► Pointer.moveTo / sendMotion │ │ │ └─ pointerResource.motion(time, sx_f, sy_f) │ │ │ │ │ ├─► Seat.notifyAxis ─► grab.axis ─► Pointer.sendAxis │ │ │ deltaMode→scale: line=×12, page=×surface-size, pixel=÷3 │ │ │ └─ pointerResource.axis(time, axis, Fixed) │ │ │ + axisSource(wheel) + frame() if v>=5 │ │ │ │ │ ├─► Seat.notifyFrame ─► pointerResource.frame() (groups events) │ │ │ │ │ └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods │ │ └─ Keyboard.sendModifiers(modsDepressed,Latched, │ │ Locked,group) │ │ │ │ Keymap path (once per WlKeyboardResource): │ │ Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt) │ │ → xkb_context_new + xkb_keymap_new_from_names (libxkbcommon WASM) │ │ → xkb_keymap_get_as_string ──► keymapString │ │ Keyboard.sendKeymap(resource): │ │ inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD │ │ keyboardResource.keymap(xkbV1, ProxyFD, size) │ └──────────────────────────────┬──────────────────────────────────────────────────────┘ │ client.connection.onFlush(wireMessages) │ ─ one big Uint32Array, FDs serialized as ProxyFD JSON ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ WIRE (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts) │ │ Browser ⇄ ReconnectingWebSocket ⇄ Internet ⇄ ws on proxy :SessionController │ │ Message layout (Uint32Array): │ │ [0] outOfBandOpcode (0 = in-band) │ │ [1] eventSerial │ │ [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned] │ │ then concatenated Wayland wire messages │ └──────────────────────────────┬──────────────────────────────────────────────────────┘ │ NativeWaylandClientSession.onMessage ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ PROXY (Node.js, compositor-proxy) │ │ │ │ SessionController wss paths: /mkfifo /mkstemp-mmap /read-fd /close-fd │ │ /read-fd-as-stream /write-fd-as-stream │ │ /signal /channel /request-keyframe │ │ │ │ mkstempMmap(ws): │ │ receives Blob → Buffer.concat → createMemoryMappedFile(buf) ← native addon │ │ returns ProxyFD { handle: , type: 'shm', host: baseURL } │ │ │ │ onWireMessageEvents: │ │ for each ProxyFD in message: │ │ nativeFd = webFS.proxyFDtoNativeFD(proxyFD) // just returns .handle │ │ └─ if proxyFD.host === baseURL → identity │ │ └─ if foreign host → spawn a bridging pipe + WebSocket │ │ messageInterceptor.interceptEvent(...) // empty stubs for kbd/ptr/touch │ │ sendEvents(wlClient, messageBuffer, fdsBuffer) ← libwayland-server addon │ │ flush(wlClient) │ └──────────────────────────────┬──────────────────────────────────────────────────────┘ │ libwayland-server sends wire msg + SCM_RIGHTS FDs ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ NATIVE (Wayland client e.g. weston-terminal, Xwayland app) │ │ wl_display.fd readable → libwayland-client demarshals │ │ wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon) │ │ wl_keyboard.key / enter / leave / modifiers → client app │ │ wl_pointer.enter / motion / button / axis / frame → client app │ └──────────────────────────────────────────────────────────────────────────────────────┘ ``` ## Key answers (quick reference) | Question | Answer | |---|---| | DOM→Wayland key event translation | `event.code` (e.g. `"KeyA"`) is looked up in `EvDevKeyCode` reverse-name-map → evdev keycode (e.g. `30`). `x11KeyCode = evdevKeyCode + 8`. DOM `.key` glyph is **ignored** — only physical `.code` is used, mirroring evdev. | | Modifier tracking | Entirely via WASM libxkbcommon state machine. `Seat.updateModifierState` calls `xkb.keyDown/Up(x11)` → `_xkb_state_update_key`, then `notifyModifiers` reads `xkb.modsDepressed/Latched/Locked/group` (each a `_xkb_state_serialize_mods` call). High-level `KeyboardModifier` bits are computed by checking xkb mod-indices resolved at keymap-compile time. | | Focus: DOM ↔ native surfaces | The `` owns keyboard focus permanently (`tabIndex=1`). `Keyboard.setFocus(surface)` emits `wl_keyboard.leave/enter` between native surfaces. Activation is driven by `XdgToplevel`/`ShellSurface` activation handlers. | | Pointer capture / window leave | `canvas.setPointerCapture` on pointerdown. `pointerleave` calls `notifyKeyboardFocusOut` (saves kbd focus, releases depressed keys, sets `needFocusInit=true` for re-sync), resets `buttonCount=0`. DnD suppresses this handler mid-drag. | | Touch event mapping | **Not implemented.** `Touch.ts` is a 46-line stub; `browser/input.ts` has no touch listeners. `wl_touch` capability is advertised if `'ontouchstart' in document.documentElement`, but no events are ever sent. Touch input flows through Pointer Events as mouse input. | | Pointer enter/leave between surfaces | `DefaultPointerGrab.focus()` runs on every motion: `Renderer.pickView` walks the view stack front-to-back, returns first view whose `Surface.isWithinInputRegion(point)` (pixman-WASM region) is true. On change, `Pointer.setFocus` emits `leave+frame` to old surface, `enter+frame` (+modifiers) to new. | | Pointer acceleration / cursor images | **No acceleration** — verbatim `offsetX/Y`. Cursor images are **CSS `cursor: url(data:...) x y, auto`**: client calls `wl_pointer.set_cursor` → `CursorRole` → draw bitmap to hidden canvas → `toDataURL()` → set on `document.body.style.cursor`. OS composites the cursor. No rotation/scale transforms. | | Scroll wheel / touchpad → `wl_pointer.axis` | `WheelEvent.deltaMode`: `DOM_DELTA_LINE × 12`, `DOM_DELTA_PAGE × surface size`, `DOM_DELTA_PIXEL ÷ 3`. Each non-zero axis → `wl_pointer.axis(time, axis, Fixed)`. v≥5 also gets `axis_source(wheel)` + `frame()`. No `axis_discrete/value120/stop`. | | WASM xkbcommon? | Yes. `libs/compositor-wasm/xkbcommon/build.sh` clones `libxkbcommon 1.5.0`, applies a patch, builds with emscripten 3.1.46, **embeds the entire xkeyboard-config database** via `--embed-file`, exports the keymap/state C API as a single-file ES module. Sibling build does the same for `pixman`. | | Keymap FD workaround for WebSocket | Direction is **browser → proxy**, not the reverse. Browser calls `inputOutput.mkstempMmap(Blob)` → opens `/mkstemp-mmap` WebSocket → proxy creates real native shm fd via `createMemoryMappedFile` addon → returns `ProxyFD {handle: , type: 'shm', host: baseURL}`. Browser sends `wl_keyboard.keymap(format, ProxyFD, size)`; the ProxyFD is **serialized as JSON** in the wire message's `fds[]` array (WebSockets can't pass FDs natively). Proxy deserializes, maps `handle` → native fd via `proxyFDtoNativeFD` (identity when `host === baseURL`), ships via `sendEvents(wlClient, msg, fds)` from the N-API addon — `libwayland-server` sends the fd via `SCM_RIGHTS` over the unix-domain socket. | ## File reference | File | Role | |---|---| | `packages/compositor/src/Seat.ts` | `wl_seat` global, multi-seat support, focus management | | `packages/compositor/src/Keyboard.ts` | `wl_keyboard` impl, keymap, modifiers, key events | | `packages/compositor/src/Pointer.ts` | `wl_pointer` impl, motion, buttons, axis | | `packages/compositor/src/Touch.ts` | `wl_touch` (46-line stub, inert) | | `packages/compositor/src/Xkb.ts` | xkbcommon WASM wrapper | | `packages/compositor/src/KeyEvent.ts` | Key event wrapper | | `packages/compositor/src/ButtonEvent.ts` | Pointer button event wrapper | | `packages/compositor/src/AxisEvent.ts` | Pointer axis (scroll) event wrapper | | `packages/compositor/src/InputQueue.ts` | Event coalescing + RAF dispatch | | `packages/compositor/src/InputOutput.ts` | FD abstraction (browser pipes / WebSocket endpoints) | | `packages/compositor/src/browser/input.ts` | DOM event listeners → Greenfield events | | `packages/compositor/src/browser/pointer.ts` | DOM pointer → wl_pointer | | `libs/compositor-wasm/` | pixman + xkbcommon WASM build | | `packages/compositor-proxy/src/wayland-server.ts` | TS wrapper around wayland-server-addon.node | | `packages/compositor-proxy/src/NativeWaylandClientSession.ts` | Per-client message interceptor | | `packages/compositor-proxy/src/io/` | WebSocket server glue (mkstemp-mmap endpoint etc.) |