1b-agent/one-os-dev-wayland-html5-compositor-greenfield-analysis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c4b45cf2180d382e2d47e6665ea28c19bb891f75
one-os-dev-wayland-html5-co…/05-input-handling.md
T

24 KiB
Raw Blame History

05 — Input Handling (Keyboard, Pointer, Touch, Focus)

This chapter covers Greenfield's input handling: keyboard (with xkbcommon in browser), pointer, touch, focus management, and how input events travel from the browser to native Linux apps via the proxy.

Headline

Greenfield is a browser-native Wayland compositor. All seat logic — keymap compilation, modifier state, pointer grabs, button/axis dispatch — runs in the browser, against a WASM build of libxkbcommon 1.5.0 + pixman. The compositor-proxy is not a compositor — it is a thin libwayland-server bridge that relays Wayland wire messages between the browser and native clients and provides file-descriptor shims.

There is no second xkbcommon on the proxy: the keymap is generated in the browser from RMLVO and shipped to the proxy as a real native shm fd via a custom /mkstemp-mmap WebSocket endpoint. wl_touch is advertised but never populated with events — touch input is collapsed into Pointer Events.

The Seat abstraction

Seat.ts (in packages/compositor/src/) is the wl_seat global. It aggregates keyboard, pointer, and touch, manages focus, and runs the input queue. Each Seat has:

  • Keyboard (always present)
  • Pointer (always present)
  • Touch (advertised only if 'ontouchstart' in document.documentElement, but inert — see §Touch below)
  • InputQueue — coalesces events and dispatches them on the next RAF
  • InputOutput — abstracts FD operations (browser pipes for in-browser apps, WebSocket endpoints for native apps via proxy)

Each seat also tracks:

  • The currently focused Surface for keyboard input
  • The currently focused Surface for pointer input (transient)
  • The active pointer grab (default, popup, drag, or resize)
  • The active keyboard grab (default or popup)

Keyboard — xkbcommon in the browser

WASM xkbcommon build

libs/compositor-wasm/xkbcommon/build.sh clones libxkbcommon 1.5.0, applies a patch (removes tests, shims strdup), builds with emscripten 3.1.46, embeds the entire xkeyboard-config database via --embed-file, exports the keymap/state C API as a single-file ES module.

The same build approach is used for pixman (for region math).

Keymap compilation flow

The browser, not the proxy, is the source of truth for the keymap. The flow is:

  1. Browser receives a keymap config (either default or user-supplied via userShell.actions.setUserConfiguration({ keyboardLayoutName })).
  2. Seat.notifyUpdateKeymap(nrmlvo) is called.
  3. xkb_context_new + xkb_keymap_new_from_names (WASM calls) compile the RMLVO into a keymap.
  4. xkb_keymap_get_as_string serializes the keymap as a long string.
  5. Keyboard.sendKeymap(resource) is called for each wl_keyboard resource:
    • inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) opens a /mkstemp-mmap WebSocket, sends the blob, and receives a ProxyFD { handle: <native_fd>, type: 'shm', host: baseURL }.
    • keyboardResource.keymap(xkbV1, ProxyFD, size) is sent to the native client.
  6. The proxy deserializes the ProxyFD, looks up the native fd via proxyFDtoNativeFD (identity when host === baseURL), ships it via sendEvents(wlClient, msg, fds) from the N-API addon — libwayland-server sends the fd via SCM_RIGHTS over the unix-domain socket.
  7. The native client's libwayland receives the keymap fd, mmaps it, and calls native xkb_keymap_new_from_string to compile its own copy.

So the keymap flows browser → proxy → native, not the reverse. This is the opposite of what you might expect (since the proxy is on the Linux host where xkbcommon natively lives), but it makes sense: the browser is the actual compositor and is the one that needs the keymap to translate DOM key events to Wayland keycodes.

DOM → Wayland key event translation

Browser DOM KeyboardEvent has event.code (e.g. "KeyA") and event.key (the resolved glyph). Greenfield uses only the physical .code, mirroring evdev:

  1. event.code is looked up in EvDevKeyCode reverse-name-map → evdev keycode (e.g. 30 for KeyA).
  2. x11KeyCode = evdevKeyCode + 8 (Linux input protocol offset).
  3. capsLock / numLock are read from event.getModifierState(...).

The DOM .key glyph is ignored — only physical .code is used, so the same physical key always produces the same evdev keycode regardless of layout. This is correct Wayland behavior.

Modifier tracking

Modifier state is tracked entirely via the WASM libxkbcommon state machine:

  1. Seat.updateModifierState calls xkb.keyDown(x11) or xkb.keyUp(x11).
  2. Internally, _xkb_state_update_key is invoked.
  3. notifyModifiers reads xkb.modsDepressed / modsLatched / modsLocked / group (each a _xkb_state_serialize_mods call).
  4. High-level KeyboardModifier bits are computed by checking xkb mod-indices resolved at keymap-compile time.
  5. Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group) sends the wl_keyboard.modifiers event to the native client.

Notable bug

Seat.ts lines 511-513 contain a copy/paste typo: modsLatched = modsLocked (should be = modsLatched). The cached keyboard.modifiers struct is unreliable but doesn't affect protocol output because notifyModifiers reads from xkb directly each time.

Pointer

DOM → Wayland pointer events

Browser DOM MouseEvent is converted to Greenfield's internal ButtonEvent / AxisEvent:

  • pointermove → createButtonEventFromMouseEvent → ButtonEvent (motion-only)
  • pointerdown / pointerup → ButtonEvent (with button + state)
  • wheel → createAxisEventFromWheelEvent → AxisEvent

Pointer enter / leave between surfaces

DefaultPointerGrab.focus() runs on every motion event:

  1. Renderer.pickView walks the view stack front-to-back.
  2. Returns the first view whose Surface.isWithinInputRegion(point) (pixman-WASM region) is true.
  3. On change, Pointer.setFocus emits wl_pointer.leave + frame to the old surface, then wl_pointer.enter + frame (+ modifiers if keyboard is grabbed) to the new surface.

Pointer capture / window leave

  • canvas.setPointerCapture is called on pointerdown.
  • pointerleave calls notifyKeyboardFocusOut (saves keyboard focus, releases depressed keys, sets needFocusInit=true for re-sync), resets buttonCount=0.
  • DnD suppresses this handler mid-drag.

Pointer acceleration / cursor images

No acceleration — verbatim offsetX/Y.

Cursor images are CSS cursor: url(data:...) x y, auto:

  1. Native client calls wl_pointer.set_cursor.
  2. The cursor surface commits a buffer.
  3. CursorRole draws the bitmap to a hidden canvas.
  4. toDataURL() produces a data URL.
  5. Set on document.body.style.cursor.

OS composites the cursor. No rotation/scale transforms.

Scroll wheel / touchpad

WheelEvent.deltaMode:

  • DOM_DELTA_LINE × 12 — discrete wheel clicks
  • DOM_DELTA_PAGE × surface size — page-sized scrolls
  • DOM_DELTA_PIXEL ÷ 3 — fine-grained touchpad

Each non-zero axis → wl_pointer.axis(time, axis, Fixed). Wayland version ≥5 also gets axis_source(wheel) + frame() (groups events).

No axis_discrete / value120 / axis_stop events are sent.

Linux button codes

DOM event.button (0=left, 1=middle, 2=right, 3=back, 4=forward) is mapped to Linux input protocol button codes via the linuxInput table in browser/input.ts.

Touch — declared but inert

Touch.ts is a 46-line stub; browser/input.ts has no touch listeners. wl_touch capability is advertised if 'ontouchstart' in document.documentElement, but no events are ever sent. Touch input flows through Pointer Events as mouse input.

For the user: if you need real touch support (e.g. for tablets), this is a feature you'll have to implement from scratch. The protocol stubs exist (via generated wl_touch bindings), but no event source populates them.

Focus management

DOM focus

The <canvas> owns keyboard focus permanently (tabIndex=1). Keyboard.setFocus(surface) emits wl_keyboard.leave / enter between native surfaces. Activation is driven by XdgToplevel / ShellSurface activation handlers.

DOM → Wayland focus

  • Browser DOM focus event → seat.notifyKeyboardFocusIn() (restores saved focus, re-syncs modifiers).
  • Browser DOM blur event → seat.notifyKeyboardFocusOut() (saves focus, releases depressed keys, sets needFocusInit=true).

Surface-level focus

  • Click on a surface → XdgToplevel activation request → desktopSurface.activate() → gainFocus() (refcounted via focusCount, fires surfaceActivationUpdated event to the user shell on 0↔1 transitions).
  • Popup grabs take keyboard + pointer focus via seat.popupGrabStart, validating the serial against the keyboard/pointer grabSerial.

Input queue

InputQueue.ts coalesces events and dispatches them on the next RAF (InputQueue.drainQueues). This batches bursts of input events (e.g. fast mouse motion) into a single dispatch, avoiding redundant work.

The queue is FIFO per event type (motion, axis, key, button). Each dispatch calls the appropriate seat.notify* method.

ASCII event-flow diagram

┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER (DOM)                                                                        │
│                                                                                      │
│  <canvas tabIndex=1>  ─┐                                                             │
│  pointermove/dn/up  ───┤── createButtonEventFromMouseEvent()  ── ButtonEvent         │
│  wheel             ────┤── createAxisEventFromWheelEvent()    ── AxisEvent           │
│  keydown / keyup   ────┤── createKeyEventFromKeyboardEvent()  ── KeyEvent           │
│                         │   • evdevKeyCode = EvDevKeyCode[event.code]                │
│                         │   • x11KeyCode   = evdevKeyCode + 8                        │
│                         │   • capsLock/numLock = event.getModifierState(...)          │
│  pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods)  │
│  focus              ────┴── seat.notifyKeyboardFocusIn()  (restores saved focus)     │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ session.inputQueue.queue{Motion,Axis,Key,Button}
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER COMPOSITOR  (InputQueue.drainQueues, scheduled on next animation frame)      │
│                                                                                      │
│  seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame             │
│         │                                                                            │
│         ├─► Seat.notifyKey      ─► Keyboard.grab.key(event)                           │
│         │                          └─ Keyboard.sendKey(event)                        │
│         │                              └─ keyboardResource.key(serial,time,evdev,st)  │
│         │                                                                            │
│         ├─► Seat.notifyButton  ─► runButtonBindings + grab.button(event)              │
│         │                          └─ Pointer.sendButton                             │
│         │                              └─ pointerResource.button(serial,time,         │
│         │                                       linuxInput[btn], state)              │
│         │                                                                            │
│         ├─► Seat.notifyMotion  ─► grab.motion ─► Pointer.moveTo / sendMotion        │
│         │                              └─ pointerResource.motion(time, sx_f, sy_f)   │
│         │                                                                            │
│         ├─► Seat.notifyAxis    ─► grab.axis ─► Pointer.sendAxis                     │
│         │       deltaMode→scale:  line=×12, page=×surface-size, pixel=÷3              │
│         │                          └─ pointerResource.axis(time, axis, Fixed)        │
│         │                              + axisSource(wheel) + frame() if v>=5         │
│         │                                                                            │
│         ├─► Seat.notifyFrame  ─► pointerResource.frame()  (groups events)           │
│         │                                                                            │
│         └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods   │
│                                  └─ Keyboard.sendModifiers(modsDepressed,Latched,   │
│                                                              Locked,group)            │
│                                                                                      │
│  Keymap path (once per WlKeyboardResource):                                          │
│   Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt)  │
│   → xkb_context_new + xkb_keymap_new_from_names  (libxkbcommon WASM)                 │
│   → xkb_keymap_get_as_string  ──► keymapString                                       │
│   Keyboard.sendKeymap(resource):                                                      │
│     inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD               │
│     keyboardResource.keymap(xkbV1, ProxyFD, size)                                    │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ client.connection.onFlush(wireMessages)
                               │  ─ one big Uint32Array, FDs serialized as ProxyFD JSON
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ WIRE  (ARQChannel = KCP-over-WebSocket, packages/compositor/src/remote/Channel.ts)    │
│   Browser  ⇄  ReconnectingWebSocket  ⇄  Internet  ⇄  ws on proxy :SessionController   │
│   Message layout (Uint32Array):                                                      │
│     [0] outOfBandOpcode (0 = in-band)                                                │
│     [1] eventSerial                                                                  │
│     [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned]                 │
│     then concatenated Wayland wire messages                                          │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ NativeWaylandClientSession.onMessage
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ PROXY  (Node.js, compositor-proxy)                                                   │
│                                                                                      │
│  SessionController wss paths:  /mkfifo /mkstemp-mmap /read-fd /close-fd              │
│                                 /read-fd-as-stream /write-fd-as-stream              │
│                                 /signal /channel /request-keyframe                   │
│                                                                                      │
│  mkstempMmap(ws):                                                                    │
│      receives Blob → Buffer.concat → createMemoryMappedFile(buf)  ← native addon    │
│      returns ProxyFD { handle: <native fd>, type: 'shm', host: baseURL }             │
│                                                                                      │
│  onWireMessageEvents:                                                                │
│      for each ProxyFD in message:                                                    │
│          nativeFd = webFS.proxyFDtoNativeFD(proxyFD)  // just returns .handle        │
│                  └─ if proxyFD.host === baseURL → identity                           │
│                  └─ if foreign host  → spawn a bridging pipe + WebSocket              │
│      messageInterceptor.interceptEvent(...)  // empty stubs for kbd/ptr/touch        │
│      sendEvents(wlClient, messageBuffer, fdsBuffer)  ← libwayland-server addon      │
│      flush(wlClient)                                                                 │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ libwayland-server sends wire msg + SCM_RIGHTS FDs
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ NATIVE  (Wayland client e.g. weston-terminal, Xwayland app)                          │
│   wl_display.fd readable → libwayland-client demarshals                              │
│   wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon)  │
│   wl_keyboard.key / enter / leave / modifiers → client app                          │
│   wl_pointer.enter / motion / button / axis / frame → client app                     │
└──────────────────────────────────────────────────────────────────────────────────────┘

Key answers (quick reference)

Question Answer
DOM→Wayland key event translation event.code (e.g. "KeyA") is looked up in EvDevKeyCode reverse-name-map → evdev keycode (e.g. 30). x11KeyCode = evdevKeyCode + 8. DOM .key glyph is ignored — only physical .code is used, mirroring evdev.
Modifier tracking Entirely via WASM libxkbcommon state machine. Seat.updateModifierState calls xkb.keyDown/Up(x11) → _xkb_state_update_key, then notifyModifiers reads xkb.modsDepressed/Latched/Locked/group (each a _xkb_state_serialize_mods call). High-level KeyboardModifier bits are computed by checking xkb mod-indices resolved at keymap-compile time.
Focus: DOM ↔ native surfaces The <canvas> owns keyboard focus permanently (tabIndex=1). Keyboard.setFocus(surface) emits wl_keyboard.leave/enter between native surfaces. Activation is driven by XdgToplevel/ShellSurface activation handlers.
Pointer capture / window leave canvas.setPointerCapture on pointerdown. pointerleave calls notifyKeyboardFocusOut (saves kbd focus, releases depressed keys, sets needFocusInit=true for re-sync), resets buttonCount=0. DnD suppresses this handler mid-drag.
Touch event mapping Not implemented. Touch.ts is a 46-line stub; browser/input.ts has no touch listeners. wl_touch capability is advertised if 'ontouchstart' in document.documentElement, but no events are ever sent. Touch input flows through Pointer Events as mouse input.
Pointer enter/leave between surfaces DefaultPointerGrab.focus() runs on every motion: Renderer.pickView walks the view stack front-to-back, returns first view whose Surface.isWithinInputRegion(point) (pixman-WASM region) is true. On change, Pointer.setFocus emits leave+frame to old surface, enter+frame (+modifiers) to new.
Pointer acceleration / cursor images No acceleration — verbatim offsetX/Y. Cursor images are CSS cursor: url(data:...) x y, auto: client calls wl_pointer.set_cursor → CursorRole → draw bitmap to hidden canvas → toDataURL() → set on document.body.style.cursor. OS composites the cursor. No rotation/scale transforms.
Scroll wheel / touchpad → wl_pointer.axis WheelEvent.deltaMode: DOM_DELTA_LINE × 12, DOM_DELTA_PAGE × surface size, DOM_DELTA_PIXEL ÷ 3. Each non-zero axis → wl_pointer.axis(time, axis, Fixed). v≥5 also gets axis_source(wheel) + frame(). No axis_discrete/value120/stop.
WASM xkbcommon? Yes. libs/compositor-wasm/xkbcommon/build.sh clones libxkbcommon 1.5.0, applies a patch, builds with emscripten 3.1.46, embeds the entire xkeyboard-config database via --embed-file, exports the keymap/state C API as a single-file ES module. Sibling build does the same for pixman.
Keymap FD workaround for WebSocket Direction is browser → proxy, not the reverse. Browser calls inputOutput.mkstempMmap(Blob) → opens /mkstemp-mmap WebSocket → proxy creates real native shm fd via createMemoryMappedFile addon → returns ProxyFD {handle: <native_fd>, type: 'shm', host: baseURL}. Browser sends wl_keyboard.keymap(format, ProxyFD, size); the ProxyFD is serialized as JSON in the wire message's fds[] array (WebSockets can't pass FDs natively). Proxy deserializes, maps handle → native fd via proxyFDtoNativeFD (identity when host === baseURL), ships via sendEvents(wlClient, msg, fds) from the N-API addon — libwayland-server sends the fd via SCM_RIGHTS over the unix-domain socket.

File reference

File Role
packages/compositor/src/Seat.ts wl_seat global, multi-seat support, focus management
packages/compositor/src/Keyboard.ts wl_keyboard impl, keymap, modifiers, key events
packages/compositor/src/Pointer.ts wl_pointer impl, motion, buttons, axis
packages/compositor/src/Touch.ts wl_touch (46-line stub, inert)
packages/compositor/src/Xkb.ts xkbcommon WASM wrapper
packages/compositor/src/KeyEvent.ts Key event wrapper
packages/compositor/src/ButtonEvent.ts Pointer button event wrapper
packages/compositor/src/AxisEvent.ts Pointer axis (scroll) event wrapper
packages/compositor/src/InputQueue.ts Event coalescing + RAF dispatch
packages/compositor/src/InputOutput.ts FD abstraction (browser pipes / WebSocket endpoints)
packages/compositor/src/browser/input.ts DOM event listeners → Greenfield events
packages/compositor/src/browser/pointer.ts DOM pointer → wl_pointer
libs/compositor-wasm/ pixman + xkbcommon WASM build
packages/compositor-proxy/src/wayland-server.ts TS wrapper around wayland-server-addon.node
packages/compositor-proxy/src/NativeWaylandClientSession.ts Per-client message interceptor
packages/compositor-proxy/src/io/ WebSocket server glue (mkstemp-mmap endpoint etc.)
Reference in new issue View Git Blame Copy Permalink