1b-agent/one-os-dev-wayland-html5-compositor-mit-compositor-analysis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
one-os-dev-wayland-html5-co…/05-input-handling.md
T

21 KiB
Raw Permalink Blame History

05 — Input Handling (Python xkbcommon + Browser WASM)

This chapter covers input handling in the MIT approach: keyboard (with xkbcommon), pointer, touch, focus management, and how input events travel between browser and native apps.

Architecture (same as Greenfield, but with improvements)

The browser is the source of truth for the keymap (same as Greenfield). The keymap is compiled in the browser using @gfld/compositor-wasm's xkbcommon WASM module, then shipped to the Python proxy as a real native shm fd via the /mkstemp-mmap WebSocket endpoint. The proxy then sends it to native apps via libwayland-server's SCM_RIGHTS FD passing.

Improvements over Greenfield:

  1. Real touch input — Greenfield has a 46-line stub that never fires
  2. Primary selection — middle-click paste, which Greenfield lacks
  3. Adaptive keyboard layout — easier to switch layouts at runtime

Browser side — Custom TypeScript (clean-room)

Seat abstraction

Same architecture as Greenfield (clean-room from spec + Wayland docs):

  • Seat.ts — wl_seat global, multi-seat support, focus management
  • Keyboard.ts — wl_keyboard impl, keymap, modifiers, key events
  • Pointer.ts — wl_pointer impl, motion, buttons, axis
  • Touch.ts — wl_touch (REAL implementation, not stub)
  • Xkb.ts — xkbcommon WASM wrapper (REUSED @gfld/compositor-wasm)
  • InputQueue.ts — event coalescing + RAF dispatch
  • InputOutput.ts — FD abstraction
  • events.ts — KeyEvent, ButtonEvent, AxisEvent wrappers

xkbcommon WASM (REUSED)

@gfld/compositor-wasm (MIT) bundles:

  • libxkbcommon 1.5.0 compiled to WASM via emsdk 3.1.46
  • Embedded xkeyboard-config database
  • Exports init() + lib.xkbcommon singleton with keymap/state functions

This is REUSED AS-IS. No custom code needed.

DOM → Wayland key event translation (clean-room)

Same approach as Greenfield (verified correct Wayland behavior):

  1. event.code (e.g. "KeyA") looked up in EvDevKeyCode reverse-name-map → evdev keycode (e.g. 30)
  2. x11KeyCode = evdevKeyCode + 8 (Linux input protocol offset)
  3. capsLock/numLock read from event.getModifierState(...)
  4. DOM .key glyph IGNORED — only physical .code used (mirrors evdev)

Modifier tracking (REUSED WASM)

Via @gfld/compositor-wasm's xkbcommon state machine:

  1. Seat.updateModifierState calls xkb.keyDown(x11) or xkb.keyUp(x11)
  2. _xkb_state_update_key invoked internally
  3. notifyModifiers reads xkb.modsDepressed / modsLatched / modsLocked / group
  4. Keyboard.sendModifiers(modsDepressed, modsLatched, modsLocked, group) sends to native client

Touch input (NEW vs Greenfield)

Greenfield has a 46-line stub for Touch.ts. Our MIT approach implements real touch:

// Pseudocode for Touch.ts (clean-room)
export class Touch {
  private touchPoints: Map<number, TouchPoint> = new Map();
  
  constructor(public seat: Seat) {
    // Listen to DOM pointer events with pointerType === 'touch'
    document.addEventListener('pointerdown', this.onPointerDown.bind(this));
    document.addEventListener('pointermove', this.onPointerMove.bind(this));
    document.addEventListener('pointerup', this.onPointerUp.bind(this));
    document.addEventListener('pointercancel', this.onPointerCancel.bind(this));
  }
  
  private onPointerDown(ev: PointerEvent) {
    if (ev.pointerType !== 'touch') return;
    // Map to wl_touch.down event
    this.touchPoints.set(ev.pointerId, { x: ev.offsetX, y: ev.offsetY });
    this.sendDown(ev.pointerId, ev.offsetX, ev.offsetY);
  }
  
  // ... etc
}

Keymap FD workaround (REUSED approach)

Same as Greenfield (see ../greenfield-analysis/05-input-handling.md):

  1. Browser compiles keymap via @gfld/compositor-wasm's xkbcommon
  2. inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) opens a /mkstemp-mmap WebSocket, sends the blob
  3. Python proxy creates a real native shm fd via os.memfd_create() (replaces Greenfield's createMemoryMappedFile native addon)
  4. Returns ProxyFD { handle: <native_fd>, type: 'shm', host: baseURL }
  5. Browser sends wl_keyboard.keymap(format, ProxyFD, size)
  6. Proxy deserializes, looks up native fd via proxyFDtoNativeFD, ships via libwayland-server's SCM_RIGHTS

Python side — pywayland + python-xkbcommon

python-xkbcommon (MIT)

The xkbcommon PyPI package (by Stephen Early, MIT-licensed) provides Python bindings to libxkbcommon. Useful if the proxy needs to compile keymaps (e.g., for XWayland apps that don't speak Wayland natively).

For most use cases, the proxy doesn't need xkbcommon — it just forwards the keymap from browser to native apps. The browser does the compilation.

Pointer (clean-room)

Same approach as Greenfield:

  • DOM pointermove / pointerdown / pointerup / wheel events
  • Mapped to ButtonEvent / AxisEvent wrappers
  • InputQueue coalesces and dispatches on next RAF
  • DefaultPointerGrab.focus() runs on every motion: Renderer.pickView walks the view stack front-to-back, returns first view whose Surface.isWithinInputRegion(point) (pixman-WASM region) is true
  • On change, Pointer.setFocus emits wl_pointer.leave + frame to old surface, then wl_pointer.enter + frame (+ modifiers if keyboard is grabbed) to new surface

Pointer capture / window leave (REUSED approach)

  • canvas.setPointerCapture on pointerdown
  • pointerleave calls notifyKeyboardFocusOut (saves keyboard focus, releases depressed keys, sets needFocusInit=true for re-sync), resets buttonCount=0
  • DnD suppresses this handler mid-drag

Cursor images (REUSED approach)

CSS cursor: url(data:...) x y, auto:

  1. Native client calls wl_pointer.set_cursor
  2. Cursor surface commits a buffer
  3. CursorRole draws the bitmap to a hidden canvas
  4. toDataURL() produces a data URL
  5. Set on document.body.style.cursor

OS composites the cursor. No rotation/scale transforms.

NEW: Use SVG cursor sprites (e.g., from the Phosphor icon set, MIT) instead of bundled PNGs (Greenfield's bundled PNGs are AGPL-contaminated since they're inside @gfld/compositor).

Scroll wheel / touchpad (REUSED approach)

WheelEvent.deltaMode:

  • DOM_DELTA_LINE × 12 — discrete wheel clicks
  • DOM_DELTA_PAGE × surface size — page-sized scrolls
  • DOM_DELTA_PIXEL ÷ 3 — fine-grained touchpad

Each non-zero axis → wl_pointer.axis(time, axis, Fixed). Wayland version ≥5 also gets axis_source(wheel) + frame() (groups events).

Linux button codes (REUSED approach)

DOM event.button (0=left, 1=middle, 2=right, 3=Back, 4=forward) mapped to Linux input protocol button codes via the linuxInput table.

Focus management (REUSED approach)

DOM focus

The <canvas> owns keyboard focus permanently (tabIndex=1). Keyboard.setFocus(surface) emits wl_keyboard.leave / enter between native surfaces. Activation is driven by XdgToplevel / ShellSurface activation handlers.

DOM → Wayland focus

  • Browser DOM focus event → seat.notifyKeyboardFocusIn() (restores saved focus, re-syncs modifiers)
  • Browser DOM blur event → seat.notifyKeyboardFocusOut() (saves focus, releases depressed keys, sets needFocusInit=true)

Surface-level focus

  • Click on a surface → XdgToplevel activation request → desktopSurface.activate() → gainFocus() (refcounted via focusCount, fires surfaceActivationUpdated event to the user shell on 0↔1 transitions)
  • Popup grabs take keyboard + pointer focus via seat.popupGrabStart, validating the serial against the keyboard/pointer grabSerial

Input queue (REUSED approach)

InputQueue.ts coalesces events and dispatches them on the next RAF (InputQueue.drainQueues). Batches bursts of input events (e.g. fast mouse motion) into a single dispatch, avoiding redundant work.

FIFO per event type (motion, axis, key, button). Each dispatch calls the appropriate seat.notify* method.

ASCII event-flow diagram

┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER (DOM)                                                                        │
│                                                                                      │
│  <canvas tabIndex=1>  ─┐                                                             │
│  pointermove/dn/up  ───┤── createButtonEventFromMouseEvent()  ── ButtonEvent         │
│  wheel             ────┤── createAxisEventFromWheelEvent()    ── AxisEvent           │
│  keydown / keyup   ────┤── createKeyEventFromKeyboardEvent()  ── KeyEvent           │
│  pointerdown (touch) ─┤── createTouchEventFromPointerEvent() ── TouchEvent (NEW)     │
│                         │   • evdevKeyCode = EvDevKeyCode[event.code]                │
│                         │   • x11KeyCode   = evdevKeyCode + 8                        │
│                         │   • capsLock/numLock = event.getModifierState(...)          │
│  pointerleave / blur ──┤── seat.notifyKeyboardFocusOut() (saves focus, resets mods)  │
│  focus              ────┴── seat.notifyKeyboardFocusIn()  (restores saved focus)     │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ session.inputQueue.queue{Motion,Axis,Key,Button,Touch}
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ BROWSER COMPOSITOR  (InputQueue.drainQueues, scheduled on next animation frame)      │
│                                                                                      │
│  seat.notifyMotion / notifyAxis / notifyButton / notifyKey / notifyFrame / notifyTouch│
│         │                                                                            │
│         ├─► Seat.notifyKey      ─► Keyboard.grab.key(event)                           │
│         │                          └─ Keyboard.sendKey(event)                        │
│         │                              └─ keyboardResource.key(serial,time,evdev,st)  │
│         │                                                                            │
│         ├─► Seat.notifyButton  ─► runButtonBindings + grab.button(event)              │
│         │                          └─ Pointer.sendButton                             │
│         │                              └─ pointerResource.button(serial,time,         │
│         │                                       linuxInput[btn], state)              │
│         │                                                                            │
│         ├─► Seat.notifyMotion  ─► grab.motion ─► Pointer.moveTo / sendMotion        │
│         │                              └─ pointerResource.motion(time, sx_f, sy_f)   │
│         │                                                                            │
│         ├─► Seat.notifyAxis    ─► grab.axis ─► Pointer.sendAxis                     │
│         │       deltaMode→scale:  line=×12, page=×surface-size, pixel=÷3              │
│         │                          └─ pointerResource.axis(time, axis, Fixed)        │
│         │                              + axisSource(wheel) + frame() if v>=5         │
│         │                                                                            │
│         ├─► Seat.notifyTouch (NEW) ─► Touch.sendDown/Motion/Up                       │
│         │                              └─ touchResource.down/motion/up/cancel         │
│         │                                                                            │
│         ├─► Seat.notifyFrame  ─► pointerResource.frame()  (groups events)           │
│         │                                                                            │
│         └─► Seat.notifyModifiers ─ xkb.keyDown/Up(x11) ─► xkb_state_serialize_mods   │
│                                  └─ Keyboard.sendModifiers(modsDepressed,Latched,   │
│                                                              Locked,group)            │
│                                                                                      │
│  Keymap path (once per WlKeyboardResource):                                          │
│   Seat.notifyUpdateKeymap(nrmlvo) → createFromNames(rules,model,layout,variant,opt)  │
│   → xkb_context_new + xkb_keymap_new_from_names  (@gfld/compositor-wasm, MIT)         │
│   → xkb_keymap_get_as_string  ──► keymapString                                       │
│   Keyboard.sendKeymap(resource):                                                      │
│     inputOutput.mkstempMmap(new Blob([utf8(keymapString)])) ──► ProxyFD               │
│       (Python proxy creates memfd via os.memfd_create())                              │
│     keyboardResource.keymap(xkbV1, ProxyFD, size)                                    │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ client.connection.onFlush(wireMessages)
                               │  ─ one big Uint32Array, FDs serialized as ProxyFD JSON
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ WIRE  (ARQChannel = KCP-over-WebSocket, custom TS, clean-room)                         │
│   Browser  ⇄  ReconnectingWebSocket  ⇄  Internet  ⇄  aiohttp on proxy :8081           │
│   Message layout (Uint32Array):                                                      │
│     [0] outOfBandOpcode (0 = in-band)                                                │
│     [1] eventSerial                                                                  │
│     [2] = nFds, then for each fd: [len][JSON bytes, 4-byte aligned]                 │
│     then concatenated Wayland wire messages                                          │
│     + Authorization: Bearer <jwt>                                                    │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ NativeWaylandClientSession.onMessage (Python)
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ PYTHON PROXY  (aiohttp + pywayland)                                                  │
│                                                                                      │
│  SessionController endpoints:  /mkfifo /mkstemp-mmap /read-fd /close-fd              │
│                                 /read-fd-as-stream /write-fd-as-stream              │
│                                 /signal /channel /request-keyframe                   │
│                                                                                      │
│  mkstempMmap(ws):                                                                    │
│      receives Blob → os.memfd_create() → write bytes → return fd                     │
│      returns ProxyFD { handle: <native_fd>, type: 'shm', host: baseURL }             │
│                                                                                      │
│  onWireMessageEvents:                                                                │
│      for each ProxyFD in message:                                                    │
│          nativeFd = webFS.proxyFDtoNativeFD(proxyFD)  # identity when host matches    │
│                  └─ if foreign host  → spawn bridging pipe + WebSocket                 │
│      pywayland per-resource dispatcher fires                                         │
│      # For seat events, dispatcher is a no-op (events just pass through)              │
│      # libwayland-server sends via wl_resource_post_event_array                      │
│      flush_clients()                                                                 │
└──────────────────────────────┬──────────────────────────────────────────────────────┘
                               │ libwayland-server sends wire msg + SCM_RIGHTS FDs
                               ▼
┌─────────────────────────────────────────────────────────────────────────────────────┐
│ NATIVE  (Wayland client, sandboxed in bubblewrap)                                     │
│   wl_display.fd readable → libwayland-client demarshals                              │
│   wl_keyboard.keymap(fd) → mmap(fd) → xkb_keymap_new_from_string (native xkbcommon)  │
│   wl_keyboard.key / enter / leave / modifiers → client app                          │
│   wl_pointer.enter / motion / button / axis / frame → client app                     │
│   wl_touch.down / motion / up / cancel → client app (NEW: actually works)            │
└──────────────────────────────────────────────────────────────────────────────────────┘

Key answers (quick reference)

Question Answer
DOM→Wayland key event translation event.code → EvDevKeyCode → evdev keycode → +8 = x11KeyCode. DOM .key glyph ignored.
Modifier tracking @gfld/compositor-wasm xkbcommon state machine (REUSED MIT)
Focus: DOM ↔ native surfaces <canvas> owns keyboard focus. Keyboard.setFocus(surface) emits leave/enter between native surfaces.
Pointer capture / window leave canvas.setPointerCapture on pointerdown. pointerleave saves focus + releases keys.
Touch event mapping NEW: REAL implementation via pointerdown/pointermove/pointerup with pointerType === 'touch'. Greenfield has only a stub.
Pointer enter/leave between surfaces DefaultPointerGrab.focus() runs on every motion. Renderer.pickView walks view stack.
Pointer acceleration / cursor images No acceleration. CSS cursor: url(data:...) x y, auto. NEW: SVG sprites (MIT) instead of bundled PNGs (AGPL).
Scroll wheel / touchpad → wl_pointer.axis WheelEvent.deltaMode: LINE×12, PAGE×size, PIXEL÷3. v≥5 gets axis_source(wheel) + frame().
WASM xkbcommon? Yes — @gfld/compositor-wasm (MIT, REUSED). libxkbcommon 1.5.0 + embedded xkeyboard-config.
Keymap FD workaround for WebSocket Browser → /mkstemp-mmap WebSocket → Python proxy os.memfd_create() → ProxyFD JSON → libwayland SCM_RIGHTS to native app.
Primary selection (middle-click paste)? NEW: implemented via zwp_primary_selection_device_manager_v1 (Greenfield lacks).

File reference

File (custom, clean-room) Role
packages/compositor/src/input/Seat.ts wl_seat global
packages/compositor/src/input/Keyboard.ts wl_keyboard impl
packages/compositor/src/input/Pointer.ts wl_pointer impl
packages/compositor/src/input/Touch.ts wl_touch impl (REAL, not stub)
packages/compositor/src/input/Xkb.ts @gfld/compositor-wasm wrapper (REUSED MIT)
packages/compositor/src/input/InputQueue.ts Event coalescing
packages/compositor/src/input/InputOutput.ts FD abstraction
packages/compositor/src/input/events.ts Event wrappers
packages/compositor/src/browser/input.ts DOM keyboard events
packages/compositor/src/browser/pointer.ts DOM pointer events
packages/compositor/src/browser/touch.ts DOM touch events (NEW)

Reused (MIT)

File Role
libs/compositor-wasm/src/index.ts xkbcommon + pixman WASM (REUSED)

Python side

Component Role
pywayland.server.Client wl_client wrapper
pywayland.server.Resource wl_resource base (per-resource dispatchers)
xkbcommon PyPI package Optional, only needed for XWayland keymap compilation
Reference in new issue View Git Blame Copy Permalink