## What it is
Gesture Synth is a chord instrument you play with your hands in front of a camera. Your left hand picks the chord — scale degree from which fingers are up, major or minor from the palm tilt. Your right hand shapes it — inversion, sevenths, octave with the thumb, filter with tilt, volume with height. A loop pedal records gesture performances into four tracks against a metronome, and everything you play is drawn back at you as a harmony-aware scene.
## The idea
I wanted an instrument where the theory is the interface: four fingers and a tilted palm is a IV major, add three fingers on the right hand and it's a IV maj7. The visuals aren't decoration — the ring walks with the chord root, the note constellation is the actual voicing, and the circle of fifths shows where you are in the key. You learn harmony by moving your hands and watching what lights up.
## How it's built
It runs from one codebase as two things: a Tauri desktop app (native audio thread, MIDI out, ffmpeg export) and a web app / installable PWA at the same URLs, offline after first load. Sessions are byte-identical across both, so a loop made in the browser opens on desktop and back again. Under the hood it spans an unusual stack:
- Rust compiled to WASM for the hot path,
- Svelte + Three.js for the reactive 3D scene,
- a Zig DSP kernel,
- a Gleam service, and
- MediaPipe hand-tracking feeding the whole thing.
The desktop build opens in Stage — a game shell with song select on the circle of fifths and rated sets; the website opens in Studio, the workstation layout. One switch in Settings moves between them.
## Try it
There's a live web build — gesture-synth.onrender.com — that runs in the browser with your camera. Give it a hand and start with a IV maj7.