Prism

A gesture-controlled color picker powered by hand tracking.

Overview

A standalone web app that uses your device camera and hand gesture detection to pick colors from the real world. Point at something, pinch to lock the color, refine brightness and saturation with hand rotation. No mouse, no sliders, just your hand and the world around you.

Built entirely client-side with Next.js, MediaPipe Tasks Vision, TypeScript, Tailwind CSS, and Canvas 2D. No backend required.

Tech

Next.js · MediaPipe Tasks Vision · TypeScript · Canvas 2D. All inference runs locally in the browser, no backend, no server, no data sent anywhere.

Solution

Index finger pointing at a red object, cursor ring filled with sampled red

Point to sample

Point your index finger at any real-world object. A cursor ring on the camera feed shows the sampled color live.

Middle finger pinched to thumb, LOCKED badge visible, color frozen

Pinch to lock

Middle finger + thumb pinch locks the current color. Your hand is free to move without losing the selection.

Hand rotated, brightness slider responding, color wheel indicator moving

Rotate to refine

After locking hue, rotate your hand to dial in brightness, then saturation, one dimension at a time.

The Problem

Every color picker works the same way, flat wheels and sliders dragged with a cursor. But inspiration comes from the physical world. The workflow is always indirect: see a color, take a photo, open an editor, eyedrop it, copy the hex. Five steps for what should be one.

Why Point-to-Sample

The first prototype mapped hand rotation to a hue wheel. It worked technically but felt wrong, you're staring at a blue wall but spinning an abstract wheel. The mental model didn't match. So I pivoted: use the index finger to point at the real world and sample the pixel directly. This immediately felt right.

But it created a new problem: raw pixel sampling was too noisy. Camera sensor noise and surface texture meant a single pixel returned wildly different values frame to frame. To stabilize the sampled color, I averaged a 25×25 pixel region (625 pixels) around the fingertip, smoothed the fingertip position with a low-pass filter, and applied temporal color smoothing across frames.

Why Middle + Thumb Pinch

The obvious pinch, index finger to thumb, was already occupied. When your index finger is pointing at something to sample a color, curling it to pinch the thumb changes where you're pointing and destroys the sample. The two gestures fought each other.

Switching the lock gesture to middle finger + thumb solved it cleanly. Your index finger stays extended and aimed at the target while the middle finger curls down to meet the thumb. Two independent actions on the same hand, no conflict.

The Four-Stage Flow

Prism breaks color selection into staged commitments. Each stage locks with a middle+thumb pinch. This staged approach solves the core UX problem: once you commit a dimension, moving your hand can't undo it.

01SamplingIndex finger points, color tracks live

02BrightnessHue locked, hand rotation adjusts brightness

03SaturationBrightness locked, rotation adjusts saturation

04DoneAll locked, copy hex or fist to reset

Four-pill stage indicator showing progression from Sampling through Done

Making It Stable

Real-time gesture detection is noisy. Three jitter problems compounded:

Stage auto-cyclingPinch fired across multiple frames, blowing through stages instantly. Fixed with sustained-pinch detection (150ms hold required) and 800ms debounce.

Lock flickeringPinch distance hovering near the threshold flipped state every frame. Fixed with hysteresis: a tighter threshold to engage (0.25) and looser to release (0.40).

Color jumpingFingertip landmark jitters a few pixels per frame. Fixed with 25×25 pixel averaging, position smoothing, and 0.85/0.15 temporal color filtering.

No single fix worked alone. The layered approach was the minimum for usability.

What I Learned

"Hysteresis over thresholds"A single threshold always flickers when input hovers near the boundary. Two thresholds (enter vs exit) is the fundamental pattern for any continuous-input UI.

"Interaction conflicts are design problems"Index+thumb pinch was the obvious gesture but conflicted with pointing. The fix wasn't technical, it was rethinking which fingers do what.

"Stability is the feature"I spent more time on smoothing and debouncing than on any feature. For gesture interfaces, if it jitters, nothing else matters.