← Back to the app
ASL Fingerspelling Recognizer
A browser tool that watches your webcam, reads the American Sign Language
fingerspelling alphabet (A–Z), and helps you practise it. It runs
entirely on your device. This page is an honest account of what it can and
can’t do.
What it is
An undergraduate student project: a single web page, no install, no
account, no server. Point a camera at your hand and it names the letter
you’re forming. On top of that sits a small set of practice modes —
a productive one (you sign, it checks) and a receptive one (it shows you a
sign, you read it).
It was built to learn how these systems work, and to be genuinely useful
to people learning fingerspelling. It is not a commercial product or
an interpreter.
What it does
- Recognises the A–Z alphabet — from a live webcam, in about half a second per letter, on desktop or phone.
- J and Z — the two moving letters, traced as a stroke rather than a held shape.
- Practice mode — pick a letter, get a live “readable / not yet” meter and a correction guide drawn on your hand.
- Challenge mode — a timed game: a random letter, a shrinking clock, a streak.
- Read mode — an animated hand spells a word, you type what you saw. Includes a Course that unlocks letters in a teaching order (easy, distinct shapes first) with progress gating.
- Spell mode — fingerspell continuously into a running transcript, with a “spell this word” drill you can point at your current lesson.
- Fluid + speak (beta) — spell a sentence, and the app tries to segment it into words and read it aloud.
What it does not do
- It is not ASL. Fingerspelling is a small part of the language — used for names, brands, and words with no sign. This app does no word-signs, no grammar, no facial expression, no body movement. A fluent signer does not spell out sentences.
- It is not native speed. It works on careful, deliberate spelling with a brief hold on each letter. Real conversational fingerspelling is a blur of transitions that a frame-by-frame classifier like this one cannot follow. That needs a different kind of model (see below).
- Digits 0–9 are not supported yet. Which means phone numbers and addresses — a huge share of real fingerspelling — don’t work.
- Names and rare words won’t auto-correct. The sentence feature leans on a dictionary; anything outside it stays as raw letters.
- It has mostly been tested by one person. Accuracy figures are from held-out data, not from many signers, hand sizes, lighting conditions, or a range of skin tones. The hand detector is Google’s pretrained model, not something this project can vouch for across all users.
- “Fluid + speak” is unproven. The pieces are built and unit-tested, but the live webcam → sentence → speech path has not been validated on varied setups. It is labelled beta for that reason.
How it works
- Hand tracking — Google’s MediaPipe
HandLandmarker finds 21 points on your hand in each video frame.
- Normalise — the points are recentred on the wrist and scaled to hand size, so distance from the camera and which hand you use don’t matter.
- Classify — a k-nearest-neighbours classifier compares that shape to a labelled reference set. Small learned “refinement heads” break the ties that k-NN gets wrong (M/N, D/O/C).
- Stabilise — a letter is only shown once the same guess holds for a few frames, so it doesn’t flicker.
On held-out test data it lands the right letter about 97% of the time,
with every letter at 92% or better. That number describes clean, careful
signing on data drawn from the same source as the training set — treat it
as a ceiling, not a promise. It is plain JavaScript: no machine-learning
framework at runtime, no build step, no network calls once the page loads.
Privacy
The camera feed is processed in your browser and never leaves your device.
Nothing is uploaded, recorded, or sent anywhere. There is no analytics, no
tracking, and no account. Your progress (letters practised, lesson unlocks)
is stored only in this browser’s local storage; clearing site data resets it.
Where it’s headed
The honest limits above point at one direction: to read fingerspelling at
real speed, the app would need a sequence model — one that looks at a
window of motion and learns the transitions between letters, rather than
classifying single frames. That has been done on-device before (the Google
ASL Fingerspelling Kaggle competition, whose data comes from 100+ Deaf
signers), and it is the natural next stage for this project.
Alongside that: digit signs, and context-aware word lists for names,
addresses, and numbers so the sentence feature stops guessing.
The bigger open question is whether fingerspelling-to-speech is even the
right problem to solve, and that is not a question code can answer. The
goal of showing this project to ASL educators and the Deaf community —
RIT/NTID in particular — is to find out, and ideally to do the next part
with people who sign, rather than for them.
Credits & data
- Hand tracking — MediaPipe Tasks (Google), loaded from a CDN.
- Training data — the ASL Alphabet dataset (grassknoted, on Kaggle), converted to hand-landmark vectors.
- Word list — an English word-frequency list derived from Peter Norvig’s n-gram data.
- Inspiration — PopSign, a Georgia Tech × RIT/NTID project, showed that a student-scale sign-language tool built with the Deaf community is possible.