SignSpeak
Real-time ASL word recognition — skeleton in, English out
The Problem
Full ASL translation — connected signing, with grammar carried by facial expression and the space around the signer — is an open research problem. SignSpeak deliberately does the part that genuinely works well: isolated, word-level recognition, in real time from a plain webcam, and stays honest about that scope.
What I Owned
Solo build, end to end: the MediaPipe landmark pipeline, the bidirectional LSTM classifier, landmark-space data augmentation, the WLASL download/training tooling, and the OpenCV live demo — plus a Claude layer that turns recognized glosses into fluent English.
Tools And Stack
- Python
- PyTorch
- MediaPipe
- OpenCV
- Claude API
How It Fits Together
The core trick: never feed pixels to the model. Every frame — recorded WLASL clip or live webcam — is reduced to a normalized skeleton of hand and upper-body landmarks, which strips away signer appearance, background, and resolution, so a model trained on WLASL can recognize you. Pipeline: webcam / WLASL video → MediaPipe Holistic → normalized skeleton → bidirectional LSTM (temporal mean-pooling) → word → optional Claude pass for fluent English.
What Changed
- 44.1% held-out test accuracy across 18 signs — ~8× the 5.6% random baseline; landmark-space augmentation plus the bidirectional model lifted it from 35.3%.
- Runs immediately: a pre-trained model ships in the repo, and the demo falls back to an offline word-join when no Claude key is set.
- Scoped as a strong proof-of-concept; the clearest next win is simply more clips per gloss to close the WLASL→webcam domain gap.