01 · Context
Connecting a recognition pipeline to a usable mobile loop
Continuous sign-language recognition must process a stream rather than a single isolated sign. The project combines an online CSLR pipeline with an iOS interface that captures input, displays translated text, and preserves a history of outputs.
The goal is research exploration, not a production accessibility claim. Real-world performance remains sensitive to lighting, occlusion, keypoint stability, and the gap between benchmark data and phone-camera input.
02 · Contribution
Team leadership, app flow, and adaptive inference experiments
I led a three-student undergraduate team, coordinated with graduate collaborators and the PI, and connected research experiments to the mobile workflow.
- Reviewed the continuous sign-language recognition literature and translated pipeline constraints into implementation tasks.
- Implemented the SwiftUI flow that sends live keypoints and resized frames to the backend.
- Displayed returned text in the app and stored translated text with timestamps for later history lookup.
- Evaluated adaptive stride and time-aware voting on the Phoenix-2014T test set.
03 · System flow
A replaceable boundary between capture, inference, and interface state
- 01Capture
The phone collects camera frames for the live translation session.
- 02Extract
MediaPipe produces body and hand keypoints alongside resized image data.
- 03Recognize
The backend runs the online CSLR pipeline and returns an evolving text result.
- 04Persist
SwiftUI displays the output and records its text and timestamp in history.
04 · Evaluation
Measured efficiency gain with an explicit WER cost
The full comparison ran both configurations serially on the same healthy GPU across all 642 Phoenix-2014T test samples. The adaptive configuration paired motion-sensitive stride selection with span-weighted voting based on real frame distance.
| Metric | Fixed stride = 1 | Adaptive + span-weighted | Change |
|---|---|---|---|
| Processed clips | 64,627 | 43,932 | −32.02% |
| End-to-end runtime | 1547.29 s | 1101.12 s | −28.84% |
| Best WER | 22.00 | 22.73 | +0.73 points |
| Equivalent throughput | 1.00× | 1.41× | +40.52% |
05 · Key decision
Adaptive sampling also required time-aware decoding
Fixed-index voting assumes evenly spaced clips. Once stride changes with motion, that assumption breaks: a fixed number of clips can cover very different real time spans. In the adaptive run, ordinary window voting degraded sharply at larger windows, while span-weighted voting restored the best adaptive WER to 22.73.
This made time-aware voting a necessary companion to adaptive sampling rather than an optional post-processing improvement.
06 · Outcome & limits
A working research loop, not a finished accessibility product
The university project concluded with a final defense in May 2026. I completed the 642-sample benchmark as a follow-up experiment after the defense; no active continuation is currently claimed.
- The iOS workflow, history flow, backend connection, and benchmark experiment are inspectable project outputs.
- Phone-camera input remains vulnerable to occlusion, lighting, and unstable keypoints.
- No usability study or reliable real-world accuracy measurement has been completed.
- The repositories remain unlinked while publication and documentation boundaries are reviewed.