01 · Context
Connecting a recognition pipeline to a usable mobile loop
Continuous sign-language recognition must process a stream rather than a single isolated sign. The project combines an online CSLR pipeline with an iOS interface that captures input, displays translated text, and preserves a history of outputs.
The goal is research exploration, not a production accessibility claim. Real-world performance remains sensitive to lighting, occlusion, keypoint stability, and the gap between benchmark data and phone-camera input.
02 · Contribution
Team leadership, app flow, and continued online-recognition research
I led a three-student undergraduate team, coordinated with graduate collaborators and the PI, and connected recognition experiments to the mobile workflow. After the formal project ended, I continued the algorithm work independently.
- Reviewed the continuous sign-language recognition literature and translated pipeline constraints into implementation tasks.
- Implemented the SwiftUI flow that sends live keypoints and resized frames to the backend.
- Displayed returned text in the app and stored translated text with timestamps for later history lookup.
- Established dense, equal-budget uniform, random, and motion-adaptive baselines for the Phoenix-2014T online pipeline.
- Designed strictly chronological scheduling experiments that separate offline teacher labels from causal deployment inputs.
03 · System flow
A replaceable boundary between capture, inference, and interface state
- 01Capture
The phone collects camera frames for the live translation session.
- 02Extract
MediaPipe produces body and hand keypoints alongside resized image data.
- 03Recognize
The backend runs the online CSLR pipeline and returns an evolving text result.
- 04Persist
SwiftUI displays the output and records its text and timestamp in history.
04 · Evaluation
Measured efficiency gain with an explicit WER cost
The current frozen comparison uses all 519 Phoenix-2014T development samples. It evaluates dense inference, equal-budget uniform sampling, and a strictly causal motion-adaptive baseline under the same model and data assets.
| Metric | Dense B0 | Equal-budget uniform B2 | Motion-adaptive A0 |
|---|---|---|---|
| Processed windows | 55,775 | 37,706 | 37,615 |
| WER | 22.2311% | 22.5514% | 22.4179% |
| Word errors | 833 | 845 | 840 |
| Relative to dense | Reference | −32.40% windows | −32.56% windows · −29.48% wall time |
05 · Key decision
Keep uniform coverage; learn only where extra computation is useful
Random sampling performed substantially worse than uniform coverage, while an offline label-derived center oracle showed that better non-uniform allocation remains possible. Simple motion statistics and exact midpoint prediction did not recover that upper bound.
This evidence changed the research direction from tuning stride thresholds to a safer structure: a uniform coverage skeleton plus a causal phase, hazard, or decoder-aware utility bonus. Every learned policy must be compared with uniform at the same per-sample budget and evaluated on WER, wall time, and latency.
06 · Latest research diagnosis
Offline utility was predictable, but the learned scheduler failed in closed loop
A 16-step future-aware rollout teacher exposed a measurable scheduling ceiling: on a 526-sample partial32 calibration smoke test, the rollout oracle improved WER by about 0.89 points over online uniform. Among causal inputs, decoder-prefix state was more informative than the tested visual history.
That signal did not survive deployment. The combined predictor reached 4.9705% closed-loop WER versus 4.7736% for online uniform, with the difference confidence interval crossing zero. Its own actions shifted the state distribution away from the teacher trajectory, identifying covariate shift—not a successful scheduler—as the current result.
07 · Outcome & limits
A working research loop, not a finished accessibility product
The university project concluded with a final defense in May 2026. I now continue the online scheduling research independently, with negative results and protocol corrections treated as first-class findings.
- The iOS workflow, history flow, backend connection, and benchmark experiment are inspectable project outputs.
- Phone-camera input remains vulnerable to occlusion, lighting, and unstable keypoints.
- No usability study or reliable real-world accuracy measurement has been completed.
- Research code and experiment artifacts remain unlinked while publication and documentation boundaries are reviewed.