# SignLink — recognising hand shapes at my first hackathon

By Arya Garg

Canonical source: https://aryagarg23.com/writing/signlink

Archive date: 2023-01 (historical site date; not a verified publication date)

Register: Technical

Tags: HACK

Section: [Hackathons](https://aryagarg23.com/about#hackathons)

## Main

Arleen Monteiro randomly invited me to BoilerMake X about four months into college. Reeve Fernandes, Arleen and I went in wanting to collect an ASL dataset and train a model on it, then put the recognition into a video call.

I had basically no idea what was possible with a computer back then. I was mostly hands off on the rest of the app. I ended up on the ML setup because that was the part I wanted to figure out.

Most of the weekend was me failing to get that version working. My laptop was too weak for what I was trying to run, and the TensorFlow and library setup kept turning into its own problem. With four or five hours left we finally stopped digging.

MediaPipe could already give us the shape of a hand, so we changed the problem. Instead of training the computer to recognise signs, we took the landmarks, calculated the finger and joint angles, and compared them against a set of hand shapes we'd defined ourselves.

It only worked for some letters, and it could break on a different hand. Anything based on movement was basically out. But the browser call worked, and the recognition worked well enough to demo.

Before that weekend, getting stuck mostly meant going deeper into whatever I'd already picked. At BoilerMake we ran out of time and had to search broader. I'd probably make some A* versus BFS analogy for that now. I definitely wasn't making that analogy four months into college.

[Hand-angle sketch: a synthetic hand runs the landmarks -> angles -> letter-dictionary pipeline with no camera. The optional camera downloads model resources and requests camera access; everything stays in your browser]

## Technical

| SignLink flow |
| --- |
| Camera |
| ↓ |
| OpenCV frame capture |
| ↓ |
| MediaPipe hand landmarks |
| ↓ |
| Relative finger / joint angles |
| ↓ |
| Hand-written pose dictionary |
| ↓ |
| Recognised letter |
| ↓ |
| PeerJS / WebRTC call interface |

![Camera frame to caption: MediaPipe landmarks, joint angles, nearest ASL pose](https://aryagarg23.com/figures/signlink/recognition_flow.png)

MediaPipe supplied 21 hand landmarks from the camera feed. The matcher used 20 of those points and calculated relative joint angles with vector geometry and trigonometry. The resulting angle vector was compared against a hand-written table of supported ASL letter poses.

The recognised pose was mapped to a letter and shown inside the browser video-call interface. PeerJS handled the peer connections and WebRTC carried the video/audio stream.

The original trained-model path was dropped during the hackathon after the team couldn't get the TensorFlow training/toolchain setup running on the available hardware. Static poses were the practical limit of the submitted version, and motion-dependent signs such as J and Z weren't covered by the same lookup method.[^signlink-technical-1][^signlink-technical-2]

[^signlink-technical-1]: [SignLink — Devpost](https://devpost.com/software/signlink)

[^signlink-technical-2]: [Original repository](https://github.com/Aryagarg23/SignLink)

## Personal

Greyhound six hours there. Thirty-six hours at Purdue hanging out with my buddies. Greyhound six hours back.

I got home at around three on Monday morning and went to class on a snow-packed day. I chase that sort of accomplishment all the time.