SignLink — recognising hand shapes at my first hackathon

Arleen Monteiro randomly invited me to BoilerMake X about four months into college. Reeve Fernandes, Arleen and I went in wanting to collect an ASL dataset and train a model on it, then put the recognition into a video call.

I had basically no idea what was possible with a computer back then. I was mostly hands off on the rest of the app. I ended up on the ML setup because that was the part I wanted to figure out.

Most of the weekend was me failing to get that version working. My laptop was too weak for what I was trying to run, and the TensorFlow and library setup kept turning into its own problem. With four or five hours left we finally stopped digging.

MediaPipe could already give us the shape of a hand, so we changed the problem. Instead of training the computer to recognise signs, we took the landmarks, calculated the finger and joint angles, and compared them against a set of hand shapes we'd defined ourselves.

It only worked for some letters, and it could break on a different hand. Anything based on movement was basically out. But the browser call worked, and the recognition worked well enough to demo.

Before that weekend, getting stuck mostly meant going deeper into whatever I'd already picked. At BoilerMake we ran out of time and had to search broader. I'd probably make some A* versus BFS analogy for that now. I definitely wasn't making that analogy four months into college.