How we built it
Our project required several technical aspects to communicate in order to create the complete tech stack. The computer vision model uses google’s mediapipe hand model interpret 2D camera information representing 3D space. Mediapipe is robust enough to handle potential occlusion of key features and transformations of the hand due to its inherent knowledge of the anatomical structure of the hand. A webcam captures the player’s gestures, which are subsequently analyzed by a python script. After interpreting the video feed, our python script emulates a keyboard typing a character in order to send commands to our videogame. The game itself was developed in Unity using C#. Along the way numerous multimedia programs were used to enhance the training, design, and presentation process. All of our graphics were made from scratch in a variety of programs. Every ASL sign in the game was hand-drawn in Procreate from reference photos. They were then imported into Adobe Photoshop where they were “retrofied” with texture, colors, and fun concepts (Note: The creators of this project do not condone the use of Comic Sans outside of ironic, quirky contexts.). Pixelated icons were made from scratch in Piskel, a browser-based pixel art program.