SignIt! An Android Game for Bilingual Play

SignIt! is a sign language-based quiz platform co-designed with the Indian Deaf and Hard-of-Hearing community. Here, I’ll describe the three primary features of SignIt!.

Feature 1: Sign language-based quizzes

The quizzes contain a series of multiple-choice questions. The question interface contains the question sign language video and a grid of option sign language videos below it. Each video contains a hint button to toggle the visibility of the corresponding English translation. This interface supports the player in discovering and learning new vocabulary in both languages while playing quizzes.

Two screenshots showing the question interface of SignIt!. In the first screenshot, the sign language video of the question is at the top with a grid of four option videos below it. In the second screenshot is the same as the first one except the English translation of the Question and third option is shown.
Question Interface of a SignIt! quiz

Feature 2: Social Gameplay

There are two modes to play socially:

  • Live mode: The live mode allows multiple players to play a quiz together in real-time.
  • Group mode: The group members play group quizzes asynchronously and compete on a common leaderboard.
A screenshot of a score screen in live mode. It contains the points awarded in the current question, total score, and a leaderboard of three players in descending order of their scores.
Live mode score screen
A screenshot of the group details screen. It shows the group image, creator, common leaderboard showing top three players, and a list of group quizzes.
Group details screen

Feature 3: Content creation

SignIt! allows players to design their own quizzes based on their interests and knowledge, and share them with others. To create questions, players have two options:

  • Create questions from scratch: The player needs to add the question and the options in both sign language and English. They also need to mark the correct answer among the options.
  • Find questions from a pre-made repository: The player can select the text of multiple questions from the repository in the app and add them to their quiz. They need to add the corresponding sign language videos for each question and their options.

Each question of a quiz must have at least two options. To record the sign language videos, the player will be guided by a video recording interface that helps them position themselves correctly before the recording starts.

Quiz creation leads to labelled sign language data collection. An interesting enhancement to this feature could be to offer customizable avatars for quiz creators. This would add personalization and help maintain anonymity by representing videos through these avatars.

A screenshot of the create quiz interface with options to change the quiz image, add quiz title, and two buttons to create and find questions respectively.
Create quiz screen
A video recording interface with the prompt 'Good' to indicate that their upper body is visible
Video recording interface

Thank you so much for reading! This work would not be possible without the support of the National Institute of Speech and Hearing (NISH). NISH is a comprehensive multi-purpose institute focusing on the identification, intervention, rehabilitation and education of individuals with disabilities.

We’re in the process of open-sourcing the code of SignIt! This platform is designed to be language-agnostic, accommodating any sign language and spoken language (both of which need to be specified while creating the quiz).

We envision organizations setting up their own storage accounts for data and hosting SignIt! independently. The anonymous data from the videos and their labels could be streamlined into an open-source repository. This could serve as a valuable resource for advancing research in sign language recognition using machine learning.