mlh-fellowship/ace

Explorer 2.1.1, Sprint1 Project: ACE (Accessible Chess Experience) by Atreya, Saumya and Suhani.

ACE - Accessible Chess Experience

ACE (Accessible Chess Experience) is a real time voice powered chess application which lets you play online with your friends. The game has features like moving pieces, finding pieces on board, getting board summary, repeating opponent moves, and much more, all implemented using voice commands.

Inspiration

We looked online and found out that two of the most popular chess websites, lichess and chess.com, have no voice options for the visually impaired. We wanted to create something, that can help those who are visually impaired to have a complete experience without the hassle on relying on screen readers for every single move. And thus originated the idea of voice powered chess, ACE.

Features

1. Play chess online with your friends using voice commands

2. Move chess piece using voice

3. Find piece on a particular square

4. Get chessboard summary

5. Get voice alerts for invalid moves, check and checkmate situations

Tech Stack

1. Frontend - ReactJS
2. Backend - NodeJS
3. Speech-To-Text - Azure Speech SDK
4. Real Time Communications - Web Sockets

Try It Yourself

Here is a link to the website

Contributing to ACE

1. ACE uses Azure text to speech API for speech recognition. The project requires a speech resource on Azure portal (to get access to SPEECH_KEY and SPEECH_REGION). However if you are interested in contributing to the project but unable to make an azure speech resource, we will share azure credentials to get you started on contrinuting to this project. You can join our discord server to get access to azure speech key and region.

2. All issues for this project are created here on this frontend repository, however you can make changes to backend as well and submit PR on backend repository as well.

Dev Setup

Prerequisites

• NodeJS
• npm
• Azure account and Azure Speech Resource

Start the server

Link to backend repository

1. Clone the above repository
2. cd into the ACE-Backend folder
3. Create a .env file and add the SPEECH_KEY and SPEECH_REGION from your azure speech service credenials. (Refer to .env.example file)
4. Go to the terminal and run the following commands

npm install
node app.js

The server will be started at localhost:8000

Start the client

1. Clone this repository
2. cd into the ACE folder
3. Open the config.js file in src/services folder
4. Replace CLIENT_URL with http://localhost:3000
5. Replace SERVER_URL with http://localhost:8000
6. Go to the terminal and run the following commands

npm install

npm start

Visit localhost:3000 to play the game

Screenshots

ace's Issues

change speak position speech

• make it speak like "white knight on d5"

Implement resign game using sockets

Allow user to resign game and deliver information to opponent

Command: Repeat Opponent Move

Repeats the Piece moved by the opponent, from which sq (source) and dest sq.

Fixing inconsistencies with dialect settings

For some reason, the game is in a german dialect on my browser (not sure if this happens for other people as well).

german.mp4

Implement chat/voice call

We can implement a chat or voice call feature so that two players can communicate with each other in real-time while playing. Maybe we can use the webspeech API to speak out the contents of the chat as well.

Create Speech+Command structure

create a structure for side buttons and connect speech with the side buttons for specific commands.

Integrate basic chess functionalities on the board

Find "XY", X = Letter, Y = Number will output the piece on the board

Output: "Black pawn" or "White Queen"

Keyboard input for moves

Add a box where the user can put in the row and column of a square to specify a square. Can be used for Find Piece and Make Move.

When those two functions are selected, maybe autofocus on the text box.

Voice prompts on repeat

Sometimes, the computer voice will repeat itself several times.
I've included an example of the voice repeating the "copy link" prompt:

repeat.mp4

Make chess two player

Make two player chess using sockets. Add link to backend repository in readme

Research the stockfish API for chess moves

Create a basic repository structure using create react app

Research about various speech recognition APIs

Create a home/landing page with a start button

Create a home/landing page with decent chess related UI and a start button which directly generates a link.

Break the move piece loop

When user makes a move, if the computer does not recognize a piece it will keep asking the user the piece name in a loop. Add a condition such that if the piece is not recognized more than three times, the loop breaks and the user is required to make the move all over again

Remove username entry

Remove the requirement for a username and allow direct link generation after pressing start on home/landing page.

Handle case when speech is not recognized or incorrect command is spoken

Computer should speak out error message in both cases

Speech misclassification

We are currently using the Azure speech API. It tends to misclassify words frequently when different accents are used. Maybe we can:

1. Further train the azure model endpoint using custom speech samples and use that endpoint in the game.
2. Hardcode some rules such as changing misclassified words like "Ivan" to E1.
3. Explore other speech APIs, for example GCP.

If anyone needs Azure credentials, please contact us here:
https://discord.gg/EmtXC9xug6

Create matches with a timer

Both players get time limit of 20 mins, the timer starts when it's player's move. When a player runs out of time, the opponent wins the game.

Sound alerts for moves/result/checks

• Sound alerts for successful/invalid move.
• Sound alerts for game result i.e. winning/losing/draw.
• Sound alerts for check/checkmate.

Add confirmation from the computer every time a user issues a move piece command.

Move Piece "K", Source: "XY", Destination: "AB"

Computer: Do you want to move "K" from "XY" to "AB"?

If the user says yes:
Make the move
If no:
Erase the move and go back to asking which piece, and source + dest sqrs

Keyboard Interrupt to stop speak positions

Currently, when speak positions is called there is no way to interrupt the computer speech. It will continue till all the pieces on the board are spoken out. To fix this, when speak positions is going on, user should be able to press esc key to break out of the function.

Icon to denote that voice input is active.

When the user presses Spacebar or voice input is active, we can add a large blinking icon of sorts to signify that the input is active.

Opponent can make moves on the player's behalf.

Maybe output something like "Not your move"

Improve UI of landing page

The landing page should consist of the rules and icon of the project. Revamp the landing page UI to make it look better.

Command: Current piece positions

Will speak out all the current piece coordinates on the board for both colours.

Move own piece

Three stage Process:

1. Computer: "Piece?"
   User: "Knight"

Note: This preferably shouldn't apply for the King and Queen since they are unique pieces. We can add this small tweak later iafter everything else is working.

2. Computer: "Source Square?"
   User: "A1"

3. Computer: "Destination Square?"
   User: "H8"

Prompt invalid move if the move is not possible, "Currently in check" if the user is in check.

Make chessboard responsive to fit on all screen sizes

Currently, the chessboard fits well on larger screens but not on smaller screens. user would need to scroll to see a complete chess board

Allow user to restart game with same opponent

Currently, when a game gets over the user does not have the option to restart the game with the same person.
We can create a workflow where when the game ends, a pop up shows where user has the option to restart the game. The opponent is asked permission for rematch, if opponent agrres, a new game starts on the same link i.e. the board resets.
Link to backend repository: https://github.com/suhanichawla/ACE-Backend

Create basic chess layout using react

Offer Draw to the player

The user should be able to offer draw to the player. If the opponent accepts draw, the user and opponent should be alerted that the game has ended in a draw
Will require socket implementation on the backend
https://github.com/suhanichawla/ACE-Backend

Freeze game board when match ends.

When the game ends (either checkmate or resignation), the game board should be frozen with no further moves being made even if both the players are still on the page.

Options for Resigning, and offering a draw.

Preferably add double confirmation. Since currently it's only two people playing via a link, resign immediately refreshes the board and a new game is started.

If a draw is offered, a draw request is sent to the opponent. If confirmed, then new game is started. Else, game continues.

Create user commands layout as a side panel with buttons