cboard-org/cboard

Augmentative and Alternative Communication (AAC) system with text-to-speech for the browser

Cboard - AAC Communication Board for browsers

Cboard is an augmentative and alternative communication (AAC) web application, allowing users with speech and language impairments (autism, cerebral palsy) to communicate with symbols and text-to-speech.

The app uses the browser's Speech Synthesis API to generate speech when a symbol is clicked. There are thousands of symbols from the most popular AAC symbol libraries to choose from when creating a board. Cboard is available in 40 languages (support varies by platform - Android, iOS, Windows).

We're using Discord to collaborate, join us at: https://discord.gg/TEH8uxh

How does it work?

This video shows Srna. She is one of the children who have received the Cboard Communicator thanks to UNICEF’s "For every child, a voice" project.

Translations

The app supports 40 languages. Languages were machine translated and require proofreading: if you want to help proofread, please use our translation management platform: https://crowdin.com/project/cboard

You do not need to be a programmer!

Translations play a major role in this project and they contribute a lot for the inclusion of children, specially in non developed countries. Please consider collaborating with us!

Translations for developers

In order to pull the latest translations from CrowdIn into the codebase, you can run yarn translations:pull. This will update all language files such as en.json as well as the central cboard.json file. Please note that this requires the CrowdIn API key to be available in the .private config file. Refer to Secrets Management. After the script completes, changes to the translation files will need to be committed to the repo by the usual process.

Getting Started

yarn start

Runs the app in development mode.
Open http://localhost:3000 to view it in the browser.

The page will reload if you make edits.
You will see the build errors and lint warnings in the console.

yarn test

Runs the test watcher in an interactive mode.
By default, runs tests related to files changed since the last commit.

Read more about testing.

yarn build

Builds the app for production to the build folder.
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.
By default, it also includes a service worker so that Cboard loads from local cache on future visits.

Cboard is ready to be deployed.

yarn build-cordova-debug

Use this to produce non-minified build for use in debugging within Cordova. It uses react-app-rewired & config-overrides.js to customize webpack operation without ejecting react.

See CBoard repo for packaging this CBoard application within Cordova.

Docker getting started

make image

Creates a Docker image with cboard built for production. The image is tagged as cboard:latest.

make run

Runs the cboard:latest Docker image on port 5000.

Secrets Management

Some external services have APIs we need to access, and these require API keys. To prevent open disclosure of these keys in the public repository, while still tracking them with the code, we encrypt some secrets into a GPG file. These files are env/local-private.gpg and env/prod-private.gpg.

In order to access the secrets, you must request the ENCRYPTION_KEY from @shaycojo and then run the decrypt script: ENCRYPTION_KEY={key-goes-here} yarn decrypt:local (or prod), which will create the file .private/local.js with the secrets in plain text where the scripts can access them. The files in .private should never be committed to the repository.

If you need to add or change a secret, make the change to the .private/local.js file, and then run the encryption script: ENCRYPTION_KEY={key-goes-here} yarn encrypt:local (or prod).

Note: These keys/secrets are not required to run or develop Cboard. They are used with scripts by some team members.

About the Cboard community

Thanks

Symbols sources

Mulberry

ARASAAC

Global Symbols

Translation

Crowdin - for providing the localization management platform.

Testing platform

Browserstack - for providing the automation infrastructure for testing.

Development

CSS Tricks - for providing feedback and support from the early stage.

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

Sponsors

Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]

📝 Legal & licenses

Copyright © 2017-2024, Assistive Technology LLC & Cboard contributors.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License version 3 as published by the Free Software Foundation.

• Code - GPLv3
• Mulberry Symbols - CC BY-SA 4.0
• ARASAAC Symbols - CC BY-NC-SA 4.0

cboard's Issues

Research Integration with CommuniKate

http://communikate.equalitytime.co.uk/

Emergency communication board

Emergency board accessible as a short-cut through the settings. Like how on iPhone you have the ungated opportunity to make an emergency call.

Disable save in <SymbolDetails />

The Label field in <SymbolDetails /> should be required. 'SAVE' button should be disabled until 'label' is truthy

Drag and drop

Hi @shayc! cboard looks like a really cool project. While I was playing around with it I figured that it would be great to have the option to reorder the symbols in the Output component after they were already selected using drag and drop. Is anyone working on such feature?

Fix Hebrew Niqqud

About Language & contribution

Hey @shayc, great work!

We analysed this app at Otsimo in office and really loved it. We made this aac app; https://github.com/otsimo/aac

After I leave Otsimo, I started working on another version and flow of AAC app (also better in performance) in my free time I would like you to check it;
https://github.com/btk/aac-native

Addition to that, I saw that this project generates its language data with translator, for example in Turkish version of it, there is problems that people in the spectrum might get effected.

In my project, we use a plain json file to store the locale data and its easier to edit. I created an issue in the project with label Hacktoberfest and asked people to translate the file into their native language.
btk/aac-native#2

In just 3 days, I saw great results. People like to contribute to get Hacktoberfest credits, also the language translation would get more accurate.

You might want to do something like this for this project. As I see you also store the data in json file. If you create a flow for people to pull request their changes or corrections instead crowdin.com, I think you will get results.

Also I personally did a lot of UX research about AAC apps if you have any questions, you can email me. Thank you.

Refactor: Board module

Further refactor of Board and BoardContainer for clarity and seperation

[material-ui] - Add Tooltips to UI interface

Add a tooltip to UI elements using material-ui Tooltip componet

All icon buttons without a visible label should be wrapped with a Tooltip componet:

• Lock/Unlock icon button
• Settings icon button
• Add icon button
• Edit icon button
• Delete icon button
• Back icon button

Integrate with Global Symbols

We want to integrate with Global Symbols - a symbol set for the Arabic community.

In the application a user should be able to search (and use) the new symbol set. The symbols should appear when a user searches for a symbol in the 'Add new button' menu.

This is the Tawasol API:

http://madaportal.org/tawasol/en/api-documentation/

The final goal is to display Arabic symbols from Tawasol API in case that the current language is an Arabic language.

Implementation steps

• check if current language is an arabic language when user tries to add a new folder or pictogram
• if arabic language is detected, search for symbols must include a call to the Tawasol API.
• Response from Tawasol must be filtered in order to display just the symbols from "source_id": "1"
• Tawasol results must be displayed in First Place

below an example of the final results:

App icon

Create an icon for Cboard!

Keyboard navigation with arrow keys

Enable navigation between symbols by using the arrow keys

Keyboard accessibility

Became aware of this awesome project from a tweet by Dan Abramov.

Had a look at it today and really awesome effort!!

I found a couple of (what I think could be) easily fixable issues that would improve the keyboard accessibility of the app.

First and foremost is the use of the following style in Board.css:

.Symbol:focus {
  outline: 0;
}

This, of course, removes the focus outline from tiles, making keyboard navigation pretty much impossible.

Is this something you want to address in the app? If so I want to see if I can get around to setting up a PR to address basic keyboard accessibility.

Support multiple grid sizes

Enhance grid capabilities to support multiple grid sizes.

Need to enable users to change grid size on demand, changing grid size should re-calculate symbols layouts.

Need to implement UI to change grid rows and columns, can be two input fields for rows/columns if a wysiwyg interface proves too difficult.

[Translation] Proof read machine translations

We've machine translated 3400 symbols from the Mulberry symbol set into 33 languages.
Most of the languages need proofreading, you can either proofread using Crowdin WYSIWYG interface (recommended) or by manually editing the .json files (no visual context - not ideal).

To proofread:

• Crowdin WYSIWYG interface
• json files: src/translations/{language}.json

Languages proofread:

• Arabic
• Bengali
• Chinese Simplified
• Czech
• Danish
• Dutch
• English
• English, United Kingdom
• Finnish
• French
• German
• Greek
• Hebrew
• Hindi
• Hungarian
• Indonesian
• Italian
• Japanese
• Khmer
• Korean
• Nepali
• Norwegian
• Polish
• Portuguese
• Romanian
• Russian
• Sinhala
• Slovak
• Spanish
• Swedish
• Thai
• Turkish
• Ukrainian
• Vietnamese

Eliminate deprecation warnings (PropTypes, createClass)

Deprecation warnings for propTypes and createClass

Warning: Accessing PropTypes via the main React package is deprecated, and will be removed in React v16.0. Use the latest available v15.* prop-types package from npm instead. For info on usage, compatibility, migration and more, see https://fb.me/prop-types-docs

Warning: Accessing createClass via the main React package is deprecated, and will be removed in React v16.0. Use a plain JavaScript class instead. If you're not yet ready to migrate, create-react-class v15.* is available on npm as a temporary, drop-in replacement. For more info see https://fb.me/react-create-class

Appears to be coming from react-grid-layout module. Related issue (PropTypes) in module repo.

Enable grid pagination

Add pagination capabilities to the grid.

We want to support non-scrolling usage via pagination. This will require the addition of two UI buttons to navigate forward/backward, and editing the component, currently we use react-grid-layout, under the hood of our custom component. RGL doesn't support pagination, so the solution should come either from a refactor to or forking react-grid-layout.

Active page should be persisted across page refreshes.

Prevent body scroll when dragging symbols

on iOS devices, when rearranging symbols the body also scrolls, preventing fluid dragging of symbols.
can try overflow: hidden on body when dragging.. not sure

Implement prettier config

https://github.com/prettier/prettier/releases/tag/1.6.0

Edit multiple symbols

We want to enable the user to edit existing symbols.
Right now the app enables the user to select multiple symbols but doesn't allow editing.

How the feature should work: once selecting the symbols (using the SELECT button), the user will be able to click on an "edit" button which will open a SymbolsDetails componet dialog, this component will present one or more symbols for edit, if there are multiple symbols, the component should be paginated using material-ui stepper, only one symbol per page, the user will be able to navigate between the symbols by clicking on "next" or "previous" buttons. When the user submits the dialog all changes will be saved (redux), if the user cancel, changes will be canceled.

FAIL src/containers/Settings/Speech/Speech.test.js ● renders without crashing

FAIL src/containers/Settings/Speech/Speech.test.js
● renders without crashing

TypeError: Cannot read property 'container' of undefined
  
  at Speech.render (src/containers/Settings/Speech/Speech.js:230:4597)
  at node_modules/react-test-renderer/lib/shallow/ReactCompositeComponent.js:795:21
  at measureLifeCyclePerf (node_modules/react-test-renderer/lib/shallow/ReactCompositeComponent.js:75:12)
  at ShallowComponentWrapper._renderValidatedComponentWithoutOwnerOrContext (node_modules/react-test-renderer/lib/shallow/ReactCompositeComponent.js:794:25)
  at ShallowComponentWrapper.performInitialMount (node_modules/react-test-renderer/lib/shallow/ReactCompositeComponent.js:361:30)
  at ShallowComponentWrapper.mountComponent (node_modules/react-test-renderer/lib/shallow/ReactCompositeComponent.js:257:21)
  at Object.mountComponent (node_modules/react-test-renderer/lib/shallow/ReactReconciler.js:45:35)
  at ReactShallowRenderer._render (node_modules/react-test-renderer/lib/shallow/ReactShallowRenderer.js:138:23)
  at _batchedRender (node_modules/react-test-renderer/lib/shallow/ReactShallowRenderer.js:85:12)
  at Object.batchedUpdates (node_modules/react-test-renderer/lib/shallow/ReactDefaultBatchingStrategy.js:60:14)
  at Object.batchedUpdates (node_modules/react-test-renderer/lib/shallow/ReactUpdates.js:97:27)
  at ReactShallowRenderer.render (node_modules/react-test-renderer/lib/shallow/ReactShallowRenderer.js:112:18)
  at ReactShallowRenderer.render (node_modules/enzyme/build/react-compat.js:171:37)
  at node_modules/enzyme/build/ShallowWrapper.js:128:26
  at ReactDefaultBatchingStrategyTransaction.perform (node_modules/react-test-renderer/lib/shallow/Transaction.js:143:20)
  at Object.batchedUpdates (node_modules/react-test-renderer/lib/shallow/ReactDefaultBatchingStrategy.js:62:26)
  at Object.batchedUpdates (node_modules/react-test-renderer/lib/shallow/ReactUpdates.js:97:27)
  at ReactShallowRenderer.unstable_batchedUpdates (node_modules/react-test-renderer/lib/shallow/ReactShallowRenderer.js:130:25)
  at performBatchedUpdates (node_modules/enzyme/build/ShallowWrapper.js:103:21)
  at node_modules/enzyme/build/ShallowWrapper.js:127:9
  at withSetStateAllowed (node_modules/enzyme/build/Utils.js:284:3)
  at new ShallowWrapper (node_modules/enzyme/build/ShallowWrapper.js:126:38)
  at shallow (node_modules/enzyme/build/shallow.js:19:10)
  at Object.<anonymous>.it (src/containers/Settings/Speech/Speech.test.js:12:23)
  at Promise.resolve.then.el (node_modules/p-map/index.js:42:16)
  at process._tickCallback (internal/process/next_tick.js:109:7)

Recording audio for symbols

Enable the recording of audio via the microphone when Adding/Editing symbols.

Can use material-icons "mic" icon for the button.
Should use Redux to save the recorded audio

Feature Request: add a second written language

it would be nice to have a setting for a second written language displayed so it can also be used as a fast visual translator.

Refactor: Split component & container

This is an omnibus issue to track conversion for major components to split the stateless UI component from the redux-connect container.

To Do:

• Board
• Settings
• Settings: Language
• Settings: Speech
• Settings: Backup
• Notification
• Header
• Grid
• ...more?

Simplify process for adding/updating translations

I'm working on setting up some scripts to help smooth for process of adding and modifying translation strings to Crowd In for approved contributors.

Right now the process is pretty clunky, and I think it can be a lot cleaner. I'm opening an issue just to share that this is WIP to avoid duplicated effort.

Enable the change of pitch and rate in speech settings screen

Add two new inputs to Settings/Speech screen that will represent pitch and rate.

Pitch and rate values can range between 0 (lowest) and 2 (highest), with 1 being the default pitch/rate for the current platform or voice.

You should use material-ui for the input fields
A slider component is most suitable for this type of settings but unfortunately material-ui beta 4 doesn't support it, can offer other components.

Walkthrough for new users

Hey all,
Congrats on the splendid project! I think y'all are doing really excellent work here. Poking around the app I found myself confused by how to access all of the functionality, and confused by what some of it does (eg, i'm very confused what the SELECT button does). I think this could be resolved pretty modestly by adding a walkthrough for first time users (as determined by a value held in localStorage or on the cookie). There are a ton of walkthrough libraries out there, such as react-joyride etc and it seems like they'd all do a great job here.

there should be a separate translation for cricket - sport and cricket - insect

I am working on the Dutch translations, but I've run into this problem. The Dutch translation for the insect should be krekel but for the sport it's just cricket.

Fix missing translations hooks in code (static strings

Settings > languages does not display all the available languages on windows/firefox

One user only had English.
I had English and French.
No error in the console

Complete development of SymbolDetails screen

Complete UI of SymbolDetails screen, mainly CSS work and remaining bugs (adding folders)

Create a welcome screen

WelcomeScreen component, displays for first time visitors (or when localStorage is purged)
It contains two pages, first page contains a short intro text welcoming the user, second page contains a form to create a profile (all client side - no server for now), should use material-ui stepper for page navigation.
UI examples: https://github.com/apl-devs/AppIntro

Form fields:

1. name
2. age
3. gender
4. language

The data will be then used to provide a more personalized experience and enable profile switching in the future

For the form fields use material-ui

Update: the WelcomeScreen component is ready, what still needs to be done is CSS styling, and connecting with a Redux store to save user profile.
Note: to style typography (font size etc.) use Typography component

After adding new symbols, selecting one of them select all the others

It happens because added symbols don't have ids so they all resolve to undefined and get selected.

More default board types: Core, Situational, Grammar

Integrate with Google Images (for personal use)

When adding/editing symbols allow the user to search for images from Google Images.
This is a new feature, please talk with me so we can come up with a UI design/direction, would love to hear your thoughts.

Very first output image doesn't display

After loading and tapping a tile, speech vocalized but associated image does not display.

Add basic auth/gating

App crashes when entering a created folder

I don't know if this functionality is supposed to exist yet.

Steps to reproduce:

1. Start from a fresh dev build (cboard 0.1.0)
2. On the homescreen, click Add button
3. Search for any picture and select it
4. Set the type to Folder
5. Save it
6. Click on the new folder

Result:
React error page with

Board.render
src/containers/Board/Board.js:212
> 212 |   const symbols = this.generateSymbols(board.symbols, board.id);

What's more, the app remains in this errored state until I call localStorage.clear() in my console (I think because board.activeBoardId points to an invalid board).

Expected behaviour:
You should be able to enter created folders.

Integrate with Cordova

To fully support offline on iOS devices we'll need to wrap the application in Apache Cordova.

Wrapping the application in Cordova means all 3400 symbols will come pre-bundled and making the app truly offline.

This will not replace our current web app but serve as an alternative and will live inside the same repo in a Cordova folder

Export / Import boards, for backup and sharing.

Initial version should just be able to output a json file and load it

Open Board Format

Implement OBF format for boards.json

Board button select/deselect on drag and drop

When dragging and dropping a board button, the button gets selected/deselected.
Correct behavior: button should not be selected/deselected after dropping.

Relevant Componets:

• BoardButton
• Grid

Create a startup screen for android/ios (web)

https://developers.google.com/web/updates/2015/10/splashscreen

Refactor app's files/components structure for better separation of concerns

Refactor components and containers for better separation of concerns.
On the way make sure to support Open Board Format (OBF) see issue: #35

symbol should speak label or text

Currently when you click on a symbol it speaks the content of the label property of symbol.
We should make changes so when a text property has a value, that value will get precedence over the label value when using speech synthesis.

The text value was planned specifically to override the label when using speech.

Should look something like this:

speakSymbol() {
  const text = symbol.text || symbol.label;
  this.speak(text);
}

Possible German vocalization setting bug

Described in channel:

see a standing problem, though - the text-to-voice bot you have has zero idea how to pronounce German. There are totally different word stresses and different pronunciations for lots of letters in German, and it doesn't know those, or at all what to do with the umlauts ('ü' comes out like a v, for example, so über is said like 'veeber'). A lot of the words will be very difficult to understand for a German speaker.

This sounds like an issue with the voice setting.

Allow full language codes for translations

Make the app support full language codes (en-UK and en-US)

Integrate arasaac.org symbol set

Integrate with ARASAAC symbols API. ARASAAC provides more than 10k symbols. This feature has huge value to our users.
Need to get a list of all symbols, translate them into 33 languages and create a dictionary. Used whenever a user wants to add/change an image.

ARASAAC:
https://beta.arasaac.org/pictograms/search/
https://github.com/juanda99/arasaac-frontend/blob/master/app/components/Pictogram/index.js

See #90 for UI example

General <Snackbar /> Component

Need to create a general (app wide) Snackbar component, to display various app-wide messages.
Should use material-ui snackbar

Integrate web app with an AAC switch - need access to one

Settings page initial setup

• add settings options to change 1) voice pitch and 2) rate
• add credits and legal information
• add basic style/layout