merkie/freespeech

A free program designed to help the non-verbal.

FreeSpeech AAC ·

FreeSpeech AAC is a free and open-source assistive communication app written in TypeScript. Talk with us on Discord!

FreeSpeech AAC is committed to providing the best AAC experience to users, this is accomplished by packaging FreeSpeech with technologies sourced from AWS, Micorosoft Cognitive Services, and Google Cloud that give users cutting-edge AI generated voices. FreeSpeech is also putting artificial intelligence to use by relying on the OpenAI GPT-3 model to handle complex textual analysis tasks along with phrase and word conjugation. FreeSpeech's client is built on top of Svelte, a blazingly fast front-end UI framework.

I created FreeSpeech initially for my sister who is nonverbal, but since then the project has been actively gaining users as the application also serves as a viable alternative to many other expensive pieces of AAC software. Keep up with the latest FreeSpeech news on the blog portion of the website!

Hosting

• Clone and enter into the project directory git clone https://github.com/merkie/freespeech freespeech && cd freespeech
• Install dependencies yarn install
• Run the dev command yarn dev

Sponsors

Special Thanks

Thank you to CNN and the BBC for covering the FreeSpeech project! Also huge thank you to Icons8 for providing all the icon assets for the project. Click on the logos to see more.

Project Authors

• Archer Calder - Founder/Lead Developer - GitHub Donate [email protected]
• Bailey Townsend - Core Team Member - GitHub Linkedin

None of this would be possible without you all. Thank you!

Made with contributors-img.

License

Free Speech AAC is MIT licensed.

freespeech-api

freespeech's Issues

On "Contributing.md" needs some data to be filled

Describe the bug
On Freespeech/Vueapp/CONTRIBUTING.MD on line 69 there is a [INSERT EMAIL ADDRESS] to be filled
To Reproduce
Steps to reproduce the behavior:

1. Go to 'Freespeech, then go to inside Vueapp folder'
2. Click on 'CONTRIBUTING.MD'
3. Scroll down to 'line 69'
4. See error: there is a [INSERT EMAIL ADDRESS]

Expected behavior
An email address to contact the 'alma máter' of this project
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: MacOSX Catalina
• Browser: Chrome
• Version: 79

Smartphone (please complete the following information):

• Device: does not apply
• OS: does not apply
• Browser: does not apply
• Version: does not apply

Additional context
Does not apply. There is no break on code. No collateral threats

Draft Overview Of Features A Backend Would Provide

This is a draft of features that creating a backend will provide. This is a high overview and is subject to change. This is non technical and only focus on end user features. These are vague on purpose. Individual issues will be created later to fully flesh out the ideas here. Suggestions are welcome, but please keep them constructive.

User account support

• Ability to create an account
• Save settings across devices, including keyboards

Keyboard store

• A place where users can post Keyboards to share with each other for free
• Ranking/Review system
• Ability to have a post with what the keyboard offers and details about it
• Trending section

Image storing

• This will also be a place to store images that are used in the custom keyboards.

This is a pretty short list and i expect it will grow. @Merkie will play a huge role in what these features will be and what they will look like along with the rest of the community.

New Editor

We're going to have to expand the current editor to support some new features.

I see two possibilities for this, the first one being a "safe" editor that will just work for any average user. This would likely consist of basically what we have now (editing tiles, adding navigation) along with a screen to manage different pages. The point of this would be that all pages are uniform. All pages are 5x12 tiles to utilize all screen space and standardize tile placements. Think of the app as typing on a keyboard, just with more buttons. If every page contained a different amount of buttons, the alignment would be off and users would have to really look at where they're typing. If all pages are 5x12, that wouldn't be an issue. Just as an example, we could standardize that the home/tiles navigation button will always be the bottom-right most tile. If all pages are 5x12 tiles, that navigation tile would always be in the same spot on the screen. If we made it so every page could have a different number of tiles, that uniformity would stop.

Another possibility is a "non-safe" editor that would give a lot more power to the users. Users could add/remove tiles at will, they could also do more powerful adjustments in the future. This gives more power to the end-user making the app more customizable, but you're risking a lack of uniformity and you're making the app more complex. I'm all down for supporting both, but I think the safe option should be the default.

Though, I think it's fitting to maybe adopt a ribbon-style editing menu like the one pictured. Would make the app feel more consistent and it would be easier to add new features I think.

Add any other context or screenshots about the feature request here.

Let me know what thoughts you guys have on this and what features you think should be added. It's a difficult balance of maintaining an easy-to-use app and an app with all the customization users might want.

Allow more than just image location for buttons

Haven't looked to see if this has been added, but why not add multiple button icon sources such as Font Awesome to present a more consistent feel for the user.

Invalid Discord invite link

Describe the bug
The invite link listed under the "Community" header in the readme (https://discord.gg/76PrNC6) is expired.

I have tried to join from both the app and the browser and the app, and both displayed the following screenshot.

Additional context
I'm new to this community and would love to gain an understanding of the current work being done in order to contribute where necessary. Thanks!

Make clear & speak buttons boxes w icons as to avoid user confusion.

Tile Screen is Not Responsive, Only Detects Device

The tile screen isn't responsive to changing the size of the window, and only changes the number of columns displayed when the device changes

Hitting a button linked to a page does not open new page in sentence building mode

Switch from Ko-Fi to Open Collective

Obviously Ko-Fi is wonderful, but contributing to the project shouldn't be just donating to a couple people. There can be more expenses that can easily be funded and tracked transparently.

Open Collective allows the project to both raise money and track expenses and allow people to request payment for expenses. All paid through a funding source through them. It helps people donating and supporting the project to know that funds are both secure and transparent. Webpack uses Open Collective and you can view their expenses here

Edit:
I know the biggest question is "why?" Well, imagine I write a backend, I host it...$25 a month it costs me. Say Merkie gets $50 a month in donations. Am I supposed to go to him every month and ask for it? I could, but that could be inefficient. Instead I could request an expense payout on Open Collective and once approved by the team, it would automatically be paid to me and added to the public expense report.

Need to get real button layout, one in place is a placeholder.

So I was considering getting an actual Speech Language Pathologist to help create a real layout for the buttons, but I am considering doing it myself. I'll make one or two versions and see what you guys think.

use github actions + now.sh to automatically publish

I can contribute to this if you're interested, now.sh uses lambda function, and it should be almost free as you have service worker implemented, (even if not, depending on how many users, it should still be cheap)

Remove "disable sentence mode" button and put it in settings. Have it on by default.

Add visual indicator when toggling edit mode

When switching between edit mode and live mode there should be some way for the user to know (like an animation, or a navbar on top) to know their in edit mode...it's already happened to me. I thought I was in live mode and I was in edit mode.

Vue Unit Testing

Unit testing needs to be implemented and would be a great contribution to the project.

Thoughts on backends?

Saw #45 and some comments there about backends so I thought I'd just bring the discussion here so everyone could talk more about it.

What backends are y'all looking at using? Just from what I've looked at, the app just seems to be a static site, so would a backend even be necessary if you just use LocalStorage to hold user preferences? If you use a static site host like now.sh or Netlify (think Netlify gives you a nice open source plan for free, should contact them) you can also host json with them for initial variables possibly.

research how swiftkey keyboard prediction and other similar tools work

maybe it'd be a good idea to help with prediction while typing? I might give this a go after a while if I get some time :)

Cannot Edit Tiles In Edit Mode On Mobile

Describe the bug
In mobile you cannot edit tiles when in edit mode. Can move them around, but not edit the contents. I suspect it has something to do with the draggable library.

To Reproduce
Steps to reproduce the behavior:

1. Launch freespeech in mobile on a mobile device.
2. Turn on custom tiles.
3. Press the pencil button on the tool bar.
4. Try to press a tile and edit

Expected behavior
That one tap would open the edit configuration screen. And dragging tiles still work. If can't do both have a toggle for mobile for dragging.

Desktop (please complete the following information):
Could sometimes open in browser mobile view. Probably be best to test/fix from mobile testing.

Smartphone (please complete the following information):

• Device: Iphone xs Max
• OS: 13.3.1
• Browser All ios browsers

VPS Hosting

Not necessarily an "issue" but just putting this here..

I can host a free VPS for this project. Capable of hosting a website or a backend, or literally whatever Merkie wants. Obviously I'd also setup SSL for him and everything.

Much love for this project. <3

Edit: Some other things included

• 1 Gbps Symmertrical connection
• Unlimited bandwidth
• Project emails (info [at] freespeech [dot] net for example) with Web UI + IMAP/SMTP Support.

edit 2: didn't see that there was already a site up, offer is still up incase you guys currently pay for a vps..

Cannot see Add a Tile button in Chrome

On the edit screen, running Chrome -
Version 96.0.4664.110 (Official Build) (64-bit)- Add a tile button is missing

GitHub Actions That Check Link And Test.

Is your feature request related to a problem? Please describe.
This would save time reviewing pull requests and help with the quality of the code we pull in.

Describe the solution you'd like
I would like for the lint section to just check. Not automatically lint their branch.
For testing i would like for it to request the author to take a look and block accepting the pr till test pass.

Describe alternatives you've considered
If a warning can't be made for linting then it automatically doing it for them works for me.

Failing to compile because of missing modules

freespeech-master % npm run build

[email protected] build /Users/patrick/Downloads/freespeech-master> vue-cli-service build
⠦ Building for production...
ERROR Failed to compile with 2 errors 20:44:13

This dependency was not found:

• !raw-loader!../../../README.md in ./node_modules/cache-loader/
  dist/cjs.js??ref--12-0!./node_modules/thread-loader/dist/cjs.js!./node_modules/babel-loader/lib!./node_modules/vuetify-loader/lib/loader.js??ref--18-0!./node_modules/cache-loader/dist/cjs.js??ref--0-0!./node_modules/vue-loader/lib??vue-loader-options!./src/views/About.vue?vue&type=script&lang=js&
  To install it, you can run: npm install --save !raw-loader!../..
  /../README.md

This relative module was not found:

• ../../../build.json in ./node_modules/cache-loader/dist/cjs.js
  ??ref--12-0!./node_modules/thread-loader/dist/cjs.js!./node_modu
  les/babel-loader/lib!./node_modules/vuetify-loader/lib/loader.js
  ??ref--18-0!./node_modules/cache-loader/dist/cjs.js??ref--0-0!./
  node_modules/vue-loader/lib??vue-loader-options!./src/views/Tile
  Pad.vue?vue&type=script&lang=js&
  ERROR Build failed with errors.
  npm ERR! code ELIFECYCLE
  npm ERR! errno 1
  npm ERR! [email protected] build: vue-cli-service build
  npm ERR! Exit status 1
  npm ERR!
  npm ERR! Failed at the [email protected] build script.
  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /Users/patrick/.npm/_logs/2020-02-17T19_44_13_385Z-debug.log

Default voice and lanuage

Every time I restart the app my voice and language defaults to Arabic (Saudi Arabia) and I live in Australia. Could we detect the devices region/language to set the default voice? Or at least default to a generic American voice

Logo Suggestions

Besides branding, I mean. It seems like it would be a nice thing to have, for favicons, app icons, etc.

Logodust is a pretty good resource for commercial and noncommercial logos.

I, personally, like this one

Roadmap

Hi, I like the idea of this project. Maybe it's a good idea to create a project for this repository so people who want to contribute know what can be done.

Use objects to store data about the tiles

Move settings modal to new page

Mainly here for roadmap, tracking, and discussion; but with more settings coming in the future, the modal for settings may get a bit....bloated. Maybe it would be a good idea to go ahead and setup it's own page route for it so we can have more freedom in the future to do more.

There should be no issue

Make a app to communicate secure on schools please

Payment doesn't work.

Who is Jan 🧐

Button tap logging

For parents/caretakers, it may be useful to understand how their user it actively using the apps, maybe having a report tool to tell us how often the buttons are being clicked could be useful, nothing too advanced...just simple counters for each button so they know which buttons are used the most and which can be replaced with new buttons due to lack of usage.

Cannot reload application from About or Settings view

Describe the bug
Click on reload in browser from Settings or About view won't reload the page and displays 404 Page not found

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://app.freespeechaac.com'
2. Go to Settings or About
3. Click on refresh in your browser
4. See error: browser will display 404 Page Not Found

Expected behavior
Application should reload page in underlying view.

Screenshots

Additional context
Web server needs a rule to rewrite all incoming request for non static resources; It should always return index.html

Divide Branches And Releases

PRs #118 and #117 both have new features that we may not entirely be ready for production, but could still be pulled into the codebase.
I propose we make a release branch. It can be something like release/x.x.x for which version, or just release-latest. This will be clean production-ready code. If there needs to be a bug fix for production, we would create it and PR to the release branch and master. This will give developers a branch of code for us all to merge things that might not quite be ready for production but are ready for a PR while giving us a clean branch for bug fixes and cutting releases from. And then just releases will come from the release or releases branches.

If this sounds good I will set up the branches inside of this repo.

choose a license

please choose a license for this project :)

Advanced button customization

For buttons, and this relates to issue #33 as well as #31, but wanted to separate it for PR related reasons. Advanced button customization, so on top of changing button size, font, icon color, text color, and even button look/feel could help tailor the app more to the user.

Button layout customization

Having the ability to have buttons is great, but sometimes there may be a need to make more common buttons, like Yes or No, a bit bigger than others. Maybe there should be way to customize the layout of buttons such as normal, wide, tile options and customize the order of them as well.

A way to stop the speaking early

Is your feature request related to a problem? Please describe.
When you click on the speak button while it is speaking it will queue it up to speak again and if pressed by accident multiple times it will speak the number of times you pressed and can't be stopped

Describe the solution you'd like
Either a button to remove all talk speak commands from queue/stop a speak early or have the speak button not work whilst already speaking would help to make this more user friendly and less prone to user error

Missing package.json file

After the latest commit I think along with node modules package.json was also deleted.
I just cloned the repo and saw that no package.json file is present

Guarding settings/add button routes

Kids like to press buttons, some more than others...maybe it would be a good idea to allow caretakers/parents to restrict access to features such as add/edit buttons, settings, etc with a 4/6 digit passcode. Once entered, it can be locked immediately, or after a set time so they can edit and save settings without worrying about the child messing up the setup.

Multiple Keyboards

Describe the solution you'd like
I would like to see multiple custom keyboards. These can be stored and selected via settings page.

Add prettier support.

Is your feature request related to a problem? Please describe.
My editor keeps changing tabs to spaces and not formatting code correctly, so it'd be great to have a prettier integration in here so I don't have to think about that.

Describe the solution you'd like
Integrate prettier into the project like in this article.

Describe alternatives you've considered
ES-Lint fix? Editor integration isn't that great though.

Ability to use lower case letters

Children who have difficulty with speech and language also often have difficulty with learning to read. These children are often taught lower case letters because most written language is in lower case letters. It would be great to have the ability to change the words so that they display in lower case, upper case, or both.

I know there is an existing request for more fuller customization, but this edit is specific and I suspect would be easier to deploy. I think the result would be that the app could also help children with word recognition (e.g., sight words for reading).

Update readme to show more important information

The readme express nothing about the project per se, it should answer some of those questions:

• What are the features and possible platforms?

• What is the aim?

• What is the to-do in future?

• How to use it?

• How to get it to run?

Also some things that would be usefull:

• Show an screencap

-Show an video or gif that show a basic usercase

Add a tile does not save

Describe the bug
Can make a tile that works but cannot save it.

To Reproduce
Steps to reproduce the behavior:

1. Go to Firefox> freespeechaac.com
2. Click on 'Edit'
3. At top, click 'Add a Tile'
4. Add one, then try to save it

Expected behavior
I expected to see a tile show up on the page

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Device name Mary
  Processor Intel(R) Core(TM) i7-8700T CPU @ 2.40GHz 2.40 GHz
  Installed RAM 16.0 GB (15.8 GB usable)
  System type 64-bit operating system, x64-based processor
  Pen and touch No pen or touch input is available for this display

Edition Windows 10 Pro
Version 20H2
Installed on ‎7/‎30/‎2020
OS build 19042.1415
Experience Windows Feature Experience Pack 120.2212.3920.0

• Browser Firefox
• Version 95.0.1 (64-bit)

Discord?

I think something like Discord could help grow the community and allow non-devs to interact with the code contributors and suggest ideas as well be more involved. Obviously it will be much smaller, but I think it could also open the road for discussion of features before they hit GH issues to allow the contributors to keep the issue board focused and clean along with helping contributors keep up with issue assignment and what they're working on so 2 people don't do the same thing.

I've went ahead and created one myself as I have a good bit of experience with creating and managing these types of communities and with the help of @fatfingers23 (if he wants) we can setup GH/Discord webhooks to allow people within the Discord community to keep up to date on what's happening on the repo itself and new commits, releases, issues, and PRs happening.

https://discord.gg/DYRBJnh

Edit: At some point, I can transfer ownership of the server to whoever on the team.

Clearing Navigation On Tile Edit Does Not Do That

Describe the bug

Currently when you go to edit a tile. There is a remove navigation button. This only clears it on the form. but not inside of Vuex. Need to remove the property from the tile data for when it's added to Vuex so it will no longer be a navigation tile

To Reproduce
Steps to reproduce the behavior:

1. Turn on edit mode
2. Set navigation if not set
3. save
4. Re open as edit
5. Click remove navigation and save.
6. Does not actually remove it and still a navigation tile

Expected behavior
When its clicked and saved it removes navigation from the tileData property for that tile.

Export/Import Custom Keyboard

Is your feature request related to a problem? Please describe.
This will allow you to save your keyboard or manually share for now. Or even back it up in the case of a software failure.

Describe the solution you'd like
As a User,
I would like to be able to export/import keyboard layouts via settings,
so that i have more control over the keyboard across devices, or can back it up.

I would like to see it export the .json file and can be "uploaded" if possible. I don't think a text field will be a good ui design for exporting and importing. Not sure if that functionality can be achieved with no back end.

So it would look like you are uploading a json file for import, or downloading it for export.

Reset button for when user has made edits to the tiles

A reset button potentially in settings to reset the custom tiles to default

List Of Voices Do Not Populate On Google Chrome Desktop

Under settings the select box to select a different voice does not get populated. This needs to be fixed in a way that does not break this functionality in other web browsers. Mainly IOS safari.

Wrap Tile.vue in a <v-Btn>

Tiles are currently not to a Web accessibility standard. Meaning some software or tools people use to access the site may not be able to use them to their full potential.

Wrapping the Tile in a button will solve that. Tile styling should not change. If that cannot be achieved then minimal changes to it please.

Language support

I think it would be nice to make an interface to help add new languages and compile a JSON in a tab on the site that shows one section at a time and requests the word that it is and what the title should be for other languages, making it easier to add new languages. This would allow people adding languages to not have to mess around with JSON (like below) and possibly break it/ find it difficult.
{ "name": "Little", "text": "Little", "image": "https://img.icons8.com/officexs/64/000000/cursor.png", "id": 1, "accent": "white", "messages": { "de": { "name": "Klein", "text": "klein" } } },
Might atempt to make a HTML mock up later to show in more detail.

Ability to save sentences

Is your feature request related to a problem? Please describe.
The ability to save sentences that are commonly used. I really do not have a great ui idea on how this is done. But i think would be a wonderful option. It would list sentences the user created and saved. While also giving them an easy ability to select them, sort, etc.

Draft Overview Of Techincal Needs From A Backend

This is a rough draft of technical aspects of a backend i would like to see us choose. I know not everything on this list is possible, but it is high points to help us decide on what framework to use. This is a very very picky list. But I'm willing to bulge on most if you have a good solution to counter what i put.

Over all about the framework.

• Must be a modern web framework. I don't wanna see any VB.NET with WCF or PHP 5.6 /s
• The backend will be a RESTFUL Json api. We will not be serving any views/guis/Vue components from this. The front end and backend will be different projects that can work together and can be decoupled.
• Designed in a way that can be consumed by Vue project or any other project that we may add in the future. e.g. Mobile apps.
• Follows a MVC, MVVC pattern. Preferably MVVC. I like ViewModels, makes it clean and easy for the view to consume i personally think.
• Support for saving files in a secure way. E.g. pictures. We will probably use something like AWS or Azure blob storage i would imagine?

Deployment/ Dev environment

• Can be hosted cheaply in the start. I'm talking free serverless or under 30 a month.
• Can scaled to multiple instances. Not to worried about this in the start with scaling to multiple. instances. But want to keep it in mind and be prepared.
• Full CI/CD pipeline support from GitHub actions or other free/included in the price. Also well documented resource.
• I don't want to see anything that comes with a 5 page setup documentation for production that includes having a file that is in the repo, but i have to ssh to the server and change credentials in that file. Full automation, i want it to push and deploy with no intervention.
• Cross platform support. This means Windows, Linux, and macos. The code will more than likely be ran from a Linux server, but i want windows users to be able to contribute as well.
• Dead simple docker setup on initial pull to master. I see us having a backend branch while getting it all setup. I also see backend being a much higher level of entry to contribute then the front end. BUT i do not want that to stop our front end contributions just cause someone can't get the code to read from their sql server. I want like one command and minimal changes to a env file for developers to get a instance of the backend up and running and they can get to doing front end without even thinking about the back end.
• If we pick serverless has to be in a way we can all design the backend on a free tier of a service.
• Something the majority of the current contributors are comfortable with or are willing to learn. e.g. May not be your first choice. But you know it and have worked in it before
• Must have unit testing support. This is an open source project and we will not have the luxery of QA testers to regression test.
• Familiar and popular language.
• Something with great documentation and stackover flow presence

Database support

• Must be model base/code first with migration support. Will find me hard press to move on this. We're not going be doing a ssh in and "Run this line of SQL before the update". Nope.
• An open source/ license free database. Sorry guys no MSSQL here. Don't think it was even in yalls mind, but you never know.🤷‍♀️

Security

• Solution for XSS scripting protection
• SQL injection prevention/ parameterized sql statements.
• A secure and modern password hashing algorithm.
• Auth0/JWT support. Bonus if it handles refresh tokens.
• I don't really want to see any self rolled security. Like some GitHub dependency with 5 stars that handles password hashing. These frameworks have already made these solutions lets use them.

Things that would make it faster and us all happy.

• Premade methods. I'm talking the whole 9 yards you find in any popular frame work
  • email confirmation tokens generation
  • users with authentication
  • built in roles
  • and so on. Almost anything you would see in a modern framework.

I'm open to all suggestions and discussion on this. I want it to be a group decision. Preferably majority decide on something. I know this is a big list and a little extreme, but as a professional software developer I've worked with PHP with no framework, Python micro frameworks, SOAP services, Django, Laravel, and dotnet core are just a few i've worked with. I've seen the good, the bad, and the failed deployment at 3am that left the site down. I want this to be a great product and will hold it to a high standard.

So with all of that said. Have a framework in mind? Great! Comment it below with a good constructive reason to why we should use it. See a comment someone made and you like that frame work? Follow these responses.

• 👍 I think this is a great framework to use and hits all the high points. I may of not used it before but i'm willing to go with it.
• ❤️ I love this framework, i use it daily or enough that i can program in it and help others.
• 👎 I do not feel like this will fit the bill.
• 😦 if someone says we should use Cobol.

If you have feed back for someone on their framework. Mention them, name what the framework is and give your constructive feed back or ask them to expand on it or give examples.

I do not want to see this as a talk down to any framework. All popular modern frameworks will more than likely fit this bill in one way or another. They all bring great things to the table and leave some behind. There will never be a perfect solution. Just looking for the best one and the one that is the most fun for us to code.