asterics/flipmouse

The repository for FLipMouse (Finger and LipMouse)

Welcome to AsTeRICS

---

AsTeRICS is a free and Open-Source graphical construction set for assistive technologies (AT). It allows the creation of flexible solutions for people with disabilities using a large set of sensors and actuators.

Possible applications are

• Computer input (mouse, keyboard, joystick)
• Environmental Control (KNX, FS20, IR, EnOcean, openHAB)
• Toys and Games (Playstation 3, computer games, RC-toys ...)
• Brain/Neural computer interfaces (Enobio, OpenVIBE, OpenEEG)
• Android Phone support (SMS, calls)
• and many more!

For more information see our website http://www.asterics.eu.

Demos (Solutions)

Check out some solutions here

Quick Build Instructions

To clone and compile the AsTeRICS framework, please execute the following steps:

1. Clone Repository (submodules will be cloned during build)

git clone https://github.com/asterics/AsTeRICS.git

2. Install the Java Development Kit (JDK, 32bit preferred) >= 8

• Ensure to set JAVA_HOME to the folder where you installed the Java JDK and add the JDK bin path to the Environment Variable Path

3. apache ant build framework (version >= 1.9.1)

• Ensure to set ANT_HOME to the folder where you installed ant and add the ant bin path to the Environment Variable Path
• The git command line tool must be available from within ant, ensure to set the Environment Variable Path to the respective bin folder.

4. Open a terminal and start the AsTeRICS Runtime Environment (ARE) by calling:

ant run

If you don't need commandline support, you can use an IDE for Java Developers, e.g. Eclipse, which already contains ant. There you can directly execute ant targets by selecting the build file build.xml and select the target of choice, e.g. run.

Documentation

• AsTeRICS Tutorial with screencasts
• Getting Started
• Tutorials
• Developer Manual
• Plugin Hellp

You can find more documentation and build plans for some hardware devices in ModelGuides and DIYGuides.

Contact

If you want to contribute to the AsTeRICS project, have questions or just need help using it don't bother to create an issue, pull request or via the contact page.

If you want to support the development of AsTeRICS you're very welcome to donate to the AsTeRICS Foundation:

   

License

This project has been partly funded by the European Commission, Grant Agreement Number 247730 in the years 2010-2012. Since 2013, further development of AsTeRICS software and hardware components is hosted at: https://github.com/asterics/AsTeRICS

ACS and NativeASAPI libraries

Licensed under LGPL

ARE-middleware, ARE-plugins, services and BNCI Suite

Licensed under a dual license MIT or GPL with CLASSPATH exception

How to apply the dual licensing

You may use these components under the terms of the MIT License, if no source code (plugins, services, libraries, ...) which is contained in your desired collection of ARE plugins and services is licensed under the GNU General Public License (GPL). In order for that, you can remove unneeded plugins or services.

Please note: There are also some plugins (e.g. MathEvaluator, VLC) which are GPL without CLASSPATH exception, so in this case your license would have to be GPL as well.

Please have a look at the individual licenses of the AsTeRICS plugins and services. You find the license files in the subfolder LICENSE of every plugin.

Disclaimer

This software is provided 'as-is', without any express or implied warranty. In no event will the authors be held liable for any damages arising from the use of this software. See DISCLAIMER

Releases

• See CHANGELOG or old Release Notes and the published releases

flipmouse's Issues

Flip/mirror X/Y axis

it should be possible to flip or mirror the X/Y axis to enable more flexibility for mounting (mounting in all rotations)

`AT SR`: add button states

I think it would be an improvement to see current button states (physical & virtual) in report raw values.
We could visualize states in the GUI and process this data otherwise.

@ChrisVeigl Do you think it would be a problem to just add another field in the report?
E.g.: <pressure>,<up>,<down>,<left>,<right>,<x>,<y>,<buttons>

Current C# GUI does not support this format, no live values are shown anymore.

Implement IR (infrared) command storage

A new feature for IR commands should store the recorded commands to the EEPROM.
Related to #4

Unclear calibration on slot-switch if gain!=50

If the sensor gain for one slot is set to 50 and for another one to != 50, calibration does not work as expected. Mostly one slot is outside deadzone, even if idle.

Problem:

FLipMouse/FLipWare/commands.cpp

Line 236 in 9b8135d

{ cx=settings.cx; cy=settings.cy; } // update calibration settings if gain settings are not default

@ChrisVeigl @klues
Is there any need to have gain settings individual for each slot?

Toggle click hold

A command should be added which does not simply hold the mouse click, but toggles hold by the same button.

Wrong component labelling for BT add-on module

The labels for components R2 und C7 are swapped - this might lead to errors when mounting the components.

Keycode lookup if esp32_mouse_keyboard aka ESP32miniBT module is connected.

Currently, it is not possible to use different keyboard layouts than US for the FLipMouse.
As we have limited resources in the FLipMouse (current master: 92% Flash, 79% static RAM usage),
we could implement a lookup of keys / text or keycodes via the ESP32 module (4MB flash -> we can store all supported keymaps of Teensyduino there).

In my opinion, we need following new features:

• Detect if a compatible ESP32 module is connected
• A dynamic table which contains the most recently used keycodes
• Update mechanism to load a new keycode for a key from ESP32
• Fallback solution (e.g. currently used static keymap) if dynamic update is not possible

Sip&Puff sensor spurious failing

@ma0027 had the problem of a failing sip&puff sensor.

Workaround: unplug & plug in for a few times.

Reason: unknown

Maybe we should add error handling for the sensor & a restart procedure of I2C.
In addition, it would be good to have error output and maybe a "storage" of error messages.

Infrared: record sometimes fails

If you record a remote, sometimes only 1 edge is recorded.

Of course, there is no IR command with 1 edge.

There should be some plausibility check of the recorded signal

HW - add HM10 Bluetooth module

Add the HM10 bluetooth module directly to the PCB on the new HW revision

Flipmouse GUI: auto-increase targets wrong slider

If configuring the cursor movement (first tab) in split X/Y mode,

the "+" symbol of the speed Y slider increases the X value, but only while auto-increasing via hovering.
If you click on the button, the correct value is increased

Couple of unclear instructions in Construction Guide

Hi,

a couple of proposed amendments to the construction guide:

• It would be prudent to make the user check values or text of the resistor banks before soldering them in as they're easily confused. There's a warning text "Take care of the text on these parts" but that's neither English nor does it give any hints what the text should be. A warning to the likes of "Before soldering make sure you're using the correct resistor bank with a value of x Ohms, verifiable with a Multimeter or by the text "xx" on the part"
• Maybe a warning to check the continuity / proper connection of the soldering points on the bottom of the board befopre soldering the pressure sensor because it covers those pins and you can't check afterwards which causes debugging headaches
• On the instructions for soldering the pin header on the daughterboard, nowhere does it mention that there's a little notch that should be aligned with the position of the header and on the illustration it's also not there - this is probably the most important of the points. Furthermore, the instruction "bend the pin header to a straight angle" is a bit confusing - you might mean "making it level/even with the board"? This isn't as big a problem because the image makes it pretty clear how it should look.

I understand that a lot of these issues are problems that only a non-experienced constructor might encounter, but in the spirit of keeping the project easily accessible by anyone and the otherwise very good Construction Guide, I personally think these should be fixed to prevent people making the same mistakes I have. Maybe I've just missed these things in the construction guide, in which case feel free to ignore this.

Thanks

FlipMouse C# GUI - List of slots is not set correctly, if settings are not loaded from mouse

See video,
if you press "no" if you get asked for loading settings from the Flipmouse, the drop down menu for available slots is empty until the buttons next/previous slot are pressed.
video_gui_bug.zip

Add AT command to switch on/off certain buzzer beeps

As requested by a user, we could implement a command to switch on or off different kinds of tone outputs:

• sip / puff
• enter or exit strong sip / puff
• calib
• change slot
• IR receive / send
• BT pairing

Suggestion:
"tone off"
AT TO 0011001100
(each bit indicates one tone type)

EDIT:
we should implement this for this repository as well as FMv2

Media keys

Teensyduino supports media keys (e.g. play/pause, next, prev.,...) with the currently used Keyboard.press() and Keyboard.release().

We should add this functionality in:

• FLipWare (keys.cpp)
• FLipMouseGUI
• FLipMouse WebGUI
• esp32_mouse_keyboard

Firmware not compilable for TeensyLC

Due to dedicated AVR libc functions (e.g. strcpy_PF), the firmware is currently not compilable for FLipmouse v2

Update landing page

Move Troubleshooting/FAQ to the Wiki

General text updates

3d-printed case: improvements

labels/tactile indicators for the buttons (at least B2 / B3) would be nice
and we should improve the 3d-print so that the M2x16 mm screws / bolts have metal nuts or a better fit for the threads

Improve batch download of slots

Currently, the GUI saves all slots in one batch, all with an "AT" command.

Maybe it should be possible to switch of the preceding "AT" for a short time during the batch download.
This would decrease the sent bytes by maximum 50% (for each command without a parameter)

Optimize EEPROM structure

Currently, the settings/slots are stored in fixed sized windows.
Improve the storage to optimize enhance the memory space layout.

GUI: losing slot settings if not every slot is stored

If the settings for one slot are not stored and you switch to another slot to configure, the settings get lost.

situation:
4 different slots

goal:
change button 2 on all slots to another action

steps:
select one slot, change the setting. Load next slot and change that (for every slot)

problem:
the changes are only stored for the last slot.

workaround:
Press "Store settings" each time you changed one slot

HW - evaluate best USB plug positioning

In many cases the USB plug on the back side might not be the best solution.
For the new hardware design, we should evaluate the best possible position of the USB plug

FlipMouse C# GUI - Problem if ID string is assigned as keyboard write

If at any button, the command "Write Text" is assigned and this text contains the word FLipMouse, it will be parsed as ID String.
This results in various errors:

• If the version is not in this string, the GUI asks for firmware updates
• The notification for loading the FLipMouse config is shown again.

The only way to avoid this error is clicking "no" if the GUI asks for loading the config from the FLipMouse.

Inconsistent EEPROM state

When programming a FLipMouse, normally there are no slots available (only the default config, not saved in EEPROM).

Using a terminal program, entering:
"AT" --> "OK"
"AT LI" --> no feedback
"AT LA" --> should write the default slot,but I get following incomplete feedback, error messages and a stuck FLipMouse:

Initializing EEPROM!
Deleting all IR slots
Error: seperator problem0
Error: seperator problem0
Error: seperator problem95
Slot:
AT AX 40
AT AY 40
AT DX 20
AT DY 20
AT MS 50
AT AC 20
AT TS 400
AT TP 600
AT WS 3
AT SP 800
AT SS 10
AT MM 1
AT GU 50
AT GD 50
AT GL 50
AT GR 50
AT RO 0
AT BT 1
AT II 
AT BM 01
AT 

Current master (v2.8.3).
It is also not working with the C# GUI.

Add AT command for renaming slots

In order to be able to easily change a slot name from WebGUI a specific AT command would be great!
Suggestion: AT SN <name> for renaming the current slot.

related: asterics/FABI#29

Macros: AT MA KW \;;CR; writing "ö" instead of ";"

I'm testing macro function via serial console AT MA KW \;;CR;:

1. Writes ö instead of ;
2. Does a right-click with the mouse as expected.

FLipMouse GUI - Managing Bluetooth devices

Currently, the FLipMouse with ESP32miniBT addon accepts every pairing request and paired devices can't be unpaired.

It would be nice to show (if available) a list of bonded devices in the FLipMouseGUI (C# & Web), including the possibility to remove one or all devices from this list.

IR: change the output method from "NOP" ASM to a PWM timer

There is the possibility to change the PWM frequency of the output pin.
According to https://pjrc.com/teensy/td_pulse.html
it is possible to adjust the frequency to 38kHz/duty cycle 50%

This would increase the clock Independence.

Saving config: performance and usability

Usability

Current Workflow

The current workflow for making configuration changes is:

• Open GUI
• Make changes in GUI
• Click "Test settings" in C# GUI, WebGUI automatically "tests" settings
• Click "Store settings" (C# GUI) / "Save" (Web GUI) in order to persist the configuration

Alternative Workflow

This would be an alternative workflow:

• Open GUI
• Make changes in GUI
• Changes are applied and persisted immediately, no further user actions are required
• There are possibilities to revert the configuration to default configuration or the starting configuration of the session.

I think most of the time the user wants to keep the new configuration, so it would be a benefit for usability if no explicit action would be required for saving the config. It's also a benefit for testing if changes (e.g. sensitivity) immediately take effect and there is no need for clicking on "Test" (currently already implemented in WebGUI).

Technical stuff

Currently it seems to be only possible to persist a config like this:

• delete all slots with AT DE
• re-save all slots
• this takes about 4 seconds

Proposed changes

• AT SA <slotname> should overwrite an existing slot with the current config, so a single AT SA command would be enough for persisting -> some milliseconds instead of 4 seconds
• I would implement WebGUI with the alternative workflow, C# GUI should stay like it is.

@ChrisVeigl, @benjaminaigner: what do you think?

Extra hole for external Sip/Puff

An extra hole should be on the bottom of the FLipmouse.

In the case of using the FLipmouse as finger operated mouse, it should be possible to route the pipe to the outside

Automatic Firmware Update

it would be great if we could enable automatic firmware updates for the BT-Add-On from the GUI (as it exists already for the Teensy firmware)

Theoretically it could work if we replace the second GND signal from the AUX connector with a connection to GPIO0 of the ESP32 and add a way to reset the ESP32 (maybe using an analog circuit)?

Then, we could forward the serial communication during the FW-upload from esptool to the ESP32 via the TeensyLC....

BLE: rate limiter / commands to addon module in general

For adapting the rate limiter, a new AT command needs to be implemented.

But in general, I would prefer to use a general AT command, which is used to send arbitrary commands to an addon board.

Suggestion: AT BC <command (according to asterics/esp32_mouse_keyboard#39
e.g., AT BC $RL80 sets the rate limiter of the esp32 addon to 80ms.

In general, these commands can/should be:

• sent from the GUI
• sent from a WebGUI
• stored in the EEPROM (we should agree on a procedure how an array of external commands can be saved in EEPROM).

openinput support

This could be a very interesting target to add to the openinput project, creating this issue to check interest and show my willingness to work on support for it.