optikey/optikey

OptiKey - Full computer control and speech with your eyes

OptiKey

OptiKey is an on-screen keyboard that is designed to help Motor Neuron Disease (MND) patients interact with Windows computers. When used with an eye-tracking device, OptiKey's on-screen keyboard allows MND patients to complete tasks, such as email composition, using only their eyes. OptiKey can also be used with a mouse or webcam. Unlike expensive and unreliable Alternative and Augmentative Communication (AAC) products that are difficult to use, Optikey is free, reliable, and easy to use.

Getting Started

The OptiKey Wiki contains OptiKey's user guides, installation and system requirements, and additional support information. OptiKey's Windows installer can be downloaded from the latest release. To get an understanding of OptiKey's use, users should watch Optikey's introduction video.

Supported Platforms

OptiKey uses the .Net 4.6 Framework and is designed to run on Windows 8 / 8.1 / 10.

Problems?

If users encounter an issue with OptiKey, such as a software crash or an unexpected behaviour, users should add an issue ticket to OptiKey's issue tracker. Users are encouraged to check if their issue is already being tracked by OptiKey before creating a new issue ticket.

The following information should be specified in an issue ticket:

• How OptiKey was being used
• What the user expected to happen
• What the user actually experienced
• Steps to reproduce the issue
• Any other information that helps to describe and/or reproduce the problem (ex. screenshots)

License

Licensed under the GNU GENERAL PUBLIC LICENSE (Version 3, 29th June 2007)

Contributing

Contributions to this project are always welcome. If you'd like to help translate Optikey to another language, fix bugs or add new features, check out our Contributing page.

Contact

To ask a question, or to discuss information that is not on the OptiKey Wiki, please use [email protected] or [email protected] to contact Julius.

optikey's Issues

BCI (brain computer interface) trigger

Off the shelf or research product?

Could approach universities to open this up to students?

From "gaze interaction and applications of eye tracking"; EEG has around 500ms latency.

Debugging display

• Debug setting
• Display points from SelectionResult event - if Debug == true
• Display points per second (in CaptureState frame?) if Debug == true

Fix issue of no selection progress on comma and space keys (and possibly others)

Touch screen trigger and point source

Fix issue of 1st key not being rendered when changing themes

Button to "rest"

• Active/Inactive - "Pause/Resume" or "Sleep/Wake" - allowing user to look around and rest
• Available on main keyboards. Probably not required in more specialist keyboards, e.g. settings, or question.

Website

ICON LINKS:
View on GitHub
Downloads - http://www.softicons.com/system-icons/windows-8-metro-invert-icons-by-dakirby309/folder-downloads-folder-icon

• Features summary on home page:
  Free
  Open source & easy to extend - want to modify how something works or looks, or need something extra? You can change it. (input sources, keyboard sets, behaviour, look and feel, etc)
  Keyboard control - simulate key strokes into any other application (e.g. write an email)
  Mouse control (coming soon)
  Speaks
  Whole word/phrase matches + auto space / auto capitalisation
  Supports multiple input sources (pointing and selection - eye trackers, etc)
  Themeable
  Customisable - many settings to change how you interact with OptiKey (e.g. changing dwell start, complete, decay timings)
  Support - just me for the moment
• USAGE GUIDE page
• TROUBLESHOOTING page

USAGE GUIDE:

GET STARTED

How to use OptiKey with an eye tracker:
Connect your tracker and start the corresponding program/programs (e.g. for the eye tracker you would start the server and then the client programs). Complete the initial procedure to get the tracker up and running; usually this involves getting yourself/your tracker/your screen in the correct position and competing a calibration. Once it's tracking your eyes reliably it's time to start OptiKey...
Once OptiKey has started, open the Management Console (ALT + M) and change the "point source" to your tracking device, and the "trigger source" to suit you (e.g. to select keys based on a prolonged look, or "dwell"). Optionally tweak the other settings and then click "OK" to save your changes. OptiKey may need to restart. Don't worry, all changes to the settings are saved and personal to you, as are any new words or phrases that you add or remove to/from the dictionary.
You should be good to go. Any problems should be displayed automatically to you. If you're not sure what to do check out the problems section below.

How do I make OptiKey speak?

HOW TO STIMULATE PHYSICAL KEYBOARD STROKES

How do I stimulate physical keyboard presses? I want to type into another application...

Do I have to type each letter in a word individually (word/phrase matching)?

Auto space & Auto capitalisation

Can I customise the dictionary?

How do I move and resize OptiKey, or change it's transparency?
OptiKey can be customised (resized, moved, or made more/less transparent) without having to switch back to conventional input devices. The layout, look & feel, and behavior of the keyboard can also be changed from the Management Console.
If you're using an eye tracker you can recalibrate from within OptiKey, as well as tell it to go to sleep so it doeACsn't distract you when you're doing something else.

How do I change the theme?

Is there a stripped down/simpler keyboard? I only want to make OptiKey speak?

Where is the rest of this guide?

This is a new project, so I'm going to enrich this website organically. If you want to know something that isn't covered here email me at [email protected] and I'll help you. I'll then add the information here if I think other people might find it useful.
Problem?

See the troubleshooting page.

TROUBLESHOOTING:

If you are installing on Windows 8 you may see a Windows SmartScreen warning saying that OptiKey is an unrecognised app - click "MORE INFO" and then "RUN ANYWAY"

I can't open the Management Console.

Eye tracker not working or stops working.

Poor eye tracking performance/accuracy.

Crash? Unexpected behaviour? Something else wrong?
If something isn't working as you'd expect, you encounter a crash, a freeze, or something else horrible then let me know and I'll get it fixed for you. I need something from you though; as much information about the problem as you can muster. Here's the ideal list of things your email will contain;

1. Logs. Developers love logs. You'll need to turn on 'Debugging mode' first (in the Management Console which is accessed by pressing ALT+M, in the 'Other' section). Turn that on then do whatever you did before to cause the unexpected problem, crash, or whatever it was, then close OptiKey. The logs are stored in a special directory which is personal to you. On Windows this will be (open File Explorer and find this directory);
   C:\Users\YOUR_USER_NAME\AppData\Roaming\JuliusSweetland\Logs
   Grab the latest log file (the modified date on the file will be the most recent) and attach it to an email to [email protected], but don't send it just yet! I also need...
2. A short description of what you were doing when the problem occurred.
3. What is the problem? A crash? OptiKey did one thing and you were expecting something else?
4. Anything else that you think is useful. Screenshots are amazing.
   I know it's demanding, but if you give me great info then I'll fix it and no-one else will have to send be that lot again.

My problem isn't answered here!
OptiKey didn't meet my needs!
I'm sorry to hear that. Please email me at [email protected].

How does the keyboard display when switching to Metro in Windows 8?

Does it get hidden, but think it is still visible?

Settings

• Make them accessible from within ETTA, e.g. new keyboards to change settings
• Edit personalised version of the dictionary:
• Remove entry (including phrases)
• Reset use count on words (individual entry/all entries)

Pause after a selection to give user a chance to register that selection has occurred before starting to record the next one

Configurable length in settings

Display MultiKeySelection capture state on View

Display frame showing input service's capture state (instead of green/red/etc maybe hide the frame unless recording, then show HTC Sense coral colour to show capturing)

Some screenshots

Remove error when trigger without active PointAndKey

A lack of cursor or key selection should hopefully be sufficient. Possibly increase point ttl to exceed average blink duration (150ms)?

Button to turn on/off MultiKeySelection

Useful when user wants to spell a word using the fixation triggers - takes twice as long with them on and can introduce errors.

Remove char? from KeyValue and just have string

Any reason to keep the Char?

Next word prediction

Mouse control

• Add keyboard for mouse clicks:

    Left
    Right
    Double (Left)
    Drag (Left down @ pos1, up @ pos2)
    Scroll horizontal/vertical by x units = 5 buttons (left, up, right, down, qty)
    3 x methods of scrolling:
        i)You select the scroll direction and then click on the target
        NO, TOO COMPLEX ii)You lock the target and then can select to apply a scroll to the target until you release the target (requires a 'scroll' button which can be locked on)
        iii)General button to repeat the last mouse action
    Repeat - FunctionKeys.ReplayLastMouseAction

• Select a key ->

    Change mode to position source = point
    Display cursor at point source position
    Change cursor to something customisable - to show selection progress. Popup at cursor position + hide cursor?
        Size from settings?
        Colour from theme?
        Set of cursor images the user can select?
    Store mouse action to perform on trigger ("Action OnNextSelection"?)
    Dwell in an area to register interest at this point
        Take the action at this point (click, scroll etc)
        Store LastMouseAction (taking action at the point) so it can be repeated
        Test and improve this to make sure it is consistent before moving on
    Add magnification functionality (setting to turn on/off, for each possible click event, e.g. on for left/right clicks, off for free scrolling)
        Display magnification of area in center of screen - screenshot or live view?
            http://stackoverflow.com/questions/11994331/magnify-a-specific-area-of-the-screen
        Display nearly full screen (with border), or over whole keyboard, with magnify and abort buttons)
            Magnify button = next click in magnification increases magnification
    Enabled/disabled state for ReplayLastMouseAction
    ScrollAmountInClicks
    Options:
        Register interest again to complete action
        Zoom in / zoom out (if zoomed in more than 1 level already)
        Back (discard mouse action and return to keyboard)

Artwork & icons

People who could help

Fraser from Cambodia?
Andy B?

Set up CI

https://travis-ci.com/
http://www.appveyor.com/

Output logic

• Method(s) to handle selection input

HandleInput(KeyValue)
Adjust output buffer and publish if Publish==true.

• output string buffer
• currentWorkLength - track length of currently output word - reset after outputting space, newline, or switching between spelled word and mulitkeyselection capture. Used when deleting back, or replacing MultiKeySelection match with other suggestion, or replacing spelled word (or partial word) with suggestion.
• Back - back 1 if lastOutput==null, else length of last output, which will either be a single keyvalue or multikeyselection.
• Back Word - always back to last line break or space. (back to last word boundary, i.e. back until the beginning of the last word.)
• suggestions - populated either from multikeyselection matches or direct calls to dictionary service to finish word when last output is a letter. when selecting a suggestion simulate back (length of lastOutput), then output suggestion, then clear suggestions.
• auto-space state (push space before current output when current output is multiKeySelection and lastOutput is not in {[\n,\s,•,«,{,©,|,[,<,®,¡,*,,¿,@,&,_,(,:,",#,/,?,.,-,'}
• auto-shift (to capitalise next output letter. User can turn it off before typing) when;
  Shift is off.
  There have been no letters output since last \n
  Last output was sentence ending punctuation (.?!) followed by at least one space.
  Evaluate after pushing any output.
• Buttons - SimulateKeyStrokes ("Publish" - bound to setting), Clear, Speak. MainVM receives all key selections and takes appropriate actions. For publish: ask user if they want to publish whole output buffer, and publish while on. OutputVM exposes method to publish whole buffer and handles publishing new selections as they are passed to it.
• passed strings are cased appropriately by MainVM (using shift state) before being passed to OutputVM
• event to request adding entry to dictionary - handled by MainVM. Detect new word when moving on from 'current'
• Publish/Pushing keys to message bus; http://inputsimulator.codeplex.com/

OutputView:

• Buffer text - don't bother highlighting the last output for the moment. The paradigm breaks down when the user back spaces by more than one output.

Write unit tests!

Add to SourceForge

Look into low cost Windows tablets for a complete system with EyeTribe

http://www.microsoftstore.com/store/msuk/en_GB/pdp/HP-Stream-7-Signature-Edition-Tablet/productID.308686300?tid=sMfzoUBir_dc&cid=5366&pcrid=50345397525&pkw=%2Bmicrosoft%20%2Bhp%20%2B7%20%2Binch%20%2Btablet&pmt=b&WT.srch=1&WT.mc_id=pointitsem_Microsoft+UK_google_HP+Stream+Tablet&WT.term=%2Bmicrosoft%20%2Bhp%20%2B7%20%2Binch%20%2Btablet&WT.campaign=HP+Stream+Tablet&WT.content=MfzoUBir&WT.source=google&WT.medium=cpc

Create and expose keyboard classes on MainVM

Keyboard classes:

• Alpha
• NumericAndSymbol1
• Symbol2
• Question
• Publish

(all derive from common Keyboard class)

Expose InputService event info as properties on MainVM

• Expose SelectionMode property on VM - (point or key selection) - push to InputService on change. This will also be used to determine if cursor is shown by OutputService (Point selection mode)
• Expose CapturingMultiKeySelection event from InputService
• Pass current position, selection progress, selection, CapturingMultiKeySelection to OutputService
• Expose Points Per Second as property on InputService for direct binding

Show selection progress and complete selections

• Display selection progress on keys if in SelectionMode == Key
• Display selection (complete) on key if SelectionMode == Key
• Stub out showing selection progress on cursor if SelectionMode == Point
• Stub out showing selection (complete) on cursor if SelectionMode == Point

Modify KeyValue class

Properties;

• string DisplayText
• string CapsDisplayText - don't set if not relevant
• FunctionKeys? FunctionKey
• string String
• VirtualKeyCode? PublishCode

Rename KeyValue class to Key class???

Throw arg exception from ctor if FunctionKey and String are null, or DisplayText is null

Add diacritic symbols

RENAME current symbol keyboards to GeneralSymbols1,2,3. Then add Diacritic1,2, etc and LogicSymbols1

Add diactritic keyboard. I guess this would be alphabet specific (e.g. latin alphabet)

A ĂĄªÃÄÅÆĀÀÁ (11)
a ăąªãäåæāàáâ
C ČÇĆ (3)
c čçć
D ĐÐĎ (3)
d đðď
E ƏĘĚÊËĒĖÈÉ (9) = 26
e əęěêëēėèé
G ĜĞĢ (3)
g ĝğģ
H ĦĤ (2)
h ħĥ
I ỊĮĪÏÎÍÌ (7)
i ịįīïîíì
K Ķ (1)
k ķ
L ŁĽĻĹ (4)
l łľļĺ
N ŇŅŃÑ (4) = 21
n ňņńñ
O ƠŒŐŌØÖÕÔÓÒ (10)
o ơœőōøöõôóò
R ŔŘ (2)
r ŕř
S ŞŠȘߌŜ (6)
s şšșßśŝ
T ŤȚŢÞ (4) = 22
t ťțţþ
U ỤƯŲŰŮŪÜÛÚÙ (10)
u ụưųűůūüûúù
W Ŵ (1)
w ŵ
Y ŶŸÝ (3)
y ŷÿý
Z ŹŻŽ (3) = 17
z źżž

Also add the following symbols somewhere;
¹
²
³

http://en.wikipedia.org/wiki/Punctuation
General typography
dagger † ‡
numero sign №
obelus ÷
ordinal indicator º ª
per mil ‰
basis point ‱
pilcrow ¶
section sign §

Currency
¤ generic currency symbol
₳ ฿ ₵ ¢ ₡ ₢ $ ₫ ₯ ₠ € ƒ ₣ ₲ ₴ ₭ ₺ ℳ ₥ ₦ ₧ ₱ ₰ £ ៛ ₹ ₨ ₪ ৳ ₸ ₮ ₩ ¥

Logic symbols
⇒ → ⊃ ⇔ ↔ ¬ ∧ ∨ ⊕ ⊻ ⊤ ⊥ F ∀ ∃ ≡ ⊢ ⊨

Solve how current position is displayed

Enabled/disabled logic for maximise/restore and repeat last mouse action buttons

License the software

http://choosealicense.com/

Enhance fixation triggers with lock on period and reset delay

1.Configure whether initial fixation (which starts the trigger) is used or not. Not using it results in behaviour where the user fixates on a key to register interest and the the progress begins from zero, rather than jumping to 25% for example.

2.Configure a delay after registering a high trigger signal, like a reset period. This prevents interest in other keys beginning immediately while the user is still registering the previous selection event and not thinking about the next selection. This may not be required if the initial fixation is discarded (above) as this would introduce a similar apparent delay between selections.

Nice to have

Features which would be nice to add at some point.

Solve how the selection progress is displayed

1. http://www.devcomponents.com/kb2/?p=1117
   
2. http://timokorinth.de/creating-circular-progress-bar-wpf-silverlight/
   
3. http://www.scottlogic.com/blog/2011/02/07/a-circular-progressbar-style-using-an-attached-viewmodel.html
   
   WPF sample project here; http://agsmith.wordpress.com/2008/04/07/propertydescriptor-addvaluechanged-alternative/

Enhance selection progress indication by shrinking text/symbol during selection

'Gaze interaction and applications of eye tracking' p.221
This technique draws in the user's attention to the centre of the key

Blink trigger

Name this project

Brainstorming...

etta
chat
chatter
babble
jabber
blabber
gibber
dart
jot
jotter / jottr
flick
saccade
eye
iris
type
assistant
take a note
take this down
-----Bro eg chatterBro
octopus
recorder
court Reporter
words
lines
speech
say
speak
speaker / speakr
transcriber / transcribr
talk
converse
conversation
communicate
squawk
squawk box
dictaphone
dicta
diction
utter
utterance
voice / my voice / the voice
my hands
expression
control
words
tarsier
dragonfly
colossal squid
four eyed fish
spookFish
bush baby
loris
eagle
hawk
buzzard
seal
jotComm
look
glance
glancer
eye glass
skim
skimr
skimmer
speed
slide
glint
connect / connection / iconnect / connectr
eye window / iWindow
spark / sparkle / the spark
flash
freeType
jotKey / dartKey / freeKey / dictaKey / flickKey
blink
gaze
watch / watcher / the watcher
escape
free / freedom
soar
independent / independence
control
autonomy
auto
power
live

Error tone when user triggers over disabled key, out of bounds, or without up to date point info

Improve accuracy of multi key selection, especially when using mouse

Detect vector changes (changes in direction) and use those to create "anchor" letters to improve the matching to dictionary words.

Check changes from last major refactor

Once over the changes I made yesterday, especially DistrictUntilChanged logic.

Show current position of gaze

• If SelectionMode == Key then use position VM property to highlight keys (if setting is on) - highlight key value in blue
• Stub out if SelectionMode == Point then use position VM property to display cursor at position in blue

Improve fixation triggers by holding onto user focus more deliberately

P218/219 of Gaze interaction and applications of eye tracking.

1. Virtual expansion of the area being selected.
2. Grab and hold - keep adding to fixation until a saccade is detected that would move focus away from the selection zone.

Package into setup project

Recalibrate from within app

• Recalibrate function from button (in "more" menu)
• Recalibrate from trigger signal - something unique when eye tracker has ceased to be useful

AudioService / AuditoryService / SoundService

• Selection (complete) - these occur at the start (always) and end (fixation triggers only at the moment). Should include starting MultiKeySelection capture - check the selection event is fired for this (it should be).
• Settings to choose sound on selection - clicks, beeps, etc
• Speak (output buffer) - uses voice settings

Improve menu layout and icons

Auditory trigger

Sound activated trigger.

Throat mounted microphone.

Would need a calibration step to get levels.

Create and display keyboard controls on MainView

• Display keyboard based on the Keyboard class property + Language setting.

ALPHAKEYBOARD (some function keys also on Numeric/Symbol keyboard, e.g. Back)

• Shift (tri-state state - reset to off after publishing key+char)
• Ctrl (tri-state + only enabled when publishing - reset to off when not publishing + reset to off after publishing key+char)
• Alt (tri-state + only enabled when publishing - reset to off when not publishing + reset to off after publishing key+char)
• Windows Key (tri-state + only enabled when publishing - reset to off when not publishing + reset to off after publishing key+char)
• MultiKeySelection allowed
• Back
• BackWord
• NewLine
• Keyboard transition (to other keyboards) buttons
• Space
• Clear (output)
• Speak (output)
• Publish (output)
• SavePhrase
• Suggestion buttons (x5)
• ("...") button - Function keyboard, Recalibrate, Settings, Dictionary management

PUBLISHKEYBOARD:

• Tab (only enabled when publishing - reset to off when not publishing)
• ESC
• Copy/Cut/Paste/SelectAll
• Tab
• Arrow keys
• PgUp/PgDown
• NumLock/ScrollLock/Pause/Break/Print Scrn/Home/End/Delete/Insert/F1-12/Menu key
• AddToDictionaryKeyboard - take any string and assume it is valid - calculate hash and ask user.
• Push key map to InputService (on load/size changed/window state (minimised/restore/other - if this doesn't fire resize)

THEME:

• 1st theme called "Sense_Dark".
• Theme exists in 1 file, which can merge in other dictionaries like this;

<ResourceDictionary xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
                    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
    <ResourceDictionary.MergedDictionaries>
      <ResourceDictionary Source="/WPFToolkit;component/DataGrid/Themes/generic.xaml" />
      <ResourceDictionary Source="/WPFToolkit;component/Calendar/Themes/generic.xaml" />
      <ResourceDictionary Source="/WPFToolkit;component/DatePicker/Themes/generic.xaml" />
    </ResourceDictionary.MergedDictionaries>
</ResourceDictionary>

• KeyboardViews should be as basic as possible. Try to put all styling/animations etc into the theme?
  Reference all resources as DynamicResources

Dynamically changing themes

Selective = http://svetoslavsavov.blogspot.co.uk/2009/07/switching-wpf-interface-themes-at.html
Remove all and re-add = http://stackoverflow.com/questions/6229724/change-theme-at-runtime

THEME RESOURCES...

KeyDefaultForegroundBrush
KeyDefaultBorderBrush
KeyDefaultBackgroundBrush

KeySelectionProgressBrush
CapturingMultiKeySelectionBrush

KeySelectionLockForegroundBrush
KeySelectionLockBorderBrush
KeySelectionLockBackgroundBrush

KeySelectionForegroundBrush
KeySelectionBorderBrush
KeySelectionBackgroundBrush

KeyDisabledForegroundBrush
KeyDisabledBorderBrush
KeyDisabledBackgroundBrush

KeyHoverForegroundBrush

PointSelectionProgressBrush
PointSelectionBrush

ScratchPadForegroundBrush
ScratchPadBorderBrush
ScratchPadBackgroundBrush

COLOURS...

• Green
  suggestions foreground
• Blue
  key highlight when user looks at key (blue border / blue foreground / blue glow around foreground /
  blue glow around border - BlurEffect class) OR perhaps another colour, e.g. yellow?
  selected/on state of some function keys, eg shift, MulitKeySelection, etc
  cursor in SelectionModes.Point
• Black
  default background
  frame when no capturing (make the frame default to transparent)
  foreground of output text
• Red
  selection progress ring
  frame when capturing
• White
  key value foreground when not selected
  background of output text
• Grey
  key borders
  whole key animated grey on selection
  disabled button foreground (e.g. CTRL is disabled when not publishing)

KEY STATES....

Using blue as the highlight/selection colour...

• Default = white foreground, black background
• Hover = blue highlighting around button (glow outside border)
• Select = blue foreground, grey background
  This shows for a short period (~500ms) on a single selection.
  This stays on for keys that are toggled on, e.g. SHIFT.
  This stays on with the addition of a blue bar under the symbol/letters when locked on, e.g. SHIFT behaving like CAPS LOCK.
• Progress = blue progress ring around key, behind key value so it doens't obscure it.
• Disabled = grey foreground, black background

Key Selection is bound to a property changed event

The Selection property need to be toggled between true and false on a Key for the animation to fire when the last/current value are the same KeyValue. The more correct approach would be to fire a Routed or CLR event and handle this with an event trigger, as the real life behaviour is an event. This may be possible, but Routed events cannot be easily accessed by child elements in a Key's template (without each element explicitly locating the parent Key object and attaching, which is ugly). Opted instead to toggle the Selection property value on each Key. Ugly, but works.

Phrases

• Button to add current output buffer to dictionary. MainVM generates hash, or error if not appropriate, then passes to Question keyboard class for display.
• Workflow: user must clear the output buffer and write phrase before saving.