CapabilityPalletDesign

Spelling mistake noted - should be *palette*

Conceptual Notes from a OSK-ng mailist discussion on this page OSK-ng wiki page. I never posted this as it needs more work

The main thing is to keep the gestures unbound from the application actions so that the user can map the gestures they want to use to specific application actions. Thus some way of listing possible actions across a11y API is needed.

There's a little fluidity in the layers but I think we want them all as they add something (I now think there are too many layers, perhaps events become privative gestures which can be aggregated into compound gestures. ). Each layer facilitates pushing logic down so it can be shared/reused by items in the higher levels. I'm also thinking of exposing this declaratively in configurations. There is flexibility as to where declarations are done, gestures, actions or application logic. For example you can define 'press' and 'release' gestures, or combine into 'hit'. Here's a couple of examples to help discussion

Single switch user using simple auto scanning:

• A switch generates 'discrete' physical events ('on' and off)
• Mapped to 'discrete' logical events unchanged
• The gesture 'press_switch' consists of a single 'on' event followed by an 'off' in a defined time.
• The gesture 'hold_switch' consists of an 'on' not followed by a 'off' for a defined time.
• The gesture 'release_switch' is a 'hold' followed by an 'off' (thus a release follows a hold).
• The 'scan' action is initiated by a 'hold' gesture and stopped by a 'release' gesture, dependant on the current state. A 'press' gesture invokes the 'activate' action on the selected item when no scan is occuring

Joystick user with no fine motor control operating on screen keyboard:

• An analogue joystick generates 'continuous' events
• Mapped to 'discrete' logical events such that a minimum value pair map to 1 of 8 directions. Less that the minimum is 'centred'.
• The gestures 'joystick_up', move_down' etc are defined (or perhaps gestures can have parameters)
• The action 'select_up' is invoked when the gesture 'joystick_up' occurs.

Events are bound to a specific device as are gestures (device is a attribute of both). Events and gestures may have parameters such as 'key_pressed: <keycode>' A pallet of gestures are defined for a user A Action set are defined for an application

A mouse is a combination of pointer + switch devices with compound gestures like right drag.

Actions can have alternative gestures. For example 'select_up' can use gestures 'joystick_up' or 'hit_key: up_arrow', a press of the up switch or a head mouse gesture (such as flick_up). Actions can be made from gestures from several devices (mouse delta: 10,10 and press button) rather than having separate gestures and logic in the application.

Gestures and actions will be state machines. I'm note sure if each key want's to be an gesture, we

I'm wondering if we could abstract scan modes into 'Action pallet' such that changing the gestures for each action changes the mode. That might be asking too much as you need to handle automatic and manual scanning. Having timer psuedo gestures may be the answer and could involve pushing state down.

Thus the action 'Next_Cell' could be defined as psuedo gesture 'scan_time_passed' in one pallet or as gesture 'Switch_1_pressed' in another. 'Start_scan' could be 'switch_held' or 'switch_pressed', 'Stop_scan' would be 'switch_released' or NULL. Each user could have a personal action pallet or perhaps even several depending on mood.

I haven't dealt with interaction with the system pointer but I guess system events can be used to define gestures within an application. Capture will perhaps be required?

The simple and more common cases can be grouped into default options in the settings UI based on enumerated devices found on the system (with reenumerate on new device). More complex custom mappings can be available via config files and/or advanced UI.

Position and application state are important.

Input methods look interesting as support speech or handwriting, http://www.mozilla.org/projects/intl/input-method-spec.html

Also see KMouseTool