Adding Accessibility to LiveCode

[trevordevore]

Has anyone given much thought to accessibility in the engine? I don't believe LiveCode can work with screen readers or other apps that interact with other applications using the accessibility protocols. I was browsing through the Cocoa docs on adding accessibility to your application and thinking about what a good approach for it would be in LiveCode. Here are the docs I was referencing:

https://developer.apple.com/library/mac ... 000118-SW1

I don't know how accessibility works on other platforms but perhaps we can start putting together a spec of how it would be implemented in LiveCode.

Thoughts?

[monte]

This question was raised some time ago in relation to iOS but I suspect it's the same on OS X. Basically native controls come with it built in and as we use emulated controls that won't work. From memory I vaguely had a plan to create a heap of transparent views that overlaid over lc objects and had their accessibility elements set. There's also some parts of the accessibility api that could be hooked up as functions or commands.

[LCMark]

As far as I'm aware the accessibility APIs on most platforms work on an abstract representation of the structure of a window - essentially you provide a 'tree' describing the types of controls, their nesting and their relationships. This is generally done separately from the view hierarchy so the fact we don't use native controls shouldn't be a problem. I think it would be a case of coming up with an appropriate property of controls which has the appropriate metadata, the engine could then use this to present the appropriate structure to the OS-specific accessibility APIs.

[trevordevore]

@runrevmark - that is my impression as well after skimming through some docs last night. I came across a sample app which may provide some guidance: Sketch+Accessibility.

https://developer.apple.com/library/mac ... TS40008920

The files under the Accessibility folder contain a FauxUIElement class that implements the NSAccessibility protocols:

https://developer.apple.com/library/mac ... rence.html

This particular file in Sketch seems to be an example of using the FauxUIElement:

https://developer.apple.com/library/mac ... lementID_7

[trevordevore]

The "Access Enabling a Cocoa Application" page provides an overview of how accessibility works and provides a table of the default accessibility support for some Cocoa classes (NSApplication, NSWindow, NSView, etc.). Perhaps that can provide some guidance:

https://developer.apple.com/library/mac ... 9-BAJJIBFA

Regarding a property of controls with the metadata - The following page has a list of constants for attributes and roles.

https://developer.apple.com/library/mac ... roup/Roles

At this point I'm not quite sure what the first step would be in planning the spec as I'm still trying to wrap my head around all of this. Perhaps just coming up with the simplest spec that would make the application/window menu, fields and push buttons accessible?

[LCMark]

@trevordevore:

How about this for an architecture for accessibility (ignoring the details of what accessibility information needs to be presented!). We add properties 'accessible' and 'accessibilityInfo' to MCObject (so present for all controls). Each control type has an inherent (simple) way of mapping properties it has to a reasonable set of access information (not using accessibilityInfo), the individual attributes of which are overridden by those in the accessInfo property (so you can customize).

From what I understand the accessibility frameworks tend to start at the root UI element and then work inwards. So things would start with a callback to (say) an MCStack asking for an accessibilityInfo array - this would actually be handled by MCObject: if the object has 'accessible' true, it builds the default accessInfo array which is then union with a set accessInfo array (if there is one - props in the set one taking precedence), then it adds any child objects which have 'isAccessible' true (my understanding here is that we don't need to build those accessInfo arrays yet, just tell the AX framework that those children are there; when it wants the info it will ask for a specific object's info).

I think this model should be general enough to cover all platforms - from the LiveCode side we are just defining a nested array of information describing the controls and such. The platform layer would then map this to specific properties and roles as appropriate. Also, by using an 'accessInfo' array it means that we can easily have platform-specific properties - i.e. ones which are untouched by the abstraction layer and just passed through to the appropriate OS.

So, all that being said, the main thing to work out is what the properties of the accessInfo array should be, how the default accessInfo arrays are built for each object type, and then how the abstracted properties we define map to the platform-specific properties for each platform.

[Martin Koob]

I am interested in this as I am developing software used by Deaf people for viewing sign language videos. One thing I was wondering if one of the types of accessibility info be in the form of video. This would be to provide an alternate version in Sign Language of control labels, control tool tips or dialog messages using short video clips. Would this be possible in the proposed framework?

Martin Koob

[trevordevore]

Martin - I believe accessibility frameworks are only to provide low-level information for screen readers or to allow other applications to input information into an app. You would probably need to manually hook up your sign-language videos yourself. Now, you may be able to make use of the accessibility information for a particular control in order to determine which video you want to show.

[trevordevore]

@runrevmark - That sounds like a good approach to me Mark. I will work on compiling an initial list of properties that would be applicable.