Retaining Backwards Compatibility in a Changing World

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, LCMark

LCMark
Livecode Staff Member
Livecode Staff Member
Posts: 1206
Joined: Thu Apr 11, 2013 11:27 am

Retaining Backwards Compatibility in a Changing World

Post by LCMark » Fri Mar 06, 2015 8:40 pm

As LiveCode moves forward we are increasingly being 'held back' to a certain degree with the need to retain (as far as is practical) existing script compatibility.

There are a raft of changes (some quite significant) which would improve LiveCode Script no-end (whether it be from the point of view of new people coming to the platform, or making it possible for existing people to write better and more robust programs), but come at a hefty risk of breaking existing scripts. Here are a few:
  • setProp / getProp handlers always firing regardless of the state of message locking (lock/unlock messages)
  • whether an array's keys are considered case-sensitive should be a property of the array, and not based on a local property
  • arrays really should throw an error when attempting to convert to a string
  • revXML and revDB functions should be returning (abstract) strings and not raw data
  • 'the result' not being a global variable and being local to handlers
  • error handling being unified as exceptions
Some sort of mechanism which means that existing scripts can run in a 'compatibility mode' to allow their authors to take advantage of new features immediately, whilst updating their existing scripts over time would be the ideal here.

One idea is to introduce an (inheritable) 'scriptVersion' property of objects. This would be a monotonically increasing integer, at each increase a number of semantics would change meaning that you would need to review the scripts of the objects for which you needed to increase its value.

Another idea is to introduce an (ever-increasing!) collection of boolean flags which could be set on objects to describe the specific semantics to which the script has been written.

On the one hand, a single scriptVersion might be too coarse. It is easy to imagine that some systems might be stuck at scriptVersion X for a long time because scriptVersions X+n all contain changes in semantics which would take a long time to patch code to support.

On the other hand, a huge array of boolean flags might be incredibly confusing (and produce an internal engine maintenance nightmare).

Perhaps there's something in between (LCB modules will eventually be intrinsically versioned - the author the module if he/she do their job correctly will be able to support API changes for as many version of the module as they wish ensuring that a program which uses version N of a module will continue to function even if version N+M is the current one).

Or perhaps we should all just 'swallow hard' and realize that it is untenable for LiveCode to continue support script semantics (many of which grew organically and were defined more than a decade ago) in a world which changes as fast as the computer industry does and all take the hit in terms of the work required to keep up to date with the latest and greatest versions and features. (Its the mixed mode here which I'm talking about - allowing existing scripts to 'forever' work alongside scripts which use much newer and perhaps substantially different functionality and semantics).

Anyway, just thought I'd post this to see what people think...

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Retaining Backwards Compatibility in a Changing World

Post by FourthWorld » Fri Mar 06, 2015 10:37 pm

That's such an awesome list of goodies that my vote goes to whatever is easier for you to implement, which I'd guess would be a single global variable.

That may be easier for us as well: we can refine some script, turn it off, see what we missed, turn it back on to do more refinement, and with that one switch we can easily check things as we go.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

malte
Posts: 1098
Joined: Thu Feb 23, 2006 8:34 pm
Location: Ostenfeld germany
Contact:

Re: Retaining Backwards Compatibility in a Changing World

Post by malte » Sat Mar 07, 2015 8:44 am

My 2 cents on this.

Whatever the change to behaviour may be, the most important thing is that it is clearly documented in the changelog.

Then it is up to the developer to decide to take the risk of switching to the more recent engine and make the required changes, or stick a little longer with the older engine.
On top of that, whatever the change in behaviour may be, there must be a benefit in doing it. If we are looking at things like changing the behaviour of revDB or revXML, what is the benefit to changing it from RAW data to an abstract string? What exactly would that mean? What is the price paid to make that change. If it only is that scripts need to be adapted, all fine with me, if clearly documented. If however access to data gets significantly slower than it was previously was, it might be worth reconsiddering if the change makes sense and leaving things as they are.

Sometimes change is inevitable and we as users need to adapt. I would dare to say that at least I would be willing to swallow rewriting old scripts (which might not be optimal anyways) if the benefit of doing so is high enough. In the rare cases were we are forced to support multiple versions of the engine, some way to reacting on the engine version would be good to have (without the need of resorting to "do" if at all possible), but even if that is not possible, there is always the option to leave older projects with older engines, or bite the sour apple and rewrite parts as necessary.

All the best,

Malte

richmond62
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 9285
Joined: Fri Feb 19, 2010 10:17 am
Location: Bulgaria

Re: Retaining Backwards Compatibility in a Changing World

Post by richmond62 » Sat Mar 07, 2015 10:47 am

Possibly, rather than JUST notes in the change-log, Clear indications within the Documentation about what has been
changed.

Currently in the Dics things are flagged as "changed" or "introduced".

"Changed" never outlines HOW a feature has been changed - that would be a very great help.

LCMark
Livecode Staff Member
Livecode Staff Member
Posts: 1206
Joined: Thu Apr 11, 2013 11:27 am

Re: Retaining Backwards Compatibility in a Changing World

Post by LCMark » Sat Mar 07, 2015 1:11 pm

@richmond62: Yes - we would, of course, have to be better at reporting 'breaking-changes'. To be fair, we have managed (for the most part) to not break things intentionally (and where we have, it has been due to us introducing bugs in almost all cases) but moving forward we are improving on the docs maintenance front. (We are currently in the process of transitioning to markdown files for all docs as the source files - thus making it a lot easier to make edits alongside the actual engine patches - so hopefully that will help a great deal moving forward). Indeed, if scriptVersion or similar is to become a reality it is going to require a higher-degree of care in the docs as otherwise everyone is going to get confused - including me and my engineers!

@malte: Each of the things I've listed above have an end benefit which far outweighs the effort of making the change (and there are more than just that list) - however, they cross-cut the entire range of engine functionality which means that we have to be careful to do it in such a way that script-writers can feasibly update their code. This is why it is important we get the granularity of any compatibility property right - too coarse and you have to change to many things to move forward, too fine-grained and it would make it a nightmare for actually updating scripts as throughout a large program you could have scripts with varying different compatibility options.

In terms of the value of each...
  • getProp/setProp: This has been long discussed on other threads - it makes setProp/getProp much more useful for the purpose of building reusable script-controls (whilst the widget architecture does supplant this to a good degree - it doesn't yet solve the problem of aggregate objects/templates being built at the script level).
  • caseSensitive an innate property of an array: The current behavior of arrays is basically wrong in this regard - if you try and use an array with a different caseSensitive from which it is created there will be bugs in your code. The current semantics complicate the engine implementation, and complicate the use by the scripter.
  • arrays cannot convert to/from strings: Again, this is more a robustness thing - arrays are not strings thus they shouldn't try to be. I've seen inordinate numbers of bugs in script come about because of an implicit conversion which is unchecked.
  • revXML and revDB: At the moment these functions expect 'native' strings which the externals treat as data - this means you miss all the benefits of unicodification and you have to do unicode by hand. This adds extra complexity to script, and extra work the engine has to do. There's no performance penalty here for making the change - the engine already constructs an abstract string, it is just it is a native string which can then be treated as data. The problem moving this forward is that (currently) to keep script-compatibility we'd have to create a separate set of 'Unicode' suffixed revDB and revXML APIs as the engine can't know from context whether you've updated the scripts using the current APIs to expect data/native strings or proper strings.
  • the result being local: A global result variable is 'okay' if you know that is how it works, but it simply is not how any other language in use works. It has an effect not just on function return and error return, but also exception handling (try-catch works quite differently in LiveCode from other languages due to it).
  • error handling unificiation: Again, discussed on a separate thread, error handling in LiveCode is incredibly confusing at the moment - for a language that is aimed to be 'easy-to-use' there needs to be a single unified approach to ensure that people can easily write robust code.
Obviously each of these has varying degrees of impact and implementation complexity (some like revDB and revXML are really quite straight-forward, others like error handling unification require a fair bit of engine work if only because of the breadth of the change), and varying degrees of benefit.

Ultimately, we don't want people to be held back from being able to access the benefit of new features we are adding (particularly widgets); but on the other hand don't want to hamstring the new functionality because we have to support old semantics. We need to find a healthy balance between the two.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Retaining Backwards Compatibility in a Changing World

Post by FourthWorld » Sat Mar 07, 2015 7:15 pm

runrevmark wrote:Ultimately, we don't want people to be held back from being able to access the benefit of new features we are adding (particularly widgets); but on the other hand don't want to hamstring the new functionality because we have to support old semantics. We need to find a healthy balance between the two.
This may not be a popular view, and I fully expect to have food thrown at me again at the next RevLive for suggesting it <g>, but I agree if we want the future of LiveCode to be longer than its past we'll have to accept that from time to time we need to update code to let the platform move forward.

The RunRev team has already maintained one of the most stellar track records in the industry for backward compatibility, and your proposal for a property to allow switching between older and newer syntax goes far beyond what many other languages provide.

Consider Python, and their sweeping revisions for v3:

Should I use Python 2 or Python 3 for my development activity?
https://wiki.python.org/moin/Python2orPython3

They call it out plain and simple in the opening line there:
Short version: Python 2.x is legacy, Python 3.x is the present and future of the language
Their community may have delivered migration tools, but AFAIK the core team provided no dual-interpreter option such as you're proposing - and yet none of those changes has kept Python from being a popular choice for an ever-larger range of projects.

Ask any VB developer about VB6 and you'll get an earful even louder than the most die-hard Python 2 fan. :)

RunRev's efforts to maintain backward compatibility are among the best the industry has seen.

IMNSHO, you needn't worry too much about moving forward as you've proposed here.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7214
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Retaining Backwards Compatibility in a Changing World

Post by jacque » Sat Mar 07, 2015 9:50 pm

This is an important enough question that I think it should go on the mailing list as well as this forum. There are lots of pro developers there who rarely or never visit the forums, and their input would be valuable. I've not yet decided what my response should be; I agree we need to move forward but I can't think what the easiest path would be to get there. A larger group for brainstorming might come up with something. I can think of several dedicated developers who will never see this question and it would be better to get their feedback earlier rather than after the fact.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

Vanceone
Posts: 37
Joined: Sat Dec 15, 2012 7:27 pm

Re: Retaining Backwards Compatibility in a Changing World

Post by Vanceone » Mon Mar 09, 2015 11:10 pm

Why not do what Apple does? They have target SDK's that you build against; so you aim for 10.7 and higher, or whatever. And then require developers to, in preferences for a project, issue a target build and maybe an SDK build (so you can use all the new stuff, but it builds an app compatible with whatever; like WinXP and Tiger, or Vista and Snow Leopard, and the equivalents for going forward.

That leaves it up to the developer what level of compatibility they want; if LiveCode builds in a feature that targets Apple's Continuity features, that could go in to a 64K build or something. And LiveCode wouldn't build it for Leopard or Windows.

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Retaining Backwards Compatibility in a Changing World

Post by mwieder » Mon Mar 09, 2015 11:40 pm

Mark-

I'm fully behind breaking backwards compatibility, within the constraints listed above. I think something like the hcaddressing flag could maintain compatibility where needed, but in any event, the stack file format should be an indicator. I can think of many other things I'd like to break besides the list above <g>, in particular that stupid thing about a trailing comma-and-space combination being a list item, but a trailing comma *not* being an item.

pkocsis
Posts: 105
Joined: Sat Apr 15, 2006 7:20 am

Re: Retaining Backwards Compatibility in a Changing World

Post by pkocsis » Tue Mar 10, 2015 10:13 am

runrevmark wrote:Ultimately, we don't want people to be held back from being able to access the benefit of new features we are adding (particularly widgets); but on the other hand don't want to hamstring the new functionality because we have to support old semantics. We need to find a healthy balance between the two.
FourthWorld wrote:This may not be a popular view, and I fully expect to have food thrown at me again at the next RevLive for suggesting it <g>, but I agree if we want the future of LiveCode to be longer than its past we'll have to accept that from time to time we need to update code to let the platform move forward.

The RunRev team has already maintained one of the most stellar track records in the industry for backward compatibility, and your proposal for a property to allow switching between older and newer syntax goes far beyond what many other languages provide.
I'm not an engine contributor, just a long time runrev/livecode user. My .02 would be close to Richard's, except maybe to go even further and not worry at all about supporting a property for 'older' syntax/behavior. Leave older syntax for older engines.

Runrev's resources are not unlimited, and many of us have waited a long time for things like documented android externals (hopefully to be supplanted by an ability to access low-level api's in LC8) and many other abilities available in each platform not directly supported in native LiveCode. I think the community should realize how detrimental and stymieing it can be to the finite LC developers to attempt to move forward with (needed) abilities coming in LC8 while still maintaining strict backwards compatibility. IMO I'll gladly take a simple list of what syntax/behavior changes have occurred, and deal with my own code while I finally get to implement some nitty-gritty's in my applications that were difficult to impossible to implement in native (pre-LC8) LiveCode.

I say we set the LC development team free.......just give me a list of what syntax/behavior changes have occurred, and I'll do my small part in keeping LC moving forward by simply changing and maintaining my own code. (GEEZ, after that diatribe, I sure hope that access to low-level api's is indeed one of the goals of LC8 :) )

LCMark
Livecode Staff Member
Livecode Staff Member
Posts: 1206
Joined: Thu Apr 11, 2013 11:27 am

Re: Retaining Backwards Compatibility in a Changing World

Post by LCMark » Tue Mar 10, 2015 10:30 am

@Vanceone: The issue of platform support is orthogonal to the script semantics changes being proposed here - in general we've done quite well at keeping support for older platforms alongside newer ones. In this case, we are talking about changing the way script actually works in some ways - making certain operations more strict for example, or tweaking some behavior to make it more 'standard' in comparison with other languages (partly for consistency with them and to make it easier to learn, but mainly because there are a few cases where LiveCode actually does things in what could be considered 'the slightly wrong way').

@mwieder: Hehe - let's perhaps avoid getting back into the delimiter discussion. I will defend to the end of my days the current behavior as it is the only *correct* one in terms of what people expect to work (although I will admit there are a few bugs in the engine and externals still which mean it seems inconsistent). That being said, expressing that particular example in that way does demonstrate it comes down to what you think your string-lists contain - which is perhaps the point of contention. If your string list is a list of strings then whitespace is probably important, if it is a list of integers then it isn't - anyway, perhaps that is a discussion for another time ;) (*starts pondering the idea of typed/structured string-lists*)

@pkocsis: Although the foreign handler support in LCB (the new language which we have created for the purposes of extending, and eventually writing, LiveCode) is still in its infancy (you can bind C functions with primitive type arguments - int, double, float, long, pointer) it is still one of the top goals to ensure it easy to interoperate with code written in other languages. In particular C, Obj-C, Java, C++ and JavaScript (although I hope we can generalize the ideas so that it is relatively straightforward to plug-in access to any language).

Obviously C (being the lowest common denominator) is the first - this works now, although it is still a little more fiddly than I'd like - Java and Obj-C are the next on the list, then probably JavaScript (for HTML5 interoperation) and finally C++. The reason C++ is the last on the list is that it is actually the most difficult to easily bind to as well as not being the 'standard' language of any of the platforms we currently support / intend to support (Java is obviously absolutely required for Android, Obj-C is absolutely required for iOS and Mac, JavaScript is absolutely required for the web).

jamsk
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 16
Joined: Tue Dec 18, 2007 4:08 am
Location: Atlanta, GA
Contact:

Re: Retaining Backwards Compatibility in a Changing World

Post by jamsk » Tue Mar 10, 2015 7:01 pm

Change is painful, and letting go of the past is difficult, but I say unshackle the team and move forward. Keep the legacy engines available for older code to run on older systems, but don't let compatibility issues interfere with moving the platform forward. OSes will continue to evolve, hardware will continue to evolve, and so must coding. Just don't 'pull an Apple' and suddenly abandon older technology.
----------------------------------------
Because I won't always be a n00b!

ChrisSheffield
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 8
Joined: Mon Apr 10, 2006 3:35 pm
Location: Idaho Falls, ID

Re: Retaining Backwards Compatibility in a Changing World

Post by ChrisSheffield » Tue Mar 10, 2015 7:08 pm

Mark (and everyone),

My main concern is not so much backwards compatibility with different LiveCode engines, but rather backwards compatible with devices (of course, maybe I'm totally missing the point to all this, and that shouldn't even be a concern).

At Read Naturally, Inc. we are currently using LiveCode strictly for iOS development. We've considered doing Android, but just haven't taken the plunge yet. We're in the education market, which, in most cases, means having to support computers/devices that are several years old. Schools in the U.S. just plain don't have the funds most of the time to keep everything "cutting edge" up-to-date. Our apps in the app store still support the iPad 1, and we'll continue to do so for as long as we can (even if it means for testing purposes we have to create two separate builds of each app because apps built with 64-bit support won't install/run on an iPad 1; we'll have to continue building 32-bit only apps for the iPad 1).

So would these proposed changes require us to ditch support for old hardware moving forward, or is that not the case, at least not yet? I realize eventually the time will come that we will have to do that, but hopefully not quite yet.

Thanks,
Chris Sheffield
Read Naturally, Inc.
Chris Sheffield
Read Naturally, Inc.
The Fluency Company

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: Retaining Backwards Compatibility in a Changing World

Post by phaworth » Tue Mar 10, 2015 7:12 pm

I'm certainly in favor of moving forward with changes like those listed.

I'm less sure about control of backwards compatibility by way of a version rather than a pick-from-the list of flags. Hard to know for sure, but if a whole bunch of changes affecting backward compatibility were to appear in a single release, it could be a pretty huge job for developers to change their code to handle all the changes at once, especially if there are some they don't really care about. I would vote for a flag for each item for maximum control, or perhaps that as well as a version compatibility setting so developers could choose one or the other.

It may be useful to provide a tool that would analyze all the scripts in a stack and highlight those that would need changes for compatibility reasons, although I suppose it would be fairly easy for developers to write their own by looking for the commands affected in any particular release.

Pete

Mikey
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 755
Joined: Fri Jun 27, 2008 9:00 pm

Re: Retaining Backwards Compatibility in a Changing World

Post by Mikey » Tue Mar 10, 2015 7:19 pm

Coarse is better, IMHO. It makes everything less subtle, weird, and special-case-prone.
Also, in a post-Builder world, in theory, backward compatibility should be more of a matter of strapping on an old module to your stack/project, etc. than it is something else. Yes, if the legacy lovers want a feature to be relevant, moving forward, they have to work on it, but bootstrapped LC makes that possible. Hopefully, Builder will make it far simpler for more folks to contribute to the code.

Forward faster is ALWAYS better. LC made itself relevant when it became the only really easy platform for developing for mobile, but it's slipping (Android support is weak, native controls will have to wait for 8, and in the meantime, our skinning tools aren't being worked on, because until 8 is closer to reality, investing in developing those tools makes less sense). Hell, if forward faster means getting rid of parts of the IDE and using publicly available tools, then do it, already.

I think everyone is keenly aware of a couple of my legacy-holdout grumbles, so PLEASE do it.

Locked

Return to “Engine Contributors”