Page 2 of 4

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 7:49 pm
by FourthWorld
ChrisSheffield wrote:So would these proposed changes require us to ditch support for old hardware moving forward, or is that not the case, at least not yet? I realize eventually the time will come that we will have to do that, but hopefully not quite yet.
In my reading these changes are internal to LiveCode, and should not affect the scope of OS versions it deploys to.

That said, we can continue to expect that from time to time supporting older versions of an OS long after the OS vendor stops supporting it may hold back essential development. We saw support for OS X 10.5 dropped several moths ago, and we can expect other older versions to be dropped in the future.

But I don't see anything from Mark here suggesting that the rate of dropping support for vendor-unsupported OS versions will in any way accelerate.

On that issue I would imagine the rate of change would be more or less the same as what we've seen in recent years, in which RunRev continues to outperforms a majority of other developers and nearly all OS vendors with the level of support they provide for older OS versions.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 8:22 pm
by asayd
I'm with Pete: Having a tool that could analyze existing code and flag potential problems would go a long way toward easing the pain of transitioning to a newer, non-backward compatible engine. And I would hope that there would be plenty of overlap between releasing the new version and continued support (I'm talking about things like bug fixes) of a legacy version.

In my world, where I'm teaching a new crop of students every year, backward compatibility is much less important than introducing them to a new, fast, robust environment.


Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 8:26 pm
by mwieder
I'm sort of hoping that someone will come up with a good argument *for* maintaining backward compatibility, because I can't think of one.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 8:35 pm
by FourthWorld
asayd wrote:I'm with Pete: Having a tool that could analyze existing code and flag potential problems would go a long way toward easing the pain of transitioning to a newer, non-backward compatible engine.
Could this be a community project? Anyone here have time an interest to lead a team on this?
And I would hope that there would be plenty of overlap between releasing the new version and continued support (I'm talking about things like bug fixes) of a legacy version.
Ideally it would seem that stable version would be v7.x. I would imagine Mark's proposals would fit well with v8.x, but I don't think he's intending to backport them to older engines.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 8:42 pm
by LCMark
I should perhaps say that this discussion that I have started is not suggesting anything 'huge' and 'monumental' is going to happen overnight. It was motivated mostly by a recent element of frustration on my part when very useful setProp/getProp tweaks surfaced again - changing those semantics is fine, but doing so in a backwards-compatible way requires doing something which really quite 'unclean' and ultimately highly inefficient. If, however, we have some sort of 'compatibility mode' (e.g. scriptVersion) then it is an incredibly easy, simple change.

We are still committed to getting 7.0 working in exactly the same way as 6.7 (except for the very few cases where it just hasn't been possible - the use of 'useUnicode' and numToChar() being the only one which comes to mind - but even that is more to do with the lack of unicode working transparently in that specific case), and 8.0 will be the same - it will be 7.0 with the extensions architecture carefully slotted into it with the intent of affecting existing script as little as possible.

We are also still going to redefine existing LiveCode Script syntax when we have a final Open Language parser implementation - and with that we will have a script translator which will rewrite scripts automatically from the old forms to the new (remember that the engine knows precisely what it thinks you mean when you write a command - even if it accepts varying ways of expressing it at the moment).

What this is more about is working out how to manage changing dynamic semantics of the language which a purely syntactic translator (which is, by definition 'static') cannot touch (for example, fixing cases where what is actually text is being treated as data and vice-versa - this is a function of where the data comes from, which in many apps could be anywhere and is particularly apparent in custom properties which prior to 7.0 do not record the necessary information about whence they came for the engine to make an informed choice). We can certainly flag syntax whose semantics might be affected by these changes - but actually 'knowing' what you mean to do in your script is perhaps something a little beyond what could realistically be done (unless a huge amount of engineering was devoted to it - a cost-benefit analysis of which I doubt would come out favourably...).

So ideally we find a way to help people start transitioning their scripts in the not to distant future - not so much syntactically but semantically - the goal being everyone's code runs faster and is more robust; and when the new syntax does finally materialize it is a matter of a simple translation process and you can happily continue but being able to take advantage of all the new features and abilities we hope that both extensions and open language will bring.

(Although that being said, there is one important syntactic change I'd like to roll into some sort of mechanism here - that is forcing you to use 'the' and 'of' - I'd very much like widgets to not only be able to export properties, but also custom chunks sooner rather than later, but there is no hope of this working unless 'the' and 'of' use are strictly enforced due to huge parsing ambiguities that would arise as a result of the current parser laxness).

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 8:48 pm
by LCMark
And me again - I started working today on classifying various bugs in Bugzilla as 'anomaly'. These are things which this thread is essentially talking about - what many would call 'bugs' or 'quirks' of LiveCode and would likely be happier if they weren't there but we cannot change without breaking existing script.

Note this is by no means complete yet - I still have to process quite a few things into bugzilla (I have a growing list on my desktop at the moment) - the initial list at the top of this post was just a taster.

Ultimately, we aren't going to change anything unless there is a significant global benefit to do so measured on the appropriate axis... Whether that be ease of use, robustness concerns, efficiency, eliminating quirks (which, by definition, make the language more difficult to pick up for those who are not part of the LiveCode-way yet - this is an important concern, ultimately LiveCode and its predecessors are a bit maverick like in their approach particularly with the 'English-like' syntax aspect so we really shouldn't be making adoption any harder than needed!), or probably several others.

@ChrisSheffield: As I already mentioned - OS / Device support is entirely orthogonal to this. We will continue to do the best we can here - if you are having a problem with iPad 1's with your app then please do file a bug and I'm sure @livecodesebastien or @livecodepanos won't mind taking a look. Apple are a bit vague about supporting older devices and it was a bit of a struggle getting combined binaries working for ARM64 against all the various iOS versions we do support - so I can't promise anything but we can at least look!

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 9:12 pm
by Mikey
Mark, I just got bitten by the inconsistency in the way items are handled vs. how the databases work, again, this last weekend, and because it was on mobile, it was two hours of poking and prodding to realize it, so please add the current item behavior vs. the database functions as one.

Even on my very oldest code (The oldest project I have goes back to 1992), I would rather have to do work to bring it forward, than have supporting it slow any future progress down.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Tue Mar 10, 2015 9:17 pm
by LCMark
@Mikey: That is one of the bugs alluded to in my response to @mwieder about delimiters... We'll look into getting that sorted if we can sooner rather than later - due to the amount of code which is out there to work-around it, it is definitely sitting on the line of something which this very thread is talking about - an anomaly which fixing would potentially break code. If you can remember the bug number related to it, please let me know - otherwise I'll dig it out in the morning and review again - at the very least it will go on the anomaly list :)

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 1:12 am
by monte
Obviously backwards compatibility is the ideal but if there are instances where the language can't progress without breaking some eggs then by all means make an omelette ;-)

I think it would be worthwhile trying to define the cases that you are declaring are anomalies and what dependencies people might have on them.

getProp/setProp - Really it's more likely that the fact the message isn't sent with lock messages false is causing bugs than anyone doing workarounds and the workarounds might be using other events to check to see if the property has changed while messages were locked. Seems to be a non issue to me.

caseSensitive an innate property of an array - Could this be some kind of dynamic property that would require matching keys with different cases to be treated caseSensitive from then on? This would mean the scripter had to set some keys while under caseSensitive and then try to get or set them while not under caseSensitive. Implemented like that it might again be a bit of a non issue.

arrays cannot convert to/from strings: This is a more interesting one because the scripter might use the same variable name for a temporary array and later for a string...

revXML and revDB: Are you suggesting the engine will work out what collation the db is using and just sort it all out for us? Sounds worthwhile to me if that's the case.

the result being local: How would a scripter make use of the global context of the result? I can't really think how this is possible other than a handler that doesn't return anything and then you get the result from the last command in the handler that did set the result...

error handling unificiation: That's going to be a biggie. I made the mistake of proposing error handling be sorted out on the dev list once... I think I'll just keep quiet ;-)

At the end of the day every time we introduce a new keyword there's a possible backwards compatibility issue. I once had to work on a project where the author used header as a variable name all over the place and it broke everything.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 1:29 am
by Mikey
delimiter related bugs: 976, 10727, 13557, 13936.

For anyone else who posted one, I apologize for missing it, but the idea is the same.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 9:42 am
by LCMark
@monte: Yes there is more work to do here, and it will take a while.

It is also important not to presuppose what may or may not be 'out in the wild' in terms of uses of existing features - just because an individual personally doesn't do things in a particular way, or expects something to work in a particular way does not mean others don't do something different. Whether something is an anomaly or not comes down to a very simple test - is the functionality required to change in a way which is not a superset of the previous behavior. i.e. If it is changed will it cause code that has been written against its current definition to work in a different way.

As it turns out there is a lot you can do and indeed we have done where new behavior is a superset of the old. Unicode would be a good example. Stacks being allowed as references in behaviors is another (this should be coming in 6.7.4-rc-2 and 7.0.4-rc-2 - it is already on develop). One of the problems has been that the engine has long been written without regard for strictness - i.e. throwing an error in all cases which have not been explicitly enumerated. (We aren't making the same mistake with LCB - in that world if in doubt we have gone for the most-strict definition and error out in all other cases, thus leaving open a much greater degree of easier evolution in the future).

Whilst I see what you are saying about getProp/setProp unfortunately it has been part of documentation and 'common-law' that you use a certain pattern to do a certain thing with custom properties for more than a decade. Yes changing it might solve other bugs related to the triggers not firing if lock messages is true and that is all well and good, but the change will for 100% certainty introduce a critical unavoidable recursion lock-out in the cases lock messages / unlock messages have been used to access the custom property store of an object in an ancestor object (the example posted on the related thread). Clearly proposed behavior is not a superset of old behavior (and indeed this is the most vexatious case because it really cause scripts which use the current pattern to lock-up if the behavior changes), and the new behavior has an incredibly unpleasant possible side-effect when run against existing code.

I do need to do some more thinking on the caseSensitive in array thing - I'm still not sure even with that idea that arrays are well enough defined. We are trying to model several distinct types of data structure with a single generic syntax here which is the problem (untypedness is nice, but does cause headaches at times) - and the question is how to ensure the engine can make the right choice in context. Really you want the array's type to be inferable from the type of the keys used to build it - but there's a problem there due to the varying ways strings can be compared (its not just caseSensitive, in the Unicode world you have formSensitive as well).

revXML is the case of making it so that rather than returning UTF-8 as native strings, the APIs should be returning abstract Unicode strings (i.e. native of UTF-16 internally as appropriate). At the moment you have to write explicit code to do conversions of the UTF-8 if you want to do text-related processing. We can easily add Unicode suffixed versions (and no doubt will), but it would be nice if there was a flag which meant you could use non-suffixed APIs but still get the Unicode behavior (after all, if you are writing new code against 7.0+ then having to append Unicode to your API calls seems a little redundent!).

revDB is similar. It is about changing the revDB APIs so that it does the mapping from the database's internal charset for a column to an abstract string for you. Its very similar to revXML in that we can easily add 'Unicode' suffixes, but ideally you wouldn't have to use those in new code.

In the array <-> string case, what I mean here is that when you try to pass an array to string context, or a string to an array context then you should get an error. At the moment the conversion is to empty in both directions which hides bugs and unhandled corner cases in code. It doesn't affect what values you can put into variables - just whether or not a type conversion is an error.

the result being local: The current global nature of 'the result' is, quite frankly, an aberration which helps no-one. It is an impediment to readability, correctness, analysis and optimization. Also if one wants multi-threadedness at some point it ceases to become global anymore, it has to be thread-local at the very least with the performance overhead that entails. It is also incredibly fragile - some commands in the engine set the result, some don't - if one doesn't now but it starts to in the future it breaks code; if you unwittingly insert a command in a handler which does set the result, or if you add a return statement to handler which is used in a context where a handler does not expect it to be set, your code breaks. A much better semantic is the one in LCB - 'the result' is always the value returned by the previously executed command, if the command did not return a result then it is empty. It is an entirely local phenomenon with no possible side-effects beyond two side-by-side commands. (Note that local result and global result written scripts can live side-by-side - when going from local result script -> global result script, you just set the global result; local result script would never touch the global result).

error handling unification: Yep - this is a biggy. The refactoring makes it a much easier project to start looking at undertaking, but there is still a fair bit of engine legwork to do.

In terms of the example of introducing new keywords causing backwards compatibility issues - then, yes, that is true... But then we are very very clear and always have been - do not use English words for your variables as your scripts might break in the future. This is why we try to encourage people as far as possible to use variable prefixes, as it almost completely eliminates the problem (tExt being an example case it doesn't catch!). (Of course this problem would be completely solved by removing case-insensitivity from the syntax - i.e. all keywords must be lower-case, all variables and handler names must contain a capital letter - I'll perhaps not propose this as an 'anomaly' though, not yet at least ;)).

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 12:15 pm
by LCMark
@Mikey: I've added the three issues listed in Bug 13936 as anomalies - 14875, 14876, 14874.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 12:27 pm
by Mikey
Mainly, please, since you've made it clear that we're stuck with this behavior, let's fix all the places in LC where it behaves the other way, especially in the database routines.

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 3:13 pm
by dunbarx
I tried to port many HC projects into LC. The process, except for small stacks, was never trivial. That I relied heavily on externals (especially Rinaldi's) notwithstanding, I can say that I was never really disappointed that there was not a foolproof "one click" translation. I simply rewrote a lot of code.

I am far more impressed that there is a one click path to build standalones onto multiple platforms.

I vote to go forward. That said, I have sold only two commercial projects, though they were large, and one was in HC. So beyond that I have only my own stuff to rework whatever might break in future engine releases. This might be far more worrisome to professionals, who do not want either to rewrite existing apps that their customers are using, in order to maintain and upgrade those apps, nor to field angry phone calls that might result if an upgrade was not tested fully.

Cannot much of the list of points of attack that Mark W. has laid out be enhanced, rather than changed, with settable properties? In this way the path forward should be painless. Of course, I am in no position to suggest that the effort to implement that is painless. Is it the relative subtlety of the proposed changes that might be difficult to identify when porting old versions to new? At least, when I was rewriting from HC to LC, even though the similarities in the code in general were pervasive, the differences and broken routines were obvious, and easily tagged.

Craig Newman

Re: Retaining Backwards Compatibility in a Changing World

Posted: Wed Mar 11, 2015 3:22 pm
by Vanceone
I like Django's system: a clear roadmap of features that will be affected in the future. Right now, I think Django is at version 1.7 (and change). Version 1.8 will in fact remove features. This was announced back when 1.6 was released. Those features work just fine in 1.7, but they issue warnings. Developers have had about 2 years to get ready for their impending departure; since version 1.8 will no longer work if you use those deprecated features.

Seems to me that a system like that would suit everyone. If you really, really need a feature to keep working like it used to, you can back the change out in the source code I suppose.

That said, I really hope that part of the array changes includes an actual ordered array along with the associative array we have now. It would make data structures so much easier! And frankly, I'd love to see some type hinting feature that we could use; it would be nice to be able to refer to a piece of data as something other than a string. Totally optional, but if you opt into it, then you can rely on it (i.e. violations would trigger on explictVariables being true). Might speed up the parser too, if it can compile something down to a type. Yes, it goes against LC's typeless system, which is why it should be optional, but man, coming from a traditional programming view and not being able to rely on my array of ints staying ints always leaves me faintly worried.