Version Control and updates - stack splitting heresy?

[markw]

Forgive any heresy inherent in my outsider perspective.

One of the biggest issues for me with getting serious about LiveCode for major commercial apps in future is the current lack of version control.

I know Monte is working on a solution that will split files out from stacks for VCS purposes but I also noted that there are plans in the near term roadmap to change the file format??

So... are there any major technical reasons for not having separate code files for separate objects for instance, or is it just historical?

Beyond finding the concept of having almost everything in one file slightly distasteful and the version control requirement, there are some other things that make having code developed and deployed in separate files quite handy:
1) iOS updates - from iOS 6.0, Apple automatically generates partial updates for you. This is entirely file based, so any file that has changed is included in the update. If all of your code is in one file then the whole thing has to be replaced in any update. Split up stacks means smaller updates.
2) Making LiveCode "live" on devices. A new cross-platform game engine called LOOM has some very nice tech that lets you see your updates immediately on external devices as you change code (it's file save based I believe). It'd be nice to add this ability to LiveCode (at least I didn't see anything to suggest it's already there). I figure that's going to be a lot easier and more responsive if it's not all one file.
3) Downloadable code - related to 2 - when not encumbered by the iOS app store restrictions it'd be great to download bits of code for various reasons without going through store updates.
4) Modularity for the web - I think a fun way to recreate what LiveCode had with browser plugins would be to use emscripten to compile the C/C++ codebase to asm.js (with appropriate implementations for graphics - maybe built on WebGL - and other functions accessible from the browser). For load time and performance reasons for apps it'd be good to be able to split them into more bits.

How many of these can already be done? Are there other/better ways to achieve them without such a radical change as splitting stacks up into their component bits?

Mark

[monte]

Regarding 1 & 3. You can use the lots of small stackfiles approach. Stackfiles are a very compact file format. Zipped they can be tiny. My work on lcVCS doesn't mean I don't like the file format. I think it's very convenient. Most of the size in a stackfile comes from imported images and with the changes released in LC 6 a referenced image is just as efficient as an imported one so keeping images and other data out of your stackfiles is a good start. Combined with the lots of small stackfiles approach and a splash stack I'd be surprised if minor updates were more than a few hundred KB...

Regarding 2 if we had a generic property listener then these updates could be sent with or without file saves as soon as the property is changed. It wouldn't matter what the file format is.

Regarding 4 It could help there but once again using referenced images and data out of your stackfile is possibly sufficient. Combine that with some compression on the stack file and most stackfiles would be much smaller than images. It would require the ability to set the filename of an image to a url though or a relative path loading just like it does in html.

[markw]

Regarding 1 & 3. You can use the lots of small stackfiles approach. Stackfiles are a very compact file format. Zipped they can be tiny. My work on lcVCS doesn't mean I don't like the file format. I think it's very convenient. Most of the size in a stackfile comes from imported images and with the changes released in LC 6 a referenced image is just as efficient as an imported one so keeping images and other data out of your stackfiles is a good start. Combined with the lots of small stackfiles approach and a splash stack I'd be surprised if minor updates were more than a few hundred KB...

OK, so this is mostly just a good practice issue rather than a structural one - nothing to stop you splitting your code up fairly arbitrarily. Can you download and run new stackfiles on-the-fly on Android? With 3 I'm mostly thinking about running A/B tests.

Regarding 2 if we had a generic property listener then these updates could be sent with or without file saves as soon as the property is changed. It wouldn't matter what the file format is.

You've lost me a bit here - I was imagining handling live updates by overwriting stackfiles. I'm now wondering if we can have a listener for updates and simply send individual scripts over the network, changing them on the other end with

Code: Select all

set the script of object to string

. That in turn leads me to wonder what stops us doing something like that on iOS right now? This is clearly against Apple policy for store apps but must be incredibly hard to police.

This sounds more like an external tool or library than an engine contribution? Except for the property listener?

Regarding 4 It could help there but once again using referenced images and data out of your stackfile is possibly sufficient. Combine that with some compression on the stack file and most stackfiles would be much smaller than images. It would require the ability to set the filename of an image to a url though or a relative path loading just like it does in html.

No point in breaking out the scripts from stackfiles here either then. You can already load images from URLs and use them right? Relative paths and network transparency (e.g. who cares if it's on my local file system or the web) would be handy add-ons though.

So, despite fairly comprehensive rejection of my original suggestion (which I was expecting), maybe some possible features to split out into separate discussions here?

[monte]

OK, so this is mostly just a good practice issue rather than a structural one - nothing to stop you splitting your code up fairly arbitrarily. Can you download and run new stackfiles on-the-fly on Android? With 3 I'm mostly thinking about running A/B tests.

Right, you could do the same on iOS too but apple will hunt you down and rip your legs off... go stack decompress(url "http:...

I'm now wondering if we can have a listener for updates and simply send individual scripts over the network, changing them on the other end with

Code: Select all

set the script of object to string

That's the idea

. That in turn leads me to wonder what stops us doing something like that on iOS right now? This is clearly against Apple policy for store apps but must be incredibly hard to police.

It depends on how much you like your legs

This sounds more like an external tool or library than an engine contribution?

Yes

Except for the property listener?

Yes... it looks like there's a property listener started for IDE only. I'd like this for a number of things including lcVCS. lcVCS could export properties as they change rather than wait to export the whole stackfile when the user is ready.

No point in breaking out the scripts from stackfiles here either then. You can already load images from URLs and use them right? Relative paths and network transparency (e.g. who cares if it's on my local file system or the web) would be handy add-ons though.

Well you can put url <url> into image X but last I checked you couldn't set the filename of image X to <url> and have the engine just handle it for you.

So, despite fairly comprehensive rejection of my original suggestion (which I was expecting), maybe some possible features to split out into separate discussions here?

Last I checked my opinion carried little weight around here so don't let my comments sway you.

[markw]

Right, you could do the same on iOS too but apple will hunt you down and rip your legs off... go stack decompress(url "http:...

I thought RunRev would have had to disable this functionality for iOS to get apps in the store - what's the difference between this and e.g. shipping your own browser with its own engine?

I'm keen on the iPad native IDE idea for LiveCode. I have Codea, which uses Lua, and the version of that which even allowed you to share code via dropbox was rejected from the store. I can't imagine them approving a LiveCode IDE that lets you run arbitrary stacks from anywhere on the web with a single line of code.

Similarly, being able to edit scripts in an iOS standalone should probably be off by default at the very least to look like LiveCode is trying to make people conform to the rules.

it looks like there's a property listener started for IDE only. I'd like this for a number of things including lcVCS. lcVCS could export properties as they change rather than wait to export the whole stackfile when the user is ready.

Hmmm, interesting built-in autosave there. Sounds like it should be reasonably easy to allow other stacks to register as listeners, is that something you're already planning on looking at? I can imagine there's quite a performance hit involved in listening to all property changes, probably needs to be handled with care.

last I checked you couldn't set the filename of image X to <url> and have the engine just handle it for you.

This is what I mean by network transparency, although the clean way to do it is to have the fundamental property of images to be url and 'set the url of image X to "file:..."' (or I guess maybe binfile: if that's still a necessary distinction) for local files with the legacy filename syntax retained as a convenience wrapper for setting file: urls.

Last I checked my opinion carried little weight around here so don't let my comments sway you.

You clearly know your LiveCode, that's enough qualification to debunk my misunderstanding of stackfile granularity. My only remaining concerns would be around the structural limitations of decomposing your stacks into lots of little files and the performance impacts of doing so. Unless I hit a problem there I've got no reason big enough to want to change the fundamental structure of a LiveCode app.

[mwieder]

I can imagine there's quite a performance hit involved in listening to all property changes, probably needs to be handled with care.

I don't think a property listener is necessary. Staging for version control should really only come into play when saving a stack. Not only would there be a performance hit, but multiple staging during the process of designing a stack's UI could well be counterproductive. Rather like issuing too many commits during the design process instead of holding off until there's something worth committing.

My only remaining concerns would be around the structural limitations of decomposing your stacks into lots of little files and the performance impacts of doing so. Unless I hit a problem there I've got no reason big enough to want to change the fundamental structure of a LiveCode app.

I'm still a fan of the native stack format on disk being individual text files (possibly in addition to the stack itself). It's still reasonably fast, and delegating the storage and retrieval to a version control system means that only the deltas will need to be saved. The advantage is the ability to work with other version control add-on tools that *only* understand text files. Within the LiveCode ecology it doesn't make much difference, and indeed the monolithic stack structure is easier to deal with, but once you start branching out to other tools the binary stack structure becomes an impediment. For instance, how do you integrate a stack into a continuous integration build process?

[markw]

Hmmm, I'd only imagined the property listener would save the files to your local drive on changes, not actually stage them (e.g. no git add) - the latter would be manual, while the former would still allow you to look at which parts you'd changed and diff your local copy vs. the repository.

Clearly it needs to be text files under version control or we can't merge.

As for CI, as long as you can fetch the source, build and run from a command line/shell then it should work fine shouldn't it?

What other tools might we not be able to use?

[mwieder]

The way I set up my version control library is that staging is automatic on saves. That's the way most IDEs handle version control. Everything else is manual. Diffing a control in memory with a saved version could be marginally useful, but I don't think I'd want to save stack changes every time I typed a key or moved a button on a stack. There's a very limited ROI on managing *every* change. Where you really want to archive things is when you commit a save, and then it makes sense to have the version control kick in as well.

Clearly it needs to be text files under version control or we can't merge.

Actually, Monte's already come up with a clever way around this.

As for CI, as long as you can fetch the source, build and run from a command line/shell then it should work fine shouldn't it?

There are two parts to CI: code checkin and acceptance test success. And I'm hard-pressed to come up with a way to automate acceptance tests without individual files.

[markw]

Is there a unit test or acceptance test framework of any kind for LiveCode? Not even knowing that much its hard to discuss that aspect. Unit tests usually use a special test build, I could imagine that being made to work with a wrapper for stackfiles and thus your stackfile granularity is important. UI acceptance tests (e.g. for a cucumber-like BDD) typically run against the standard executable, possibly with a library injected. Seems like it should be possible to make them work without caring about the format on disk at all.

[mwieder]

There isn't right now - it would have to be command-line driven and ideally acceptance tests would be separate from the stack code itself. I'd love to see things move in this direction, and I don't think it's feasible without breaking into separate files.

[monte]

I think your right... we wouldn't want saves on every property change. I was thinking of creating an array of property changes and then when saving it just saves the changed bits.

One of the problems with having the engine just move (or have an option) to save as text files in order to support VCS is there's so much data that you want to trim out of a stackfile when saving for a VCS. For example, you don't want to save stack rect because every move on screen will cause a merge conflict so you just save width and height. What it you'r stack's resizable and has a resize stack handler? Well you want to reset it to a default size during the export or there will be merge conflicts everywhere. That's why I implemented the lcVCSExport message. Then there's stuff like text you've entered during development that is really only session data. In the end I realised that what we really need is more support from the engine to make a script solution as fast as possible.

BTW I know nothing about CI so I can't comment there more than wondering if the acceptance tests could be run on an engine like the old CGI where there's an executable script that can load stack files.

[mwieder]

BTW I know nothing about CI so I can't comment there more than wondering if the acceptance tests could be run on an engine like the old CGI where there's an executable script that can load stack files.

Just started a new topic for this.

Monte-

The basics of continuous integration are that it's mostly like a push to a remote repository... you push to the CI server, but before the code gets merged into the remote system it has to pass predefined tests. The tests are designed to do things like make sure changes don't break existing functionality, cause compile errors, etc. If the tests pass then the code is checked in and all is well; if the tests fail then notifications are sent to the authorities. You have the option of running the tests at checkin time and/or at regular intervals on the stored codebase. We used to run CI tests to ensure that our online database was still accessible online - if it ever failed we'd get emails or sms messages immediately so someone could put it back up asap.

[monte]

OK, that's all in line with my uneducated assumptions. I know GIT has local hooks that could be used for some basic automated testing. It also has server side hooks that could be used to integrate the kind of service you're talking about. It would be nice if RunRev were able to build such a server and allow us to hook our repos up to it so we could ensure our stuff passed the CI tests before issuing a pull request but I guess it doesn't make much difference if it's before or after as long as this is an automated step before anyone from RunRev need to look at the pull request. I guess it's a bit complicated because in theory the tests need to be run on so many machines...

[mwieder]

Actually, for C code this is easy. Github already has Travis intances running - I think you just have to load one.

But I'd like there to be automated tests for stacks as well, and for all the CI servers I know of, that means a commandline interface to load, run, and build.

[monte]

Right... hence the CGI topic