develop branch

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, LCMark

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: develop branch

Post by peter-b » Sat Jan 31, 2015 7:08 pm

mwieder wrote:Getting back to building the develop branch after not touching things for a while.

It's no longer possible to just 'make all'... you have to do a 'make bootstrap' first.
And bootstrap has a new requirement that flex be available.
Afterwards it does build properly.
We have a new tool in the source tree that's a bit expensive to compile (because it gets incrementally compiled and recompiled in 4 stages). The tool's then used to compile a bunch of other stuff. Most of the time, we don't need to do the complicated 4-stage bootstrap compile dance, so it only needs to get run if you're compiling from scratch for the first or if some basic changes occur in the system. Hence separate "bootstrap" and "all" targets.

Actually you now need both flex *and* bison.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Feb 01, 2015 3:42 am

Hence separate "bootstrap" and "all" targets.
Makes sense, but shouldn't the bootstrap dependencies just be taken care of by the make files?
Actually you now need both flex *and* bison.
<g> either bison was already in my distro (mint 17) or I had previously installed it.

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: develop branch

Post by peter-b » Sun Feb 01, 2015 8:23 am

mwieder wrote:
Hence separate "bootstrap" and "all" targets.
Makes sense, but shouldn't the bootstrap dependencies just be taken care of by the make files?

I spent quite a while trying to decide if it could be done, but if I allowed "make all" to also bootstrap then I always ended up with a nasty circular dependency. Feel free to fiddle with it and submit patches. :)

Edit: Not just a circular dependency; it also ended up recompiling lots of stuff that didn't need it.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Jun 21, 2015 4:47 pm

It seems the build process has changed.
Significantly.
Is the proper procedure for building the develop branch now

./config.sh
make

and make clean/bootstrap is no longer required/possible? The Makefile is significantly different.

The _build and _cache directories are now obsolete and replaced by build-linux-x86_64?
If so, I have to change my post-build installation scripts.

My screen is filling up with even more warnings about unused/uninitialized parameters, e.g.

../../thirdparty/libsqlite/src/qry_dat.cpp:678:3: warning: control reaches end of non-void function [-Wreturn-type]
../../engine/src/mode_installer_lnx.cpp:240:6: warning: unused parameter ‘p_message’ [-Wunused-parameter]
./../engine/src/sysdefs.h:547:14: warning: unused parameter ‘size’ [-Wunused-parameter]
inline void *operator new (size_t size, void *p)

The develop branch now seems to build LC 8.
What are the proper branches for building the LC6.7 / LC 7.x builds?

¯\_(ツ)_/¯

LCfraser
Livecode Staff Member
Livecode Staff Member
Posts: 71
Joined: Thu Nov 28, 2013 11:18 am
Contact:

Re: develop branch

Post by LCfraser » Sun Jun 21, 2015 4:59 pm

Hi Mark,

We've moved our build system to a tool called Gyp from Google - instead of having 5+ different sets of project files that need to be updated when we make a change, we use Gyp to generate makefiles for Android and Linux, Xcode projects for OSX and iOS and Visual Studio projects for Windows.

For Linux, you should only need to do "make" - the configure step should happen automatically. The interesting top-level targets are:

make config-linux-x86_64
make compile-linux-x86_64
make all-linux-x86_64

The first step runs Gyp to do the configuration; the second compiles the engine; and the third does both. For 32-bit builds, replace x86_64 with x86.

Now that we have Gyp-ified the build system, I managed to sneak in a couple of enhancements - the manual fetch prebuilts step is no longer necessary as it gets done automatically as does the bootstrap step for the LCB compiler.

Our main development branch is "develop" - this will always be the version of LiveCode under active development. LiveCode 7 is available on develop-7.0 while 6.7 is on develop-6.7.

As you've noticed, we've turned up the level of warnings from the compiler - with the eventual aim of fixing them! There is a bit of refinement needed; we should turn off the warnings for third-party components.

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Jun 21, 2015 5:12 pm

Fraser-

Thanks for the quick response.
This does indeed look like progress, and I like the added refinements.

It would be nice to know when something like this happens, as it now screws up my development process and I need to adjust my build scripts accordingly.

What is the proper branch for submitting pull requests now?
Should all pull requests still go against the develop branch? The codebase is significantly different there, and I'd worry that porting bugfixes and enhancements wouldn't be at all straightforward. This concerns me since the 7.x/8.x builds are so unstable on linux.

Will all binaries now end up in the build-linux-x86_64 directory structure no matter what build version?

LCfraser
Livecode Staff Member
Livecode Staff Member
Posts: 71
Joined: Thu Nov 28, 2013 11:18 am
Contact:

Re: develop branch

Post by LCfraser » Sun Jun 21, 2015 5:22 pm

Sorry for not letting you know - I got rather blinkered by making sure everything was working in-house!

Pull requests should go against develop if they are adding a new feature. Bug fixes for 6.7/7.0 should go against the respective develop-x.y branch. The codebase for 7.0 and 8.0 is fairly similar - the in-engine changes are mostly confined to the MCWidget class and supporting infrastructure, plus some changes to libfoundation to support new LCB features. The LCB toolchain is entirely separate from the engine - separate enough that I've been considering converting some of our Perl build tools to LCB (don't hold your breath though ;)).

The Gyp-ification has happened to all of our active branches so 6.7, 7.0 and 8.0 put their build products in the same places. One thing that might be worth noting is that a make will create a directory called "linux-x86_64-bin" containing all the build artifacts that are used to build our installers. In other words, it doesn't contain any of the intermediate build products.

Let me know if you have any other questions!

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Jun 21, 2015 5:29 pm

Sorry for not letting you know - I got rather blinkered by making sure everything was working in-house!
Of the two, I'd rather take the hit and have things working properly in-house.
converting some of our Perl build tools
Yes, please

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Jun 21, 2015 10:24 pm

<sigh>

livecode: develop-6.7$ make
make config-linux-x86_64
make[1]: Entering directory `/home/mwieder/livecode'
./config.sh --platform linux-x86_64
gyp/gyp --format make --depth . --generator-output build-linux-x86_64/livecode -DOS=linux -Dtarget_arch=x86_64
gyp: prebuilt/libcef.gyp not found (cwd: /home/mwieder/livecode)
gyp: prebuilt/libopenssl.gyp not found (cwd: /home/mwieder/livecode)
make[1]: *** [config-linux-x86_64] Error 1
make[1]: Leaving directory `/home/mwieder/livecode'
make: *** [all-linux-x86_64] Error 2

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sat Jun 27, 2015 8:35 pm

Well, this is getting ridiculous.

There's no master branch anymore on github, which is causing git some confusion.
I created a new repository from github and I can now build 8.0 and 7.0 engines on the develop and develop-7.0 branches.
So far so good.
But since both of those are pretty painful to use on linux, I need to build from the 6.x tree.

I can *not* build the develop-6.7 branch because the prebuilt directory is missing libcef.gyp (it seems prebuilt is no longer a submodule?)

Luckily I kept a copy of my old master branch in a separate directory, so I can still build LC 6.6.3, fwiw.

Also, the ide directory ends up with an environment_log.txt file after building.
This should either be in the repository or (more likely) in the .gitignore file. (or possibly deleted at the end of the build process?)

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: develop branch

Post by peter-b » Sat Jun 27, 2015 11:27 pm

mwieder wrote:I can *not* build the develop-6.7 branch because the prebuilt directory is missing libcef.gyp (it seems prebuilt is no longer a submodule?)
It is still a submodule -- in the develop-6.7 branch. You need to run "git submodule update --init --recursive" after checking out the develop-6.7 branch.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Jun 28, 2015 2:42 am

Peter- I had actually tried that before (or at least I could swear I had). I cloned the repo from scratch again:
(have I mentioned how much I hate submodules?)

git clone --recurse-submodules https://github.com/runrev/livecode.git
cd livecode
git fetch
git checkout develop-6.7
git submodule update --init --recursive
make

# success... I built 6.7.6rc2, ran my installation scripts and it launched successfully.
# ok - let's go back to the develop branch then

git checkout develop

livecode: develop-6.7$ git co develop
error: The following untracked working tree files would be overwritten by checkout:
prebuilt/libcef.gyp
prebuilt/libcurl.gyp
prebuilt/libopenssl.gyp
Please move or remove them before you can switch branches.
Aborting

livecode: develop-6.7$ git status
On branch develop-6.7
Your branch is up-to-date with 'origin/develop-6.7'.
nothing to commit, working directory clean

So...
While the prebuilt directory exists in the develop-6.7 branch, it's not in the other branches.
And this causes warnings about a non-empty directory, and also messes up the git status.
And prevents switching between branches properly and committing.

(have I mentioned how much I hate submodules?)

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: develop branch

Post by peter-b » Sun Jun 28, 2015 9:36 am

Yeah, submodules are a real pain. :(

Within LiveCode, most of the developers seem to use multiple checkouts to get around the issues. For example, I have ~/git/livecode for working on LC 7+, and ~/git/livecode-6.7 for working on LC 6.7.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: develop branch

Post by mwieder » Sun Jun 28, 2015 5:59 pm

Yeah, I'm trying to avoid that because of all the duplication, but it's getting less and less possible to do so.

Must make merging streams a pain on your end. I take it everyone has remotes set up for localhost repos as well.
Maybe the readme needs some updating?
Speaking of which, I notice that the README.md files in all the submodules are empty.

The lack of a master branch still just seems weird. I assume that a master branch will appear once we get an actual release of 8.0.

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: develop branch

Post by peter-b » Sun Jun 28, 2015 10:38 pm

mwieder wrote:Yeah, I'm trying to avoid that because of all the duplication, but it's getting less and less possible to do so.

Must make merging streams a pain on your end. I take it everyone has remotes set up for localhost repos as well.
Maybe the readme needs some updating?
Speaking of which, I notice that the README.md files in all the submodules are empty.

The lack of a master branch still just seems weird. I assume that a master branch will appear once we get an actual release of 8.0.
You'll have to ask Mark -- he's the person who decided that the branch usually called "master" elsewhere is called "develop" in LiveCode. :)

Under current plans, when the 8.0 branch starts (probably at 8.0.0 rc1) the current "develop" branch will become "develop-8.0". LiveCode 8.1 development will then (probably) take place on "develop".
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

Locked

Return to “Engine Contributors”