Using debug versions of mobile engines

[monte]

The only way I seem to be able to test a mobile engine is to build a release version then put it in the Runtime folder in my extensions... There must be an easier way than that. In particular a way to load the debug version which runs the simulator when built... I think it just needs a stackfile to load... I tried adding an environment variable TEST_STACK that I saw in an old post but that didn't help... any tips?

[LCMark]

Running iOS engines is a bit different from the desktop as (even in the simulator) the app has to be 'installed' by Xcode first. The way we do this is to add the stack we want to test to the project (to be included in the standalone-mobile-community target), search for 'iphone_test' and then replace the filename appropriately.

This works well for both device and simulator (although the latter could work with the TEST_STACK environment variable, since the simulator shares the host filesystem).

[monte]

Ah... ok, I just named the file iphone_test.livecode and added it to bundle resources...

Idea... seeing as we are all mostly forking off master what about each merge of a release branch on master the iPhone_test file (could it be mobile_test and cover android too?) is reset to a vanilla stack (or some default test stack) then we make our changes to the engine and the test stack and include them both in the pull request.. you setup your repo to keep yours on merge so when you pull in the changes you keep the vanilla version. However, the test stack is there in the history. I'd suggest lcVCS except in this case we don't want to merge anyway... we just want to keep track of people's test stacks in the repo history.

[LCMark]

The concern I'd have with that approach would be that it is a bit fiddly then to extract the test stacks - ideally, we'd build up the test stacks in a library to executed as part of a testing phase on built executables.

Indeed, the plan we currently have for such things is that when you submit a pull-request for a bug, if possible along with it you'd submit a test-case in the appropriate folder in the repo (something under tests/). The test stack (or file - we have a system we're developing that builds test cases from annotated text files containing livecode scripts, as well as stacks) could then be run on every build to make sure things that have been fixed aren't unfixed (thinking about it, some would have to be marked as 'non-automatable' as there are things which it is difficult to test without a human involved).

Also, to help with documentation and such, we're currently integrating a system using markdown for generating the release-notes automatically. Again, with a pull-request we'll ask that you also include an appropriate markdown file (in the appropriate place - docs/notes/...) documenting any changes / new features / etc.

[monte]

Hmm... so to test the changes locally we will need to make one stack then to submit the pull request another? Is it possible to have a path in the engine to a current test (like iPhone_test) which we then move before the pull request? Also it would be good if Hanson was setup to handle these test files as well as stackfiles in a report.

[LCMark]

The test system will be integrated into the repository, so you'll be able to execute a test (having built a suitable engine) by just running a command from the command-line - i.e. the test-case you can use to test the bug fix can be the same as the one you put in the repository.

Now that being said, ideally you'd easily be able to tell Xcode (or whatever) to run the engine with the test-case so debugging is easier, we'll have to look into how that might be achieved.

In terms of interaction with bugzilla, then if you are fixing a bug and were the person who discovered it there wouldn't really be a need for you to submit a stack there since you'd be submitting a test-case as part of the fix.

If you weren't fixing the bug, then I guess you'd submit a stack in the usual way. The rationale here being that what the test-case should be is heavily influenced by what the bug is that is actually fixed and so it should probably be the engineer that fixes it that writes the test case - in many cases a stack that exhibits a bug isn't necessarily minimal, nor specific enough to be rendered as an automated test.

(I'm musing here to a certain degree - a bit more thought is needed).

[monte]

[quote]Now that being said, ideally you'd easily be able to tell Xcode (or whatever) to run the engine with the test-case so debugging is easier, we'll have to look into how that might be achieved.[/qoute]

Right... that's where I was going but I said testing locally instead of what I meant in running a debug build with your test case automatically (aka what you get with the iPhone_test stack).