Project: Performance Benchmarking Toolkit

This forum is a working group for community contributors to the LiveCode IDE included in the main LiveCode distribution.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9801
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by FourthWorld » Thu Nov 20, 2014 4:12 pm

I think the main thing being requested there is to be very careful of separating display/rendering performance from other measurements related to the inner workings of the engine. I think as long as we're not displaying progress info during the test we're probably getting the same benefit as with Server.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: Project: Performance Benchmarking Toolkit

Post by peter-b » Thu Nov 20, 2014 5:30 pm

FourthWorld wrote:I think the main thing being requested there is to be very careful of separating display/rendering performance from other measurements related to the inner workings of the engine. I think as long as we're not displaying progress info during the test we're probably getting the same benefit as with Server.
Correct.

Additionally:

1. Gathering high-quality profiling data can only be done when the engine *only* runs the benchmark and quits, without doing anything else.

2. Automated comparison of performance results between multiple versions of LiveCode is easier when there aren't many manual steps.

For profiling I would have to convert any benchmark stacks into server scripts anyway. It makes my life easier if you write them that way to start with.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Project: Performance Benchmarking Toolkit

Post by [-hh] » Thu Nov 20, 2014 9:48 pm

Richard,
could you make a utility, please, that converts any testscript into a server script (file), so that, whoever uses it, can choose to test on both, server and desktop?

I know you can do this much faster and more elegant than I do ...

Hermann
shiftLock happens

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9801
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by FourthWorld » Thu Nov 20, 2014 9:58 pm

[-hh] wrote:Richard,
could you make a utility, please, that converts any testscript into a server script...
Any is a tall order. For anything I could write a converter for in a reasonable amount of time, the author could just write the original with a shebang at the top even faster.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Project: Performance Benchmarking Toolkit

Post by [-hh] » Thu Nov 20, 2014 10:27 pm

Sorry, I'm not deep enough in server things. Do you mean with the latter (shebang) not a "real" serving but run the server-executable via shell? Is it also that what Peter meant?

I also would like to know your opinion about how good it is possible, to exactly separate the time needed by apache. Take an additional apache-timing by "ab"? I never really worked with that, only played around (and wasn't convinced after that).
shiftLock happens

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9801
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by FourthWorld » Thu Nov 20, 2014 10:51 pm

"Shebang" (sometimes called "hashbang") refers to the Unix convention for specifying an interpreter for a script by putting that app's path in the script's first line following "#!".

The ability for LC Server scripts to run from the command line with shebangs pointing to the engine, coupled with the ability to have scripts there outside of "<lc?"..."?>", tags was restored to the engine earlier this year, making a lot of command-line work much simpler:
http://livecode.com/blog/2014/03/06/livecode-server/
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Project: Performance Benchmarking Toolkit

Post by [-hh] » Thu Nov 20, 2014 11:24 pm

I used that already, it is especially good for testing server scripts before using them in the webserver environment or have a quick job done from terminal.

That's why I asked: I can't see any advantage when doing benchmarking this way compared to a desktop run (with locked screen and/or messages).
Also I cannot test visual displays with that (bezier paths), what is pretty much also low-level engine(?)
And a simple desktop button from any LC version could also launch, run and close same or different other desktop versions just alike server versions.
It is also possible to give a "nice" to the desktop versions before running the tests.

What is after all the difference, why should we use LC-server?
shiftLock happens

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: Project: Performance Benchmarking Toolkit

Post by peter-b » Fri Nov 21, 2014 9:45 am

[-hh] wrote:Also I cannot test visual displays with that (bezier paths), what is pretty much also low-level engine(?)
You'll need to use the IDE for any benchmark involving a graphical user interface (GUI). Those will be quite a lot harder to get accurate profiles for though. :(
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9801
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by FourthWorld » Fri Nov 21, 2014 4:58 pm

@peter-b: Thanks for your interest in this. Very promising.

In reworking my test script for LC Server compatibility I discovered two things:
1. revAvailableHandlers isn't available in the Server engine (not a biggie by itself, but:)
2. When a handler can't be found in a text script, the script doesn't compile at all, so that rather than generating any info like "Handler not found" or anything else which might guide us to a solution, it simply doesn't run. Is that a bug or just a necessary limitation?

My revised script is below. I saved it as "t.lc", and stored a copy in two folders, one for v6.7.1rc2 and v7.0.1rc2. I was curious about memory usage in addition to performance, so I ran each test with:

Code: Select all

/usr/bin/time -v ./t.lc
Here are the relevant portions of each output from "time", run on a 3.0GHz Haswell:
6.7.1 rc2:
User time (seconds): 3.54
System time (seconds): 0.01
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.56
Maximum resident set size (kbytes): 12212
Minor (reclaiming a frame) page faults: 1169
Voluntary context switches: 2
Involuntary context switches: 255

7.0.1 rc2:
User time (seconds): 13.62
System time (seconds): 0.09
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.76
Maximum resident set size (kbytes): 16732
Minor (reclaiming a frame) page faults: 7021
Voluntary context switches: 2000
Involuntary context switches: 280
Given the nature of CGIs, this impact on scalability is clear.

I very much appreciate the team looking into this. Hopefully we can see a v6.7 with backported 64-bit compatibility to tide us over while v7 is being optimized.

Script:

Code: Select all

#!livecode-server

--=======================================================================--
-- Quickie Benchmark Suite
-- Richard Gaskin, Fourth World Systems
-- Placed into the public domain on 15 November, 2014
--
-- This script is self-contained, and can be put into a button
-- and clicked to run.
-- 
-- Tbe constants define parameters for the script.
-- Script-locals provide vars that can be reused among
-- the various commands.
-- 
-- The flow is simple enough:
-- BenchmarkAll sets up a header for the output and then
-- finds the handlers in the script that begin
-- with "Test_", and runs each one.
-- BenchmarkTest runs the actual handler within a wrapper that
-- provides metrics.
-- 
-- This particular set of handlers tests things I commonly do in
-- server scripts, but can be replaced with anything you need to
-- measure.
--
-- Please post any feedback, improvements, or suggestions to the
-- working group forum for Malte's Performance Benchmarking project:
-- http://forums.livecode.com/viewtopic.php?f=67&t=22072
--=======================================================================--

constant kTestSuiteIdentifier = "Common Server Tasks"
constant kBaseIterations = 100 -- common number of iterations for each test
constant kFileIterations = 1000 -- number of iterations for each file IO test

local sReport -- container for test results
local sTestList -- used in list-related tests
local sTestA -- used for array-related tests
local sTestFilePath -- used for file-related tasks
local sTemp -- scratch for misc needs (such as line numbers)


BenchmarkAll the long name of me


    -- Simple way to handle bulk tests:
on BenchmarkAll
   SetBenchmarkHeader
   --
   -- Test each "Test_*" handler in script of pObj:
   put MyTestHandlers() into tTestList
   repeat for each item tTest in tTestList
         BenchmarkTest ("Test_"& tTest)
   end repeat   
   --
   put sReport
end BenchmarkAll


function MyTestHandlers
    return "BuildList,LineOffset,LineAccessByNumber,LineAccessForEach,"\
      &"ArraySplit,EncodeArray,DecodeArray,ArrayAccess,Merge,"\
      &"BuildFilePath,FileTextWrite,FileTextRead,FileBinWrite,FileBinRead"
end MyTestHandlers


-- Header with basic system info:
on SetBenchmarkHeader
   put "Performance Benchmarking: "& kTestSuiteIdentifier &cr\
   &"LiveCode Version: "& the version &cr\
   &"System Version: "& the platform && systemVersion()&cr\
   &"--"&cr into sReport
end SetBenchmarkHeader


-- Common handler allows easy authoring of specific tests,
-- acknowledging that it includes the overhead of dispatching.
--  But since the purpose of this testing is to compare
-- relative performance, and since dispatch is the fastest way
-- to route handling to arbitrary commands, it's hoped that the
-- use of it here will be forgiven:
on BenchmarkTest pCmd
   put "Running "& pCmd &"..."&cr
   put the millisecs into t
   if "File" is in pCmd then
      repeat kFileIterations
         dispatch pCmd
      end repeat
   else
      repeat kBaseIterations
         dispatch pCmd
      end repeat
   end if 
   put the millisecs - t into t
   --
   put pCmd &": "& t &" ms" &cr after sReport
end BenchmarkTest



on Test_BuildList
   put empty into sTestList
   repeat with i = 1 to 1000
      put "SomeStuff" && i &cr after sTestList
   end repeat
end Test_BuildList


on Test_LineOffset
   put empty into sTemp
   put the number of lines of sTestList into tMax
   repeat with i = 1 to tMax
      put lineoffset("SomeStuff "& i, sTestList) &cr after sTemp
   end repeat
   delete last char of sTemp -- trailing CR
end Test_LineOffset


on Test_LineAccessByNumber
   repeat for each line tNum in sTemp
      get line tNum of sTestList
   end repeat
end Test_LineAccessByNumber

on Test_LineAccessForEach
   repeat for each line tLine in sTestList
      get tLine
   end repeat
end Test_LineAccessForEach


on Test_ArraySplit
   put sTestList into sTestA
   split sTestA by cr
end Test_ArraySplit


on Test_EncodeArray
   put arrayEncode(sTestA) into sTemp
end Test_EncodeArray


on Test_DecodeArray
   put arrayDecode(sTemp) into sTestA
end Test_DecodeArray


on Test_ArrayAccess
   repeat for each key tKey in sTestA
      get sTestA[tKey]
   end repeat
end Test_ArrayAccess


on Test_Merge
   put "Merge data from array here: [[  GetArrayValueForMergeTest(tKey) ]]" into tString
   repeat for each key tKey in sTestA
      get merge(tString)
   end repeat 
end Test_Merge

-- Needed when testing older engines because there was a bug that
-- prevents proper evaluation of array expressions between double
-- brackets:
function GetArrayValueForMergeTest pKey
   return sTestA[pKey]
end GetArrayValueForMergeTest


on Test_BuildFilePath
   put specialFolderPath("temporary") &"/MalteBenchmarkTestData" into sTestFilePath   
end Test_BuildFilePath


on Test_FileTextWrite
   put sTestList into url ("file:"& sTestFilePath)
end Test_FileTextWrite


on Test_FileTextRead
   put url ("file:"& sTestFilePath) into sTestList
end Test_FileTextRead


on Test_FileBinWrite
   put sTestList into url ("binfile:"& sTestFilePath)
end Test_FileBinWrite


on Test_FileBinRead
   put url ("binfile:"& sTestFilePath) into sTestList
end Test_FileBinRead
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

malte
Posts: 1098
Joined: Thu Feb 23, 2006 8:34 pm
Location: Ostenfeld germany
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by malte » Fri Nov 21, 2014 9:34 pm

Sorry I chime in late. Here are my 2 cents regarding faceless vs. UI driven testing... I think we need both. One of the important factors on a server / CGI installation is the startuptime. (Has anybody measured that already? Of course it shall be easier to test the low level operations in a faceless environment, however there are many many things that need the UI to be tested. From creating / destroying controls over everything that requires a render to things that are rude in the IDE itself (revUpdateGeometry has always been a bugger that way).

Also I would like to say that I really really appreciate all of you joining and especially am thankful that the RR engineers are chiming in as well. Thank you!

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9801
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by FourthWorld » Sun Nov 23, 2014 5:57 pm

malte wrote:Sorry I chime in late. Here are my 2 cents regarding faceless vs. UI driven testing... I think we need both.
I believe the folks at RunRev agree. The reason for the emphasis on core language features rather than rendering is that they have a few options already in development to boost rendering speed, so while those are being worked on they want to identify other opportunties for performance gains for all platforms, including faceless along with GUI ones, such as we're currently doing here.
One of the important factors on a server / CGI installation is the startuptime. Has anybody measured that already?
I haven't yet isolated startup time. but a simple script that just quits should be a reasonable starting point - I'll report back when I get some time to run that.

This morning I ran another test using the server script I posted recently and running it under strace:

Code: Select all

strace -c ./t.lc
The strace command by default outputs systems calls the app passed to it makes to the OS. With the -c flag it outputs only a summary of counts, and when i run it here with v6.7.1 and 7.0.1 I see about an order of magnitude more calls made to the system in v7. Oddly, I also see an error reported related to file I/O in v6.7.1, so I'll have to dig a little deeper to see what that's about.

I think this optimization effort is very important, time-consuming but with high ROI. While LC Server may not be an important revenue stream in itself, in this "Third Platform" era of computing it plays an important role in tying together all of the other platforms LC supports. Optimization across all these platforms is important, but in a world with plenty of server language options we need to get LC performance at least on par with PHP in order for its many advantages to become discoverable.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Project: Performance Benchmarking Toolkit

Post by [-hh] » Sun Nov 23, 2014 9:31 pm

Simply some "facts" from me, I don't interpret this. May be the measurement is not stable enough.
Nevertheless interesting. I repeated the test several times, had also some other rankings, but this is most frequent:

Calling Richard's server scripts (I added Test_ArrayCombine)
running under LC 6.7.1 (from my attached stack) livecode-community-server 6.7.1
(livecode-server was not available on my products page).

10 secs by shebang "#!" is slightly *slower* than
9 secs by livecode-server <file with tags "<? ... ?>" > or
9 secs by GUI

21 secs by GUI-LC 7.01 is by far slowest, but the factor of slowdown from 6.7.1 to 7.0.1 is smaller than with my GUI test of 'byte handling'.

So using GUI is even advantageous from my tests (may be the LC app has better memory handling?).

Code: Select all

====================================================================
Sun, 23 Nov 2014 20:49:57 +0100
==[ 5 ]=============================================================
Start	 Sun, 23 Nov 2014 20:47:16 +0100
End	 Sun, 23 Nov 2014 20:47:26 +0100
--> Total sum of single tasks: 9.902 seconds
Performance Benchmarking: Common Server Tasks
--> LC-Server[ #! ]: 6.7.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 78 ms
Test_LineOffset: 3039 ms
Test_LineAccessByNumber: 551 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 51 ms
Test_ArrayCombine: 39 ms
Test_EncodeArray: 7 ms
Test_DecodeArray: 80 ms
Test_ArrayAccess: 16 ms
Test_Merge: 378 ms
Test_BuildFilePath: 8 ms
Test_FileTextWrite: 2444 ms
Test_FileTextRead: 59 ms
Test_FileBinWrite: 3127 ms
Test_FileBinRead: 17 ms
==[ 6 ]=============================================================
Start	 Sun, 23 Nov 2014 20:47:57 +0100
End	 Sun, 23 Nov 2014 20:48:06 +0100
--> Total sum of single tasks: 8.879 seconds
Performance Benchmarking: Common Server Tasks
--> LC-Server[ <?lc ]: 6.7.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 74 ms
Test_LineOffset: 2995 ms
Test_LineAccessByNumber: 514 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 56 ms
Test_ArrayCombine: 45 ms
Test_EncodeArray: 8 ms
Test_DecodeArray: 83 ms
Test_ArrayAccess: 17 ms
Test_Merge: 368 ms
Test_BuildFilePath: 7 ms
Test_FileTextWrite: 2107 ms
Test_FileTextRead: 56 ms
Test_FileBinWrite: 2522 ms
Test_FileBinRead: 19 ms
==[ 7 ]=============================================================
Start	 Sun, 23 Nov 2014 20:48:32 +0100
End	 Sun, 23 Nov 2014 20:48:41 +0100
--> Total sum of single tasks: 8.581 seconds
Performance Benchmarking: Common Server Tasks
--> LC [ GUI ] 6.7.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 75 ms
Test_LineOffset: 2872 ms
Test_LineAccessByNumber: 506 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 52 ms
Test_ArrayCombine: 43 ms
Test_EncodeArray: 7 ms
Test_DecodeArray: 89 ms
Test_ArrayAccess: 18 ms
Test_Merge: 379 ms
Test_BuildFilePath: 9 ms
Test_FileTextWrite: 2043 ms
Test_FileTextRead: 58 ms
Test_FileBinWrite: 2398 ms
Test_FileBinRead: 24 ms
==[ 8 ]=============================================================
Start	 Sun, 23 Nov 2014 20:49:28 +0100
End	 Sun, 23 Nov 2014 20:49:50 +0100
--> Total sum of single tasks: 21.376 seconds
Performance Benchmarking: Common Server Tasks
--> LC [ GUI ] 7.0.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 282 ms
Test_LineOffset: 8729 ms
Test_LineAccessByNumber: 5634 ms
Test_LineAccessForEach: 53 ms
Test_ArraySplit: 123 ms
Test_ArrayCombine: 272 ms
Test_EncodeArray: 88 ms
Test_DecodeArray: 118 ms
Test_ArrayAccess: 63 ms
Test_Merge: 887 ms
Test_BuildFilePath: 9 ms
Test_FileTextWrite: 2358 ms
Test_FileTextRead: 264 ms
Test_FileBinWrite: 2462 ms
Test_FileBinRead: 34 ms
====================================================================
Sun, 23 Nov 2014 20:49:57 +0100
====================================================================
Here the testing stack, in case someone would like to repeat these tests in his environment.
Attachments
Performance2_7.livecode.zip
(11 KiB) Downloaded 375 times
shiftLock happens

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: Project: Performance Benchmarking Toolkit

Post by peter-b » Mon Nov 24, 2014 9:39 am

malte wrote:One of the important factors on a server / CGI installation is the startuptime. (Has anybody measured that already?
I've already tested this. 7.0 starts up marginally faster than 6.7, although TBH the times are close enough that it's not a significant difference.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: Project: Performance Benchmarking Toolkit

Post by peter-b » Mon Nov 24, 2014 9:42 am

FourthWorld wrote:I haven't yet isolated startup time. but a simple script that just quits should be a reasonable starting point - I'll report back when I get some time to run that.
This is pretty effective:

Code: Select all

echo > empty.lc
livecode-community-server empty.lc
N.b. you can't do this:

Code: Select all

touch empty.lc
livecode-community-server empty.lc
because of Bug 14029.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2262
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Project: Performance Benchmarking Toolkit

Post by [-hh] » Mon Nov 24, 2014 2:23 pm

Fourthworld wrote:I haven't yet isolated startup time
Just to avoid misunderstanding for not-this-very-expert readers as me:
You haven't yet isolated startup time for measuring time, but it's currently *NOT* included in the overall time I reported. The startup enlarges this "Total sum" to the difference between the Start and End time stamps (first two lines).
Currently these are seconds only, but it shows already that this iadditional time is by far not this much time as needed for running server in CGI mode by Apache.
shiftLock happens

Locked

Return to “IDE Contributors”