Project: Performance Benchmarking Toolkit
Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller
-
- VIP Livecode Opensource Backer
- Posts: 9867
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: Project: Performance Benchmarking Toolkit
I think the main thing being requested there is to be very careful of separating display/rendering performance from other measurements related to the inner workings of the engine. I think as long as we're not displaying progress info during the test we're probably getting the same benefit as with Server.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
Re: Project: Performance Benchmarking Toolkit
Correct.FourthWorld wrote:I think the main thing being requested there is to be very careful of separating display/rendering performance from other measurements related to the inner workings of the engine. I think as long as we're not displaying progress info during the test we're probably getting the same benefit as with Server.
Additionally:
1. Gathering high-quality profiling data can only be done when the engine *only* runs the benchmark and quits, without doing anything else.
2. Automated comparison of performance results between multiple versions of LiveCode is easier when there aren't many manual steps.
For profiling I would have to convert any benchmark stacks into server scripts anyway. It makes my life easier if you write them that way to start with.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com
-
- VIP Livecode Opensource Backer
- Posts: 2262
- Joined: Thu Feb 28, 2013 11:52 pm
- Location: Göttingen, DE
Re: Project: Performance Benchmarking Toolkit
Richard,
could you make a utility, please, that converts any testscript into a server script (file), so that, whoever uses it, can choose to test on both, server and desktop?
I know you can do this much faster and more elegant than I do ...
Hermann
could you make a utility, please, that converts any testscript into a server script (file), so that, whoever uses it, can choose to test on both, server and desktop?
I know you can do this much faster and more elegant than I do ...
Hermann
shiftLock happens
-
- VIP Livecode Opensource Backer
- Posts: 9867
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: Project: Performance Benchmarking Toolkit
Any is a tall order. For anything I could write a converter for in a reasonable amount of time, the author could just write the original with a shebang at the top even faster.[-hh] wrote:Richard,
could you make a utility, please, that converts any testscript into a server script...
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
-
- VIP Livecode Opensource Backer
- Posts: 2262
- Joined: Thu Feb 28, 2013 11:52 pm
- Location: Göttingen, DE
Re: Project: Performance Benchmarking Toolkit
Sorry, I'm not deep enough in server things. Do you mean with the latter (shebang) not a "real" serving but run the server-executable via shell? Is it also that what Peter meant?
I also would like to know your opinion about how good it is possible, to exactly separate the time needed by apache. Take an additional apache-timing by "ab"? I never really worked with that, only played around (and wasn't convinced after that).
I also would like to know your opinion about how good it is possible, to exactly separate the time needed by apache. Take an additional apache-timing by "ab"? I never really worked with that, only played around (and wasn't convinced after that).
shiftLock happens
-
- VIP Livecode Opensource Backer
- Posts: 9867
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: Project: Performance Benchmarking Toolkit
"Shebang" (sometimes called "hashbang") refers to the Unix convention for specifying an interpreter for a script by putting that app's path in the script's first line following "#!".
The ability for LC Server scripts to run from the command line with shebangs pointing to the engine, coupled with the ability to have scripts there outside of "<lc?"..."?>", tags was restored to the engine earlier this year, making a lot of command-line work much simpler:
http://livecode.com/blog/2014/03/06/livecode-server/
The ability for LC Server scripts to run from the command line with shebangs pointing to the engine, coupled with the ability to have scripts there outside of "<lc?"..."?>", tags was restored to the engine earlier this year, making a lot of command-line work much simpler:
http://livecode.com/blog/2014/03/06/livecode-server/
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
-
- VIP Livecode Opensource Backer
- Posts: 2262
- Joined: Thu Feb 28, 2013 11:52 pm
- Location: Göttingen, DE
Re: Project: Performance Benchmarking Toolkit
I used that already, it is especially good for testing server scripts before using them in the webserver environment or have a quick job done from terminal.
That's why I asked: I can't see any advantage when doing benchmarking this way compared to a desktop run (with locked screen and/or messages).
Also I cannot test visual displays with that (bezier paths), what is pretty much also low-level engine(?)
And a simple desktop button from any LC version could also launch, run and close same or different other desktop versions just alike server versions.
It is also possible to give a "nice" to the desktop versions before running the tests.
What is after all the difference, why should we use LC-server?
That's why I asked: I can't see any advantage when doing benchmarking this way compared to a desktop run (with locked screen and/or messages).
Also I cannot test visual displays with that (bezier paths), what is pretty much also low-level engine(?)
And a simple desktop button from any LC version could also launch, run and close same or different other desktop versions just alike server versions.
It is also possible to give a "nice" to the desktop versions before running the tests.
What is after all the difference, why should we use LC-server?
shiftLock happens
Re: Project: Performance Benchmarking Toolkit
You'll need to use the IDE for any benchmark involving a graphical user interface (GUI). Those will be quite a lot harder to get accurate profiles for though.[-hh] wrote:Also I cannot test visual displays with that (bezier paths), what is pretty much also low-level engine(?)
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com
-
- VIP Livecode Opensource Backer
- Posts: 9867
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: Project: Performance Benchmarking Toolkit
@peter-b: Thanks for your interest in this. Very promising.
In reworking my test script for LC Server compatibility I discovered two things:
1. revAvailableHandlers isn't available in the Server engine (not a biggie by itself, but:)
2. When a handler can't be found in a text script, the script doesn't compile at all, so that rather than generating any info like "Handler not found" or anything else which might guide us to a solution, it simply doesn't run. Is that a bug or just a necessary limitation?
My revised script is below. I saved it as "t.lc", and stored a copy in two folders, one for v6.7.1rc2 and v7.0.1rc2. I was curious about memory usage in addition to performance, so I ran each test with:
Here are the relevant portions of each output from "time", run on a 3.0GHz Haswell:
I very much appreciate the team looking into this. Hopefully we can see a v6.7 with backported 64-bit compatibility to tide us over while v7 is being optimized.
Script:
In reworking my test script for LC Server compatibility I discovered two things:
1. revAvailableHandlers isn't available in the Server engine (not a biggie by itself, but:)
2. When a handler can't be found in a text script, the script doesn't compile at all, so that rather than generating any info like "Handler not found" or anything else which might guide us to a solution, it simply doesn't run. Is that a bug or just a necessary limitation?
My revised script is below. I saved it as "t.lc", and stored a copy in two folders, one for v6.7.1rc2 and v7.0.1rc2. I was curious about memory usage in addition to performance, so I ran each test with:
Code: Select all
/usr/bin/time -v ./t.lc
Given the nature of CGIs, this impact on scalability is clear.6.7.1 rc2:
User time (seconds): 3.54
System time (seconds): 0.01
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.56
Maximum resident set size (kbytes): 12212
Minor (reclaiming a frame) page faults: 1169
Voluntary context switches: 2
Involuntary context switches: 255
7.0.1 rc2:
User time (seconds): 13.62
System time (seconds): 0.09
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.76
Maximum resident set size (kbytes): 16732
Minor (reclaiming a frame) page faults: 7021
Voluntary context switches: 2000
Involuntary context switches: 280
I very much appreciate the team looking into this. Hopefully we can see a v6.7 with backported 64-bit compatibility to tide us over while v7 is being optimized.
Script:
Code: Select all
#!livecode-server
--=======================================================================--
-- Quickie Benchmark Suite
-- Richard Gaskin, Fourth World Systems
-- Placed into the public domain on 15 November, 2014
--
-- This script is self-contained, and can be put into a button
-- and clicked to run.
--
-- Tbe constants define parameters for the script.
-- Script-locals provide vars that can be reused among
-- the various commands.
--
-- The flow is simple enough:
-- BenchmarkAll sets up a header for the output and then
-- finds the handlers in the script that begin
-- with "Test_", and runs each one.
-- BenchmarkTest runs the actual handler within a wrapper that
-- provides metrics.
--
-- This particular set of handlers tests things I commonly do in
-- server scripts, but can be replaced with anything you need to
-- measure.
--
-- Please post any feedback, improvements, or suggestions to the
-- working group forum for Malte's Performance Benchmarking project:
-- http://forums.livecode.com/viewtopic.php?f=67&t=22072
--=======================================================================--
constant kTestSuiteIdentifier = "Common Server Tasks"
constant kBaseIterations = 100 -- common number of iterations for each test
constant kFileIterations = 1000 -- number of iterations for each file IO test
local sReport -- container for test results
local sTestList -- used in list-related tests
local sTestA -- used for array-related tests
local sTestFilePath -- used for file-related tasks
local sTemp -- scratch for misc needs (such as line numbers)
BenchmarkAll the long name of me
-- Simple way to handle bulk tests:
on BenchmarkAll
SetBenchmarkHeader
--
-- Test each "Test_*" handler in script of pObj:
put MyTestHandlers() into tTestList
repeat for each item tTest in tTestList
BenchmarkTest ("Test_"& tTest)
end repeat
--
put sReport
end BenchmarkAll
function MyTestHandlers
return "BuildList,LineOffset,LineAccessByNumber,LineAccessForEach,"\
&"ArraySplit,EncodeArray,DecodeArray,ArrayAccess,Merge,"\
&"BuildFilePath,FileTextWrite,FileTextRead,FileBinWrite,FileBinRead"
end MyTestHandlers
-- Header with basic system info:
on SetBenchmarkHeader
put "Performance Benchmarking: "& kTestSuiteIdentifier &cr\
&"LiveCode Version: "& the version &cr\
&"System Version: "& the platform && systemVersion()&cr\
&"--"&cr into sReport
end SetBenchmarkHeader
-- Common handler allows easy authoring of specific tests,
-- acknowledging that it includes the overhead of dispatching.
-- But since the purpose of this testing is to compare
-- relative performance, and since dispatch is the fastest way
-- to route handling to arbitrary commands, it's hoped that the
-- use of it here will be forgiven:
on BenchmarkTest pCmd
put "Running "& pCmd &"..."&cr
put the millisecs into t
if "File" is in pCmd then
repeat kFileIterations
dispatch pCmd
end repeat
else
repeat kBaseIterations
dispatch pCmd
end repeat
end if
put the millisecs - t into t
--
put pCmd &": "& t &" ms" &cr after sReport
end BenchmarkTest
on Test_BuildList
put empty into sTestList
repeat with i = 1 to 1000
put "SomeStuff" && i &cr after sTestList
end repeat
end Test_BuildList
on Test_LineOffset
put empty into sTemp
put the number of lines of sTestList into tMax
repeat with i = 1 to tMax
put lineoffset("SomeStuff "& i, sTestList) &cr after sTemp
end repeat
delete last char of sTemp -- trailing CR
end Test_LineOffset
on Test_LineAccessByNumber
repeat for each line tNum in sTemp
get line tNum of sTestList
end repeat
end Test_LineAccessByNumber
on Test_LineAccessForEach
repeat for each line tLine in sTestList
get tLine
end repeat
end Test_LineAccessForEach
on Test_ArraySplit
put sTestList into sTestA
split sTestA by cr
end Test_ArraySplit
on Test_EncodeArray
put arrayEncode(sTestA) into sTemp
end Test_EncodeArray
on Test_DecodeArray
put arrayDecode(sTemp) into sTestA
end Test_DecodeArray
on Test_ArrayAccess
repeat for each key tKey in sTestA
get sTestA[tKey]
end repeat
end Test_ArrayAccess
on Test_Merge
put "Merge data from array here: [[ GetArrayValueForMergeTest(tKey) ]]" into tString
repeat for each key tKey in sTestA
get merge(tString)
end repeat
end Test_Merge
-- Needed when testing older engines because there was a bug that
-- prevents proper evaluation of array expressions between double
-- brackets:
function GetArrayValueForMergeTest pKey
return sTestA[pKey]
end GetArrayValueForMergeTest
on Test_BuildFilePath
put specialFolderPath("temporary") &"/MalteBenchmarkTestData" into sTestFilePath
end Test_BuildFilePath
on Test_FileTextWrite
put sTestList into url ("file:"& sTestFilePath)
end Test_FileTextWrite
on Test_FileTextRead
put url ("file:"& sTestFilePath) into sTestList
end Test_FileTextRead
on Test_FileBinWrite
put sTestList into url ("binfile:"& sTestFilePath)
end Test_FileBinWrite
on Test_FileBinRead
put url ("binfile:"& sTestFilePath) into sTestList
end Test_FileBinRead
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
Re: Project: Performance Benchmarking Toolkit
Sorry I chime in late. Here are my 2 cents regarding faceless vs. UI driven testing... I think we need both. One of the important factors on a server / CGI installation is the startuptime. (Has anybody measured that already? Of course it shall be easier to test the low level operations in a faceless environment, however there are many many things that need the UI to be tested. From creating / destroying controls over everything that requires a render to things that are rude in the IDE itself (revUpdateGeometry has always been a bugger that way).
Also I would like to say that I really really appreciate all of you joining and especially am thankful that the RR engineers are chiming in as well. Thank you!
Also I would like to say that I really really appreciate all of you joining and especially am thankful that the RR engineers are chiming in as well. Thank you!
-
- VIP Livecode Opensource Backer
- Posts: 9867
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: Project: Performance Benchmarking Toolkit
I believe the folks at RunRev agree. The reason for the emphasis on core language features rather than rendering is that they have a few options already in development to boost rendering speed, so while those are being worked on they want to identify other opportunties for performance gains for all platforms, including faceless along with GUI ones, such as we're currently doing here.malte wrote:Sorry I chime in late. Here are my 2 cents regarding faceless vs. UI driven testing... I think we need both.
I haven't yet isolated startup time. but a simple script that just quits should be a reasonable starting point - I'll report back when I get some time to run that.One of the important factors on a server / CGI installation is the startuptime. Has anybody measured that already?
This morning I ran another test using the server script I posted recently and running it under strace:
Code: Select all
strace -c ./t.lc
I think this optimization effort is very important, time-consuming but with high ROI. While LC Server may not be an important revenue stream in itself, in this "Third Platform" era of computing it plays an important role in tying together all of the other platforms LC supports. Optimization across all these platforms is important, but in a world with plenty of server language options we need to get LC performance at least on par with PHP in order for its many advantages to become discoverable.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
-
- VIP Livecode Opensource Backer
- Posts: 2262
- Joined: Thu Feb 28, 2013 11:52 pm
- Location: Göttingen, DE
Re: Project: Performance Benchmarking Toolkit
Simply some "facts" from me, I don't interpret this. May be the measurement is not stable enough.
Nevertheless interesting. I repeated the test several times, had also some other rankings, but this is most frequent:
Calling Richard's server scripts (I added Test_ArrayCombine)
running under LC 6.7.1 (from my attached stack) livecode-community-server 6.7.1
(livecode-server was not available on my products page).
10 secs by shebang "#!" is slightly *slower* than
9 secs by livecode-server <file with tags "<? ... ?>" > or
9 secs by GUI
21 secs by GUI-LC 7.01 is by far slowest, but the factor of slowdown from 6.7.1 to 7.0.1 is smaller than with my GUI test of 'byte handling'.
So using GUI is even advantageous from my tests (may be the LC app has better memory handling?).
Here the testing stack, in case someone would like to repeat these tests in his environment.
Nevertheless interesting. I repeated the test several times, had also some other rankings, but this is most frequent:
Calling Richard's server scripts (I added Test_ArrayCombine)
running under LC 6.7.1 (from my attached stack) livecode-community-server 6.7.1
(livecode-server was not available on my products page).
10 secs by shebang "#!" is slightly *slower* than
9 secs by livecode-server <file with tags "<? ... ?>" > or
9 secs by GUI
21 secs by GUI-LC 7.01 is by far slowest, but the factor of slowdown from 6.7.1 to 7.0.1 is smaller than with my GUI test of 'byte handling'.
So using GUI is even advantageous from my tests (may be the LC app has better memory handling?).
Code: Select all
====================================================================
Sun, 23 Nov 2014 20:49:57 +0100
==[ 5 ]=============================================================
Start Sun, 23 Nov 2014 20:47:16 +0100
End Sun, 23 Nov 2014 20:47:26 +0100
--> Total sum of single tasks: 9.902 seconds
Performance Benchmarking: Common Server Tasks
--> LC-Server[ #! ]: 6.7.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 78 ms
Test_LineOffset: 3039 ms
Test_LineAccessByNumber: 551 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 51 ms
Test_ArrayCombine: 39 ms
Test_EncodeArray: 7 ms
Test_DecodeArray: 80 ms
Test_ArrayAccess: 16 ms
Test_Merge: 378 ms
Test_BuildFilePath: 8 ms
Test_FileTextWrite: 2444 ms
Test_FileTextRead: 59 ms
Test_FileBinWrite: 3127 ms
Test_FileBinRead: 17 ms
==[ 6 ]=============================================================
Start Sun, 23 Nov 2014 20:47:57 +0100
End Sun, 23 Nov 2014 20:48:06 +0100
--> Total sum of single tasks: 8.879 seconds
Performance Benchmarking: Common Server Tasks
--> LC-Server[ <?lc ]: 6.7.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 74 ms
Test_LineOffset: 2995 ms
Test_LineAccessByNumber: 514 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 56 ms
Test_ArrayCombine: 45 ms
Test_EncodeArray: 8 ms
Test_DecodeArray: 83 ms
Test_ArrayAccess: 17 ms
Test_Merge: 368 ms
Test_BuildFilePath: 7 ms
Test_FileTextWrite: 2107 ms
Test_FileTextRead: 56 ms
Test_FileBinWrite: 2522 ms
Test_FileBinRead: 19 ms
==[ 7 ]=============================================================
Start Sun, 23 Nov 2014 20:48:32 +0100
End Sun, 23 Nov 2014 20:48:41 +0100
--> Total sum of single tasks: 8.581 seconds
Performance Benchmarking: Common Server Tasks
--> LC [ GUI ] 6.7.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 75 ms
Test_LineOffset: 2872 ms
Test_LineAccessByNumber: 506 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 52 ms
Test_ArrayCombine: 43 ms
Test_EncodeArray: 7 ms
Test_DecodeArray: 89 ms
Test_ArrayAccess: 18 ms
Test_Merge: 379 ms
Test_BuildFilePath: 9 ms
Test_FileTextWrite: 2043 ms
Test_FileTextRead: 58 ms
Test_FileBinWrite: 2398 ms
Test_FileBinRead: 24 ms
==[ 8 ]=============================================================
Start Sun, 23 Nov 2014 20:49:28 +0100
End Sun, 23 Nov 2014 20:49:50 +0100
--> Total sum of single tasks: 21.376 seconds
Performance Benchmarking: Common Server Tasks
--> LC [ GUI ] 7.0.1-rc-2 / System: MacOS 10.10.1
--> [Mac mini/2.5 GHz i5/8 GB RAM]
--> TestIterations: 100 / FileIterations: 1000
Test_BuildList: 282 ms
Test_LineOffset: 8729 ms
Test_LineAccessByNumber: 5634 ms
Test_LineAccessForEach: 53 ms
Test_ArraySplit: 123 ms
Test_ArrayCombine: 272 ms
Test_EncodeArray: 88 ms
Test_DecodeArray: 118 ms
Test_ArrayAccess: 63 ms
Test_Merge: 887 ms
Test_BuildFilePath: 9 ms
Test_FileTextWrite: 2358 ms
Test_FileTextRead: 264 ms
Test_FileBinWrite: 2462 ms
Test_FileBinRead: 34 ms
====================================================================
Sun, 23 Nov 2014 20:49:57 +0100
====================================================================
- Attachments
-
- Performance2_7.livecode.zip
- (11 KiB) Downloaded 492 times
shiftLock happens
Re: Project: Performance Benchmarking Toolkit
I've already tested this. 7.0 starts up marginally faster than 6.7, although TBH the times are close enough that it's not a significant difference.malte wrote:One of the important factors on a server / CGI installation is the startuptime. (Has anybody measured that already?
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com
Re: Project: Performance Benchmarking Toolkit
This is pretty effective:FourthWorld wrote:I haven't yet isolated startup time. but a simple script that just quits should be a reasonable starting point - I'll report back when I get some time to run that.
Code: Select all
echo > empty.lc
livecode-community-server empty.lc
Code: Select all
touch empty.lc
livecode-community-server empty.lc
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com
-
- VIP Livecode Opensource Backer
- Posts: 2262
- Joined: Thu Feb 28, 2013 11:52 pm
- Location: Göttingen, DE
Re: Project: Performance Benchmarking Toolkit
Just to avoid misunderstanding for not-this-very-expert readers as me:Fourthworld wrote:I haven't yet isolated startup time
You haven't yet isolated startup time for measuring time, but it's currently *NOT* included in the overall time I reported. The startup enlarges this "Total sum" to the difference between the Start and End time stamps (first two lines).
Currently these are seconds only, but it shows already that this iadditional time is by far not this much time as needed for running server in CGI mode by Apache.
shiftLock happens