Project: Performance Benchmarking Toolkit

This forum is a working group for community contributors to the LiveCode IDE included in the main LiveCode distribution.

Moderators: Klaus, FourthWorld, heatherlaine, kevinmiller

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2588
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by mwieder » Wed Jan 07, 2015 11:19 pm

I'm coming late into this thread, and admittedly haven't read too much of the multiple pages here. I normally have a healthy distrust of benchmarks, but I'm happy to see work being done along those lines.

I *do* want to point out that a single pass isn't going to provide much in the way of useful data, though. You're going to be at the whims of cacheing and initial setup and other ephemeral things that affect timing. The benchmarks would be much better off running a number of times through a repeat loop (without the setup steps as much as possible) and then averaging the result at the end. Statistically you'll get a more realistic result that way.

peter-b
Posts: 182
Joined: Thu Nov 20, 2014 2:14 pm
Location: LiveCode Ltd.

Re: Project: Performance Benchmarking Toolkit

Post by peter-b » Mon Jan 26, 2015 4:24 pm

Just a quick follow-up from the development team here at LiveCode.

Internally, we're aware that there are some synthetic workloads that benchmark worse on 7.0.x than on 6.7.x. However, we're still looking for evidence of significant, widespread degradation in performance on real-world workloads. It's going to be very hard for us to justify spending a large amount of engineering effort on performance improvements if its unlikely to have significant impact across the user base.

If you've got an example of a practical, real-world workload that has slowed down a lot when moving from 6.7.x to 7.0.x, please file a bug report. I'm happy for you to contact me directly (peter@livecode.com) if you have stacks or datasets that you can't share on the bug tracker.

A very small number of people have been reporting a huge amount of slowdown moving from 6.7.x to 7.0.x on real-world or even trivial workloads. This has been reported specifically for LiveCode server in CGI mode, running on Dreamhost servers. This was reported as Bug 13983. We know that these users have been affected quite badly, and even completely trivial scripts are now taking multiple seconds to complete.

I've therefore performed some local testing to see if I can identify a problem.

I used this test script:

Code: Select all

<?lc
local tLoop
repeat with tLoop = 1 to 1000
    put "Hello world! It's " & the long time & linefeed
end repeat
?>
First, I tested running in "normal" server mode, using the command-line:

Code: Select all

$ time /path/to/livecode-community-server test.lc > /dev/null
Secondly, I tested running in a synthetic CGI environment, using a wrapper script that sets the CGI environment variables.

The wrapper script is called "test_server_dummy.sh":

Code: Select all

#/bin/bash

CONTENT='username=My%20Username&password=Not%20My%20Password&form_submitted=true'

export AUTH_TYPE=""
export CONTENT_LENGTH=${#CONTENT}
export CONTENT_TYPE="application/x-www-form-urlencoded"
export GATEWAY_INTERFACE="CGI/1.1"
export PATH_INFO=""
export PATH_TRANSLATED=$(readlink -f $1)
export QUERY_STRING=""
export REMOTE_ADDR="127.0.0.1"
export REMOTE_HOST="localhost"
export REMOTE_IDENT=""
export REMOTE_USER=""
export REQUEST_METHOD="POST"
export SCRIPT_NAME=""
export SERVER_NAME="localhost"
export SERVER_PORT="80"
export SERVER_PROTOCOL="HTTP/1.1"
export SERVER_SOFTWARE="bash"

echo "$CONTENT" | $SERVER $PATH_TRANSLATED
The test was run in CGI mode using the command line:

Code: Select all

$ time SERVER="/path/to/livecode-community-server" ./test_server_dummy.sh test.lc
Each test was run three times and the third run-time was used (in order to ensure that all caches were hot and that I wasn't benchmarking my HDD's fragmentation).

The results are as follows:

Code: Select all

    7.0.2 rc 1        i686      0.013 s
    7.0.2 rc 1 cgi    i686      0.017 s
    7.0.2 rc 1        x86-64    0.019 s
    7.0.2 rc 1 cgi    x86-64    0.022 s
    6.7.2 rc 2        i686      0.008 s
    6.7.2 rc 2 cgi    i686      0.012 s
As you can see:
  • the cost of running in CGI mode is unchanged from 6.7.x to 7.0.x
  • running in 64-bit mode is slower that running in 32-bit mode (which is to be expected, because 64-bit requires moving roughly twice as much data around all of the time)
  • 7.0.x is slower that 6.7.x for this task but it is *not* suddenly taking seconds where it previously took milliseconds.
These results suggest that there is performance degradation in 7.0.x relative to 6.7.x, but there may be other factors causing the extreme slowdown that some users are reporting.

For people who are looking for a hosted, fully-supported LiveCode 7 solution, we are expecting to start providing LiveCode 7 hosting soon via On-Rev.
LiveCode Open Source Team — @PeterTBBrett — peter.brett@livecode.com

geoffcanyon
Posts: 15
Joined: Thu Aug 15, 2013 9:25 am

Re: Project: Performance Benchmarking Toolkit

Post by geoffcanyon » Tue Nov 17, 2015 4:38 pm

In my testing, there are multiple, significant slowdowns moving from 6.7.3 to 7.0.3, and (again, in my testing) *no* speed-ups, so calling any benchmark "synthetic" is misleading at best. Fundamentally, 7.0.3 seems to be anywhere from 40% to 100+% slower than 6.7.3. Examples below, all performed on an up-to-date Mac:

on mouseUp
put the long seconds into T
repeat with i = 1 to 20000000

end repeat
put the long seconds - T into T
put T
end mouseUp

7.0.3 takes about 60% longer to run than 6.7.3 -- 1.6 seconds vs. 1.0 seconds.

on mouseUp
put the long seconds into T
repeat with i = 1 to 5000000
get random(123456)
end repeat
put the long seconds - T into T
put T
end mouseUp

7.0.3 takes about 2.7x as long as 6.7.3 -- 1.71 seconds vs. 0.63

on mouseUp
put the long seconds into T
repeat with i = 1 to 10000000
get 3+3
end repeat
put the long seconds - T into T
put T
end mouseUp

7.0.3 takes about 2.6x as long as 6.7.3 -- 3.17 seconds vs. 1.19 seconds

on mouseUp
   repeat with i = 1 to 1000
      put i,"" after L
   end repeat
   put the long seconds into T
   repeat with i = 1 to 100000
      get item 200 + random(600) of L 
   end repeat
   put the long seconds - T into T
   put T
end mouseUp

7.0.3 takes about 17x as long as 6.7.3 -- 3.45 seconds vs. 0.18 seconds

I'm just coming up with these tests at random. I have yet to find anything where 7.0.3 is as fast as 6.7.3, let alone faster.

bn
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3193
Joined: Sun Jan 07, 2007 9:12 pm
Location: Bochum, Germany

Re: Project: Performance Benchmarking Toolkit

Post by bn » Tue Nov 17, 2015 9:07 pm

Hi Geoff,
These are my results on a not to upToDate Mac

Note the improvements in some of the tests over the 7/8 series

6.7.8-rc-3 7.0.3 7.1.0 8.0.0-dp-9
Test 1 1.10 1.60 1.64 1.61
Test 2 0.75 1.71 1.87 0.96
Test 3 1.33 3.17 3.40 1.82
Test 4 0.23 3.45 3.51 3.02

tried to edit the format, the Forum wouldn't let me.
Kind regards

Bernd

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 5462
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by FourthWorld » Tue Dec 15, 2015 3:45 pm

I've read the team's been hard at work optimizing a few of the most commonly-used aspects of chunk expressions and other such language elements we use, so now that v8.0dp11 is out I re-ran the tests I'd been using on earlier versions with this new build.

The benchmark script I used is the one in this post:
http://forums.livecode.com/viewtopic.ph ... 15#p113997

Here are this morning's results running the two builds I'd been testing with earlier (6.7 and 7.0.1rc2) along with v8.0dp11:

Performance Benchmarking: Common Server Tasks
LiveCode Version: 6.7.1-rc-2
System Version: Linux Linux 3.13.0-71-generic
--
Test_BuildList: 72 ms
Test_Lineoffset: 2544 ms
Test_LineAccessByNumber: 518 ms
Test_LineAccessForEach: 8 ms
Test_ArraySplit: 35 ms
Test_EncodeArray: 9 ms
Test_DecodeArray: 46 ms
Test_ArrayAccess: 19 ms
Test_Merge: 220 ms
Test_BuildFilePath: 0 ms
Test_FileTextWrite: 2 ms
Test_FileTextRead: 1 ms
Test_FileBinWrite: 2 ms
Test_FileBinRead: 1 ms


Performance Benchmarking: Common Server Tasks
LiveCode Version: 7.0.1-rc-2
System Version: Linux Linux 3.13.0-71-generic
--
Test_BuildList: 197 ms
Test_Lineoffset: 7568 ms
Test_LineAccessByNumber: 4684 ms
Test_LineAccessForEach: 41 ms
Test_ArraySplit: 75 ms
Test_EncodeArray: 58 ms
Test_DecodeArray: 79 ms
Test_ArrayAccess: 48 ms
Test_Merge: 620 ms
Test_BuildFilePath: 22 ms
Test_FileTextWrite: 96 ms
Test_FileTextRead: 156 ms
Test_FileBinWrite: 80 ms
Test_FileBinRead: 14 ms


Performance Benchmarking: Common Server Tasks
LiveCode Version: 8.0.0-dp-11
System Version: Linux Linux 3.13.0-71-generic
--
Test_BuildList: 98 ms
Test_Lineoffset: 2521 ms
Test_LineAccessByNumber: 319 ms
Test_LineAccessForEach: 17 ms
Test_ArraySplit: 57 ms
Test_EncodeArray: 42 ms
Test_DecodeArray: 63 ms
Test_ArrayAccess: 27 ms
Test_Merge: 349 ms
Test_BuildFilePath: 14 ms
Test_FileTextWrite: 94 ms
Test_FileTextRead: 77 ms
Test_FileBinWrite: 78 ms
Test_FileBinRead: 14 ms

Overall v8 is faster than v7 in all tested operations, and in the areas with the greatest impact (lineoffset and accessing data at a specified line number) performance is even a bit faster than in v6.7.

Well done Mr. Waddingham, Ali, Peter, and the others who had a hand in this. Looking forward to delivering wonderfully performant apps with v8!
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

bhall2001
Posts: 99
Joined: Thu Oct 30, 2014 3:54 pm

Re: Project: Performance Benchmarking Toolkit

Post by bhall2001 » Tue Dec 15, 2015 5:46 pm

WOW! Looks like I will be moving to 8.0 ASAP for a couple of my projects that are in development. That's really impressive!

Bob Hall

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2588
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by mwieder » Tue Dec 15, 2015 8:15 pm

Yeah, those are impressive results.
I'm a bit dismayed by the file operations, but I trust that a similar optimization effort would take care of that quickly as well.

adventuresofgreg
Posts: 286
Joined: Tue Oct 28, 2008 1:23 am
Contact:

Re: Project: Performance Benchmarking Toolkit

Post by adventuresofgreg » Sun Dec 20, 2015 6:03 pm

Hello: Thanks for that test Richard. I have an app written with 5.1.1, and I am finding that either v7 OR v8 are running CONSIDERABLY SLOWER. Almost 3 times slower in fact, than v5, which I imagine would be similar speed to v6. (running on a MacPro).

My app does a LOT of array spitting, and text file reads - both of which, according to Richard's tests are considerably slower than v6.

Post Reply

Return to “IDE Contributors”