"new" JSON library fastJson

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

"new" JSON library fastJson

Post by bhall2001 » Sat Oct 03, 2015 3:19 am

The other day I was having some fun comparing easyJson and libJson. I could see that a merged library would result in better overall JSON performance. I also believe there may be some tweaks that can be made to get a little bit more performance as well. Welcome fastJson.

The Project is on Github (url below). Both easyJson and libJson calls will work and my intention is that fastJson should be a drop in replacement for existing stacks using either of these libraries (it's worked on my projects as far as I can see). There's a new function isJson() for JSON validation. I've included the stack I'm using for speed comparisons in the Github repository. You'll find a brief overview of the test stack in the stack script.

The library has an overview as well that explains how to use it. If you've used either easyJson or libJson just keep doing what you know. It should just work (only a little faster). Help/Suggestions are very welcome.

https://github.com/bhall2001/fastjson

Bob

pink
Posts: 272
Joined: Wed Mar 12, 2014 6:18 pm

Re: "new" JSON library fastJson

Post by pink » Sun Oct 04, 2015 5:13 pm

I ran my own speed tests.

I have a program that accesses several CouchDB databases, downloads each record (in JSON format), converts them to arrays, then there's some jiggery-pokery to extract some information. Then the array is converted back into a JSON doc and uploaded to a new database.

I ran 10 tests using easyJson which ranged from 20231 to 21013 milliseconds. Average 20716.

I ran 10 tests using fastJson which ranged from 13849 to 15772 milliseconds. Average 14509.

Now obviously there's more going on here than just the conversion of JSONs to and from arrays, but there is an obvious speed bump. Should I get up the gumption to piece out the speeds I will.

Well done Bob, thanks.
Greg (pink) Miller

MadPink, LLC
I'm Mad, Pink and Dangerous to Know

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Sun Oct 04, 2015 9:22 pm

Thanks pink for running your tests! I was seeing similar results and it's nice to see that it's looking like I moved in the right direction.

Did you run the tests with JSON validation on or off? If you had it on, could you run the same tests again with validation checking off? I'm curious what you get with your test suite with that (this obviously assumes you're sure of getting a well formed JSON in but this should yield an even better result).

The call would be...
jsonToArray(pJsonData,,true)

Bob

pink
Posts: 272
Joined: Wed Mar 12, 2014 6:18 pm

Re: "new" JSON library fastJson

Post by pink » Mon Oct 05, 2015 4:52 pm

I ran the tests again turning validation off and the speed does noticeably increase.

I ran 10 tests using easyJson which ranged from 20231 to 21013 milliseconds. Average 20716.
I ran 10 tests using fastJson which ranged from 13849 to 15772 milliseconds. Average 14509. (with validation ON)
I ran 10 tests using fastJson which ranged from 11899 to 13444 milliseconds. Average 12203. (with validation OFF)

For illustrative purposes, using fastJson decreases the amount of time the process takes in my program by 30% with validation on, and 40% with validation off.

Again, it's important to note that other actions are taking place in the above tests. There are about 700 JSON to Array conversions, and about 150 Array to JSON conversions that take place.
Greg (pink) Miller

MadPink, LLC
I'm Mad, Pink and Dangerous to Know

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Mon Oct 05, 2015 6:40 pm

Greg,

Any chance your can test with your code and the new LC8 importJson/ExportJson. I have already with my script. I'd be curious what you get for numbers.

Bob

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Mon Oct 05, 2015 7:02 pm

I'm modifying the library a bit as well as doing a round of Livecode 8 testing.

Here's what I have for numbers. I am taking a complex JSON datafile from an application that I've written which is roughly 94k of JSON. There are almost 24,000 tokens in the file to parse. My test stack first converts the file to an Array, then converts the array back to a JSON. The number of milliseconds to perform each step is recorded. I'm comparing easyJson, libJson, and fastJson. Here's a table of an average of runs.

file size: 94k & 23816 tokens to parse

Code: Select all

                       
                          easyJson         libJson        fastJson(w/validation)  fastJson (no validation)  
JSON to Array               4258              1040           990                              759
Array to JSON               172                2782           174                              174
Now, I've also done a round with fastJson and LC8's JSON library. This library I found out is based on easyJson logic. Also, for some reason (yet to be determined), my 94k JSON file will not run through the libraries without reporting invalid JSON (which I think I'm narrowing in on).

But more interesting, I can load in the array up in my test stack and create a JSON file with LC8's library as a new file. When I do this, file size and tokens go up by a significant amount. I'm working on diff-ing the files now and can see some, well, differences. I need some more time to look at things before I make any conclusions though. But for now, as we stand.

file size: 151k & 37626 tokens to parse

Code: Select all

                           easyJson      libJson      fastJson       fastJson (no)              LC8
JSON to Array  not a typo-->30071           3389         1499                1137                  2121
Array to JSON              229              3647         229                  229                  242
There you have it, converting a JSON to an array and back to JSON test results.
Bob

pink
Posts: 272
Joined: Wed Mar 12, 2014 6:18 pm

Re: "new" JSON library fastJson

Post by pink » Mon Oct 05, 2015 8:28 pm

I must be missing something here... this is the function I'm using to try and use the Json library:

Code: Select all

function arrayFromJson @pJson
     return JsonImport(pJson)
end arrayFromJson
I keep getting this error:

Code: Select all

stack "couchdbKatie"; execution error at line 808 (extensionL error occured with domain) near "runtime", char 13
Greg (pink) Miller

MadPink, LLC
I'm Mad, Pink and Dangerous to Know

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Tue Oct 06, 2015 5:44 pm

When I did my test stack I did not try to call jsonImport from within another function. I just called it directly replacing the arrayFromJason call all together. Maybe (but don't know why, calling with a referenced parameter is causing a problem.

Also, run the json file through jsonlint and confirm it is valid. I have found in the last day or 2 that both easyJson and libJson are very loose in determining what is "valid" json. My new favorite online free linter is http://www.freeformatter.com/json-validator.html. It's fast and you can make it not as strict.

Give that a try.

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Tue Oct 06, 2015 9:12 pm

For anyone following along here, I just uncovered a significant issue with both libJson and easyJson and how they handle numbers. Both routines rely on the "is number" function of Livecode to determine if a data element is a number or a string. Well, turns out Livecode, being the verbose language that we love, returns true if you ask if any of the following "is number" -- infinity, zero, one, two, three, four, five, six, seven, eight, nine and ten (from what I can see). If you have a data element that is intending to be a string but contains one of these words, the existing Json libraries will create a malformed Json string.

I have the fix in fastJson. And of course adding in this extra check resulted in loosing about very small percentage of speed (15ms in my 94k test file). BUT, we are getting a much better Json string as a result.

Bob

dave.kilroy
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 858
Joined: Wed Jun 24, 2009 1:17 pm
Location: Plymouth, UK
Contact:

Re: "new" JSON library fastJson

Post by dave.kilroy » Wed Oct 07, 2015 2:06 am

Hi Bob - thank you for your work in speeding-up and beefing-up json in LiveCode

What you noticed with words 'five', 'six' etc returning values as numbers - that is because they are built-in LiveCode constants! But if you try the 'is a number' test against strings (with the word 'five' enclosed in quotation marks) then LiveCode resolves this as a string

If this is a potential fragility in managing json then there are other constants you might want to check for as well - for the list of built-in constants try "put the constantnames" in the messagebox

Kind regards

Dave
"...this is not the code you are looking for..."

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Wed Oct 07, 2015 4:57 pm

Thanks for the heads up on the constants. I did see the "is a number" does do what I need, for the most part. I'm a little concerned that this behavior might change as it's different than all the other isNumber methods.

Also, there are 2 constants missing from the list (and I've reported as a bug this am -- "inf", "infinity"
follow along here: http://quality.runrev.com/show_bug.cgi?id=16160

I've also discovered that there is a bug in LC with Infinity as it relates to conditionals. It never evaluates to true (I think). Have some fun, go to this bug report and download the test stack (ok, it's really not that much fun but shows the issue ;-)
http://quality.runrev.com/show_bug.cgi?id=16162

The other issue that I came across in the older libraries is how they handled numbers starting with a 0. I've got it worked out in fastJson. Not pretty, but works and is as fast as I can make it I think. Sometimes fast solutions are not pretty. Ahh, that takes me back to 1979 when I was coding in assembly on the 68000...

Bob

pink
Posts: 272
Joined: Wed Mar 12, 2014 6:18 pm

Re: "new" JSON library fastJson

Post by pink » Wed Oct 07, 2015 5:09 pm

the on ething I have found most odd is that infinity does not appear in the documentation at all

my concern is, are there other non-documented pitfalls?
Greg (pink) Miller

MadPink, LLC
I'm Mad, Pink and Dangerous to Know

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: "new" JSON library fastJson

Post by FourthWorld » Wed Oct 07, 2015 7:41 pm

That infinity is a defined constant doesn't seem a pitfall as much as feature, the bug being that it's undocumented.

For the most part the Dictionary is current, and with new changes made since the project was moved to Github it's generally kept current with new tokens, as Dictionary updates are a part of the defined process.

That said, now and then we may find an older token not documented. AFAIK most have been caught, but if you find one please submit a report and it should be added in the next build or two after (depending on where they are in the dev cycle at the moment the report is submitted).
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

pink
Posts: 272
Joined: Wed Mar 12, 2014 6:18 pm

Re: "new" JSON library fastJson

Post by pink » Wed Oct 07, 2015 9:55 pm

out of curiosity, is it possible to override a built-in constant?

e.g. define one to be "one"
Greg (pink) Miller

MadPink, LLC
I'm Mad, Pink and Dangerous to Know

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: "new" JSON library fastJson

Post by FourthWorld » Wed Oct 07, 2015 11:05 pm

Not yet, but the ability to override built-in properties has been so frequently requested that hopefully we'll have that one day, and perhaps when we do then overriding constants shortly afterward.

This is a tricky thing, though, in terms of compatibility: what happens when you use someone else's library that redefines a constant?

Using getProp or setProp for built-in properties in custom controls like the DataGrid would be a godsend, and limited enough in scope to only affect the object itself.

But once we step outside of behaviors specific to custom controls, there be dragons.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”