"new" JSON library fastJson
Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller
"new" JSON library fastJson
The other day I was having some fun comparing easyJson and libJson. I could see that a merged library would result in better overall JSON performance. I also believe there may be some tweaks that can be made to get a little bit more performance as well. Welcome fastJson.
The Project is on Github (url below). Both easyJson and libJson calls will work and my intention is that fastJson should be a drop in replacement for existing stacks using either of these libraries (it's worked on my projects as far as I can see). There's a new function isJson() for JSON validation. I've included the stack I'm using for speed comparisons in the Github repository. You'll find a brief overview of the test stack in the stack script.
The library has an overview as well that explains how to use it. If you've used either easyJson or libJson just keep doing what you know. It should just work (only a little faster). Help/Suggestions are very welcome.
https://github.com/bhall2001/fastjson
Bob
The Project is on Github (url below). Both easyJson and libJson calls will work and my intention is that fastJson should be a drop in replacement for existing stacks using either of these libraries (it's worked on my projects as far as I can see). There's a new function isJson() for JSON validation. I've included the stack I'm using for speed comparisons in the Github repository. You'll find a brief overview of the test stack in the stack script.
The library has an overview as well that explains how to use it. If you've used either easyJson or libJson just keep doing what you know. It should just work (only a little faster). Help/Suggestions are very welcome.
https://github.com/bhall2001/fastjson
Bob
Re: "new" JSON library fastJson
I ran my own speed tests.
I have a program that accesses several CouchDB databases, downloads each record (in JSON format), converts them to arrays, then there's some jiggery-pokery to extract some information. Then the array is converted back into a JSON doc and uploaded to a new database.
I ran 10 tests using easyJson which ranged from 20231 to 21013 milliseconds. Average 20716.
I ran 10 tests using fastJson which ranged from 13849 to 15772 milliseconds. Average 14509.
Now obviously there's more going on here than just the conversion of JSONs to and from arrays, but there is an obvious speed bump. Should I get up the gumption to piece out the speeds I will.
Well done Bob, thanks.
I have a program that accesses several CouchDB databases, downloads each record (in JSON format), converts them to arrays, then there's some jiggery-pokery to extract some information. Then the array is converted back into a JSON doc and uploaded to a new database.
I ran 10 tests using easyJson which ranged from 20231 to 21013 milliseconds. Average 20716.
I ran 10 tests using fastJson which ranged from 13849 to 15772 milliseconds. Average 14509.
Now obviously there's more going on here than just the conversion of JSONs to and from arrays, but there is an obvious speed bump. Should I get up the gumption to piece out the speeds I will.
Well done Bob, thanks.
Greg (pink) Miller
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
Re: "new" JSON library fastJson
Thanks pink for running your tests! I was seeing similar results and it's nice to see that it's looking like I moved in the right direction.
Did you run the tests with JSON validation on or off? If you had it on, could you run the same tests again with validation checking off? I'm curious what you get with your test suite with that (this obviously assumes you're sure of getting a well formed JSON in but this should yield an even better result).
The call would be...
jsonToArray(pJsonData,,true)
Bob
Did you run the tests with JSON validation on or off? If you had it on, could you run the same tests again with validation checking off? I'm curious what you get with your test suite with that (this obviously assumes you're sure of getting a well formed JSON in but this should yield an even better result).
The call would be...
jsonToArray(pJsonData,,true)
Bob
Re: "new" JSON library fastJson
I ran the tests again turning validation off and the speed does noticeably increase.
I ran 10 tests using easyJson which ranged from 20231 to 21013 milliseconds. Average 20716.
I ran 10 tests using fastJson which ranged from 13849 to 15772 milliseconds. Average 14509. (with validation ON)
I ran 10 tests using fastJson which ranged from 11899 to 13444 milliseconds. Average 12203. (with validation OFF)
For illustrative purposes, using fastJson decreases the amount of time the process takes in my program by 30% with validation on, and 40% with validation off.
Again, it's important to note that other actions are taking place in the above tests. There are about 700 JSON to Array conversions, and about 150 Array to JSON conversions that take place.
I ran 10 tests using easyJson which ranged from 20231 to 21013 milliseconds. Average 20716.
I ran 10 tests using fastJson which ranged from 13849 to 15772 milliseconds. Average 14509. (with validation ON)
I ran 10 tests using fastJson which ranged from 11899 to 13444 milliseconds. Average 12203. (with validation OFF)
For illustrative purposes, using fastJson decreases the amount of time the process takes in my program by 30% with validation on, and 40% with validation off.
Again, it's important to note that other actions are taking place in the above tests. There are about 700 JSON to Array conversions, and about 150 Array to JSON conversions that take place.
Greg (pink) Miller
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
Re: "new" JSON library fastJson
Greg,
Any chance your can test with your code and the new LC8 importJson/ExportJson. I have already with my script. I'd be curious what you get for numbers.
Bob
Any chance your can test with your code and the new LC8 importJson/ExportJson. I have already with my script. I'd be curious what you get for numbers.
Bob
Re: "new" JSON library fastJson
I'm modifying the library a bit as well as doing a round of Livecode 8 testing.
Here's what I have for numbers. I am taking a complex JSON datafile from an application that I've written which is roughly 94k of JSON. There are almost 24,000 tokens in the file to parse. My test stack first converts the file to an Array, then converts the array back to a JSON. The number of milliseconds to perform each step is recorded. I'm comparing easyJson, libJson, and fastJson. Here's a table of an average of runs.
file size: 94k & 23816 tokens to parse
Now, I've also done a round with fastJson and LC8's JSON library. This library I found out is based on easyJson logic. Also, for some reason (yet to be determined), my 94k JSON file will not run through the libraries without reporting invalid JSON (which I think I'm narrowing in on).
But more interesting, I can load in the array up in my test stack and create a JSON file with LC8's library as a new file. When I do this, file size and tokens go up by a significant amount. I'm working on diff-ing the files now and can see some, well, differences. I need some more time to look at things before I make any conclusions though. But for now, as we stand.
file size: 151k & 37626 tokens to parse
There you have it, converting a JSON to an array and back to JSON test results.
Bob
Here's what I have for numbers. I am taking a complex JSON datafile from an application that I've written which is roughly 94k of JSON. There are almost 24,000 tokens in the file to parse. My test stack first converts the file to an Array, then converts the array back to a JSON. The number of milliseconds to perform each step is recorded. I'm comparing easyJson, libJson, and fastJson. Here's a table of an average of runs.
file size: 94k & 23816 tokens to parse
Code: Select all
easyJson libJson fastJson(w/validation) fastJson (no validation)
JSON to Array 4258 1040 990 759
Array to JSON 172 2782 174 174
But more interesting, I can load in the array up in my test stack and create a JSON file with LC8's library as a new file. When I do this, file size and tokens go up by a significant amount. I'm working on diff-ing the files now and can see some, well, differences. I need some more time to look at things before I make any conclusions though. But for now, as we stand.
file size: 151k & 37626 tokens to parse
Code: Select all
easyJson libJson fastJson fastJson (no) LC8
JSON to Array not a typo-->30071 3389 1499 1137 2121
Array to JSON 229 3647 229 229 242
Bob
Re: "new" JSON library fastJson
I must be missing something here... this is the function I'm using to try and use the Json library:
I keep getting this error:
Code: Select all
function arrayFromJson @pJson
return JsonImport(pJson)
end arrayFromJson
Code: Select all
stack "couchdbKatie"; execution error at line 808 (extensionL error occured with domain) near "runtime", char 13
Greg (pink) Miller
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
Re: "new" JSON library fastJson
When I did my test stack I did not try to call jsonImport from within another function. I just called it directly replacing the arrayFromJason call all together. Maybe (but don't know why, calling with a referenced parameter is causing a problem.
Also, run the json file through jsonlint and confirm it is valid. I have found in the last day or 2 that both easyJson and libJson are very loose in determining what is "valid" json. My new favorite online free linter is http://www.freeformatter.com/json-validator.html. It's fast and you can make it not as strict.
Give that a try.
Also, run the json file through jsonlint and confirm it is valid. I have found in the last day or 2 that both easyJson and libJson are very loose in determining what is "valid" json. My new favorite online free linter is http://www.freeformatter.com/json-validator.html. It's fast and you can make it not as strict.
Give that a try.
Re: "new" JSON library fastJson
For anyone following along here, I just uncovered a significant issue with both libJson and easyJson and how they handle numbers. Both routines rely on the "is number" function of Livecode to determine if a data element is a number or a string. Well, turns out Livecode, being the verbose language that we love, returns true if you ask if any of the following "is number" -- infinity, zero, one, two, three, four, five, six, seven, eight, nine and ten (from what I can see). If you have a data element that is intending to be a string but contains one of these words, the existing Json libraries will create a malformed Json string.
I have the fix in fastJson. And of course adding in this extra check resulted in loosing about very small percentage of speed (15ms in my 94k test file). BUT, we are getting a much better Json string as a result.
Bob
I have the fix in fastJson. And of course adding in this extra check resulted in loosing about very small percentage of speed (15ms in my 94k test file). BUT, we are getting a much better Json string as a result.
Bob
-
- VIP Livecode Opensource Backer
- Posts: 858
- Joined: Wed Jun 24, 2009 1:17 pm
- Location: Plymouth, UK
- Contact:
Re: "new" JSON library fastJson
Hi Bob - thank you for your work in speeding-up and beefing-up json in LiveCode
What you noticed with words 'five', 'six' etc returning values as numbers - that is because they are built-in LiveCode constants! But if you try the 'is a number' test against strings (with the word 'five' enclosed in quotation marks) then LiveCode resolves this as a string
If this is a potential fragility in managing json then there are other constants you might want to check for as well - for the list of built-in constants try "put the constantnames" in the messagebox
Kind regards
Dave
What you noticed with words 'five', 'six' etc returning values as numbers - that is because they are built-in LiveCode constants! But if you try the 'is a number' test against strings (with the word 'five' enclosed in quotation marks) then LiveCode resolves this as a string
If this is a potential fragility in managing json then there are other constants you might want to check for as well - for the list of built-in constants try "put the constantnames" in the messagebox
Kind regards
Dave
"...this is not the code you are looking for..."
Re: "new" JSON library fastJson
Thanks for the heads up on the constants. I did see the "is a number" does do what I need, for the most part. I'm a little concerned that this behavior might change as it's different than all the other isNumber methods.
Also, there are 2 constants missing from the list (and I've reported as a bug this am -- "inf", "infinity"
follow along here: http://quality.runrev.com/show_bug.cgi?id=16160
I've also discovered that there is a bug in LC with Infinity as it relates to conditionals. It never evaluates to true (I think). Have some fun, go to this bug report and download the test stack (ok, it's really not that much fun but shows the issue
http://quality.runrev.com/show_bug.cgi?id=16162
The other issue that I came across in the older libraries is how they handled numbers starting with a 0. I've got it worked out in fastJson. Not pretty, but works and is as fast as I can make it I think. Sometimes fast solutions are not pretty. Ahh, that takes me back to 1979 when I was coding in assembly on the 68000...
Bob
Also, there are 2 constants missing from the list (and I've reported as a bug this am -- "inf", "infinity"
follow along here: http://quality.runrev.com/show_bug.cgi?id=16160
I've also discovered that there is a bug in LC with Infinity as it relates to conditionals. It never evaluates to true (I think). Have some fun, go to this bug report and download the test stack (ok, it's really not that much fun but shows the issue
http://quality.runrev.com/show_bug.cgi?id=16162
The other issue that I came across in the older libraries is how they handled numbers starting with a 0. I've got it worked out in fastJson. Not pretty, but works and is as fast as I can make it I think. Sometimes fast solutions are not pretty. Ahh, that takes me back to 1979 when I was coding in assembly on the 68000...
Bob
Re: "new" JSON library fastJson
the on ething I have found most odd is that infinity does not appear in the documentation at all
my concern is, are there other non-documented pitfalls?
my concern is, are there other non-documented pitfalls?
Greg (pink) Miller
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
-
- VIP Livecode Opensource Backer
- Posts: 9802
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: "new" JSON library fastJson
That infinity is a defined constant doesn't seem a pitfall as much as feature, the bug being that it's undocumented.
For the most part the Dictionary is current, and with new changes made since the project was moved to Github it's generally kept current with new tokens, as Dictionary updates are a part of the defined process.
That said, now and then we may find an older token not documented. AFAIK most have been caught, but if you find one please submit a report and it should be added in the next build or two after (depending on where they are in the dev cycle at the moment the report is submitted).
For the most part the Dictionary is current, and with new changes made since the project was moved to Github it's generally kept current with new tokens, as Dictionary updates are a part of the defined process.
That said, now and then we may find an older token not documented. AFAIK most have been caught, but if you find one please submit a report and it should be added in the next build or two after (depending on where they are in the dev cycle at the moment the report is submitted).
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
Re: "new" JSON library fastJson
out of curiosity, is it possible to override a built-in constant?
e.g. define one to be "one"
e.g. define one to be "one"
Greg (pink) Miller
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
MadPink, LLC
I'm Mad, Pink and Dangerous to Know
-
- VIP Livecode Opensource Backer
- Posts: 9802
- Joined: Sat Apr 08, 2006 7:05 am
- Location: Los Angeles
- Contact:
Re: "new" JSON library fastJson
Not yet, but the ability to override built-in properties has been so frequently requested that hopefully we'll have that one day, and perhaps when we do then overriding constants shortly afterward.
This is a tricky thing, though, in terms of compatibility: what happens when you use someone else's library that redefines a constant?
Using getProp or setProp for built-in properties in custom controls like the DataGrid would be a godsend, and limited enough in scope to only affect the object itself.
But once we step outside of behaviors specific to custom controls, there be dragons.
This is a tricky thing, though, in terms of compatibility: what happens when you use someone else's library that redefines a constant?
Using getProp or setProp for built-in properties in custom controls like the DataGrid would be a godsend, and limited enough in scope to only affect the object itself.
But once we step outside of behaviors specific to custom controls, there be dragons.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn