"new" JSON library fastJson

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Fri Oct 16, 2015 1:14 pm

I was able to find the LC Dictionary file in LC8. I'm also happy to report that the 0.3.1 branch of fastJson was able to convert the JSON to an array and back again! Converting the 9.8MB JSON file to an array took about 24 seconds on my 2007 iMac. This is by far the largest JSON file I've run through the library and I'm happy that it read in the file. From what I can see the data appears to be correct. Let me know how the data looks on your side once you get it into a stack.

Now, kicking the array back out to a JSON was surprising slow (42 seconds). In my other tests, ARRAY to JSON is usually much faster than the JSON to array. So, I'm off to see why this is slower than expected.

Bob

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Fri Oct 16, 2015 1:45 pm

Well, hold on... while it looks like the library did parse the file and my initial look at the data I thought was fine but I think there's an issue. I'm going need to do some more detailed analysis before I make my final conclusion that we have a valid array in LC.

Bob

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: "new" JSON library fastJson

Post by phaworth » Fri Oct 16, 2015 5:12 pm

Hi Bob,
I copy/pasted the new code into my app.

I ran jsonToArray against the dictionary data with false for the unicode parameter and true for the skip validation. It took anywhere from 10 to 25 seconds seconds but unfortunately, the results weren't correct.

The array had one subkey in it, "docs", which had a subkey "doc". WIth the doc subkey I think there should have been a number of numbered keys, each with one set of information from the dictionary file. Instead, I got no numbered keys and just the information for one doc entry, for the sum function.

If I run it with the skip valdation parameter set to false, the array is not created, my input data is changes to "false", but there is no error message of any sort.

Hope that helps track down what's going on.

Pete

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Fri Oct 16, 2015 10:28 pm

Been working on this today. The issue is that the data is not valid JSON.

a) I need better error handling in the code. isJson() determines that the JSON is not valid.
b) confirmed the json is not valid by running it through a known json linter.

fastJson isJson() determines that the {[]} are off by 10.
The linter kicks out so far at line 96535 with the following (just a snippet)

Code: Select all

			"description":"This simple library gives access to the LiveCode Builder canvas syntax from LiveCode Script by wrapping in a few simple handlers.\n\n\tExample usage:\n\n\t  canvasCreate 100, 100\n\t  canvasSetColor 1, 0, 0, 1\n\t  canvasFillCircle 50, 50, 25\n\t  canvasApplyToImage \"image 1\"\n      canvasDestroy\n\n   This creates a canvas of size 100x100, fills a circle of radius 25 at\n   the centre, then copies the contents of the canvas to 'image 1' (which\n   must already exist)."
		}]
	}
	]
}var dictionary_data =
{
	"docs":[{
		"display name":"LiveCode Script",
Notice the "var dictionary_data=" this is javascript in the middle of the json. I'm trying to figure out to get the dictionary into a single valid JSON data structure. I'm working on doing lines 2-96535 as a "part 1" making it valid JSON and then I'll send it through the library.

I'm also going to work on the error handling of isJson() so that the library gracefully fails out if JSON is not valid.

Bob

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: "new" JSON library fastJson

Post by phaworth » Fri Oct 16, 2015 11:21 pm

Thanks Bob.
I did remove an initial var statement in line 1 of the file, didn't realize there was another one. Let me try removing all the var statements before passing it off to jsonToArray.

Yes, it woul definitly help if there was more in the way of an error message, even if just a line number where the failure occurred.

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: "new" JSON library fastJson

Post by phaworth » Fri Oct 16, 2015 11:49 pm

Hi Bob, just been looking at this. I think you might be using a different file than me. Mine ends at line 96361 and I can't find another instance of a var declaration after the first line. Also, the var dictionary data in your snippet is on line 1 of my file. I'm getting the file mentioned by bn in a recent post, running OSX. What platform are you on? I really hope the dictionary isn;t different on different platforms!
pete

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Sat Oct 17, 2015 4:27 pm

HMM... let me look into it. I have no idea what's up with that. I am on Mac OS 10.10.5 and I'm using LC 7.1 Indy but did get the Dictionary JSON from LC8 DP6 Community. I have a bouncy icon yelling at me to upgrade my LC which I will do momentarily and see what's in there for a dictionary.

Pete, THANK YOU! THANK YOU! THANK YOU!

Your desire to parse the dictionary resulted in me finding some issues in the logic of the Tokenizer that I'm not sure we would of come across. Good news is that I can now parse the Dictionary JSON and am VERY CONFIDENT the resulting array is accurate and correct. VERY COOL! Dang, I actually had to put on my thinking cap for this one :lol:

Version 0.4.0 of fastJson is up on Github. I've also updated my silly little test stack. As I've convinced myself that fastJSON is faster then the other libraries, you will see the other libraries are now removed from the test stack. Also, there are fields to show the array and JSON created from the test file. (note: only shows first 20k characters in fields).

https://github.com/bhall2001/fastjson/tree/0.4.0

You can download just the library by clicking on it, then copying and pasting the code into your library stack.

I will look into what's up with my test dictionary (I hope my fault), and please, if you see anything, let me know so I can get it fixed.

Bob

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Sat Oct 17, 2015 4:48 pm

Pete, user error. I did a copy/paste in ATOM and must of double pasted the dictionary file. Thank goodness!

I've attached a screen capture of my test stack's results of parsing the dictionary (old and tired 2007 iMac). 405707 tokens got parsed in 18.5 seconds and 12.9 seconds to go back to an array. I'll take it!

If you do use the test stack to do this, be patient at the end, it takes almost as long for LC to populate the text fields as it does to run through the library...

Bob
Image

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: "new" JSON library fastJson

Post by phaworth » Sat Oct 17, 2015 5:43 pm

Bob,
Looks like it works fine now on the dictionary, thanks a lot for the fixes. It took around 25 seconds on my latest model iMac in LC8/OSX 10.10 but it is a huge file.

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: "new" JSON library fastJson

Post by phaworth » Sat Oct 17, 2015 5:53 pm

After successfully parsing the dictionary file, I was prowling around the array and came across jsonImport and jsonExport as part of a JSON library which looks like it was introduced in LC8. Out of curiosity, I used jsonImport to parse the dictionary file. It looks like it produced the same results as fastJSON but took about 20% longer (30 seconds vs 25 seconds)
Pete

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Sat Oct 17, 2015 11:42 pm

I've noticed in my testing a strange behavior and am wondering if someone might be able to steer me in a direction to track down the issue.

When I use my test stack on the dictionary here's what I've noticed.
Launch LC 7.
Launch test stack
open Livecode Dictionary json file and run tests.

Speed is VERY fast on Json to Array.

ANY subsequent run of the library I see a significant slow down in performance (dictionary file is like 3x slower all additional runs).

Any idea what I should be looking for that would cause this? I don't even know where to start.

Thanks in advance,
Bob

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: "new" JSON library fastJson

Post by phaworth » Sun Oct 18, 2015 12:40 am

You know that happened to me too but I thought it was a factor of what else was happening on my computer at the time. I have a test stack of my own that I used to figure out how to get what I need from the dictionary so I just went back and rant a few times and sure enough, it takes about 3 times as long on the second try. I notice you create a script local variable named sJSON. I wonder if that should be emptied/deleted after your processing is complete?
Pete

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: "new" JSON library fastJson

Post by mwieder » Sun Oct 18, 2015 1:34 am

Bob-

your jsonValue function is doing some weird things in the default case... it's working with a variable named pText, but there isn't one.
You're also (but this doesn't affect the speed tests) grabbing the binfile contents of the json file twice in uiLoadData. The second one doesn't seem necessary at all.

Allow me to push the idea that enabling "Strict Compilation Mode" in the IDE's preferences is a good idea <g> to prevent things like this.

I grabbed the LC8 dictionary file, so my figures will be slightly different from LC7 (and I'm running the tests on linux Mint 17), but here's what I get:

Token Count: 406306

(all times in milliseconds)
JSON to Array: 15273 33938 33974 34300 34336
Array to JSON: 6175 6099 6048 6696 6905

Got no idea currently on why runs after the first take twice as long.

bhall2001
Posts: 107
Joined: Thu Oct 30, 2014 3:54 pm
Location: Manchester, NH

Re: "new" JSON library fastJson

Post by bhall2001 » Sun Oct 18, 2015 8:34 pm

Just created branch 4.0.1.

Includes fix to pText, declaration of local vars for strict compilation, and most importantly, THE FIX TO "RUNS FAST ONCE THEN NOT" :D

Here's what caused the issue. If the JSON data contained a data pair of "name":"", Livecode would forever run the library slow. I added in a special case looking for "" as a data element and handling that separate from a regular string.

Works great now. Nice and fast all the time. If you do install this, make sure you quite and re-launch Livecode.

https://github.com/bhall2001/fastjson/tree/0.4.1

Bob

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: "new" JSON library fastJson

Post by mwieder » Mon Oct 19, 2015 12:11 am

Oh, that's bizarre. Now I gotta clone the repo and see where you changed that.
Is that a namespace collision between the JSON data and the LC keywords?
Is it that you can't have an array key named "name"?

Whatever, it sounds like it's worthy of a bug report.

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”