0.2 s to add a new element to a large array NOW WITH GRAPH!
Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller
0.2 s to add a new element to a large array NOW WITH GRAPH!
It takes a fifth of a second to add the millionth entry into an array.
Slow byte strings and slow arrays make for very boring days.
Slow byte strings and slow arrays make for very boring days.
Last edited by DarScott on Sat May 24, 2014 1:37 am, edited 1 time in total.
Re: 0.2 s to add a new element to a large array
My mistake. That is not the time to add the millionth element. That is the time to add the 561110th element of my test array.
Adding elements gets very slow between 553000 and 563000. And it somewhat slow over a wider range of 533000 through 563000.
Adding elements gets very slow between 553000 and 563000. And it somewhat slow over a wider range of 533000 through 563000.
Re: 0.2 s to add a new element to a large array
Here is a plot of the time to build a array with integers as keys and integers as values:
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Could Operating System background operations be the real culprit?
If you repeat this test immediately after rebooting your computer,
Will you get the same results?
Al
If you repeat this test immediately after rebooting your computer,
Will you get the same results?
Al
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Here is the same data displayed with lines instead of dots:
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Hi, Al! Here is what I got with less stat gathering overhead and after a reboot:
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Almost 15 seconds of delay!!!
What would be doing the engine in those 15 seconds?
Grabbing a big chunk of memory?
Creating indexes for accessing array elements?
The answer is in the source...
The engine source code!
Al
![Confused :?](./images/smilies/icon_confused.gif)
What would be doing the engine in those 15 seconds?
Grabbing a big chunk of memory?
Creating indexes for accessing array elements?
The answer is in the source...
The engine source code!
![Shocked :shock:](./images/smilies/icon_eek.gif)
Al
Re: 0.2 s to add a new element to a large array NOW WITH GRA
I'm guessing that as each hash bucket gets filled, a bigger size and maybe bigger kind of bucket is created. They are good hashes, so they fill about the same time.
Last edited by DarScott on Sat May 24, 2014 9:30 pm, edited 1 time in total.
Re: 0.2 s to add a new element to a large array NOW WITH GRA
The time to access the array is constant, about 3 microseconds for me, but a little higher in the 500,000 to 600,000 zone.
Re: 0.2 s to add a new element to a large array NOW WITH GRA
It took 16 minutes to union two non-overlapping 500,000 element arrays. So, that is not a solution.
Another possibility would be to make an array of a thousand entries each with a thousand entries. I'd rather avoid that.
Maybe build with split?
Another possibility would be to make an array of a thousand entries each with a thousand entries. I'd rather avoid that.
Maybe build with split?
Re: 0.2 s to add a new element to a large array NOW WITH GRA
The new arrayEncode(x,"7.0") packs arrays into half the size as before, but takes over twice as long. I'm good with that.
However, the arrayDecode(xe) is still slow. I would prefer it to be the faster one.
However, the arrayDecode(xe) is still slow. I would prefer it to be the faster one.
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Did you have a Test stack for this bug?
Al
Al
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Hi, Al!
I was about to say that I'm not sure it is a bug. It certainly is a nasty quirk. However, it has been slowing down today, and that is weird. It now takes 33 seconds. I'm up to 192 MB but I don't know what is normal for 7.0.
I have a stack. Not very clean. I have added some other buttons to see about a workaround and get other timing data. I'm currently looking at keeping a compressed empty array (integer-indexed) of size 650,000 or more in a custom property and popping it out every time I need an integer-indexed array larger than, say, 300,000.
It doesn't do the graph, it just puts csv data into a field. I did that as something quick to get a graph.
Would that be helpful? Is it possible to put stacks on the forum?
I was about to say that I'm not sure it is a bug. It certainly is a nasty quirk. However, it has been slowing down today, and that is weird. It now takes 33 seconds. I'm up to 192 MB but I don't know what is normal for 7.0.
I have a stack. Not very clean. I have added some other buttons to see about a workaround and get other timing data. I'm currently looking at keeping a compressed empty array (integer-indexed) of size 650,000 or more in a custom property and popping it out every time I need an integer-indexed array larger than, say, 300,000.
It doesn't do the graph, it just puts csv data into a field. I did that as something quick to get a graph.
Would that be helpful? Is it possible to put stacks on the forum?
-
- VIP Livecode Opensource Backer
- Posts: 3901
- Joined: Sat Mar 24, 2007 2:54 am
- Location: Palo Alto
Re: 0.2 s to add a new element to a large array NOW WITH GRA
Yep, just zip them up first and they have to be under 256k(?)Would that be helpful? Is it possible to put stacks on the forum?
Use the same method as posting pics.
Simon
I used to be a newbie but then I learned how to spell teh correctly and now I'm a noob!
Re: 0.2 s to add a new element to a large array NOW WITH GRA
I tried building an million-element array with empty elements, indexed by consecutive integers. I compress( arrayEncode( it, "7.0")) and it got down to 1012 bytes! I saved that in a property. Now, for arrays with similar keys, I can pull this out for arrays bigger than 345,000 entries. (I'm thinking of making the saved one smaller so arrayDecode() is faster. It is slow for large arrays.)
Anyway, I was able to build my array in 6.4 seconds instead of the 24 seconds up.
I'll get the stack up soon.
Anyway, I was able to build my array in 6.4 seconds instead of the 24 seconds up.
I'll get the stack up soon.