Why Array over list

Got a LiveCode personal license? Are you a beginner, hobbyist or educator that's new to LiveCode? This forum is the place to go for help getting started. Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Why Array over list

Post by FourthWorld » Tue Oct 25, 2016 6:08 pm

Custom properties ans property sets can be used to express arrays well enough:

Code: Select all

  set the foo[1] of this stack to (the foo[1] of this stack + 1)
That can be useful when one needs only 1 or 2 dimensions, and when binding data specifically to an object is important.

For general file or network I/O, rather than binding to an object an array of any depth can be converted to a bytestream easily with arrayEncode, and converted back to an array with arrayDecode, e.g.:

Code: Select all

put arrayEncode(tMyArray) into url ("binfile:/home/User/path/to/file.lson")
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Why Array over list

Post by AxWald » Wed Oct 26, 2016 3:03 pm

Hi,

seems you guys are much faster than me - while I wrote my latest post so many new things arrived ...

OK, let's see what's up:
the array_bin of this stack[1]
is wrong. OK.

Richard means:
FourthWorld wrote:Custom properties ans property sets can be used to express arrays nicely:

Code: Select all

add 1 to the uArray[1] of this stack
I try this using my array of integers:

Code: Select all

put the array_bin of this stack into myArray
put myArray[1]     --  testing what we have
-- works; -> 3

add 1 to the array_bin[1] of this stack
-- doesn't work; -> Script compile error:
--                  Error description: Chunk: can't create a variable with that name (explicitVariables?)

-- You shall no resist my coding powerz:
put the array_bin of this stack into myArray
add 1 to myArray[1]
set the array_bin of this stack to myArray
put the array_bin of this stack into myArray
put myArray[1]
--  works; it's now -> 4
Hmmm. Works, but elegant is other ...
dunbarx wrote:I see what you are trying to get, but this is not valid syntax. You cannot get the value of a custom property and an array element at the same time.
I see. And I have just learned to love the power of custom properties ...

Anyways. What's next? The manipulation of the elements of the array, i.e. adding random(3) to each of it. That adding a constant to all of it is very fast ("add 42 to myArray"), that's great.
But usually I have to do something else with array like data, conditional things at most ("if myArray[5] > myArray[7] then put 1 into myArray[23] else put 0 into myArray[23]").

So "add random(3) to myArray[x]" could be any function that manipulates the data in a key.
The "list" example in my prev post gives these results:

Code: Select all

LST 1000000 / 985 / 1884
LST 1000000 / 987 / 1884
LST 1000000 / 988 / 1885
LST 1000000 / 990 / 1900
LST 1000000 / 992 / 1903
The second number is the first item of the list, and, as expected, it increases for random(3) at each run.
(the first number is the items affected, the last number is the millisecs used)

What do I have to do now to change the data in the array? Craigs example always gives me the initial data back when I add "put myArray[1] && myArray[2]" as last line ...

Some time later: Found what was wrong in my prev post.
It's because "i" is the number of the key in the array, the debugger tells me. So the correct way is (the line in the loop):

Code: Select all

put MyArray[i] + random(3) into MyArray[i]
\o/ \o/ \o/
And the array solution is actually ~334 millisecs faster, for 1000000 keys/ items ;-)

After some more thinking, this looks most impressive:
Looping using the fast "for each" construction, but still having the comfort of the "with i = 1 to X" way - i as counter, and being able to modify the data on the fly! This is something to consider in the future, for me ...

Have fun!
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Why Array over list

Post by FourthWorld » Wed Oct 26, 2016 4:16 pm

AxWald wrote:Richard means:
FourthWorld wrote:Custom properties ans property sets can be used to express arrays nicely:

Code: Select all

add 1 to the uArray[1] of this stack
I try this using my array of integers:

Code: Select all

put the array_bin of this stack into myArray
put myArray[1]     --  testing what we have
-- works; -> 3

add 1 to the array_bin[1] of this stack
-- doesn't work; -> Script compile error:
--                  Error description: Chunk: can't create a variable with that name (explicitVariables?)

-- You shall no resist my coding powerz:
put the array_bin of this stack into myArray
add 1 to myArray[1]
set the array_bin of this stack to myArray
put the array_bin of this stack into myArray
put myArray[1]
--  works; it's now -> 4
Hmmm. Works, but elegant is other ...
Indeed, working on variables directly is much simpler. And you're right: my example was in error; I've corrected it to avoid confusing others who may come across this thread in the future.

Custom properties can be useful when you truly need to bind data to specific objects. But even then, it's often simpler to pull the array out, work on it, and tuck it back into the object when you're done.

If you need persistence but don't need it bound to a specific object, arrayEncode and arrayDecode will let you serialize the array for file and network I/O far more efficiently than XML or even JSON, and sometimes even more efficiently than when using a stack file for storage (depending on the size and depth of the array/property set).
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

dunbarx
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10305
Joined: Wed May 06, 2009 2:28 pm

Re: Why Array over list

Post by dunbarx » Wed Oct 26, 2016 5:25 pm

Richard makes a point in that custom properties are "normally" bound to an object. Of course, in actual fact, they surely are. I think he meant that such properties "naturally" belong to an object in that their functionality makes the most sense within the confines of that object

But I use the stack itself as a reservoir for custom properties, sort of like a library, because the stack is above all cards and controls. I keep them there, and never think of them as bound to anything at all, any such relationship being irrelevant. So I could get "the doSomething of this stack" and have it work anywhere within that stack.

One could create a master property stack, put it in use at the beginning of each session, and store custom properties for all open stacks. Of course, now you have to name that stack explicitly:
"the doSomething of stack customPropStack".

Maybe this is beyond the law of diminishing returns.

Craig

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Why Array over list

Post by FourthWorld » Wed Oct 26, 2016 5:40 pm

I used to use stack files for preferences and other array-based storage, and still do when I need to also store native LC objects. But over time I found array globals simpler, if for no other reason than that I don't need to think about stack naming, purging, or other such things.

But both work well enough. And stack files can store ANYTHING, images, buttons, fields, widgets, so they can be uniquely good containers for a wide range of apps.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Why Array over list

Post by AxWald » Wed Oct 26, 2016 6:11 pm

Hi,

will come back later to your posts - I agree with a lot of it!

Just 3 simple questions meanwhile that came up when trying to use an array in my current project (Oh, the deadline!). I have a simple table-like data structure, tab-return delimited. Basically: ("->" for tab)

Code: Select all

Bread -> 2 -> 1.24
Vine -> 6 -> 5.00
Cheese -> 1 -> 2.10
Articles with amounts and prices ...

1.) I split the variable containing the data by row. Possible that it changes my sort?
2.) Now I can get "item 2 of MyVar[2]" = 6. But I cannot delete MyVar[2], right?
3.) How would I need to split to be able to access the "6" with MyVar[2][2]?
(split by return and tab doesn't work)

Thx for answers, would help a lot!

Have fun!
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

dunbarx
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10305
Joined: Wed May 06, 2009 2:28 pm

Re: Why Array over list

Post by dunbarx » Wed Oct 26, 2016 6:34 pm

1- Sorting is not supported. Do not rely on it.

2- No. You can get myVar[2], but the way you have it, the entire string is the associated element. You can put that back into the clear and get item 2, or you can set up the array in the beginning with each of those terms as separate "dimensions". And you need to say:

Code: Select all

delete variable myArray[2]
3- AxWald, you are a man after my own heart. Make a small field with your crudités platter of data. Make a button. Put this into its script:

Code: Select all

on mouseUp
   put fld 2 into temp
   set the itemDel to tab
   repeat with y = 1 to the number of lines of temp
          do "put item 3 of line y of temp into myArray" &  "[" & "item 1 of line y of temp" & "][" & "item 2 of line y of temp" & "]"
       end repeat
end mouseUp
Step through the handler. Have fun.

Craig

EDIT; It seems this can be shortened:

Code: Select all

        do "put item 3 of line y of temp into myArray[item 1 of line y of temp][item 2 of line y of temp]"

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Why Array over list

Post by AxWald » Wed Oct 26, 2016 9:18 pm

Craig.

This nearly broke my brain. As far as I understand it's like this:

Code: Select all

do "put item 3 of line y of temp into myArray[item 1 of line y of temp][item 2 of line y of temp]"
translates to:

Code: Select all

put "1.24" into myArray["Bread"]["2"]
for line 1 of my crudités platter.

This creates an array with an entry named "Bread" and two values, "2" and "1.24", Var watcher:

Code: Select all

v  Bread
       2     1.24
*Scratchhead*
Nope, it created an entry "Bread" with a subentry "2" and a value "1.24" for this! Right?

Does this mean I always have a key/ value pair? With just 1 value? OMG, I really need to search for some manual "arrays made easy, special edition for databankster". Cannot help to think I have a massive misconception of it ;-)

But not today anymore! Have fun, & sleep well!
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Why Array over list

Post by FourthWorld » Wed Oct 26, 2016 10:11 pm

AxWald wrote:Does this mean I always have a key/ value pair? With just 1 value? OMG, I really need to search for some manual "arrays made easy, special edition for databankster". Cannot help to think I have a massive misconception of it ;-)
Here's one stab at it:

LiveCode Script's arrays are associative arrays, similar to what you'd find in other languages that support associative arrays, like Perl, Python and others. Some languages call them "dictionaries", but the general structure and use is more or less the same for all associative arrays. They may differ in specifics, esp. limits and types, so what follows is for LC only.

An associative array is a collection of key-value pairs.

The key can be any string up to 255 chars, NULLs and CRs best avoided (the dictionary suggests sticking to alphanumeric characters, which is generally good guidance).

The value can be any string, or any binary data, or even another array. There's a logical limit of 4 GB to an array element, but in practical terms you're likely to run out of memory for other reasons before you reach that one.

Other than the syntax you already know, that's most of everything you need to know to use arrays. :)

But sometimes it can help to use them well to be able to picture what's going on under the hood:

When a value is put into an array element, the key is run through a simple hash function to determine a memory bucket to store it in, which each bucket being a collection of pointers to the values. The hash is designed to distribute element pointers across buckets somewhat evenly, favoring access time at the cost of any internal representation of order.

For example, given a set of elements named "bob", "ellen", "elmer", and "ethan", if our hash merely used the first letter to assign an element to one of four buckets, we'd be top-heavy in "e" with only one element in "b", and two buckets would be empty. The hash used in LC is smarter, attempting to assign buckets somewhat evenly so we might wind up with "bob" in bucket 4, "ellen" in bucket 1, "ethan" in bucket 3, and "elmer" in bucket 4. The hash isn't concerned with order, it's focused on even distribution.

This internal hashing is not something we normally need to think about, but helps explain why there is no sense of order to the elements of an associative array, in contrast the indexed arrays found in some languages where keys are always sequential integers.

When integers are used in an associative array, they're merely strings; e.g. "1" might just as well be "one" or "uno" or even "bob". Numeric keys may useful for us conceptually, like being able to get and set values sequentially in our scripts. But the internal mechanism that keeps track of keys doesn't care whether the string contains only numerals or anything else. The key of an associative array element is always just a string to the engine, and all keys are stored internally by hash value, not any numeric value that may be in the string.

This hashing also explains why the time needed to find an element doesn't change much even as the array grows, because the location is determined by the hash of the key, pointing directly to a bucket through which the value can be found. This contrasts with things like delimited lists, which require the engine to evaluate each character sequentially, counting delimiters as it goes.

In the engine, an array's collection of memory pointers can occupy many locations, so there is no single contiguous block of memory representing it. This means that we have no way to save an array in its native form, so we need to serialize it, to turn the sequence of pointers to disparate locations into a single contiguous stream of bytes.

And that takes us to arrayEncode, which converts the pointer collection as the engine handles the array into a bytestream that can be saved to disk or sent over a network. The arrayDecode reverses that process, converting the bytestream back into a collection of pointers in memory ready for the engine to use efficiently.

Hope some of this background is useful. The Wikipedia page on associative arrays provides more detail:
https://en.wikipedia.org/wiki/Associative_array
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Contact:

Re: Why Array over list

Post by mwieder » Thu Oct 27, 2016 12:16 am

Note that associative arrays are also known as hashes in some languages (notably ruby).

And some examples might help as well.

Code: Select all

put "carrot" into bugsbunny["password"]
put "magic helmet" into elmerfudd["password"]
now you can get

Code: Select all

put elmerfudd["password"]
and get "magic helmet"

OK - that's first order arrays and so far pretty simple.

Now let's expand on that:

Add more data to the two arrays:

Code: Select all

put "hole in the ground" into bugsbunny["home"]
Put the two arrays into a parent array:

Code: Select all

put bugsbunny into character["protagonist"]
put elmerfudd into character["antagonist"]
Now we have an array made of two arrays.
And we can get the data:

Code: Select all

put character["protagonist"]
gives us an array. so does

Code: Select all

put character["antagonist"]
although it's a different array with different data.

And now digging deeper,

Code: Select all

put character["antagonist"]["password"]
gives us "magic helmet"

Code: Select all

put character["protagonist"]["home"]
gives us "hole in the ground"

You can skip the intermediate steps and say

Code: Select all

put "anvil" into character["coyote"]["tool"]

dunbarx
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10305
Joined: Wed May 06, 2009 2:28 pm

Re: Why Array over list

Post by dunbarx » Thu Oct 27, 2016 12:19 am

Richard was the one, along with Jacque, who disabused me of the notion that I could:

Code: Select all

on mouseUp
   put "D" into myArray{"A"]["B"]["C"]
   put myArray{"B"] into xx
end mouseUp
Nothing appears in xx. This is because I have broken the rules about LC arrays. There is a hierarchy of dependent key/element pairs, but not the ability to extract "intermediate" elements from myArray per se. There are no intermediate elements, only those hierarchal relationships.

But this is relatively advanced stuff, and must not deter you from using arrays. You just need practice. Stick with simple ones. Go back to the word counter. Create "lists" in array form and both push and pull data in and out. Look once more at what Hermann did, which gives you a hint of what arrays look like from a slightly higher altitude. In a sense, what he gave you was a way to look at array variables (they are just variables, you know) in a way more similar to an "ordinary" variable.

Then when you need to, and the need will arrive one day, start in with additional levels in your array.

Craig

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Contact:

Re: Why Array over list

Post by mwieder » Thu Oct 27, 2016 2:23 am

but

Code: Select all

on mouseUp
   put "D" into myArray["A"]["B"]["C"]
   put myArray["A"] into aa
   put myArray["B"] into bb
   put myArray["C"] into cc
   put aa is an array & cr after msg
   put bb is an array & cr after msg
   put cc is an array & cr after msg
end mouseUp
(edited because of copy-pasta error)
Last edited by mwieder on Thu Oct 27, 2016 2:49 am, edited 1 time in total.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Why Array over list

Post by FourthWorld » Thu Oct 27, 2016 2:33 am

dunbarx wrote:Richard was the one, along with Jacque, who disabused me of the notion that I could:

Code: Select all

on mouseUp
   put "D" into myArray{"A"]["B"]["C"]
   put myArray{"B"] into xx
end mouseUp
Nothing appears in xx. This is because I have broken the rules about LC arrays. There is a hierarchy of dependent key/element pairs, but not the ability to extract "intermediate" elements from myArray per se. There are no intermediate elements, only those hierarchal relationships.
Ah, but there you do extract the value of myArray["B"] into xx. It's just that the value is an array rather than a string, so xx becomes an array. If you did this it may become clearer:

Code: Select all

on mouseUp
   put "D" into myArray{"A"]["B"]["C"]
   put myArray["A"]["B"] into xx
   put the keys of xx
end mouseUp
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Contact:

Re: Why Array over list

Post by mwieder » Thu Oct 27, 2016 2:51 am

Code: Select all

put myArray["B"] into xx
still won't give you an array. you need

Code: Select all

put myArray["A"]["B"] into xx

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 10043
Joined: Sat Apr 08, 2006 7:05 am
Contact:

Re: Why Array over list

Post by FourthWorld » Thu Oct 27, 2016 4:14 am

D'oh! Of course. Thanks, Mark - I've updated my post so it doesn't confuse anyone else dropping in here.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Post Reply