filter enhancements

[Martin Koob]

[quote="monte"]Is there any reason with and without can't work?

Code: Select all

filter thelist with item 3 of each is "foo"... 

From the standpoint of someone who is the average LiveCode developer (i.e. someone's whose primary(or only) programming language is LiveCode and who is not someone with the programming skills to contribute to the engine) I would say that 'where' gives a better idea of what will be done than than 'with'. Also I think 'where' is better than 'by' in this regard too.

A LiveCoder who reads the first part of the sentence and takes it literally "Filter the list with item 3..." may understand it as lists with item 3 will be filtered.
"Filter thelist with item 3 of each is "foo" now is harder to take literally and understand.

Taken literally "Filter thelist where item 3 of each is "foo" makes sense.
Also "Filter thelist where item 3 of each is not "foo" also makes sense when taken literally than "filter thelist" without item 3 of each is "foo"

I am not sure how many average users read the engine contributors forum and would be involved in this discussion but I think it would be good to get their(our) input on some of these syntax changes.

Perhaps before Jan gets down to working on the project it would be a good idea to have a post in one of the general forums or on the use list summarizing the possible syntaxes and get the feedback from the broader community as to what would make the most sense to them.

[edgore]

Martin, I just started reading this sub forum on a whim last week, previously having figured that the discussion would be all technical stuff about github and c coding. I was very glad to stumble across this discussion, which covers something that would be *very* useful to me on a daily basis. I will be looking over the posts in the sub forum (and the IDE one once it starts to take off) going forward, since I always love to throw in my two cents on things I don't really understand.

Another filter option that would be very useful for me, that might fit into this topic, is the ability to filter out duplicates from a list with the same ability to specify the granularity and scope of the duplicate removal (by line, by specific item, multiple specific items (item 1 to 4,etc.) and get back the list with the duplicate items removed. This is another thing that I have to do all the time, and while I have my own functions to do it I am sure it would be faster and make for more readable code if it was implemented as part of an overall filter command update.

[mwieder]

Ed-

I always love to throw in my two cents on things I don't really understand.

LOL. Au contraire... I always find you have something useful to contribute.

Re: removing duplicates, I think this is a nice idea to add to the pile - as an analogy, SQL supports this with the "distinct" keyword.
Maybe "unique" is a more natural way of describing this, though:

Code: Select all

filter unique lines of x into tNoDuplicatesHere

[LCMark]

Entering the discussion slightly late (so apologies if I've misunderstoond)... The proposed extension to filter here (where clause) is actually a lot more general, indeed it doesn't need to be a command at all and is, in fact, an extension of chunk expressions:

Code: Select all

put the items of tItemList where each begins with "baz" into tVar
repeat for each item tItem in tItemList where each begins with "baz"
  ...
end repeat

Here, the general syntactic form is:

Code: Select all

  the <plural chunk type> of <expr> where <expr of each>

And filter just becomes a slightly more convenient way of writing certain variants:

Code: Select all

filter tList with "x*"
   is just syntactic sugar for
get the lines of tList where each matches wildcard pattern "x*" -- this assumes we have a 'matches' operator ;)

One thing I've never been entirely happy with was the use of a global 'each' variable for these cases (particularly since it causes re-entrancy problems when you start using functions that also use each in the each expression). One option would be to make 'each' local to the clause, or maybe consider slightly more englishy form:

Code: Select all

get the lines of tList where the line matches wildcard pattern "x*"
put the items of tList where the round of the item is not 0 into tMyVar

[Martin Koob]

@edgore

I read the Engine Contributors forum regularly even though it is often over my head. I do it for two reasons. One is I want to see where LiveCode is headed and what features are coming down the pipe. Two is even if many things are over my head perhaps if I read enough of it I might start to understand it.

I guess three is that It is a good way to learn how to use the new features that are coming soon and about exiting features and their current use and limitations.

This is getting off topic but to bring it back to the discussion at hand.

@everyone

I did not really understand the filter command and have not used it. I am sure I have done workarounds to accomplish the same result. Reading this post has helped me understand how it is used currently and how it may work in the future once Jan's project gets implemented.

I want to add a few more comments as to why I think 'where' is better than 'with' Reading how filter is currently implemented the filter with form makes sense(is english-like) as there is a simple 'parameter'.

Code: Select all

filter myVariable with "A?2"

It is kind of like saying 'Filter mixture with fine sieve', which makes sense.

But 'with' does not make sense(is not english like) when you have a clause or clauses as 'parameters'

Code: Select all

filter thelist with item 3 of each is "foo"...

That would be like saying "Filter mixture with diameter of particle is 5 mm."
It would be more english-like to say "Filter mixture where diameter of particle is 5 mm".
Still not perfect english but I think reading the line with 'where' the average LiveCode user could more easily intuit what was happening.

[DarScott]

We can think of an each expression as a sort of lambda expression. That is, a function is implied with one parameter named each.

So...

Code: Select all

sort x by item 3 of each

implies...

Code: Select all

 function metric each
   return item 3 of each
end metric

which is used in the sort. The same would apply with the filter that Jan is encouraging.

When in an expression the "lamba" of a sort has precedence rules that might be confusing especially since since one function might be defined using another. Which each is what? In lambda expressions names are used. In english anaphora, referring back often uses "the" which is heavily loaded in LiveCode, but might work here, we can make a distinction between "the line' and 'the item', but that will have a problem when referring back to two items. We need to tinker with that.

[edgore]

runrevmark,

Would your implementation allow for something like:

put the lines of tItemList where item 3 is ""baz" into tVar

(sorry, but for some reason the bbcode buttons aren't working in Safari today)

That is is the sort of thing that I am interested in doing.

Reading your example:

put the items of tItemList where each begins with "baz" into tVar

makes me think that if tItemList has 6 items per line and 60 lines that I would get back a list, one item per line, of all the items in any of the lines of tItemList that started with "baz".

I like the idea of making this an extension of the existing chunking expressions, I just want to understand how it would work.

[jacque]

Another average user who reads this forum to see where we're headed.

I am more inclined to keep the language as consistent as possible. If I consider the filtering expression as a unit in and of itself, "with" always makes sense to me:

filter tList with "x*"
filter tList with (item 2 is x)
filter mixture with (diameter of particle is 5 mm)

But "where" is okay too and maybe should be a synonym.

[DarScott]

@jacque

The problem with keeping the same word is that it is ambiguous. Is this a regex match or a boolean test? (I'm assuming filter is expanded to regex, simplifying the language.)

Maybe "with" can mean match the word or regex, and "where" can mean when the expression is true. We can keep "without". Now, you can do anything with "where" that you can do with "with", but some folks might be comfortable with "with" and have used it.

[DarScott]

One of the things discussed is adding the chunk type that is being filtered. That is, the data can be interpreted as lines and those filtered, or it can be interpreted as items and those filtered. That can apply to the filter command or it can apply to (distracting) chunk expression variation.

When we look at chunk expressions, we might have something like this:

Code: Select all

get the lines of x where each is greater than 1

The 'the line of' signals that this is a selection on lines. It works like filter in a way. Each line is put into the variable named 'each'. The boolean expression to the right of 'where' is evaluated for that line. If it is true, the line is kept, otherwise it is discarded. The result is a line list.

Maybe it is like this (ignoring my coding typos):

Code: Select all

 get empty
repeat for each line eachLineOfX of x
  if eachLineOfX is greater than 1 then put eachLineOfX & LF after it
end repeat
delete char -1 of it

But if it had been 'the items of", then items are pulled from x and tested in the same way. The result is an item list.

An extension of that can convert one chunk list to another chunk list. We can add 'as items" or 'as elements', but maybe that is distracting at this point.

(The word 'where' is a valuable asset, so we might want to consider if there is a better place for it, but it seems good here, so this will compete well for the word.)

Though 'each' has some history in 'sort', we don't have to use 'each'. Maybe 'each line' would be better or 'the line' would be better. Or any of those can be allowed. We need to consider the consequences. One of the potential problems is a selection chunk expression inside the each expression of another selection chunk expression. I think that can be readily examined.

[phaworth]

No recent posts on this thread so wondering where this change stands. Personally, I think this is the most generally useful engine change I've seen discussed so I hope it gets implemented in some form.
Pete