filter enhancements

Moderators: Klaus, FourthWorld, heatherlaine, kevinmiller, LCMark

Janschenkel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 977
Joined: Sat Apr 08, 2006 7:47 am
Location: Aalst, Belgium
Contact:

Re: filter enhancements

Post by Janschenkel » Tue May 28, 2013 9:25 pm

I just filed bug 10928 regarding the inconsistency between 'sort items' and the chunk hierarchy.
Once we get a clarification from the team about this, we can deliver an implementation of 'filter items' that is consistent with 'sort items'.
Now I'm going back to study the 'matchText' implementation to see how best to implement pattern matching...

Jan Schenkel.
Quartam Reports & PDF Library for LiveCode
www.quartam.com

LCMark
Livecode Staff Member
Livecode Staff Member
Posts: 1033
Joined: Thu Apr 11, 2013 11:27 am

Re: filter enhancements

Post by LCMark » Wed May 29, 2013 10:20 am

I just filed bug 10928 regarding the inconsistency between 'sort items' and the chunk hierarchy.
The current action of 'sort', whilst simplistic does seem reasonable to a certain degree. There's no real difference between line-delimited lists and item-delimited lists in the engine - they just have a different delimiter. If you want more hierarchical action then really the syntax should reflect that:

Code: Select all

sort items of lines of tVar
filter items of lines of tVar matching "foo*"

Janschenkel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 977
Joined: Sat Apr 08, 2006 7:47 am
Location: Aalst, Belgium
Contact:

Re: filter enhancements

Post by Janschenkel » Wed May 29, 2013 9:04 pm

Okido - though I prefer 'in' to 'of' in that syntax for improved human parsing.
So the target syntax for the 'filter' command becomes

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } <filter_pattern> | [ not ] matching [ pattern ] <regex_pattern> } [ into <container> ] 
Phew. Now to break that into separate incremental modifications to submit to the 'develop' branch :-)

Jan Schenkel.
Quartam Reports & PDF Library for LiveCode
www.quartam.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2826
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Wed May 29, 2013 9:15 pm

I like the BNF syntax. Especially the [into <container>], which I am just now noticing.
And I want to encourage you not to overlook being able to tie the sort command into that as well (from bug 9980).

Edit: why not "{ lines | items | items in lines } [in | of]" so that either is acceptable?

Janschenkel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 977
Joined: Sat Apr 08, 2006 7:47 am
Location: Aalst, Belgium
Contact:

Re: filter enhancements

Post by Janschenkel » Wed May 29, 2013 9:53 pm

@mwieder
I assume you refer to bug 9959 - Extend Filter and Sort commands with 'into' clause ?
I've written recursive descent parsers and used ANTLR in the past, but I haven't puzzled out the LiveCode parser yet.
But once I've figured it out for the 'filter' command, I might take a stab at the 'sort' command next :-)

Jan Schenkel.
Quartam Reports & PDF Library for LiveCode
www.quartam.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2826
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Wed May 29, 2013 10:48 pm

9959 - that's the one.
I haven't puzzled out the LiveCode parser yet.
yeah... it's a twisty maze of little passages. I've been trying to work my way through it as well.
Each command has a parse method and an eval method.
Sort::parse is in cmds.cpp and you'll have to add ST_INTO to the switch statement.
Filter::parse is in cmdsf.cpp.
After that it gets a bit gnarly.

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: filter enhancements

Post by phaworth » Thu May 30, 2013 1:12 am

mwieder wrote: I *would*, however, like to take to extrapolation to array syntax into a separate thread and separate pull requests, because I think there's a lot of work to do on arrays and a lot of pitfalls along the way. Dealing with arrays of a single dimension looks easy enough by expanding the current syntax, but gets very sticky when going to multiple dimensions. And I'd like to avoid going down a path now that would have to be reverted.

Hi Mark,
I agree it's a separate exercise to deal with arrays. For what it's worth, I've been working on a handler that would allow me to filter generic arrays and found I needed the following input:

- the value to be searched for (of course)
- whether I'm looking for the value as a key name or a key value
- the "depth", for want of a better word, of the key I want to search.

That was sufficient for my needs but it's quite possible there needs to be more flexibility

So if I had an array:

[1]
[abc]
[1] = "foo"
[2] = "bar"
[def]
[1] = "mark"
[2] = "pete
[2]
[abc]
[1] = "pete"

I can search for "pete" as a level 3 key value and get
[1]
[def]
[2]="pete"
[2]
[abc]
[1]="pete

I can search for "abc" as a level 2 key name which would give me
[1]
[abc]
[1]="foo"
[2]="bar"
[2]
[abc]
[1]="pete

Pete

rkriesel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 67
Joined: Thu Apr 13, 2006 6:25 pm

Re: filter enhancements

Post by rkriesel » Thu May 30, 2013 9:16 am

Janschenkel wrote:

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } <filter_pattern> | [ not ] matching [ pattern ] <regex_pattern> } [ into <container> ] 
A simpler form obviates introducing "matching," and negating it, instead introducing the simpler term "regex:"

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } [ regex ] <filter_pattern> [ into <container> ] 
Do you agree, Jan?

LCMark
Livecode Staff Member
Livecode Staff Member
Posts: 1033
Joined: Thu Apr 11, 2013 11:27 am

Re: filter enhancements

Post by LCMark » Thu May 30, 2013 11:00 am

@rkriesel: One of the advantages of introducing 'matching' is that it has synergy with a future 'matches' operator. If we were going to distinguish between old and new with an adjective then I'd suggest 'pattern' rather than 'regex' - as that is what a regular expression actually is (a pattern for matching strings).

Janschenkel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 977
Joined: Sat Apr 08, 2006 7:47 am
Location: Aalst, Belgium
Contact:

Re: filter enhancements

Post by Janschenkel » Thu May 30, 2013 1:25 pm

@rkriesel
I feel it more aptly differentiates the filtering behaviours if we introduce the 'matching' keyword for this purpose.
The adjective 'regex' sounds out-of-place as other regex-related functions use 'match' in their name.
Using the adjective 'pattern' to differentiate would be confusing as we're calling the wildcard string a 'filterPattern' in the documentation.

So I'd rather stick with the last syntax proposal...

Jan Schenkel.
Quartam Reports & PDF Library for LiveCode
www.quartam.com

LCMark
Livecode Staff Member
Livecode Staff Member
Posts: 1033
Joined: Thu Apr 11, 2013 11:27 am

Re: filter enhancements

Post by LCMark » Thu May 30, 2013 2:30 pm

Using the adjective 'pattern' to differentiate would be confusing as we're calling the wildcard string a 'filterPattern' in the documentation.
Well, we could just rename 'filterPattern' in the dictionary ;)

However, I do think 'filter items of ... [not] matching ...' reads better (and is more descriptive) than 'filter items of ... (with | without)' so I'm, personally, perfectly happy with that.

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2826
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Thu May 30, 2013 6:50 pm

I do think it's going to be very tricky explaining that

filter tVar matching "*text"

is different from

filter tVar with "*text"

so maybe enforcing the modifier "pattern" or "regex", as in

filter tVar matching pattern ".*text"

is necessary.

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: filter enhancements

Post by phaworth » Thu May 30, 2013 7:20 pm

mwieder wrote:I do think it's going to be very tricky explaining that

filter tVar matching "*text"

is different from

filter tVar with "*text"

so maybe enforcing the modifier "pattern" or "regex", as in

filter tVar matching pattern ".*text"

is necessary.
My preference would be:

filter tVar with pattern "*text"

Having to use "matching" instead" of "with" seems like an unnecessary complication to me. However, I think some extra bias should be given to the opinions of the folks who will be doing the coding so I'll leave it at that!

Pete

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2826
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Thu May 30, 2013 7:56 pm

Here's the problem with that:

filter tVar with pattern "*text"

will return empty.
If you mean "pattern" to be a regex string then it won't match anything.
If you mean "with pattern" to do what "with" currently does, then it will break existing scripts.
If you want to match the string "text" inside tVar then your regex would need to be ".*text"

I put that example in my post to illustrate why it's going to be tricky explaining why "matching" and "with" are different.

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: filter enhancements

Post by phaworth » Thu May 30, 2013 8:51 pm

mwieder wrote: I put that example in my post to illustrate why it's going to be tricky explaining why "matching" and "with" are different.
OK, let's try again, ignoring what the regexp is.

filter tVar with pattern <regexp>
filter tVar with <filter>

Doesn't break any existing scripts and eliminates having to remember when to use "with" and when to use "matching". No need to explain the difference between "matching" and "with" because "matching" doesn't exist any more. Only explanation is that "pattern" indicates the use of a regexp and that doesn't seem tricky to understand.

Post Reply

Return to “Engine Contributors”