Page 3 of 7
Re: filter enhancements
Posted: Tue May 28, 2013 9:25 pm
by Janschenkel
I just filed
bug 10928 regarding the inconsistency between 'sort items' and the chunk hierarchy.
Once we get a clarification from the team about this, we can deliver an implementation of 'filter items' that is consistent with 'sort items'.
Now I'm going back to study the 'matchText' implementation to see how best to implement pattern matching...
Jan Schenkel.
Re: filter enhancements
Posted: Wed May 29, 2013 10:20 am
by LCMark
I just filed bug 10928 regarding the inconsistency between 'sort items' and the chunk hierarchy.
The current action of 'sort', whilst simplistic does seem reasonable to a certain degree. There's no real difference between line-delimited lists and item-delimited lists in the engine - they just have a different delimiter. If you want more hierarchical action then really the syntax should reflect that:
Code: Select all
sort items of lines of tVar
filter items of lines of tVar matching "foo*"
Re: filter enhancements
Posted: Wed May 29, 2013 9:04 pm
by Janschenkel
Okido - though I prefer 'in' to 'of' in that syntax for improved human parsing.
So the target syntax for the 'filter' command becomes
Code: Select all
filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } <filter_pattern> | [ not ] matching [ pattern ] <regex_pattern> } [ into <container> ]
Phew. Now to break that into separate incremental modifications to submit to the 'develop' branch
Jan Schenkel.
Re: filter enhancements
Posted: Wed May 29, 2013 9:15 pm
by mwieder
I like the BNF syntax. Especially the [into <container>], which I am just now noticing.
And I want to encourage you not to overlook being able to tie the sort command into that as well (from bug 9980).
Edit: why not "{ lines | items | items in lines } [in | of]" so that either is acceptable?
Re: filter enhancements
Posted: Wed May 29, 2013 9:53 pm
by Janschenkel
@mwieder
I assume you refer to
bug 9959 -
Extend Filter and Sort commands with 'into' clause ?
I've written recursive descent parsers and used
ANTLR in the past, but I haven't puzzled out the LiveCode parser yet.
But once I've figured it out for the 'filter' command, I might take a stab at the 'sort' command next
Jan Schenkel.
Re: filter enhancements
Posted: Wed May 29, 2013 10:48 pm
by mwieder
9959 - that's the one.
I haven't puzzled out the LiveCode parser yet.
yeah... it's a twisty maze of little passages. I've been trying to work my way through it as well.
Each command has a parse method and an eval method.
Sort::parse is in cmds.cpp and you'll have to add ST_INTO to the switch statement.
Filter::parse is in cmdsf.cpp.
After that it gets a bit gnarly.
Re: filter enhancements
Posted: Thu May 30, 2013 1:12 am
by phaworth
mwieder wrote:
I *would*, however, like to take to extrapolation to array syntax into a separate thread and separate pull requests, because I think there's a lot of work to do on arrays and a lot of pitfalls along the way. Dealing with arrays of a single dimension looks easy enough by expanding the current syntax, but gets very sticky when going to multiple dimensions. And I'd like to avoid going down a path now that would have to be reverted.
Hi Mark,
I agree it's a separate exercise to deal with arrays. For what it's worth, I've been working on a handler that would allow me to filter generic arrays and found I needed the following input:
- the value to be searched for (of course)
- whether I'm looking for the value as a key name or a key value
- the "depth", for want of a better word, of the key I want to search.
That was sufficient for my needs but it's quite possible there needs to be more flexibility
So if I had an array:
[1]
[abc]
[1] = "foo"
[2] = "bar"
[def]
[1] = "mark"
[2] = "pete
[2]
[abc]
[1] = "pete"
I can search for "pete" as a level 3 key value and get
[1]
[def]
[2]="pete"
[2]
[abc]
[1]="pete
I can search for "abc" as a level 2 key name which would give me
[1]
[abc]
[1]="foo"
[2]="bar"
[2]
[abc]
[1]="pete
Pete
Re: filter enhancements
Posted: Thu May 30, 2013 9:16 am
by rkriesel
Janschenkel wrote:
Code: Select all
filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } <filter_pattern> | [ not ] matching [ pattern ] <regex_pattern> } [ into <container> ]
A simpler form obviates introducing "matching," and negating it, instead introducing the simpler term "regex:"
Code: Select all
filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } [ regex ] <filter_pattern> [ into <container> ]
Do you agree, Jan?
Re: filter enhancements
Posted: Thu May 30, 2013 11:00 am
by LCMark
@rkriesel: One of the advantages of introducing 'matching' is that it has synergy with a future 'matches' operator. If we were going to distinguish between old and new with an adjective then I'd suggest 'pattern' rather than 'regex' - as that is what a regular expression actually is (a pattern for matching strings).
Re: filter enhancements
Posted: Thu May 30, 2013 1:25 pm
by Janschenkel
@rkriesel
I feel it more aptly differentiates the filtering behaviours if we introduce the 'matching' keyword for this purpose.
The adjective 'regex' sounds out-of-place as other regex-related functions use 'match' in their name.
Using the adjective 'pattern' to differentiate would be confusing as we're calling the wildcard string a 'filterPattern' in the documentation.
So I'd rather stick with the last syntax proposal...
Jan Schenkel.
Re: filter enhancements
Posted: Thu May 30, 2013 2:30 pm
by LCMark
Using the adjective 'pattern' to differentiate would be confusing as we're calling the wildcard string a 'filterPattern' in the documentation.
Well, we could just rename 'filterPattern' in the dictionary
However, I do think 'filter items of ... [not] matching ...' reads better (and is more descriptive) than 'filter items of ... (with | without)' so I'm, personally, perfectly happy with that.
Re: filter enhancements
Posted: Thu May 30, 2013 6:50 pm
by mwieder
I do think it's going to be very tricky explaining that
filter tVar matching "*text"
is different from
filter tVar with "*text"
so maybe enforcing the modifier "pattern" or "regex", as in
filter tVar matching pattern ".*text"
is necessary.
Re: filter enhancements
Posted: Thu May 30, 2013 7:20 pm
by phaworth
mwieder wrote:I do think it's going to be very tricky explaining that
filter tVar matching "*text"
is different from
filter tVar with "*text"
so maybe enforcing the modifier "pattern" or "regex", as in
filter tVar matching pattern ".*text"
is necessary.
My preference would be:
filter tVar with pattern "*text"
Having to use "matching" instead" of "with" seems like an unnecessary complication to me. However, I think some extra bias should be given to the opinions of the folks who will be doing the coding so I'll leave it at that!
Pete
Re: filter enhancements
Posted: Thu May 30, 2013 7:56 pm
by mwieder
Here's the problem with that:
filter tVar with pattern "*text"
will return empty.
If you mean "pattern" to be a regex string then it won't match anything.
If you mean "with pattern" to do what "with" currently does, then it will break existing scripts.
If you want to match the string "text" inside tVar then your regex would need to be ".*text"
I put that example in my post to illustrate why it's going to be tricky explaining why "matching" and "with" are different.
Re: filter enhancements
Posted: Thu May 30, 2013 8:51 pm
by phaworth
mwieder wrote:
I put that example in my post to illustrate why it's going to be tricky explaining why "matching" and "with" are different.
OK, let's try again, ignoring what the regexp is.
filter tVar with pattern <regexp>
filter tVar with <filter>
Doesn't break any existing scripts and eliminates having to remember when to use "with" and when to use "matching". No need to explain the difference between "matching" and "with" because "matching" doesn't exist any more. Only explanation is that "pattern" indicates the use of a regexp and that doesn't seem tricky to understand.