filter enhancements

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, LCMark

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Thu May 30, 2013 9:57 pm

Only explanation is that "pattern" indicates the use of a regexp and that doesn't seem tricky to understand.
Well, I guess I disagree that it "doesn't seem tricky".
You understand the context because you've been following this discussion.
Now remove the forum context and try to explain why 'pattern "*text"' isn't valid.

I think "regex" rather than "pattern" enforces the cognitive translation, but there are hurdles in making this natural-language-compatible. I suppose having a regex string in there is disconcerting enough, but I worry that anyone just coming into the filter command isn't going to make the distinction that "pattern" means a regex pattern. I'd like the regex syntax to be jarring enough that it's unlikely that anyone would make a mistake and use the wrong form.

The fact that

Code: Select all

filter tVar with "text*"
and

Code: Select all

filter tVar with pattern "text*"
return different results isn't at all obvious, IMO.
Of course, I'd say the same about

Code: Select all

filter tVar matching "text*"
I think I'm gradually coming around to Dick Kreisel's syntax. If someone's sophisticated enough to craft a valid regex string, they won't be put off by having to use the word "regex".

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: filter enhancements

Post by phaworth » Thu May 30, 2013 10:46 pm

mwieder wrote: I think I'm gradually coming around to Dick Kreisel's syntax. If someone's sophisticated enough to craft a valid regex string, they won't be put off by having to use the word "regex".
I agree with all that. I like Dick's idea too even though it gets away for the "English language" basis of LC.

Pete

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Fri May 31, 2013 12:40 am

Well, there's not much English-language-like in regex.

monte
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 1564
Joined: Fri Jan 13, 2012 1:47 am
Contact:

Re: filter enhancements

Post by monte » Fri May 31, 2013 7:21 am

I don't really see the big problem introducing `matching` for the regex form. I do wonder if skipping ahead to the filter map reduce phase resolves the problem for us....

Code: Select all

filter lines of tVar by matchText(each,".*text")
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

Janschenkel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 977
Joined: Sat Apr 08, 2006 7:47 am
Location: Aalst, Belgium
Contact:

Re: filter enhancements

Post by Janschenkel » Fri May 31, 2013 10:00 am

Alright let's see if we can find a middle ground here so I only have to change the parser once - at the risk of bikeshedding.
Since we all agree on the '[ { lines | items | items in lines } of ]' part as well as the '[ into <container> ]' part, I'll leave these out of the discussion below.

Idea

We could treat 'with' and 'matching' as synonyms, and of course likewise 'without' and 'not matching'.
And to clarify the 'pattern' to use, we introduce 'wildcard' as adjective for the old filter pattern mechanism, and 'regex' for regular expression pattern matching.

Then the script line:

Code: Select all

filter tText with "*text"
would be equivalent to both of the following verbose variants:

Code: Select all

filter tText with wildcard pattern "*text"
filter tText matching wildcard pattern "*text"
And the script line:

Code: Select all

filter tText without "*text"
would be equivalent to both of the following verbose variants:

Code: Select all

filter tText without wildcard pattern "*text"
filter tText not matching wildcard pattern "*text"
Now to use regular expression matching you could write either of the following:

Code: Select all

filter tText with regex pattern ".*text"
filter tText matching regex pattern ".*text"
And you can reverse the filtering by writing either of the following:

Code: Select all

filter tText without regex pattern ".*text"
filter tText not matching regex pattern ".*text"
Syntax

The BNF for this part of the 'filter' command would then become:

Code: Select all

filter ... <container_or_exp> { with | without | [ not ] matching } [ { wildcard | regex } pattern ] <pattern_exp> ... 
Resulting in the complete BNF for the 'filter' command as follows:

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { with | without | [ not ] matching } [ { wildcard | regex } pattern ] <pattern_exp> [ into <container> ] 
This gives us backward compatibility and allows each developer to match it up with their programming style (pun intended).
And Mark Waddingham can decide which forms survive in the 'new syntax' project and how it's converted ;-)

Boot note: related language items

Remember that while wildcard matching is not as powerful as regex matching, it has better performance, so we may want to carry this over to related language items already mentioned before.

We could introduce a 'matches' operator with similar options

Code: Select all

if tText matches wildcard pattern "*text" then...
if tText matches regex pattern ".*text" then ...
And extend the 'replace' command likewise with both wildcard and regex matching if it makes sense.

Jan Schenkel.
Quartam Reports & PDF Library for LiveCode
www.quartam.com

phaworth
Posts: 592
Joined: Thu Jun 11, 2009 9:51 pm

Re: filter enhancements

Post by phaworth » Fri May 31, 2013 4:42 pm

Janschenkel wrote:Alright let's see if we can find a middle ground here so I only have to change the parser once - at the risk of bikeshedding.
Since we all agree on the '[ { lines | items | items in lines } of ]' part as well as the '[ into <container> ]' part, I'll leave these out of the discussion below.
Thanks for being proactive about this Jan and trying to keep everyone happy. And thanks for introducing me to a new term, bikeshedding. Hadn't come across that before although I'm familiar with the principle.

I'm happy to accept whatever syntax you think works best and provides the functionality we all want. My input has just been on the way I would ideally prefer to write the new filter statement in a script but it's not a major issue if I have to write "filter tVar matching <regexp>" instead of "filter tVar with regexp <regexp>". In fact, I think it may make life more difficult if there are too many synonyms, especially for newcomers, so I'd suggest that, as the "project leader" on this one, you pick the syntax you like best and go with it.

Pete

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Fri May 31, 2013 5:20 pm

OK - "bikeshedding" was new to me too.

I think I'd prefer the BNF form of

Code: Select all

[ [regex] pattern ] <pattern_exp> ... 
I don't see the need to introduce the "wildcard" modifier into this - it's already implicitly there, and I'm not convinced it adds either readability or more information to the pattern. If the parser needs it for orthogonality with "regex" then I'd vote for making the modifier optional:

Code: Select all

[ [{ wildcard | regex }] pattern ] <pattern_exp> ... 
...and I just want to make sure that "filter tVar without empty" still works with the new syntax, because I do that all the time.
"filter tVar without wildcard pattern empty" just seems silly.

Other than that minor change, thanks for the summation - looks good to me.

rkriesel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 118
Joined: Thu Apr 13, 2006 6:25 pm

Re: filter enhancements

Post by rkriesel » Sat Jun 01, 2013 6:34 am

Janschenkel wrote: We could treat 'with' and 'matching' as synonyms, and of course likewise 'without' and 'not matching'.
And to clarify the 'pattern' to use, we introduce 'wildcard' as adjective for the old filter pattern mechanism, and 'regex' for regular expression pattern matching.
...
The BNF for this part of the 'filter' command would then become:

Code: Select all

filter ... <container_or_exp> { with | without | [ not ] matching } [ { wildcard | regex } pattern ] <pattern_exp> ... 
Thanks, Jan. I like the idea -- except for one word: "pattern." Because offers no information beyond what we'd get from [ wildcard | regex ], I'd be reluctant to force coders to code it.

Is there a policy about LiveCode syntax that applies to no-impact terms in commands? Or does "pattern" here have potential value I'm missing?

-- Dick

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Sat Jun 01, 2013 3:33 pm

@Dick - should the BNF pattern be more like this?

Code: Select all

[ [regex] [pattern] ] <pattern_exp> ... 
I assumed that "pattern" would be optional as well. If a developer is more comfortable writing "regex pattern" instead of "regex" to make the code more (for them) readable, then I don't think there should be a penalty for adding the ignored word.

rkriesel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 118
Joined: Thu Apr 13, 2006 6:25 pm

Re: filter enhancements

Post by rkriesel » Sun Jun 02, 2013 2:55 am

mwieder wrote:@Dick - should the BNF pattern be more like this?

Code: Select all

[ [regex] [pattern] ] <pattern_exp> ... 
I assumed that "pattern" would be optional as well. If a developer is more comfortable writing "regex pattern" instead of "regex" to make the code more (for them) readable, then I don't think there should be a penalty for adding the ignored word.
@ahmark - If it were up to me now, I'd say the word "pattern" without the word "regex" is too broad for using it here to imply "regex."

So the BNF fragment would be

Code: Select all

[ regex | regex pattern ] <pattern_exp>
But I'm not a regex aficionado. I don't recognize how strongly the word "pattern" implies "regex" in other circles. To me the word "pattern" alone in this context suggests the "filterPattern" we have now.

-- Dick

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Sun Jun 02, 2013 3:51 am

To me the word "pattern" alone in this context suggests the "filterPattern" we have now.
Yes, that's exactly what I meant. Sorry if that wasn't clear. So

filter tVar with "*text"
and
filter tVar with pattern "*text"
would be indentical with "pattern" just for readability as needed, and

filter tVar with regex ".+text"
and
filter tVar with regex pattern ".+text"
would likewise be identical.

rkriesel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 118
Joined: Thu Apr 13, 2006 6:25 pm

Re: filter enhancements

Post by rkriesel » Sun Jun 02, 2013 6:55 am

It's so -%$#@- dark in this bikeshed! I can't believe they didn't include motion-sensing eco-friendly mood lighting in here.

monte
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 1564
Joined: Fri Jan 13, 2012 1:47 am
Contact:

Re: filter enhancements

Post by monte » Sun Jun 02, 2013 9:52 am

Lol.. Jan.. just do what I do. Pick one of:
- Implement it, send the pull request, do whatever runrevmark wants to make him happy with it then tell people it's done.
- Post here to discuss the idea, do whatever runrevmark wants to make him happy, implement it, send the pull request.

It's good that everyone puts their opinion out there but unless we think of these forums as a discussion between the contributor and RunRev with some suggestions from the sidelines then we will all too often find ourselves in very long threads that don't go anywhere.
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

Janschenkel
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 977
Joined: Sat Apr 08, 2006 7:47 am
Location: Aalst, Belgium
Contact:

Re: filter enhancements

Post by Janschenkel » Sun Jun 02, 2013 11:58 am

Heh, it's good to see no one took my bikeshedding remark the wrong way. I certainly didn't mean to scoff anyone contributing to this thread.
I'm used to designing API's for libraries and frameworks, but not a programming language. When it's a library, you can deprecate a mistake in a later version and provide an alternative command or function. But if it's part of the language itself, the pressure is on to get it 'right' the first time around.
So here's what I'll go for and leave up to the RunRev team to tweak:

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { with | without | [ not ] matching } [ { wildcard | regex } [ pattern ] ] <pattern_exp> [ into <container> ] 
Now everyone can be as verbose or as concise as they want to be, and the old syntax still works. And people looking to filter with regex can write the following variants in their scripts:

Code: Select all

filter tText with regex ".*text"
filter tText matching regex ".*text"
filter lines of tText matching regex pattern ".*text"
Chances are good that my scripts will look like the second or third variant because I have the memory of a goldfish and would rather not open the dictionary again to see the specific rules of the 'filter' command and how they apply to what I was trying to achieve there - self-documenting code, right? ;-)

Jan Schenkel.
Quartam Reports & PDF Library for LiveCode
www.quartam.com

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3581
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: filter enhancements

Post by mwieder » Sun Jun 02, 2013 4:39 pm

@Jan- the BNF syntax looks good. Just go with Monte's advice.

Yeah, the problem with language design is that you can't easily back out of something once it's in place.
So just build it, release it into the wild, and let the team deal with the pull request.

Locked

Return to “Engine Contributors”