filter enhancements

[Janschenkel]

I just filed bug 10928 regarding the inconsistency between 'sort items' and the chunk hierarchy.
Once we get a clarification from the team about this, we can deliver an implementation of 'filter items' that is consistent with 'sort items'.
Now I'm going back to study the 'matchText' implementation to see how best to implement pattern matching...

Jan Schenkel.

[LCMark]

I just filed bug 10928 regarding the inconsistency between 'sort items' and the chunk hierarchy.

The current action of 'sort', whilst simplistic does seem reasonable to a certain degree. There's no real difference between line-delimited lists and item-delimited lists in the engine - they just have a different delimiter. If you want more hierarchical action then really the syntax should reflect that:

Code: Select all

sort items of lines of tVar
filter items of lines of tVar matching "foo*"

[Janschenkel]

Okido - though I prefer 'in' to 'of' in that syntax for improved human parsing.
So the target syntax for the 'filter' command becomes

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } <filter_pattern> | [ not ] matching [ pattern ] <regex_pattern> } [ into <container> ] 

Phew. Now to break that into separate incremental modifications to submit to the 'develop' branch

Jan Schenkel.

[mwieder]

I like the BNF syntax. Especially the [into <container>], which I am just now noticing.
And I want to encourage you not to overlook being able to tie the sort command into that as well (from bug 9980).

Edit: why not "{ lines | items | items in lines } [in | of]" so that either is acceptable?

[Janschenkel]

@mwieder
I assume you refer to bug 9959 - Extend Filter and Sort commands with 'into' clause ?
I've written recursive descent parsers and used ANTLR in the past, but I haven't puzzled out the LiveCode parser yet.
But once I've figured it out for the 'filter' command, I might take a stab at the 'sort' command next

Jan Schenkel.

[mwieder]

9959 - that's the one.

I haven't puzzled out the LiveCode parser yet.

yeah... it's a twisty maze of little passages. I've been trying to work my way through it as well.
Each command has a parse method and an eval method.
Sort::parse is in cmds.cpp and you'll have to add ST_INTO to the switch statement.
Filter::parse is in cmdsf.cpp.
After that it gets a bit gnarly.

[phaworth]

I *would*, however, like to take to extrapolation to array syntax into a separate thread and separate pull requests, because I think there's a lot of work to do on arrays and a lot of pitfalls along the way. Dealing with arrays of a single dimension looks easy enough by expanding the current syntax, but gets very sticky when going to multiple dimensions. And I'd like to avoid going down a path now that would have to be reverted.

Hi Mark,
I agree it's a separate exercise to deal with arrays. For what it's worth, I've been working on a handler that would allow me to filter generic arrays and found I needed the following input:

- the value to be searched for (of course)
- whether I'm looking for the value as a key name or a key value
- the "depth", for want of a better word, of the key I want to search.

That was sufficient for my needs but it's quite possible there needs to be more flexibility

So if I had an array:

[1]
[abc]
[1] = "foo"
[2] = "bar"
[def]
[1] = "mark"
[2] = "pete
[2]
[abc]
[1] = "pete"

I can search for "pete" as a level 3 key value and get
[1]
[def]
[2]="pete"
[2]
[abc]
[1]="pete

I can search for "abc" as a level 2 key name which would give me
[1]
[abc]
[1]="foo"
[2]="bar"
[2]
[abc]
[1]="pete

Pete

[rkriesel]

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } <filter_pattern> | [ not ] matching [ pattern ] <regex_pattern> } [ into <container> ] 

A simpler form obviates introducing "matching," and negating it, instead introducing the simpler term "regex:"

Code: Select all

filter [ { lines | items | items in lines } of ] <container_or_exp> { { with | without } [ regex ] <filter_pattern> [ into <container> ] 

Do you agree, Jan?

[LCMark]

@rkriesel: One of the advantages of introducing 'matching' is that it has synergy with a future 'matches' operator. If we were going to distinguish between old and new with an adjective then I'd suggest 'pattern' rather than 'regex' - as that is what a regular expression actually is (a pattern for matching strings).

[Janschenkel]

@rkriesel
I feel it more aptly differentiates the filtering behaviours if we introduce the 'matching' keyword for this purpose.
The adjective 'regex' sounds out-of-place as other regex-related functions use 'match' in their name.
Using the adjective 'pattern' to differentiate would be confusing as we're calling the wildcard string a 'filterPattern' in the documentation.

So I'd rather stick with the last syntax proposal...

Jan Schenkel.

[LCMark]

Using the adjective 'pattern' to differentiate would be confusing as we're calling the wildcard string a 'filterPattern' in the documentation.

Well, we could just rename 'filterPattern' in the dictionary

However, I do think 'filter items of ... [not] matching ...' reads better (and is more descriptive) than 'filter items of ... (with | without)' so I'm, personally, perfectly happy with that.

[mwieder]

I do think it's going to be very tricky explaining that

filter tVar matching "*text"

is different from

filter tVar with "*text"

so maybe enforcing the modifier "pattern" or "regex", as in

filter tVar matching pattern ".*text"

is necessary.

[phaworth]

I do think it's going to be very tricky explaining that

filter tVar matching "*text"

is different from

filter tVar with "*text"

so maybe enforcing the modifier "pattern" or "regex", as in

filter tVar matching pattern ".*text"

is necessary.

My preference would be:

filter tVar with pattern "*text"

Having to use "matching" instead" of "with" seems like an unnecessary complication to me. However, I think some extra bias should be given to the opinions of the folks who will be doing the coding so I'll leave it at that!

Pete

[mwieder]

Here's the problem with that:

filter tVar with pattern "*text"

will return empty.
If you mean "pattern" to be a regex string then it won't match anything.
If you mean "with pattern" to do what "with" currently does, then it will break existing scripts.
If you want to match the string "text" inside tVar then your regex would need to be ".*text"

I put that example in my post to illustrate why it's going to be tricky explaining why "matching" and "with" are different.

[phaworth]

I put that example in my post to illustrate why it's going to be tricky explaining why "matching" and "with" are different.

OK, let's try again, ignoring what the regexp is.

filter tVar with pattern <regexp>
filter tVar with <filter>

Doesn't break any existing scripts and eliminates having to remember when to use "with" and when to use "matching". No need to explain the difference between "matching" and "with" because "matching" doesn't exist any more. Only explanation is that "pattern" indicates the use of a regexp and that doesn't seem tricky to understand.