Implementing full regex in LiveCode
Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Implementing full regex in LiveCode
This has been discussed on the forum a few times. LiveCode allows searching text using regex and identifying patterns found, but it misses a crucial part of standard regex, and that is replacement, using "back referencing". So you can search for the most complex patterns but you need a lot of work to replace that pattern.
I have managed to do what I have wanted but using a lot of acrobatics, multiple loops, etc. And this spoils the other beautiful features of LiveCode that save so much time normally.
The only solution right now as far as I can see is Thierry's sunnYrex:
https://sunny-tdz.com/livecode/sunnyrex
but it would be good to have this built into LiveCode.
What is required in having this implemented? Is it a matter of implementing the latest PCRE?
For those who are much cleverer than me on this list but have declared their fear of regex, if you support me in getting this implemented in LiveCode I promise to write a simple guide to regex and to make some tutorial videos and I promise you will love using it.
Sorry if too much rant!
Kaveh
I have managed to do what I have wanted but using a lot of acrobatics, multiple loops, etc. And this spoils the other beautiful features of LiveCode that save so much time normally.
The only solution right now as far as I can see is Thierry's sunnYrex:
https://sunny-tdz.com/livecode/sunnyrex
but it would be good to have this built into LiveCode.
What is required in having this implemented? Is it a matter of implementing the latest PCRE?
For those who are much cleverer than me on this list but have declared their fear of regex, if you support me in getting this implemented in LiveCode I promise to write a simple guide to regex and to make some tutorial videos and I promise you will love using it.
Sorry if too much rant!
Kaveh
Kaveh
-
- Livecode Opensource Backer
- Posts: 9375
- Joined: Fri Feb 19, 2010 10:17 am
- Location: Bulgaria
Re: Implementing full regex in LiveCode
Speaking as a total 'Regex peasant' (meaning I, finally, learnt what Regex really
means, rather than the vague, woolly idea I had previously):
https://en.wikipedia.org/wiki/Regular_expression
I can see that this is a 'must' for LiveCode.
Even if it is connected with Chomsky's Generative Grammar, which, as any Cognitive Grammatician
knows (see my MA Thesis: https://www.dropbox.com/s/ksziiwmi20mop ... f.zip?dl=0 ) does NOT really explain language that well.
But, as computers can ONLY work in some sort of quasi-mathematical, mechanistic framework,
Regex is probably the best we've got.
means, rather than the vague, woolly idea I had previously):
https://en.wikipedia.org/wiki/Regular_expression
I can see that this is a 'must' for LiveCode.
Even if it is connected with Chomsky's Generative Grammar, which, as any Cognitive Grammatician
knows (see my MA Thesis: https://www.dropbox.com/s/ksziiwmi20mop ... f.zip?dl=0 ) does NOT really explain language that well.
But, as computers can ONLY work in some sort of quasi-mathematical, mechanistic framework,
Regex is probably the best we've got.
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Re: Implementing full regex in LiveCode
That's some thesis! Do I understand it correctly that you are saying the syntax is not user friendly? If that is the case I fully agree with you. But it is *powerful* and *fast*.
If LiveCode had full regex implemented, then we could put a great front end on it so it becomes less frightening. I have done a bit of that already and LiveCode is a great tool for that. But I can't go further till I can actually replace what has been found on the fly.
Currently the best tool we have is
https://regex101.com/r/BivK6T/1
that I use a lot. But you still have to type the gobbledygook and it becomes unreadable quickly. And virtually impossible to debug. I want to make a better tool in LiveCode.
If LiveCode had full regex implemented, then we could put a great front end on it so it becomes less frightening. I have done a bit of that already and LiveCode is a great tool for that. But I can't go further till I can actually replace what has been found on the fly.
Currently the best tool we have is
https://regex101.com/r/BivK6T/1
that I use a lot. But you still have to type the gobbledygook and it becomes unreadable quickly. And virtually impossible to debug. I want to make a better tool in LiveCode.
Kaveh
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Implementing full regex in LiveCode
The PCRE library in LiveCode does indeed support backreferences. Although PCRE regex statements, like the rest of Perl, are in a write-only language.
put matchtext("abca", "(a)(b)(c)\g-3") # returns true
put matchtext("abca", "(a)(b)(c)\g-2") # returns false
put matchtext("abca", "(a)(b)(c)\g-3") # returns true
put matchtext("abca", "(a)(b)(c)\g-2") # returns false
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Re: Implementing full regex in LiveCode
Hi Mark
Thanks for that. So we can use back-referencing in searching, but what I am looking for is to be able to replace text, e.g. to change:
so looking to use something like
LiveCode does not allow \2 \1. The replace text has to be normal text, not regex.
You have to use
first, then remember the start and end of the numbers and digits, then go back and replace. And if there are multiple case you need a loop. I would so love to be able to use standard regex.
Thanks for that. So we can use back-referencing in searching, but what I am looking for is to be able to replace text, e.g. to change:
Code: Select all
1234ABC > ABC 1234
678XYZ > XYZ 678
Code: Select all
put replaceText(fld 1,(\d+)([A-Z]+),\2 \1)
You have to use
Code: Select all
matchchunk
Kaveh
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Implementing full regex in LiveCode
Ah. Got it. So your problem here isn't that LiveCode's PCRE implementation doesn't support backreferences, but rather that the replaceText command could benefit from the replacement string *also* supporting regex (and being able to leverage the backreferences from the match expression).
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Re: Implementing full regex in LiveCode
I bow to your superior articulation sire.
Yes, that is exactly what I meant. I hope you agree it would be a real boost to have it. I am not sure of the complications in implementing it.
Yes, that is exactly what I meant. I hope you agree it would be a real boost to have it. I am not sure of the complications in implementing it.
Kaveh
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Implementing full regex in LiveCode
Yep - that would add some significant power to text processing. Meanwhile, I think Thierry has done an excellent job with his regex extension library.
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Re: Implementing full regex in LiveCode
He certainly has and he has been amazingly helpful on these forums. But this is a basic requirement in regex and it needs to be within LiveCode.
Kaveh
-
- VIP Livecode Opensource Backer
- Posts: 3581
- Joined: Mon Jan 22, 2007 7:36 am
- Location: Berkeley, CA, US
- Contact:
Re: Implementing full regex in LiveCode
Code: Select all
1234ABC > ABC 1234
678XYZ > XYZ 678
Code: Select all
function convertBackwards pInputString
local tResultString
local tNumber, tText
if matchtext(pInputString, "(\d+)([A-Z]+)", tNumber, tText) then
put tText && tNumber into tResultString
end if
return tResultString
end convertBackwards
PowerDebug http://powerdebug.ahsoftware.net
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
PowerTools http://www.ahsoftware.net/PowerTools/PowerTools.irev
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Re: Implementing full regex in LiveCode
Thanks Mark for taking the time to write that. That is a minimal example. My problem is when these are embedded in text and I want to replace all, so the text might be:
I want to turn it into
In BBEdit, a simple regex will do it and extremely fast. In LiveCode I need a loop to find each original case, extract it, use your macro to convert it, and then put it back in the same location in the text. I could write a macro for that but my bigger concern is that the speed may not be what a native regex engine gives me.
This is to be used on large chunks of text and on the fly while editing, so speed is essential. Love to hear opinions.
Code: Select all
Lorem ipsum dolor sit amet, 1234ABC consectetuer
adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore
zzril delenit 678XYZ augue duis dolore te feugait nulla
facilisi.
Code: Select all
Lorem ipsum dolor sit amet, ABC 1234 consectetuer
adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore
zzril delenit XYZ 678 augue duis dolore te feugait nulla
facilisi.
This is to be used on large chunks of text and on the fly while editing, so speed is essential. Love to hear opinions.
Kaveh
Re: Implementing full regex in LiveCode
Just curious kaveh, have you actually tested how long it did take? I don't do a lot with replacing text, but have looped through large amounts of it on the fly and it usually doesn't take more than micro-seconds to do.
Aside from that, even if the way you write it takes too long for you, others may know how to speed up your code so it becomes no issue.
Just my .02, which is now .0000000000002 due to inflation
-
- Livecode Opensource Backer
- Posts: 508
- Joined: Sun Dec 18, 2011 7:23 pm
- Location: London
- Contact:
Re: Implementing full regex in LiveCode
thank you Bogs. I have not tested it for speed and I have to say it's pretty fast. I will get a chance to do it in the next couple of weeks and will report back.
Kaveh
-
- VIP Livecode Opensource Backer
- Posts: 2262
- Joined: Thu Feb 28, 2013 11:52 pm
- Location: Göttingen, DE
Re: Implementing full regex in LiveCode
If the unit is still the same then that's not inflation but enormous deflationbogs wrote:Just my .02, which is now .0000000000002 due to inflation.
shiftLock happens
Re: Implementing full regex in LiveCode
The only thing deflating around here is my ego hee hee