Implementing full regex in LiveCode

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: Klaus, FourthWorld, heatherlaine, robinmiller, kevinmiller

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Implementing full regex in LiveCode

Post by kaveh1000 » Sat Jul 06, 2019 10:55 am

This has been discussed on the forum a few times. LiveCode allows searching text using regex and identifying patterns found, but it misses a crucial part of standard regex, and that is replacement, using "back referencing". So you can search for the most complex patterns but you need a lot of work to replace that pattern.

I have managed to do what I have wanted but using a lot of acrobatics, multiple loops, etc. And this spoils the other beautiful features of LiveCode that save so much time normally.

The only solution right now as far as I can see is Thierry's sunnYrex:

https://sunny-tdz.com/livecode/sunnyrex

but it would be good to have this built into LiveCode.

What is required in having this implemented? Is it a matter of implementing the latest PCRE?

For those who are much cleverer than me on this list but have declared their fear of regex, if you support me in getting this implemented in LiveCode I promise to write a simple guide to regex and to make some tutorial videos and I promise you will love using it. :-)

Sorry if too much rant!

Kaveh
Kaveh

richmond62
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 3351
Joined: Fri Feb 19, 2010 10:17 am
Location: Bulgaria

Re: Implementing full regex in LiveCode

Post by richmond62 » Sat Jul 06, 2019 12:43 pm

Speaking as a total 'Regex peasant' (meaning I, finally, learnt what Regex really
means, rather than the vague, woolly idea I had previously):

https://en.wikipedia.org/wiki/Regular_expression

I can see that this is a 'must' for LiveCode.

Even if it is connected with Chomsky's Generative Grammar, which, as any Cognitive Grammatician
knows (see my MA Thesis: https://www.dropbox.com/s/ksziiwmi20mop ... f.zip?dl=0 ) does NOT really explain language that well.

But, as computers can ONLY work in some sort of quasi-mathematical, mechanistic framework,
Regex is probably the best we've got.

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Re: Implementing full regex in LiveCode

Post by kaveh1000 » Sat Jul 06, 2019 1:51 pm

That's some thesis! Do I understand it correctly that you are saying the syntax is not user friendly? If that is the case I fully agree with you. But it is *powerful* and *fast*.

If LiveCode had full regex implemented, then we could put a great front end on it so it becomes less frightening. I have done a bit of that already and LiveCode is a great tool for that. But I can't go further till I can actually replace what has been found on the fly.

Currently the best tool we have is

https://regex101.com/r/BivK6T/1

that I use a lot. But you still have to type the gobbledygook and it becomes unreadable quickly. And virtually impossible to debug. I want to make a better tool in LiveCode.
Kaveh

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2786
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Implementing full regex in LiveCode

Post by mwieder » Sun Jul 07, 2019 10:58 pm

The PCRE library in LiveCode does indeed support backreferences. Although PCRE regex statements, like the rest of Perl, are in a write-only language.

put matchtext("abca", "(a)(b)(c)\g-3") # returns true
put matchtext("abca", "(a)(b)(c)\g-2") # returns false

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Re: Implementing full regex in LiveCode

Post by kaveh1000 » Mon Jul 08, 2019 1:55 pm

Hi Mark

Thanks for that. So we can use back-referencing in searching, but what I am looking for is to be able to replace text, e.g. to change:

Code: Select all

1234ABC > ABC 1234
678XYZ > XYZ 678
so looking to use something like

Code: Select all

put replaceText(fld 1,(\d+)([A-Z]+),\2 \1)
LiveCode does not allow \2 \1. The replace text has to be normal text, not regex.

You have to use

Code: Select all

matchchunk
first, then remember the start and end of the numbers and digits, then go back and replace. And if there are multiple case you need a loop. I would so love to be able to use standard regex.
Kaveh

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2786
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Implementing full regex in LiveCode

Post by mwieder » Mon Jul 08, 2019 4:49 pm

Ah. Got it. So your problem here isn't that LiveCode's PCRE implementation doesn't support backreferences, but rather that the replaceText command could benefit from the replacement string *also* supporting regex (and being able to leverage the backreferences from the match expression).

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Re: Implementing full regex in LiveCode

Post by kaveh1000 » Mon Jul 08, 2019 4:54 pm

I bow to your superior articulation sire.

Yes, that is exactly what I meant. I hope you agree it would be a real boost to have it. I am not sure of the complications in implementing it.
Kaveh

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2786
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Implementing full regex in LiveCode

Post by mwieder » Mon Jul 08, 2019 5:31 pm

Yep - that would add some significant power to text processing. Meanwhile, I think Thierry has done an excellent job with his regex extension library.

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Re: Implementing full regex in LiveCode

Post by kaveh1000 » Mon Jul 08, 2019 8:10 pm

He certainly has and he has been amazingly helpful on these forums. But this is a basic requirement in regex and it needs to be within LiveCode.
Kaveh

mwieder
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2786
Joined: Mon Jan 22, 2007 7:36 am
Location: Berkeley, CA, US
Contact:

Re: Implementing full regex in LiveCode

Post by mwieder » Tue Jul 09, 2019 12:14 am

Code: Select all

1234ABC > ABC 1234
678XYZ > XYZ 678
If that's literally all you need then how about something like

Code: Select all

function convertBackwards pInputString
   local tResultString
   local tNumber, tText
   if matchtext(pInputString, "(\d+)([A-Z]+)", tNumber, tText) then
      put tText && tNumber into tResultString
   end if
   return tResultString
end convertBackwards

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Re: Implementing full regex in LiveCode

Post by kaveh1000 » Tue Jul 09, 2019 7:32 am

Thanks Mark for taking the time to write that. That is a minimal example. My problem is when these are embedded in text and I want to replace all, so the text might be:

Code: Select all

Lorem ipsum dolor sit amet, 1234ABC consectetuer
adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore
zzril delenit 678XYZ augue duis dolore te feugait nulla
facilisi.
I want to turn it into

Code: Select all

Lorem ipsum dolor sit amet, ABC 1234 consectetuer
adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore
zzril delenit XYZ 678 augue duis dolore te feugait nulla
facilisi.
In BBEdit, a simple regex will do it and extremely fast. In LiveCode I need a loop to find each original case, extract it, use your macro to convert it, and then put it back in the same location in the text. I could write a macro for that but my bigger concern is that the speed may not be what a native regex engine gives me.

This is to be used on large chunks of text and on the fly while editing, so speed is essential. Love to hear opinions.
Kaveh

bogs
Posts: 3545
Joined: Sat Feb 25, 2017 10:45 pm

Re: Implementing full regex in LiveCode

Post by bogs » Tue Jul 09, 2019 10:50 am

kaveh1000 wrote:
Tue Jul 09, 2019 7:32 am
I could write a macro for that but my bigger concern is that the speed may not be what a native regex engine gives me.
Just curious kaveh, have you actually tested how long it did take? I don't do a lot with replacing text, but have looped through large amounts of it on the fly and it usually doesn't take more than micro-seconds to do.

Aside from that, even if the way you write it takes too long for you, others may know how to speed up your code so it becomes no issue.

Just my .02, which is now .0000000000002 due to inflation :P
Image

kaveh1000
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 266
Joined: Sun Dec 18, 2011 7:23 pm
Location: London
Contact:

Re: Implementing full regex in LiveCode

Post by kaveh1000 » Tue Jul 09, 2019 10:55 am

thank you Bogs. I have not tested it for speed and I have to say it's pretty fast. I will get a chance to do it in the next couple of weeks and will report back.
Kaveh

[-hh]
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 2068
Joined: Thu Feb 28, 2013 11:52 pm
Location: Göttingen, DE

Re: Implementing full regex in LiveCode

Post by [-hh] » Tue Jul 09, 2019 1:42 pm

bogs wrote:Just my .02, which is now .0000000000002 due to inflation.
If the unit is still the same then that's not inflation but enormous deflation ;-)
shiftLock happens

bogs
Posts: 3545
Joined: Sat Feb 25, 2017 10:45 pm

Re: Implementing full regex in LiveCode

Post by bogs » Tue Jul 09, 2019 3:25 pm

The only thing deflating around here is my ego hee hee :D
Image

Post Reply

Return to “Talking LiveCode”