Implementing full regex in LiveCode

[kaveh1000]

This has been discussed on the forum a few times. LiveCode allows searching text using regex and identifying patterns found, but it misses a crucial part of standard regex, and that is replacement, using "back referencing". So you can search for the most complex patterns but you need a lot of work to replace that pattern.

I have managed to do what I have wanted but using a lot of acrobatics, multiple loops, etc. And this spoils the other beautiful features of LiveCode that save so much time normally.

The only solution right now as far as I can see is Thierry's sunnYrex:

https://sunny-tdz.com/livecode/sunnyrex

but it would be good to have this built into LiveCode.

What is required in having this implemented? Is it a matter of implementing the latest PCRE?

For those who are much cleverer than me on this list but have declared their fear of regex, if you support me in getting this implemented in LiveCode I promise to write a simple guide to regex and to make some tutorial videos and I promise you will love using it.

Sorry if too much rant!

Kaveh

[richmond62]

Speaking as a total 'Regex peasant' (meaning I, finally, learnt what Regex really
means, rather than the vague, woolly idea I had previously):

https://en.wikipedia.org/wiki/Regular_expression

I can see that this is a 'must' for LiveCode.

Even if it is connected with Chomsky's Generative Grammar, which, as any Cognitive Grammatician
knows (see my MA Thesis: https://www.dropbox.com/s/ksziiwmi20mop ... f.zip?dl=0 ) does NOT really explain language that well.

But, as computers can ONLY work in some sort of quasi-mathematical, mechanistic framework,
Regex is probably the best we've got.

[kaveh1000]

That's some thesis! Do I understand it correctly that you are saying the syntax is not user friendly? If that is the case I fully agree with you. But it is *powerful* and *fast*.

If LiveCode had full regex implemented, then we could put a great front end on it so it becomes less frightening. I have done a bit of that already and LiveCode is a great tool for that. But I can't go further till I can actually replace what has been found on the fly.

Currently the best tool we have is

https://regex101.com/r/BivK6T/1

that I use a lot. But you still have to type the gobbledygook and it becomes unreadable quickly. And virtually impossible to debug. I want to make a better tool in LiveCode.

[mwieder]

The PCRE library in LiveCode does indeed support backreferences. Although PCRE regex statements, like the rest of Perl, are in a write-only language.

put matchtext("abca", "(a)(b)(c)\g-3") # returns true
put matchtext("abca", "(a)(b)(c)\g-2") # returns false

[kaveh1000]

Hi Mark

Thanks for that. So we can use back-referencing in searching, but what I am looking for is to be able to replace text, e.g. to change:

Code: Select all

1234ABC > ABC 1234
678XYZ > XYZ 678

so looking to use something like

Code: Select all

put replaceText(fld 1,(\d+)([A-Z]+),\2 \1)

LiveCode does not allow \2 \1. The replace text has to be normal text, not regex.

You have to use

Code: Select all

matchchunk

first, then remember the start and end of the numbers and digits, then go back and replace. And if there are multiple case you need a loop. I would so love to be able to use standard regex.

[mwieder]

Ah. Got it. So your problem here isn't that LiveCode's PCRE implementation doesn't support backreferences, but rather that the replaceText command could benefit from the replacement string *also* supporting regex (and being able to leverage the backreferences from the match expression).

[kaveh1000]

I bow to your superior articulation sire.

Yes, that is exactly what I meant. I hope you agree it would be a real boost to have it. I am not sure of the complications in implementing it.

[mwieder]

Yep - that would add some significant power to text processing. Meanwhile, I think Thierry has done an excellent job with his regex extension library.

[kaveh1000]

He certainly has and he has been amazingly helpful on these forums. But this is a basic requirement in regex and it needs to be within LiveCode.

[mwieder]

Code: Select all

1234ABC > ABC 1234
678XYZ > XYZ 678

If that's literally all you need then how about something like

Code: Select all

function convertBackwards pInputString
   local tResultString
   local tNumber, tText
   if matchtext(pInputString, "(\d+)([A-Z]+)", tNumber, tText) then
      put tText && tNumber into tResultString
   end if
   return tResultString
end convertBackwards

[kaveh1000]

Thanks Mark for taking the time to write that. That is a minimal example. My problem is when these are embedded in text and I want to replace all, so the text might be:

Code: Select all

Lorem ipsum dolor sit amet, 1234ABC consectetuer
adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore
zzril delenit 678XYZ augue duis dolore te feugait nulla
facilisi.

I want to turn it into

Code: Select all

Lorem ipsum dolor sit amet, ABC 1234 consectetuer
adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore
zzril delenit XYZ 678 augue duis dolore te feugait nulla
facilisi.

In BBEdit, a simple regex will do it and extremely fast. In LiveCode I need a loop to find each original case, extract it, use your macro to convert it, and then put it back in the same location in the text. I could write a macro for that but my bigger concern is that the speed may not be what a native regex engine gives me.

This is to be used on large chunks of text and on the fly while editing, so speed is essential. Love to hear opinions.

[bogs]

I could write a macro for that but my bigger concern is that the speed may not be what a native regex engine gives me.

Just curious kaveh, have you actually tested how long it did take? I don't do a lot with replacing text, but have looped through large amounts of it on the fly and it usually doesn't take more than micro-seconds to do.

Aside from that, even if the way you write it takes too long for you, others may know how to speed up your code so it becomes no issue.

Just my .02, which is now .0000000000002 due to inflation

[kaveh1000]

thank you Bogs. I have not tested it for speed and I have to say it's pretty fast. I will get a chance to do it in the next couple of weeks and will report back.

[[-hh]]

Just my .02, which is now .0000000000002 due to inflation.

If the unit is still the same then that's not inflation but enormous deflation

[bogs]

The only thing deflating around here is my ego hee hee