Stemming algorithms in LC?

[grimaldiface]

Has anyone written any stemming algorithms in lc? (example, porter stemmer). I tried translating the porter algorithm a while back, but it didn't work quite right and I gave up. I eventually started just piping text through an external perl script to stem, but it would make distribution of my program a lot simpler if I could just include the stemmer directly in my stack.

I found this thread from a while back: http://mail.on-rev.com/forums/viewtopic ... a3fb#p1536 . The links no longer work and nothing else came up in my searches.

Any help would be much appreciated! Thanks!

-Phil

[dunbarx]

Hi.

There might be something already written, but this is a simple thing to do in LC./

That is, if I understand what you want, to take a list such as:

stem
stemmed
stemming
distemper

and extract the common string among them, that is, "stem"

Craig Newman

EDIT.
Cavalier always. Simple, but may take a bit of thinking and effort.

[grimaldiface]

Yes, stemmers extract the stem of a word. More info here: http://tartarus.org/martin/PorterStemmer/

And a description of the algorithm here: http://tartarus.org/martin/PorterStemmer/def.txt

The most basic algorithm is the porter. I tried doing it a while ago. It was pretty good, but my output didn't always produce the correct result. I honestly found it quite difficult, and don't feel like doing it again if someone else has already gone through the trouble

[Simon]

Hi Phil,
The stack in that post was made by Eric Chatonet who has since passed away.
I have been able to find traces of communications about it but no actual stack. The last I find is from Richard Gaskin (Fourthworld) in 2012. In it he says he contacted Eric's estate asking for an MIT licence version but the thread ends there. He also mentions a second stack made by someone called "Andrew", the stack apparently is English only (Eric's was about 5 languages). I'm not clear but Andrews maybe Porter2.

Luckily Richard show up here often so he may chime in.

Simon
Edit: Richard is one of this forums moderators, so you can just click fourthworld up at the top and email him.

[paul_gr]

Found this 2005 PorterStemmer stack in an old archive.
Is this the one you are looking for?

Paul

[FourthWorld]

Good find, Paul. Where did you turn that up? I had to write Ken for a copy last year when I was exploring this.

Ken's is pretty good, but it's Porter1 rather than the newer Porter2. And unlike Eric's, it doesn't have Porter's Romance language stemmers, just US English.

If any of you have time to bring it up to Porter2, and/or add the Romance stemmers, please post it.

Porter1 isn't bad as-is; Porter himself notes that it only improves about 5% of words, which is largely why he's put so little effort into it since.

Anyone know of any other stemmers in the xTalk world, such as Lovins or Paice/Husk? In my circumstance speed is more important than reduced index set size (up to a point, of course <g>), and I wonder if Lovins' is faster - Ken's Porter implementation has few opportunities for optimization, and offhand I didn't see any that could speed it up as much as I need for frequently-added data.

[paul_gr]

Good find, Paul. Where did you turn that up? I had to write Ken for a copy last year when I was exploring this.

Found it on an old drive gathering dust; unfortunately no links.

Paul

[grimaldiface]

Awesome! Thank you Paul!