Improoving LC Forum Search algo - in sequence ?

Working group for improving LiveCode's User Guide, Dictionary, and other learning resorces.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Locked
liveme
Posts: 240
Joined: Thu Aug 27, 2015 5:22 pm
Location: down under

Improoving LC Forum Search algo - in sequence ?

Post by liveme » Thu Mar 11, 2021 3:33 am

Since forum is as usefull as Documentation, I posted here
Hi Team,

I'm wondering if LC forum search online could not improove its search Algorythm...its been a while since this keeps comming back to my mind.

The following happened again just today....
From my mobile I was looking for topics on contact apps, so typed in :

Code: Select all

 "livecode contact"
which google returned me an intersting Topic, from the LC forum.
(PC being more practical for reading, I usely send the message to Telegram and click again when on PC.)

but since the Google 2 words only search was efficient to return the Forum page.. when facing my monitor I just typed in the same 2 words in the Forum page that was already opened...
Since it returned many pages - 68!! - but not the Topic named "How to access build in contact app" in the top firsts... :(
I just started to include more KEY words :

Code: Select all

access build+in contact.
..
no better results in first page ! :roll:

So I decided to type in the whole Topic Title in the exact same way it was displayed on my mobile phone:

Code: Select all

  "How to access build-in contact app"
and again...my Topic Title cant make it in the Top results not even at the botom of the first page / 5 pages :shock:
I decided to chang symbol - for + since it sometimes affect search results ...but no better.

So my question is :
can't LC modify its Forum Search Algorithm

so that it :
>>> prioritize any sequences of 2 or more words in the strict same order ???
(and then..in an inversed or mix order)
i.e typing

Code: Select all

contact build-in"
or

Code: Select all

"contact build+in access"
should logicaly be returning "ANY" Topic Titles containing those same words as "How to access build+in contact app"..and similar among the top results, nop ?

One usually input SEARCH criterias with 2 or 3 words...
- and then include more words or other ones to reduce the number of answers returned .
-or lean towards similar possible topics.
..and if not satified.. finaly "goes for Google search" :D

I Know LC has some pretty good search features included in its language - though the Forum is likely not built using LC.
Indexing the Title Column should be included, if not already, but the search algo still needs some efforts !

Else, I find the Simple Search text very poor as for the previous search example,
I guess 90% of the time it is just useless to me by returning too vague results.
Which I beleive by being "more strict" it could return more relevant Topics and waste less resources at the end !
.
(the Advanced search is as inefficient as far as I tested it with the same sequence ..and + or not + :oops:
*it still returned 5 pages and the Wanted Topic is still not in the first one...

Code: Select all

+access+build+in+contact
How about using some A.I :shock: :P ?
I Dont know what is on the road for this improovment...but I find it a key gamer to all !

Code: Select all

"+my+2+cents" 
Thanks for reading ! Sorry for crying !
:idea: :wink:

AxWald
Posts: 578
Joined: Thu Mar 06, 2014 2:57 pm

Re: Improoving LC Forum Search algo - in sequence ?

Post by AxWald » Thu Mar 11, 2021 1:02 pm

Hi,

I stopped using the forum search ages ago - in any phpBB forum. Those PHP guys seem to have strange ideas about searching.

But especially in this forum, Google works fine - the search term:

Code: Select all

site:forums.livecode.com contact
gives exactly your desired result.

Calling such a search via plugIn or text editor is easy. Have fun!
All code published by me here was created with Community Editions of LC (thus is GPLv3).
If you use it in closed source projects, or for the Apple AppStore, or with XCode
you'll violate some license terms - read your relevant EULAs & Licenses!

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Improoving LC Forum Search algo - in sequence ?

Post by bogs » Thu Mar 11, 2021 1:45 pm

Actually. I think Andy P. created just such a plugin which searches all kinds of things, not just the forum. If you made it a standalone, it would probably work on mobile as well.

https://github.com/AndyPiddock/LiveCode-Multi-Search
Image

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7214
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Improoving LC Forum Search algo - in sequence ?

Post by jacque » Thu Mar 11, 2021 7:20 pm

I tried a Google search for "livecode contact" and did not see any forum results, I only saw how to contact LC itself.

Are you sure the post title was "build-in"? Normally the English would be "built-in".
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

liveme
Posts: 240
Joined: Thu Aug 27, 2015 5:22 pm
Location: down under

Re: Improoving LC Forum Search algo - in sequence ?

Post by liveme » Fri Mar 12, 2021 5:07 am

The thing si I'm trying to use the less google-made tools as possible...(privacy adicted I guess)

@jaque : Don't know what browser you'd be using ?
I'm on Brave - but duckduckgo or else cant - yet - compete - with TyranausaurusGoogulus
..so whenever I need I temporarly switch the engine when there is no satistyting results with others.

True the word is built-in ...but my search returned this :
Image

if the forum PHP page search algo code can be modified,
its probably using siomething like SQL
...then I have the beleif it can be improoved ! :roll:
Attachments
aaa google result.png

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: Improoving LC Forum Search algo - in sequence ?

Post by bogs » Fri Mar 12, 2021 11:49 am

liveme wrote:
Fri Mar 12, 2021 5:07 am
The thing si I'm trying to use the less google-made tools as possible...(privacy adicted I guess)

@jaque : Don't know what browser you'd be using ?
I'm on Brave - but duckduckgo or else cant - yet - compete - with TyranausaurusGoogulus
..so whenever I need I temporarly switch the engine when there is no satistyting results with others.
Erm, the entry AxWald posted for use in Google works just fine in DuckDuckGo's search, and even if DDG was a little less stellar than Big G, it would still be light years ahead of any BB board's built in search I've ever seen heh.
aPic_DDG-search.png
I been searching...
If AxWald hadn't already suggested using a real search engine, I would have, really any of the search engines will work this way.
Image

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7214
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: Improoving LC Forum Search algo - in sequence ?

Post by jacque » Fri Mar 12, 2021 7:39 pm

In the original post you said you searched for "livecode contact" which doesn't work very well. In the image you posted I see that the actual search terms were "livecode access contact". The browser doesn't matter really, the important thing is that you use Google search.

But the problem here is that the PHP code that runs this forum is a free package used by many forums everywhere. The search algorithm is part of the package and that is what needs to be updated. I agree that the current implementation needs work.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Improoving LC Forum Search algo - in sequence ?

Post by FourthWorld » Fri Mar 12, 2021 9:35 pm

PHPBB provides only a frontend to MySQL's fulltext search. It's pretty good, but isn't nearly as richly indexed as the Akamai engine behind DuckDuckGo, or Google's engine, or the one Microsoft built for Bing.

The R&D needed for truly superior search is an ongoing quest where the only real competitors are a handful of the world's most powerful software companies.

There's a lot more that could be said about search, including solutions halfway between MySQL's fulltext engine and the giants like Google, such as the Apache Lucene project and others.

And you're right about LC possibly playing a role in making a useful search engine, esp. when an embedded one is needed for local document stores where Lucene would be overkill. Our community has a couple different stemmers for indexing, and arrays can provide efficient access to inverted indices. I've made a few embeddable search engines in LC over the years, and am making another for my web site this summer.

But the challenge with adding either a very expensive custom solution here or the free and open Lucene, is still cost: integrating external search into the PHPBB framework is a non-trivial task.

Much more cost-effective and more social is to ask members here for help with a given problem. Humans remain excellent resources where insight may be as useful as memory.

So what were you looking for?

Are you interested in making a contact app, or a CRM, or integrating with existing one?
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

liveme
Posts: 240
Joined: Thu Aug 27, 2015 5:22 pm
Location: down under

Re: Improoving LC Forum Search algo - in sequence ?

Post by liveme » Sat Mar 13, 2021 2:40 am

But the challenge with adding either a very expensive custom solution here or the free and open Lucene, is still cost:
Except for the side of not having to maintain an out-of-the box built forum system, I would have though an LC built forum with a DB would not be so hard or expensive to build, including the content search system.
Although I never really run some calc to see how much i'd cost...
beside offering a real live sample of what LC can make combined to PHP I m a bit surprised not to see it hasnt been done since LC is probably well equiped to run its own LC made Forum....
Any forum could serve as a model, and beside cloning few pages behaviour, I would not think its an harduous task to build one in full LC/PHP codes..
DB Hosting would probably be around 8us$ a month + yearly free SSL. against full control of your forum !
Would the project idea sounds crazy ?
(beside, LC could ReBrand it and "resell" their own forum ready to install package..)
for use in Google works just fine in DuckDuckGo's search,
interesting, I never digged much to know about whats behinf all those search engines...DDG is okay for me, I just "feel" GG returned more updated or richest replies - but thats likely coz they kno wme more after years of usage too :lol:
making a contact app, or a CRM, or integrating with existing one?
Yop, I'd be interesting in installing my own made contact App, I just dont know how doable it is to import all contact from the supplied android contact app.
I would add one or two new features to the simple contact app that I have always missed and still cant find after comparing the most 15 popular alternatives...
thats roughly the plan :)

Beside I'm into coding my first ever self-use DB app, all those months learning the Tip of the iceberg prooved to be useful and now taking benfit from that...feels great !

Locked

Return to “Documentation and Tutorials”