Web crawler

[buchacho]

Is anyone aware of a LiveCode web crawler? I have a website I am interested in backing up and want to have it do things with the data it crawls, such as categorizing, tagging, storing in a database structure, etc based on the content. I don't have a really concrete idea of what it will do, I am wondering if there are any projects people are working on her or have seen, and it sounds like something interesting to try to develop with LiveCode. Any suggestions on how to crawl and parse the pages?

[FourthWorld]

It's a lot of work, esp. handling robots.txt correctly and making sure you're kind to other people's server resources, but doable. I found this book helpful - nothing LC specific, but full of good advice that's easily adaptable to LC:
http://shop.oreilly.com/product/9781593273972.do