theotherbassist wrote:hh, really helpful. thanks. Honestly, it's quick enough for my purposes with either version. I'm sort of waiting on the "official" release of 8 to switch over--but that reservation is more just out of habit than anything else. Most of the "wait" time in my stack so far comes from pulling the contents of individual nodes from RSS feed XML data anyway--titles, descriptions, urls, etc. Do you think that it would be faster to save all the XML as local files and *then* read in the node content? What I'm doing now is pulling individual xml node contents directly from the web locations. Maybe making everything local in one fell swoop would cut some ms from the repeat functions that access the nodes.
The difference between download+process vs down first and process later is merely the time spent on disk I/O, likely just a couple milliseconds at most.
But the advantage of downloading first is that you have the opportunity to process a second time if needed without having to download again. The download is the longest part.
In LiveNet's RSS viewer (in the LC IDE see Plugins->GoLiveNet, or in older versions GoRevNet) I wound up breaking the process up into three steps:
1. Download the feeds to a local cache
This gives me the freedom to use different schedules for different feeds. For example, the feed for this forum and the use-livecode discussion list update every few minutes, but I only bother downloading the feed for the LC blog once a day since it rarely changes more than that. This not only saves my processor time, but is kinder to the servers providing the feeds. For all the over-design of RSS, I do try to honor the update frequency.
2. Parse into common elements
In the olden days RSS was "Really Simple Syndication", and it was true to its name, a joy to work with. Then that acronym was coopted by the RDF crew to serve double-duty as their "RDF Site Summary", which is not only needlessly complicated but alters the original focus from syndication to summarization. So not only are RSS feeds today radically different in structure (replete with evidence that few authors of RSS generators are able to read specs - "Hey guys, ever heard of CDATA?"), but also bifurcating the mandate of the format.
All that is just to say that RSS is complicated to parse, but despite the best efforts of the RDF gang to pervert the format, and the laudible efforts of the Atom crew to correct those mistakes, in the end we find just enough similarity among the various RSS formats that relevant parts can be extracted and put into a unified form.
I prefer using arrays to work with them, so storing them uses LSON (LiveCode serialized arrays such as you get with arrayEncode). LSON data is superfast to pull and decode from disk, and since the result is an array it's very efficient to work with for extraction and rendering.
3. Render for display
Most RSS feeds generated today are made with the dismaying presumption that you'll be using a multi-megabytes-of-RAM browser engine to render it. If you use an embedded browser object you'll find many aspects of dealing with today's often-broken RSS feeds a bit easier.
But if you want a stripped version of the content more akin to RSS' original mandate, giving you more control over its display, a little extra work can turn the dizzying mix of plain text, misunderstood/misapplied CDATA, and HTML you'll find in modern feeds into LC's htmlText, which can be dropped into a field.
All that said, I broke the process into three steps in large part for the unique needs of LiveNet: I wanted the shortest time I could get between opening the card and displaying the aggregate feeds, and I wanted to minimize traffic on the RSS hosts as much as possible.
So in LiveNet steps 1 and 2 above are done on a machine in my office, and the aggregated LSON file uploaded in compressed form every few minutes to one of my public servers, where LiveNet then grabs it whenever the user opens LiveNet.
That's probably more steps than you'd want for a local aggregator, but I do think there's benefit in using separate download schedules for different feeds, to be both more efficient for the user and kinder to the RSS host.
Whether you parse it on download or after probably doesn't matter much, but I would suggest parsing the content into an array for more efficient handling. And if you use different download schedules for each feed, it may be useful to store each in its own array, serialized to disk with arrayEncode.
You could probably dispose of the original XML once you have it converted to an array, but a habit from my analytics work compels me to keep sources in their original form just in case I later decide to do some other parsing on them. Probably not needed with ephemeral data like RSS, but sometimes old habits are hard to break.
