filter an array with numeric bounds

[n.allan]

Hello all. I am trying to store the points of an svg mercator map in an array. Is there a fast way to only retrieve the key values that are within the bounds of my stack?

For example...I know the maximum and minimum and minimum latitude and longitude that my stack can display. I would like to store the svg data in such a manner that it can be queried and return a list within these values.

I am thinking...put all the points into an sqlite data base and return values WHERE tx > minlon and tx < maxlon etc...

I would like to do this with an array in memory for speed. Would there be a massive speed gain using an array in memory? The downside is there is a huge amount of data for the whole world, as you can imagine.

Any ideas?

[shaosean]

You would have to sort the keys of the array and then (more than likely) loop over them to find the keys that fit between your min and max values and then store them for reference (quickest way would be to slap them in a new array).. You can see your memory usage will be slightly higher this way and there could be a performance hit if you min value is high up in the array (once you hit the max value, you can bail out of the loop).. I think both ways have their pros and cons, so whatever you feel more comfortable doing is probably the better way..

[Mark]

Hi,

You can do a repeat-for-each loop or put the keys into a variable and do a numeric sort on the variable or use sqlite. A repeat-for-each is very fast and it takes a real lot of data for repeat-for-each to be beaten by sqlite. Sorting might slow it down, but if you have a lot of data and if each query retrieves a relatively small number of data points, then you might win some time by sorting.

If each of these option take longer than 1 or 2 seconds, you might want to create a php or onrev script on a web server and send a query to the server, because servers are usually very fast.

Since you use the word "filter", I should probably add that using the filter command, or any other regex features, would be very inefficient.

Kind regards,

Mark

[n.allan]

I was afraid that the loop was needed. That is the way I am doing it at the moment and its producing an unacceptable delay looping thru about 480000 points. I will have a tinker with the syntax and see which method is the fastest.

Thanks for the suggestions.

[FourthWorld]

Half a million evaluations will take some time. How much of a delay are you seeing? If you post your code we may be able to see other opportunities for optimizations.

[n.allan]

I am currently using the loop method:

This is very elementary but I don't have the code to hand....I have the following for example:

Code: Select all

local tSVG,tValidLon
   put 10 into tSVG["X"][90]
   put 11 into tSVG["X"][90.1]
   put 12 into tSVG["X"][90.2]
   put 13 into tSVG["X"][90.3]
   put 14 into tSVG["X"][90.4]
   put 15 into tSVG["X"][90.5]
   put 16 into tSVG["X"][90.6]
   put 17 into tSVG["X"][90.7]
   put 18 into tSVG["X"][90.8]
   put 19 into tSVG["X"][90.9]
   put 20 into tSVG["X"][91]
   
   repeat with tX = 90 to 91 step 0.1 -- loop thru all the keys of tSVG["X"]
      if ((tSVG["X"][tX] >= 13) and (tSVG["X"][tX] <= 16)) then put tX & return after tValidLon 
   end repeat
   put tValidLon -- tValidLon now contains a list of keys who's values are within the bounds of 13 and 16
end test

Just by asking the question, I have spotted some glaring errors in the way I have created my SVG array. I will go back to the drawing board and map out my SVG data differently.

[Mark]

Hi,

Repeat with is very slow. Instead try this:

Code: Select all

repeat for each key myKey in tSVG["X"]
   if tX >= 90 and tX < 91 and myKey >= 13 and myKey <= 16 then
    put tX & cr after tValidon
  end if
end repeat

Kind regards,

Mark

[n.allan]

Thanks. Ill bear that in mind. If I mapped out the array properly I could probably use the offsetLine function on the keys of tSVG[x] too. Thanks for the suggestions.

[Mark]

Hi,

If you're going to use offset, you'll need to do a numeric sort first. I don't know how that is going to affect performance with a very large dataset. It might be worth a try.

Kind regards,

Mark