crash on repeated filters with large text

[gmccarthy]

I have a been getting crashes everytime in livecode 8 when repeating filters of a large text more than about 5 times.
The text is over 60000 lines with about 20 tabbed items per lines.
The first 2 tab delimited items of each line are a name and an ID.
Sometimes I need to get the line for a name or for an ID.
There may be repeated uses of the same name -with different IDs.
I need to filter the text for certain strings in item 1 or 2 of the text.
In certain situations I have to many strings to filter for, not just one (which seems to be stable).
When I need a repeat loop with the filtering more than about 5 times to gather information for more than 5 strings the repeated filters end up causing a crash.

I have reported this as a bug. http://quality.livecode.com/show_bug.cgi?id=17841

Any ideas what I might do while waiting for a fix?
Should I consider building arrays from the data instead, although it would have to be done after starting up the start to take into account the new data file which might is typically altered several times per day.

[jacque]

It sounds like a memory limitation, which wouldn't really be a bug. It's just that you're trying to stuff more data into RAM than the capacity of the computer will hold. I had the same issue when trying to load too many very large images at once, and altering my script fixed it. But we'd need to see your handler to know for sure.

If you aren't putting empty into variables the handler no longer needs, that would be the first thing to try. Re-using variables (replacing their contents) may do much the same thing. How is your repeat loop and filtering process structured?

[AxWald]

Hi,

never tried "filter" in such a situation - and why do you wait during the repeat?

Code: Select all

repeat with tCount = 1 to tRepeatFilterNum
      wait tRepeatFilterNum ticks
      put empty into tOutput
      put tRepeatFilterNum into fld "RepeatFilterNumCount"
      filter lines of tPlayers with wildcard pattern ("*" & tFilter & tab & "*") into tOutput
   end repeat

This code doesn't make sense to me. Care to explain?

Code: Select all

  put empty into tOutPut
    repeat for each line MyLine in tPlayers
        if offset(tFilter & tab,MyLine) is not "0" then
            put MyLine & return after tOutPut
        else next repeat
    end repeat

This gives you a variable containing all lines that match your search string - and I'm quite sure there'll be no crashes.

For really big tData I'd write the found lines to a file, this is even faster (see here).

But I assume your problem is different. To judge this more input is required

Have fun!