LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!
I need to search a field for a substring which may be repeated many times. I am only interested in the last occurrence of the sub string and I wish to avoid searching the very last line.
repeat for each line tLine in gDatabuffer
If item 1 of tLine ="$GPGGA" then put tLine into tFixData
end repeat
which is a traditional forward pass through the data and it also includes the last line of data in the search which I do not want included. I have tried to write a similar loop using negative indexes but the syntax has defeated me.
to exclude the last line and to proceed from the line last line - 1 to the first line try this (same as Jeff's, Jeff just forgot to exclude the last line)
put the number of lines of gDatabuffer into tLastLine
put 1 into tCounter
repeat for each line tLine in gDatabuffer
if tCounter = tLastLine then exit repeat -- exit before processing the last line
If item 1 of tLine ="$GPGGA" then put tLine into tFixData
add 1 to tCounter
end repeat
you will notice the speed gain of this only if you have a fair amount of lines (guessing > 500 to 1000)
regards
Bernd
I promised to report back...
I have tried the code snippets above and they all seem to work fast enough for the task which is to scan a block of less than 20 lines of CSV text.
Unfortunately my code has a bug which is resisting my attempts to locate. The code below is inside a routine that is called once a second, its purpose is to extract a single line of data from the block presented and then process the fields. It works for a while say 30 - 60 seconds but then it stops. I have proved that the repeating routine is still being called and that none of the sub routines are causing the failure. Its as if the if statement at line 5 fails on one read and then remains in a failed state until the routine is stopped and started. Please will you assess my mess? For example should I be checking that item 9 is a number before I do a comparison. Thanks.
put the number of lines of gDatabuffer into tLastLine
put 1 into tCounter
repeat for each line tLine in gDatabuffer
if tCounter = tLastLine then exit repeat -- exit before processing the last line
If item 1 of tLine ="$GPGGA" AND item 7 of tLine= "1" AND item 9 of tLine <gHDOP then
beep
--store the start time if required
if gStartTime="000000" then
put item 2 of tLine into gStartTime
--answer gstartTime
end if
DisplayTime (item 2 of tLine)
DisplayPosition tLine
If gLastFix<>"" then SetDistanceMoved gLastFix, tLine
put item 8 of tLine into fld"satcount"
put item 9 of tLine into fld"HDop"
put tLine into gLastFix
CalculateAvSpeed item 2 of tLine
--exit repeat
ELSE if item 1 of tLine ="$GPGGA" then
put tLine & "XXX" after gRejectedFixs
beep
beep
beep
end if
add 1 to tCounter
end repeat
Simon,
I don't see anything that is wrong as far as syntax is concerned. I reread your initial post and you just wanted the most recent occurence of the event. The way it is now you do actually start with the oldest event assuming that the most recent lines are added after the older ones. Does that matter? If so I would go for Jeff's "repeat with i = the number of lines - 1 down to 1" and exit if you found the most recent occurence, which would be the first one meeting your conditions, I suppose.
Of course if the global gDataBuffer is empty then nothing happens even though you enter the handler. If gDatabuffer is just empty lines you would enter the repeat loop.
What happens to gHDOP? Do any of the lines you would want show up in gRejectedFixs?
I would change this but just out of a habbit, probably nothing wrong with it (I sometimes get confused by the many ways you can do a conditional with if, then,else
ELSE
if item 1 of tLine ="$GPGGA" then
put tLine & "XXX" after gRejectedFixs
beep
beep
beep
end if
end if
Sorry, maybe someone else sees something obvious, but at this point I would check how the globals are filled. You are shure that your handler is indeed called?
regards
Bernd
In the way of things I realised that I only wanted a single line of data, the GPS position fix. This is reported once a second along with quite a lot of other data. I realised that once I had extracted the newest fix I could set the global to "" ready for refilling by the serial port reader. Once it was running the search routine only had to parse a maximum of ten lines so I decided that I would stick with searching forwards. I was also confused by the user guide which states that the earliest position in the data should be requested first in any chunk expressions. Anyway that seems a long time ago....
The global gHDOP is used as part of the selection of valid data, the nearer it is to 1 the better the accuracy of the fix data. I was concerned that as it was my only numeric comparison that it might be causing the problems. However, it is not the source of the problems. The code following the ELSE does not report any "missing" messages, when it fails the whole IF construct stops operating.
I am sure the handler is being called as I put a heartbeat indicator on the card and the following code is just before the IF construct that is causing the problems:
-- Modulate the heart beat
If the hilite of btn"heartbeat" is false then
set the hilite of btn"heartbeat" to true
Else
set the hilite of btn"heartbeat" to false
end if
The code I posted included much of my debug code put in to trace the bug I touched on above. When the bug bit it seems that the whole IF ELSE ENDIf construct was being ignored. RunRev would only re-start conducting the IF test once the repeating call to the routine had been stopped and then restarted. I have spent this afternoon tracking this bug down and have decided that it is probably caused by odd data being stored in the global gDataBuffer : every so often the code that reads the serial port writes extra linefeeds Hex(0D) into the variable and I think that these were causing a problem with the detection of line ends within the loop. The odd thing is that no errors were posted and the construct just went to sleep. I decided to rewrite the portion of the code that extracts the data from the buffer and this is what I have come up with:
put 1 into tPTR
put the number of characters of gDatabuffer into tCharCount
Repeat until tPTR = tCharCount
put character tPTR of gDataBuffer into theChar
--put "Position " & tPTR & " " & theChar & LF after fld"buffer"
If theChar = "$" then
--we are at the start of a data word, so read the header word
--and see if it is the one required
put character tPTR to tPTR+5 of gDataBuffer into tID
if tID="$GPGGA" then
--extract the word from the buffer
put False into tISLast
Repeat Until tIsLast=True
put character tPTR of gDataBuffer into theChar
If theChar="*" then
--end of data sentence, so exit the loop
put theChar after tDataSentence
put True into tIsLast
Else
put theChar after tDataSentence
end if
put tPTR+1 into tPTR
end Repeat
end if
exit repeat
end if
put tPTR+1 into tPTR
end Repeat
--debug code
This code snip just ignores and LF or CRs and searches on the data for key characters. I would be interested in how it may be improved as I feel some of it is rather clumsy and overall it is far to complex. It searches through the global gDataBuffer seeking the ID word "$GPGGA" it then reads the ID word and all the characters following up to and including the "*" character. I need to add more code to protect against the case where the sentence starts near the end of the buffer and is incomplete i.e no "*" is found; in this situation the data sentence is completed at the start of the next set of data passed as gDataBuffer is set to "" on each pass.
on extractData
put line 1 to - 2 of gDataBuffer into tMyData -- leave out last line
put lineOffset("$GPGGA",tMyData) into tLineNo
if tLineNo > 0 then -- found "$GPGGA"
put line tLineNo of tMyData into tLineToAnalyse
put offset("*",tLineToAnalyse) into tEndOfInfo
if tEndOfInfo = 0 then
--- no "*" found, proceed appropriately
else
put char 1 to tEndOfInfo of tLineToAnalyse into tLineToAnalyse -- cut off after "*"
-- do something to item 1
-- do something to item x
-- etc
end if -- tEndOfInfo = 0
end if -- tLineNo > 0
end extractData
I dont know if this helps. But you might also look at a the offset function that you could use in a repeat loop.
You could use above posted code in a repeat loop by taking (in case you found a line)
put line tLineNo + 1 to line - 2 of gDataBuffer into tMyData
and test anew for "$GPGGA"
maybe you could post 10-20 lines of sample data and an example of what your extracted and rejected data is supposed to look like, also some typical "dirty" lines of valid data.
regards
Bernd
Dear Bernd,
I have posted some sample data below. I have two routines running on repeat timers the first repeating every 250 ms, this reads the contents of the serial buffer and appends it to the global gDataBuffer. The second routine is where the problem occurs, it is called once per second and processes the contents of the data buffer. If all is well it ignores the last line, which is often incomplete and when it has processed all the other lines they are deleted leaving the part line still in the buffer. The next read operation will complete the line and all will be well. While attempting to track down what was causing the with each line construct to fail I added code that reported the number of lines in the data buffer followed by the buffer contents. When eight lines were present the code ran as expected, however once in a while 7 lines were reported (despite there being eight) and the routine failed. Close inspection of the data revealed that normally the line ends were indicated by two Hex(0D) characters but every so often three Hex(0D) were present. When there were three Hex(0D) characters present the "with each line" construct hangs. It will restart after a period of time and all is well if the timer is canceled and the routine is recalled.
I have attached two text files. The first is recorded from the routine that reads the serial port, the second is the same data as read by the second routine but with the following "//////////////FLThe buffer contains 8 lines of data" inserted to indicate the buffer start and ends. You will notice that extra blank lines are present indicating the third hex(0D) character.
I tried to attache the two files but .txt files are not allowed! I have put them in a zip and displayed some different data from the two recording points below:
RECORDED BY THE SERIAL PORT READ ROUTINE
056,33*70
Simon,
is it correct that Hex 0D is 13 and that ascii 13 is your linedelimiter? ASCII 13 beeing a return?
Assuming this, your could set the "lineDelimiter" to that. Or you could clean the data with a replace before processing.
Just trying to understand the data structure. I suppose you do a binary read?
regards
Bernd
Simon,
could you make a binary file of your buffered data and zip it and uploaod it?
like this before Rev does anything to the data, i.e. in a field Rev changes the linedelimiter or when writing to a file Rev changes the lineendings to the lineendings of the platform.
All this is avoided by writing it as a binary file. This should do the trick:
on mouseUp
put specialfolderpath ("desktop") & "/" into aPath
put "BinaryExportPortData" after aPath
put myGlobalPortData into URL ("binfile:" & aPath)
end mouseUp