Refactoring a recursive handler

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Refactoring a recursive handler

Post by Simon Knight » Thu Apr 07, 2022 9:01 am

Hi,
I struggle with recursion and have a recursive handler, that Ken Ray of Sons of Thunder published, that I would like to refactor. At present the handler works well but it saves data to a script local variable which I access via a second function. I would like to convert the handler to a function but my own attempts have given odd results so I have binned them. Here is the original script along with two helper functions.

Code: Select all

local sListOfFolders

on GetListOfFolders whatFolder
   set the defaultFolder to whatFolder
   put whatFolder  & return after sListOfFolders
   put the folders into tDirList
   repeat with n = 2 to the number of lines of tDirList
      GetListOfFolders (whatFolder & "/" & (line n of tDirList))
   end repeat
end GetListOfFolders

Function FoldersFound
   If the last char of sListOfFolders is cr then delete the last char of sListOfFolders
   return  sListOfFolders
end FoldersFound

On ZeroFoldersFound
   put empty into sListOfFolders
end ZeroFoldersFound
Any ideas?
Thanks for reading.
best wishes
Skids

stam
Posts: 2634
Joined: Sun Jun 04, 2006 9:39 pm
Location: London, UK

Re: Refactoring a recursive handler

Post by stam » Thu Apr 07, 2022 9:48 am

The question is, as always, what is your goal?

Are you out to simply create a list of folders?
Or is this ultimately to create a list of files within the chosen folder and all its subfolders?

If you want a function to do the latter, you can use what i posted in an earlier post of yours.

The function below will create a list of all absolute paths to each file within the chosen folder and all its subfolders - with the the option to filter the file list for certain string patterns, such as file extensions, but which you can simply pass as a wildcard "*" to list all files:

Code: Select all

function recursiveFilesWithPattern pFolder, pPattern
     Local tPaths
     filter files(pFolder) with pPattern
     repeat for each line tFile in it
          put pFolder & slash & tFile & cr after tPaths
     end repeat
     filter folders(pFolder) without ".."
     repeat for each line tFolder in it
          put recursiveFilesWithPattern(pFolder & slash & tFolder, pPattern) after tPaths
     end repeat
     if the last char of tPaths is return then delete the last char of tPaths
     return tPaths
end recursiveFilesWithPattern
If on the other hand your goal is purely to create a list of folders the above may serve as a starting point; i don't have the to do this now but can do at some point if you've got nowhere.

HTH
Stam

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Thu Apr 07, 2022 9:51 am

OK so I decided I should not just give up. I think the following works :

Code: Select all

on mouseUp pButtonNumber
   answer folder "Select a folder." as sheet
   if the result is "Cancel" then exit  mouseUp
   put it into tSelectedFolder
   put empty into tList
   put FolderList (tSelectedFolder,tList) into tList
end mouseUp

Function FolderList pTopFolder,pList
   put pTopFolder  & return after tList
   put folders (pTopFolder) after tSubFolders
   repeat with n = 2 to the number of lines of tSubFolders
      put FolderList (pTopFolder & slash & (line n of tSubFolders),tList) after tList
   end repeat
   return tList
end FolderList
The found the solution by trial and error as I still find that recursion is really confusing.
best wishes
Skids

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Thu Apr 07, 2022 9:53 am

Hi Stam,

Our posts crossed. However you are correct in the end I need to generate a list of files so I will take a close look at your code.
best wishes
Skids

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Thu Apr 07, 2022 10:09 am

Looking at Stam's code and my own attempt it looks that the way to think about recursion is bottom up. Both codes work down to the lowest branch and return a single name. This gets past back up the chain and is added to the list as it rises.

#Stam - my parent application requires a list of all image files and bases its filter on over two hundred file extensions which are stored as part of the application. I could put the filter within your code but I think I would prefer to keep it as a general function that can be used in any appliction and apply the filter on the list that gets returned. Its probably worth mentioning that I am working with up to 100,000 image files so I am being very careful about how I access and modify them.

Thanks
S
best wishes
Skids

bn
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 3990
Joined: Sun Jan 07, 2007 9:12 pm
Location: Bochum, Germany

Re: Refactoring a recursive handler

Post by bn » Thu Apr 07, 2022 11:11 am

Hi Simon,

here is some recycled code of mine to get at the file paths of a folder and all its subfolders

Code: Select all

on mouseUp
   local tMyFiles, tStartFolder, tNumFiles
   answer folder "where to start"
   if it is empty then exit mouseUp
   put it & slash into tStartFolder
   put getAllFolders(tStartFolder) into tMyFiles
   put the number of lines of tMyFiles into tNumFiles
   put cr & tNumfiles && "Files" after msg
end mouseUp

function getAllFolders, pStartFolder
   local tOldDefault, tFolderRootSearch, tTime, tCollector, tAllFiles, doAllTheseFiles, tFolderArray, tFoldersToSearch
   put the defaultfolder into tOldDefault
   put pStartFolder into tFolderRootSearch
   
   put the milliseconds into tTime
   
   -- begin of search for subfolders
   put "" into tFolderArray
   put 1 into tFolderArray[tFolderRootSearch]
   
   local tLookForMoreFolders, tFoldersTemp, tSubTemp
   
   repeat -- will exit when no more folders are found
      put "" into tFoldersToSearch
      
      put the keys of tFolderArray into tLookForMoreFolders
      repeat for each line aFolder in tLookForMoreFolders
         if tFolderArray[aFolder] is not "" then put aFolder & cr after tFoldersToSearch
         put "" into tFolderArray[aFolder] -- so next time this path will not be searched
      end repeat
      
      if tFoldersToSearch is "" then exit repeat -- no folder found that has not been looked at
      
      repeat for each line aFolder in tFoldersToSearch
         set the defaultfolder to aFolder
         put "" into tFolderArray[aFolder] -- so next time this path will not be searched
         put the folders into tFoldersTemp
         
         -- filter out operating system folders on mac (like "..", or starting with "." to make it invisible)
         filter tFoldersTemp without ".*"
         -- filter without application folders, on the mac application may be pakages
         filter tFoldersTemp without "*.app"
         
         if tFoldersTemp is "" then next repeat
         
         repeat for each line aSubFolder in tFoldersTemp
            put aFolder & aSubFolder & "/" into tSubTemp
            put 1 into tFolderArray[tSubTemp] -- next time this path will be searched, at the same time the path is a new key in the array
         end repeat
      end repeat
   end repeat
   
   put the keys of tFolderArray into tCollector
   sort tCollector
   -- end of finding all subfolders
   -- tCollector contains now the path to all subfolders including the rootfolder
   -- now get all the files and the filepaths
   
   repeat for each line thePath in tCollector
      set the defaultfolder to thePath
      put the files into tAllFiles
      filter tAllFiles without ".*"
      if tAllFiles is not "" then
         repeat for each line aFile in tAllFiles
            put thePath & aFile & return after doAllTheseFiles
         end repeat
      end if
   end repeat
   delete last char of doAllTheseFiles -- return
   set the defaultFolder to tOldDefault
   put the milliseconds - tTime && "ms"
   return doAllTheseFiles
end getAllFolders
It took about 2 minutes to find all the 649713 files for my user account. That is a very fast SSD on a new MacBook Pro.
It took about 8 seconds to find 75780 files in a large directory.

I did not test a recursive version. In my experience recursive code is a tad slower than linear with repeat.
Although I used array which tend to get slow above 40 - 50 k entries.

Recurse and the world curses with you :)

Kind regards
Bernd

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Refactoring a recursive handler

Post by FourthWorld » Thu Apr 07, 2022 11:23 am

I/O strongly benefits from error checking.

The recursion with common directory walkers is often caused by code that predates OSX and doesn't account for permissions:

viewtopic.php?f=104&t=36903#p214210
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

stam
Posts: 2634
Joined: Sun Jun 04, 2006 9:39 pm
Location: London, UK

Re: Refactoring a recursive handler

Post by stam » Thu Apr 07, 2022 1:23 pm

Thanks Richard - you are undoubtedly right... but does this apply to user-level stuff?

How common is it that a user will not have read access to his/her files and folders?
I understand write access can be weird, but i would expect read access to be almost a non-issue on mac/win at least... am i very wrong?

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Thu Apr 07, 2022 2:35 pm

# Stam

I tried your code and there is a bug somewhere as every so often two file paths are listed on the same line.

Here is a short extract from part of the list returned from the recursive script, the error appears to occur when the folder path changes:

Code: Select all

/Volumes/Image_Library_SSD/Photo-Library/2022/2022-02-12/2022-02-12-162128-P1072190-Salli-Gainford-Photo-Shoot-Dancer-DCG9-.RW2.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-02-12/2022-02-12-170710-P1072196-Salli-Gainford-Photo-Shoot-Dancer-DCG9-.RW2.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-02-12/2022-02-12-143111-P1072144-Salli-Gainford-Photo-Shoot-Dancer-DCG9-.RW2.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-02-12/2022-02-12-160911-P1072184-Salli-Gainford-Photo-Shoot-Dancer-DCG9-.RW2.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-02-12/2022-02-12-120356-P1071871-Salli-Gainford-Photo-Shoot-Dancer-DCG9-.RW2.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-02-12/2022-02-12-154305-P1072152-Salli-Gainford-Photo-Shoot-Dancer-DCG9-.RW2.dop/Volumes/Image_Library_SSD/Photo-Library/2022/2022-03-15/2022-03-15-132912-RX000028-Dredger-Barge-River-Trent-Gunthorpe-Boat-.JPG.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-03-15/2022-03-15-133137-RX000035-Landscape-Heckdyke-Farm-Building-Barn-.DNG.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-03-15/2022-03-15-131516-RX000014-Natural-History-Flower-.JPG.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-03-15/2022-03-15-131355-RX000010-Natural-History-Ivy-.DNG.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-03-15/2022-03-15-133051-RX000033-Sluice-River-Trent-Heckdyke-Landscape-.JPG.dop
/Volumes/Image_Library_SSD/Photo-Library/2022/2022-03-15/2022-03-15-132459-RX000019-River-Trent-Bank-Gunthorpe-Landscape-.JPG.dop
# Bernd
Hi and thanks for your code. Based on my experience with attempting to understand and debug recursive handlers I think I will stick with loops especially if there is no time penalty. There seems little point in producing code that is difficult to understand, debug and that may be slower than the simple stuff.

#ForthWorld
I hear and will heed your warning. It is not just permissions that cause issues as I have struggled with characters in file names and the length allowed in path names and filenames and that is just on Mac OS. I think pathnames may be several thousand characters long with filenames limited to 240 characters but I have not found any definitive documentation. As to permissions not even Apple seem to understand them - for example who or what is "wheel" .

My present "policy" is to only use ascii characters a to Z plus the hyphen and I am experimenting with the following when using URL commands

Code: Select all

function EncodeFilePath pFilePath
   put the URLEncode of pFilePath  into tURLEdncodeFilePath
   
   replace "+" with "%20" in tURLEdncodeFilePath  
   replace "|" with "%7C" in tURLEdncodeFilePath  

   return tURLEdncodeFilePath  
end encodeFilePath
which is a simplified version of code in the dictionary :

Code: Select all

function urlEncodeRFC pString
    if pString is strictly a string then
        put textEncode(pString,"UTF-8") into pString
    end if
    put URLEncode(pString) into pString
    replace "+" with "%20" in pString
    return pString
end urlEncodeRFC
I'm not sure when I need to use UTF-8 encoding, possibly when dealing with remote servers.

best wishes
S
best wishes
Skids

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Refactoring a recursive handler

Post by FourthWorld » Thu Apr 07, 2022 5:08 pm

stam wrote:
Thu Apr 07, 2022 1:23 pm
Thanks Richard - you are undoubtedly right... but does this apply to user-level stuff?

How common is it that a user will not have read access to his/her files and folders?
I understand write access can be weird, but i would expect read access to be almost a non-issue on mac/win at least... am i very wrong?
If it's failing, it's failing.

I've seen many cases where permissions were the problem.

But whatever the cause, LiveCode and the OS want to tell us what it is.

Checking "the result" will let LiveCode tell you that an issue happened, and including a call to sysError() in any error-reporting output will let the OS give you the precise error code telling you what it is.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Thu Apr 07, 2022 5:18 pm

Richard wrote :

"Checking "the result" will let LiveCode tell you that an issue happened"

Which commands do you have in mind ? For example I am about to use the "rename" command to change the name of files on a single volume. The file path/name have been gained using the "Files" command. I wonder if it is safe to assume (what am I asking!) that the text names from the Files command is compatible with the "Rename" command? Where as I know that the list is not compatible with the URL commands as spaces and other legal characters cause problems. (The dictionary makes no mention of "the result" of a rename command.)

best wishes
S
best wishes
Skids

stam
Posts: 2634
Joined: Sun Jun 04, 2006 9:39 pm
Location: London, UK

Re: Refactoring a recursive handler

Post by stam » Thu Apr 07, 2022 7:52 pm

Simon Knight wrote:
Thu Apr 07, 2022 2:35 pm
# Stam

I tried your code and there is a bug somewhere as every so often two file paths are listed on the same line.
This is what happens when I try to be too clever lol... the bug is the final line getting rid of the extra CR at the end of the list.

I added that to a working version knowing that it produces and empty row at the end, but that then causes issues when called recursively (although I’m not really clear what the issue is!). Removing that fixes the issue.

This works:

Code: Select all

function recursiveFilesWithPattern pFolder, pPattern
     Local tPaths
     filter files(pFolder) with pPattern
     repeat for each line tFile in it
          put pFolder & slash & tFile & cr after tPaths
     end repeat
     filter folders(pFolder) without ".."
     repeat for each line tFolder in it
          put recursiveFilesWithPattern(pFolder & slash & tFolder, pPattern) after tPaths
     end repeat
     return tPaths
end recursiveFilesWithPattern
Don't you just love recursive functions...
Last edited by stam on Thu Apr 07, 2022 9:23 pm, edited 2 times in total.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Refactoring a recursive handler

Post by FourthWorld » Thu Apr 07, 2022 9:15 pm

Simon Knight wrote:
Thu Apr 07, 2022 5:18 pm
Richard wrote :

"Checking "the result" will let LiveCode tell you that an issue happened"

Which commands do you have in mind ?
The first instance of reaching out to the file system for any file system object, such as a file or folder.

In this case, I'd check "the result" after the calls to "the files" and "the folders" functions.
For example I am about to use the "rename" command to change the name of files on a single volume. The file path/name have been gained using the "Files" command. I wonder if it is safe to assume (what am I asking!) that the text names from the Files command is compatible with the "Rename" command?
I'm a big fan of resolving any known issues before introducing other potential issues.

Once you've got this directory walker working well, you'll already know that the list you have is for files that exist and in folders you can access.

It wouldn't hurt to check "the result" after a rename, and may be useful to account for potential issues where the file exists and is accessible but cannot be renamed.

Where as I know that the list is not compatible with the URL commands as spaces and other legal characters cause problems.
Is it? The directory listing functions should be providing their list in urlEncoded format. At least that's true with the "detailed" form. If you anticipate problems with special chars in names it may be useful to use the "detailed" form to ensure urlEncoded return values.
(The dictionary makes no mention of "the result" of a rename command.)
If the method by which the engine reports errors when attempting to rename a file isn't listed in the Dictionary entry for the "rename" command, that may simply be an oversight.

It's hard to imagine any I/O operation for which the engine provides no means for error checking.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Fri Apr 08, 2022 8:21 am

Re: using the URL command. In the past I have encountered problems with spaces and certain other characters e.g. the vertical bar in filenames. I don't remember all the details but from memory the "Files" command returns file names that includes space characters. If these are used elsewhere, I think with the URL command the spaces are replaced with plus characters and the file is not found. This is where my code snip originates :

Code: Select all

function EncodeFilePath pFilePath
   put the URLEncode of pFilePath  into tURLEdncodeFilePath
   
   replace "+" with "%20" in tURLEdncodeFilePath  
   replace "|" with "%7C" in tURLEdncodeFilePath  

   return tURLEdncodeFilePath  
end encodeFilePath
Having written the above I am uncertain of the exact details so will create a test stack and report back as my notes just say the handler should be used with the browser object.
best wishes
Skids

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Refactoring a recursive handler

Post by Simon Knight » Fri Apr 08, 2022 10:48 am

Well I have been unable to break either the Rename or URL commands; yet only the other day I'm sure I saw an error generated by filenames with multiple + characters in the name as listed by livecode.

S
best wishes
Skids

Post Reply

Return to “Talking LiveCode”