File commands stops working

[Simon Knight]

I am using LC 9.6.7. on an Apple Mac and am seeing issues with the files command. I suspect that its an issue with objects in a certain folder but as yet I have not been able to identify a cause. I wonder if others have encountered problems with the command?

The issue in detail

Code in a sub stack builds a list of folder names using regression (Code adapted from a handler published by Ken Ray some time ago):

Code: Select all

on GetListOfFolders whatFolder
   set the defaultFolder to whatFolder
   put whatFolder  & return after sListOfFolders
   put the folders into tDirList
   repeat with n = 2 to the number of lines of tDirList
      GetListOfFolders (whatFolder & "/" & (line n of tDirList))
   end repeat
end GetListOfFolders

This works and the script local variable sListOfFolders is populated.

Next another handler on the substack processes the list and adds details of certain file types to an array.

Code: Select all

function ReadRAWfileIFDdata pSelectedFolder
   ## The only function that is exposed to the application using this sub stack
   ##  Build an array of all the images in the folders/s
   put empty into sListOfFolders
   If there is a folder pSelectedFolder then
      ## Build a list of folders
      GetListOfFolders pSelectedFolder
      If the last char of sListOfFolders is cr then delete the last char of sLi
      repeat for each line tFolder in sListOfFolders
         put GetFileInformation (tFileInformationA, tFolder) into tFileInformationA
      end repeat
      put SortDataByCreationMinutes (tFileInformationA) into tImageFileDataA
      Return tImageFileDataA
   else
      answer "Invalid Folder path passed to EXIF reading sub-stack"
      Return "Error"
   end if
end ReadRAWfileIFDdata

Private function GetFileInformation @pFileInfoA, pFolder
   ##########################################################################
   #
   # Populates an array with information about the files stored in a folder 
   # and the first generation of child folders.
   #
   # pFileInfoA : Array of information, to which new information is added.
   # pFolder : the full path of the folder to be searched.
   ##########################################################################
   put the keys of pFileInfoA into tKeys
   put the number of lines of tKeys into tFileCounter

##******* The next line causes a problem ********##
   put files(pFolder,detailed) into tFiles
  

   repeat for each line tFile in tFiles
      --add one to tProgressCounter
      put item 1 of tFile into tFileName
      ## Filter out unwanted files by exiting loop and moving to next file
      If tFileName begins with "." then next repeat
      
      put GetFileExtn(tFileName) into tExtn
      If tExtn is "xmp" then next repeat
      
      -- Check that file is image file else next repeat
      if IsImageFile(tExtn) is False then next repeat
      
      ## process a valid image file 
      add one to tFileCounter
      put word 1 of tFileName into tNameAndExtn
      
      
      ## Read Meta data in two Tiff File tags from raw files
      ## Load the Raw File into a variable
      put pFolder & slash & tFileName into tNameRawFile
      
      ## Call handler chain that reads the specific Tiff File Tags
      ## Reads the tags in Image File Directory Zero of the image files
      put ReadIfdZeroTags (tNameRawFile,tExtn,sDelimiter) into tFileTagDataA
      /* Store the data decoded from the image file into the main array */
      repeat for each line tKey in the keys of tFileTagDataA
         put tFileTagDataA[tKey] into pFileInfoA[tFileCounter][tkey]
      end repeat
      
      put pFolder into pFileInfoA[tFileCounter]["Path"]
      
      put tFileName into pFileInfoA[tFileCounter]["FileName"]
      put GetFileExtn(tFileName) into pFileInfoA[tFileCounter]["extn"]
      put GetRootName(tFileName) into pFileInfoA[tFileCounter]["RootName"]
      
      put item 4 of tFile into tCreationSeconds
      put tCreationSeconds into pFileInfoA[tFileCounter]["CreationSeconds"] -- 1591869930
      
      put tFileTagDataA ["ByteCountJPEG"] into pFileInfoA[tFileCounter]["ByteCountJPEG"]
      put tFileTagDataA ["PtrToJPEG"]+1 into pFileInfoA[tFileCounter]["JPEGStartByte"]
      
      
      if tFileTagDataA ["ValidFileType"] then
         -- use the meta data read from the EXIF data
         put tFileTagDataA ["CaptureDate"] & sDelimiter & tFileTagDataA ["CaptureTime"] into pFileInfoA[tFileCounter]["DTG"]
         put tFileTagDataA ["CaptureDate"] into pFileInfoA[tFileCounter]["sqlDate"]
         put tFileTagDataA ["CaptureDate"] into pFileInfoA[tFileCounter]["NewFolderName"]  --! DUPLICATION
         put tFileTagDataA ["Orientation"] into pFileInfoA[tFileCounter]["Orientation"]
         put tFileTagDataA ["CaptureTime"] into pFileInfoA[tFileCounter]["CaptureTime"]
      else
         -- unable to read data from the file so use the filesystem data
         put GetDTG (tCreationSeconds, "DTG") into pFileInfoA[tFileCounter]["DTG"] -- YYYY-MM-DD-HHMMSS
         put GetDTG (tCreationSeconds, "txtDate") into pFileInfoA[tFileCounter]["txtDate"] -- Thursday, June 11, 2020
         put GetDTG (tCreationSeconds, "sqlDate") into pFileInfoA[tFileCounter]["sqlDate"] -- YYYY-MM-DD
         put GetDTG (tCreationSeconds, "NewFolderName") into pFileInfoA[tFileCounter]["NewFolderName"] -- YYYY-MM-DD
         --put GetDTG (tCreationSeconds, "HMS") into pFileInfoA[tFileCounter]["CaptureTime"] -- HH:MM:SS
      end if
      --put GetDTG (tCreationSeconds, "MINS") into pFileInfoA[tFileCounter]["Minutes"] -- Time in minutes of day
      
      --put (tCreationSeconds DIV 60) into pFileInfoA[tFileCounter]["CreationMinutes"]
      put ConvertDate (pFileInfoA[tFileCounter]["sqlDate"]) into pFileInfoA[tFileCounter]["DisplayDate"] -- DD/MM/YYYY
      
      # Note the examples assume that a hyphen is selected as the separator character
      put "Array count is : " & tFileCounter && " Folder : " & pFolder  & CR after field "debug" of card "DateStampAndCopyFiles" of stack "DropZoneRename"
   end repeat
   
   return pFileInfoA
end GetFileInformation

The logic appears sound and the array does get populated. However at some point the command

Code: Select all

put files(pFolder,detailed) into tFiles

stops returning a list of files and the IDE while allowing edits and test runs is no longer able to save the stacks. Whan a save is attempted it posts a warning that it is unable to open stack . The only solution I have found is to quite Livecode and restart.

Screen shot of a break point just after the files command. Note that tFiles is empty and the folder is named 2021-06-13

Screen shot of the folder 2021-06-13 as displayed in the Finder

Stopping the debug run and attempting to save the stacks results in this error message:

It appears that the failed command has killed all file operations in the IDE.

I am checking the contents of the folders to see if there is anything odd about them.

Any thoughts ?

[stam]

Hi Simon - do you mean that you are wanting to recursively files within subfolders of a given directory to create a list of files of a certain type?

[Simon Knight]

Hi Stam,

do you mean that you are wanting to recursively files within subfolders of a given directory to create a list of files of a certain type

Yes, I want to list image files.

However I have discovered that the fault lies elsewhere as the code processes all the files when I disable some of the data reading handlers. The interesting thing is that an error is not being thrown.

I'll post my findings if I find what is going on.

[stam]

I didn't really go through your code, but in case there's an issue with generating full file paths for each each file, maybe this will be of help:

I had to solve this some time ago - recursively delve into a folder and its subfolders to create a list of absolute filepaths for each individual file matching a certain text pattern such as an extension.

Code: Select all

function recursiveFilesWithPattern pFolder, pPattern
     Local tPaths
     filter files(pFolder) with pPattern
     repeat for each line tFile in it
          wait 0 with messages
          put pFolder & slash & tFile & cr after tPaths
     end repeat
     filter folders(pFolder) without ".."
     repeat for each line tFolder in it
          wait 0 with messages
          put recursiveFilesWithPattern(pFolder & slash & tFolder, pPattern) after tPaths
     end repeat
     if the last char of tPaths is return then delete the last char of tPaths
     return tPaths
end recursiveFilesWithPattern

This will give a return delimited list of the absolute filepaths for each file matching the given pattern, recursing through all subfolders.
I guess for your purposes you could just pass "*.*" as the pattern to get all files, or change the filter line to check for multiple filetypes etc.

Let me know if you find this helpful -- or not
Stam

[Simon Knight]

Problem solved.

The bug has been in my code for some years but has become apparent when trying to process large numbers of files. The code reads bytes from files and uses toe open file command. Unfortunately I had omitted a close file command meaning that at some point the file system stopped working.

Code: Select all

Private function ReadFileBytes pfileName, pStartByte, pByteCount, pReadType
   /* pfileName is the full path to a file
   LowByte and HighByte are integers specifying bytes requied
   Note: codes adds a byte offset to account for JPEG files.
   N.B. This handler only decoded bytes or ASCII it is up to the calling 
   handler to interpret the data that is read
   */
   put 0 into tOffset
   set itemdel to "."
   put the last item of pFileName into tExtn
   If tExtn is "jpg" then put 12 into tOffset
   If tExtn is "CR3" then put 320 into tOffset
   
   if there is a file pfilename then
      Open File pfileName for binary read
      Switch pReadType 
         Case "uInt1"
            read from file pfileName at (pStartByte+tOffset) for pByteCount uInt1
            /* Bytes are decoded to decimal and placed in varaible as comma sep data */
            break
         Case "ASCII"
            read from file pfileName at (pStartByte+tOffset) for pByteCount
            break
      end Switch
      --A simple fix!!!!!!!! :
      Close File pfileName 
   else
      answer "An error has occurred in handler ReadFileBytes attempting to read " && pFileName
   end if
   
   return it
end ReadFileBytes

[FourthWorld]

Also check folder permissions.

All I/O needs error-checking, seeing if "the result" is empty and calling sysError() if not to get the OS error code that will tell you exactly what happened.

If permissions were not the issue here, sooner or later they will be. I've seen this with directory walking many times over many years, head-scratching over recursion because the walker is unable to keep walking due to permissions.

[Simon Knight]

Great tip, thanks. I'll add some error checking.