FTP download?

[adeam]

Last week, I bought a license of LiveCode 4.5.2 which I would like to test for my first project. My project is to create a download utility from our ftp. I don't know where to start, but basically, I would like to download all the files/folders (including files in the subfolders) in our FTP site on specific path and save them to my C: drive, maintaining the folder structure. If the file to be downloaded exists on the target folder, compare their file creation date, if they are the same, skip, otherwise copy. I think the difficult part here, how can we download the file and keep their original creation date? Because, I believe, when you download, it changes the file's creation date with current date. I would like to have the output for Windows and Mac.

Shown below the summary of the requirements:
• Read the files and folders recursively on our ftp site
• Download the files and keep their original creation date
• Get file's creation date, both in FTP and local file, and compare
• Should work on Windows and Mac

Hope you can help me.

Thanks.

[Klaus]

Hi adeam,

please check all terms in the dictionary for further info:

• Read the files and folders recursively on our ftp site

You can use
libURLSetFTPListCommand "NLST"
to get a file and folder listing for a give ftp URL.

• Download the files and keep their original creation date

Read the creation date from the file on the ftp server and after download with
e.g. libURLDownloadToFile
you could use shell("touch...")(on a Mac, no idea about Windows) to change the creation date to the one on the ftp server.

• Get file's creation date, both in FTP and local file, and compare

See above and use:
the long files
to get a listing of your local files including creation/modification dates

Except the shell command "touch" this will work on Mac and Win!


Best

Klaus

[adeam]

Thanks a lot for your help, Klaus.

I will try to use the commands you've mentionned, it may take some time for me to do it as I am still new with LiveCode but hope will complete it. BTW, I've already made this program but using other Windows program (no output to Mac) and I've also used the touch.exe from http://www.touchdotexe.com. It's good to know there's a mac version of touch as well.

In case some more problem, I'll post them here.

Regards.

[Klaus]

Hi adeam,

you're welcome!

Hint:
On OS X, the shell command "touch" is already built into the OS!
Like many other nice and helpful thingies!


Best

Klaus

[adeam]

Thanks Klaus.

[FourthWorld]

If you plan to support drag-and-drop I'd be interested in learning how you derive paths to items dragged from your window to the desktop.

I have a request for this at http://quality.runrev.com/qacenter/show_bug.cgi?id=8634 but haven't been able to find a way to handle that in the meantime.

[adeam]

Hi FourthWorld, at the moment, I have no plan to support drag-and-drop ...

Klaus, I'm almost complete with the ftp download, that is to read the files and folders recursively, extract, compare dates and download the file and then update the modification date by touch.exe.

During my debug, the flow is correct for each download but when I run without debugging, it can't update the modification date because it runs asynchronously (the file is not yet completely downloaded and it starts updating the file's modification date) .

Is there a command to make the download synchronously? Or do you have a sample on how to trap if it downloaded the file completely, then run the update on modification date?

Thanks.

[Klaus]

Hi adeam,

did you check "libUrlDownLoadToFile" in the dictionary?

You can define a "callback" message, in which you can check the state of the download
and only start to "touch" when the file has finished downloading.
Sorry, no script at hand at the moment.

Out of my head it is somehting like this:
Both of these handlers have to be in the same script (card/button/whatever)

Code: Select all

on mouseup
  libURLDownloadToFile "ftp://...","Path_to_file_on_harddisk...","downloadComplete"
end mouseup

command downloadComplete the_loaded_url,tDownloadstatus
  switch tDownloadstatus
   case "error"
   case "timeout"
    ## etc check "UrlStatus" in the doctionary for more info
     answer "Damn, not works!"
     exit to top
  break
  case "cached"
    ## file has been successfully downloaded
    put url the_loaded_url into url("binfile:" & path_on_harddisk)
   ## or "file:..." for text files
   ##  TOUCH NOW and HERE! :)
   break
  end switch
end downloadComplete

You get the picture


Best

Klaus

[adeam]

Thanks Klaus for the code.

I've inserted it into my code, run, then it gave me the same result as before, it downloaded all the files but didn't update the modification date. When I trace the value of tDownloadstatus, it contains "downloaded" (not "cached"), so I changed it, see code below. Run it again, still, it doesn't work. Perhaps the variables inside the function "downloadComplete" were empty? If it is so, how do we pass the variables? Or does it work on a loop?

Excuse me for my ignorance, I also don't understand the line "put url the_loaded_url into url("binfile:" & path_on_harddisk)", because even I commented it, the files were downloaded. Do I need to keep it?

Code: Select all

on mouseup
  put "C:/Temp/LiveCode/touch.exe " into winTouch
  -- read all the files from ftp and put it into field "ftpFileList"
  repeat for each line tLine in field "ftpFileList"
     -- get the values of myFtpFileName, myLocalFileName, timestamp 
     libURLDownloadToFile myFtpFileName, myLocalFileName, "downloadComplete"
  end repeat
  answer "Download completed."
end mouseUp

command downloadComplete the_loaded_url,tDownloadstatus
  switch tDownloadstatus
   case "error"
   case "timeout"
     ## etc check "UrlStatus" in the doctionary for more info
     answer "Damn, not works!"
     exit to top
     break
  case "downloaded"    --"cached"
      ## file has been successfully downloaded
      put url the_loaded_url into url("binfile:" & path_on_harddisk)
      ## or "file:..." for text files
      ## TOUCH NOW and HERE! :)

      launch winTouch & timestamp & " " & myLocalFileName
      break
  end switch
end downloadComplete

[Klaus]

Hi adeam,

you might need to set upo and use local variables, that can be accessed in both handlers in the same script!

Code: Select all

local winTouch
local tFileList
local tTimeStamp

on mouseup
  put "C:/Temp/LiveCode/touch.exe " into winTouch
  -- read all the files from ftp and put it into field "ftpFileList"

## SEE BELOW!!!
## Now set up a variable containing_
##  the ftp filenname TAB the local file name
## file 2 on ftp TAB file 2 on disk
## etc... 
## We name this var tFileList
 put ftpFIle1 & TAB & lcaolFile1 into tFileList
 put CR & ftpFile2 & TAB & localFile2 after tFileList
  repeat for each line tLine in field "ftpFileList"
     -- get the values of myFtpFileName, myLocalFileName, timestamp 
     libURLDownloadToFile myFtpFileName, myLocalFileName, "downloadComplete"
   ###!!! This will save all downloaded files into the SAME file on your harddisk: myLocalFileName!!!!!
   ### You will want to manage this manually somehow!
  end repeat
##  answer "Download completed."
## We are not ready yet!!!
answer "Download has started!"
end mouseUp

command downloadComplete the_loaded_url,tDownloadstatus
  switch tDownloadstatus
   case "error"
   case "timeout"
     ## etc check "UrlStatus" in the doctionary for more info
     answer "Damn, not works!"
     exit to top
     break
  case "downloaded"    
  case "cached"
      ## file has been successfully downloaded
     ##  put url the_loaded_url into url("binfile:" & path_on_harddisk)

   ## Sorry, my fault, I mixed this somehow witht he "load" command!
   ### "the_loaded_url" = "myFtpFileName" in the other handler!
   ### "path_on_harddisk" = myLocalFileName the other handler!
   ## The file(s) have already been saved to hadrddisk, sou you shoud keep track of them,
   ## maybe in another variable in this order see above: 
   ### FTPFile TAB local filename
   ## This way you could get the according local file with a give url like in this case

    set itemdel to TAB
    put item 2 of line (lineoffset(the_loaded_url,tFileList) into myLocalFileName

      ## or "file:..." for text files
      ## TOUCH NOW and HERE! :)

      launch winTouch & tTimeStamp & " " & myLocalFileName

    ### !!!! tTimeStamp still needs to be initialised somehwere!
      break
  end switch
end downloadComplete

Get the picture?


Best

Klaus

[adeam]

Thanks again Klaus, I will try your suggestion later. I'm a bit confused about putting all the code under the case "cached" and not under "downloaded". Is it correct?

Anyhow, I was also looking on the command "wait until the status is downloaded", see code below. But it was hanging. I was trying to make the code synchronous because when I'm debugging it, step by step, it was working but when I run it w/o debug, it hangs. What's wrong with the wait command below?

Code: Select all

on mouseup
  put "C:/Temp/LiveCode/touch.exe " into winTouch
  -- read all the files from ftp and put it into field "ftpFileList"

  -- now, read all the files to be downloaded below
  repeat for each line tLine in field "ftpFileList"
     -- get the values of myFtpFileName, myLocalFileName, timestamp 

     libURLDownloadToFile myFtpFileName, myLocalFileName, "downloadComplete"
     
     wait until URLStatus(myFtpFileName) = "downloaded"

     launch winTouch & timestamp & " " & myLocalFileName

  end repeat
  answer "Download completed."
end mouseUp

command downloadComplete the_loaded_url,tDownloadstatus
  switch tDownloadstatus
   case "error"
   case "timeout"
     ## etc check "UrlStatus" in the doctionary for more info
     answer "Damn, not works!"
     exit to top
     break
  case "downloaded"    --"cached"
      ## file has been successfully downloaded
      ##put url the_loaded_url into url("binfile:" & path_on_harddisk)
      ## or "file:..." for text files
      ## TOUCH NOW and HERE! :)

      break
  end switch
end downloadComplete

[Klaus]

Hi adeam,

plase check my script, it handles "cached" in the same way as "downloaded"!
Check "swithc" in the dictionary!

NEVER EVER do, won't work and give you probably an endless loop!:
...
wait until URLStatus(myFtpFileName) = "downloaded"
...
Check the dictionary for "libUrlDownloadToFile" or ""urlstatus"!

This is the way the libURL works!
Load a file to your harddisk and get a noticication (callback message) after the download is finished.
THEN you can do whatever you want with the loaded file(s)!

And for my solution you need a "reference" sheet where you noted the FTPFile and LOCALFilepath pairs
(one pair per line, delimited with TAB,) so you can check where a FTP has been saved to file to let it be "touched"!


Best

Klaus

[adeam]

Thanks again Klaus. I will do it after I finished some other urgent work. Hopefully, in 2 days... i'll come back again.
Cheers.

[adeam]

Hi Klaus,

I've implemented your code and it's now working - downloading then afterwards, update the file's modification date. Thanks.

Now, I don't know where to put the command - answer "Download completed!"? Is there a command or function when the callback function ends? We have noted that "Download has started!" but where to put "Download completed!"? Thanks again.

BTW, I used comma for itemDel as it was used in the previous lines.

Code: Select all

local winTouch
local tFileList
local tTimeStamp

on mouseup
  put "C:/Temp/LiveCode/touch.exe " into winTouch
  -- read all the files from ftp and put it into field "ftpFileList"

## SEE BELOW!!!
## Now set up a variable containing_
##  the ftp filenname TAB the local file name
## file 2 on ftp TAB file 2 on disk
## etc... 
## We name this var tFileList
put ftpFIle1 & TAB & lcaolFile1 into tFileList
put CR & ftpFile2 & TAB & localFile2 after tFileList
  repeat for each line tLine in field "ftpFileList"
     -- get the values of myFtpFileName, myLocalFileName, timestamp 

     put tFileList & myFtpFileName & "," & myLocalFileName & "," & tTimeStamp & CR into tFileList

     libURLDownloadToFile myFtpFileName, myLocalFileName, "downloadComplete"

   ###!!! This will save all downloaded files into the SAME file on your harddisk: myLocalFileName!!!!!
   ### You will want to manage this manually somehow!

  end repeat

  ##  answer "Download completed."
  ## We are not ready yet!!!
  answer "Download has started!"
end mouseUp

command downloadComplete the_loaded_url,tDownloadstatus
  switch tDownloadstatus
   case "error"
   case "timeout"
     ## etc check "UrlStatus" in the doctionary for more info
     answer "Damn, not works!"
     exit to top
     break
  case "downloaded"    
  case "cached"
      ## file has been successfully downloaded
     ##  put url the_loaded_url into url("binfile:" & path_on_harddisk)

   ## Sorry, my fault, I mixed this somehow witht he "load" command!
   ### "the_loaded_url" = "myFtpFileName" in the other handler!
   ### "path_on_harddisk" = myLocalFileName the other handler!
   ## The file(s) have already been saved to hadrddisk, sou you shoud keep track of them,
   ## maybe in another variable in this order see above: 
   ### FTPFile TAB local filename
   ## This way you could get the according local file with a give url like in this case

    --set itemdel to TAB
      put item 2 of (line (lineoffset(the_loaded_url, tFileList)) of tFileList) into tLocalFileName
      put item 3 of (line (lineoffset(the_loaded_url, tFileList)) of tFileList) into tTimeStamp 
      
   ## TOUCH NOW and HERE! :)

      launch winTouch & tTimeStamp & " " & tLocalFileName

      break

  end switch
end downloadComplete

Regards.

[Klaus]

Hi adeam,

out of my head:
You could delete the line(s) with the already downloaded and touched files,
so you can check if that list is empty, then the download is complete.

Code: Select all

...
    put lineoffset(the_loaded_url, tFileList) into tLine
     put item 2 of line tLine of tFileList into tLocalFileName
     put item 3 of line tLine) of tFileList into tTimeStamp      
     ## TOUCH NOW and HERE! :)
      launch winTouch & tTimeStamp & " " & tLocalFileName

    ## Now delete all handeld line(s)
     delete line tLine of tFileList
     if tFilelist = empty then
         answer "Dwonload complete"
     end if
     break
  end switch
end downloadComplete

Best

Klaus