Reading file and folder dates

[marksmithhfx]

I was also thinking of something even more basic:

What if the first time a file was saved it was ”Data0000000000000001.dat” and then each further save from any source just increments that by 1. You don’t really need to care about the date or time so long as you know that you have the newest master data.

Hi Kelly, I really gave this a lot of thought and even implemented a solution using actual dates converted into seconds, but for several reasons found it less than ideal.

1. You need to constantly rename files (either with the date as seconds or a sequential number).

2. That means you also have to keep track of old file names and delete them when they become outdated ie. when you upload mydata5 you need to delete mydata4. Using date as seconds in the filename is even trickier.

Then I hit on a LC method of converting local time to zulu time (used by dropbox) and decided to go with that solution for 2 reasons:

1. I don't have to keep track of the sequencing -- the clock does it for me.

2. I don't have to rename any files (except the backup file I make) or delete older ones.

I'll be posting a full solution using file dates and a single naming scheme in the next few messages.

But many thanks for the suggestion. It was definitely worth looking into and probably taught me a lot about this file syncing challenge.

Mark

[marksmithhfx]

Dropbox data file syncing protocol — Part 1

Caveats - designed for synching data across a single users iOS devices (not multi-user)

Req’s: a dropbox account (free), an “app” on dropBox (see https://www.dropbox.com/developers/refe ... oper-guide). (An “app” in this case is just a designated folder on dropbox for your app. However, there is a special process to setup this folder), an “access token” to access your “app” folder (you create this in your “app console” on dropbox). See also the DEMO_DROPBOX_EXAMPLE from Bangkok which really helped me get started moving data to/from my dropbox “app” folder. It’s a great demo stack. viewtopic.php?f=11&t=33901&p=206382&hil ... ry#p206382

Once you have your access key you are good to go. For convenience I stored it as a custom property in my app.

To implement the file swapping protocol (so each device always has the latest copy of the data) you will need to know 3 things:

1. The last modified file date on dropbox
2. The last modified file date of your local copy.
3. A protocol for comparing these dates (at the correct time) to determine whether files should be uploaded or downloaded to/from dropbox.

Keep in mind, dropbox files are stored using Zulu time (sometimes referred to as UTC or GMT but adjusted for seasonality). You can type “zulu time” into Google to see what the current Zulu time is in your area. LC provides a generic solution for converting to/from Zulu time which I document below.

You will need to include the following LC libraries:

JSON Library
Remote Debugger (recommended during testing)
Dropbox Library
mergBgTask — for background processing
tsNet — to capture and respond to network traffic errors
mergNotify — to know when your app is suspended and resumed

The basic file transfer protocol is this: when your app is suspended, upload the latest copy of the app data. When the app is resumed, check the dropbox file date and see if it is more recent than your local copy. If it is, download it replacing the local copy. Also, when your app is started (after being shut down) treat it as resuming and check dropbox for updates.

So in summary, there is only 1 time you need to be concerned about moving data from local storage to dropbox: that is when the app is suspended or quit (but we only have to respond to “suspended” since you have to suspend an app to quit it.). There are 2 times where you need to check the file on dropbox to see if it needs downloading: when the app is started, and when the app is resumed after being suspended.

Code examples for doing this are presented in parts 2, 3 and 4 of this thread.

[marksmithhfx]

Dropbox data file syncing protocol — Part 2 (see Part 1 for an introduction)

1. Here is how I setup my app to receive notification that it is being suspended (or resumed):

Code: Select all

on openStack
   mergNotify "UIApplicationWillResignActiveNotification" -- enables the app to detect when it is moving into the background
   mergNotify "UIApplicationDidBecomeActiveNotification"  -- and when it is being brought back to the foreground
end openStack

And here is how I respond to these messages:

Code: Select all

on UIApplicationWillResignActiveNotification
   -- iOS only
   -- we are being put into the background
   -- lets push the current changes to dropbox (using mergBgTask) -- iOS only
   if gCloudService then
      mergBgTaskStart
      put the result into tTaskID
      doDropBoxPUSH -- our background task
      mergBgTaskStop tTaskID
   end if
end UIApplicationWillResignActiveNotification

on UIApplicationDidBecomeActiveNotification
   -- iOS only
   -- we are coming back from the beyond
   if gCloudService then
      doDropBoxPULL 
   end if
end UIApplicationDidBecomeActiveNotification

Notice I have an if-statement which only executes the push/pull if “CloudServices” are enabled. I allow the user to turn this on or off as they wish. Sometimes they may not be in range of a wifi link or 5G connection and for performance reasons, want to turn it off.

Inside the handlers (doDropBoxPUSH and doDropBoxPULL) I do 4 things:

1. determine the date of the dropbox file
2. determine the date of the local file
3. Adjust accordingly and make comparisons between the two file dates
4. Call a handler to move the file as appropriate

I will post code for each of the above in Parts 3 and 4 of this thread.

[marksmithhfx]

Dropbox data file syncing protocol — Part 3

1. Here is how I determine the dropBox file date (DbDate):

Code: Select all

  // first, make sure we have DB credentials
   put the cpAccessToken of this stack into tAccessToken -- dropbox access token
   
   if tAccessToken is empty then
      answer "Access token required"
      exit to top
   end if
   
   // second, get the last modified date in DB 
   put true into pRecursive
   put false into pIncludeMediaInfo
   
   dropboxListFolder tAccessToken, "", pRecursive, pIncludeMediaInfo
   
   if it is empty then
      answer "doDropBoxPUSH: Access Token not valid"
      exit to top
   end if
   
   put JsonImport(it) into tArray
   
   -- process the Array to find when "todoitems.sqlite" was last modified
   
   repeat for each key tKey in tArray["entries"]
      if tArray["entries"][tKey]["name"] = “your-app-filename.ext” then
         // grab the file date
         put tArray["entries"][tKey]["client_modified"] into tString -- last modified date YYYY-MM-DDTHH:MM:SSZ — note the “T” in the middle, and “Z” at the end
         put offset ("T",tString) into tPos
         if tPos > 1 then
            put char 1 to tPos-1 of tString into tDate -- YYYY-MM-DD
            replace "-" with "/" in tDate -- YYYY/MM/DD
            set itemdelimiter to "/"
            put char 3 to 4 of tDate into YR -- YY
            delete item 1 of tDate -- MM/DD
            put "/" & YR after tDate -- MM/DD/YY
            put char tPos+1 to -2 of tString into tTime -- HH:MM:SS
            put tDate & " " & tTime into DbDate -- MM/DD/YY HH:MM:SS
            convert DbDate to seconds
            subtract 15 from DbDate  -- a fudge factor
            // explanation: when closing the app (and pushing the latest version to DB), if the app is re-opened
            // quickly it is reporting (or seeing) a newer version of the data in dropbox. Even though this is
            // the exact same copy that was just pushed to dropbox a second ago. To prevent this, I am retarding
            // the DB date by 15 seconds, so it is not seen as "newer" then the copy just uploaded. 
            exit repeat
         end if
      end if
   end repeat

2. Here is how I determine the local file date (iOSDate):

Code: Select all

  put files (specialfolderpath("documents"), "detailed") into tTemp
   filter lines of tTemp with "your-app-filename.*" -- doesn't work with file extension included
   set itemdelimiter to comma
   put item 5 of tTemp into iOSDate -- item 5 contains the last modified date in seconds
   convert iOSDate from seconds to internet date
   put char -4 to -1 of iOSDate into timezoneOffset
   put char -5 of iOSDate into tzSign
   // add timezoneoffset to iOSDate
   convert iOSDate to seconds
   put char 1 to 2 of timezoneOffset into HH
   put char 3 to 4 of timezoneOffset into MM
   put (HH * 3600) + (MM * 60) into timezoneOffset -- converted to seconds
   // since the offset determines how to go from Zulu to local time, to do the opposite (local to Zulu) we
   // need to switch the sign
   switch tzSign
      case "+"
         put iOSDate - timezoneOffset into iOSDate 
         break
      case "-"
         put iOSDate + timezoneOffset into iOSDate
         break
   end switch

[marksmithhfx]

Dropbox data file syncing protocol — Part 4

All that is left is to compare the dates and call the appropriate upload/download handler (because I do this from different parts of the program — resume or suspend — I use the appropriate test in each location):

Code: Select all

   if iOSDate > DbDate then
      uploadiOSfile tAccessToken 
   end if

   if iOSDate < DbDate then
      downloadDBfile tAccessToken
   end if

Here is how I upload the file to dropBox

Code: Select all

command uploadiOSfile pAccessToken
   // upload the iOS file to dropbox
   
   put specialFolderPath("documents") & "/your-app-filename.ext" into tSourceFile -- file on iOS
   
   put "overwrite" into pMode  
   put False into pAutorename 
   put False into pAllowedSharedFolder 
   put True into pMute
   put "/your-app-filename.ext" into pPath
      
   put URL ("binfile:"&tSourceFile) into pData -- read the source
   dropboxUpload pAccessToken, pPath, pMode, pAutorename, pMute, pData -- upload the data
   
   put the result into tResult
   
   if tResult is not empty then
      answer error "UploadiOSFile: " & tResult
      exit to top
   end if
end uploadiOSFile

And here is how I download the file from dropBox:

Code: Select all

command downloadDBfile pAccessToken   
   //dload the dropbox file
   put specialFolderPath("documents") & "/your-app-filename.ext" into tTargetFile
   put "/your-app-filename.ext" into pPath -- the path to the file on dropbox
 
      -- if this is an SQLite file, you should probably close it first
      if gConnectID is a number and gConnectID is among the items of revOpenDatabases() then
         revCloseDatabase gConnectID
         put empty into gConnectID
      end if

     -- -- backup the target first
      rename specialFolderPath("documents") & "/your-app-filename.ext" to specialFolderPath("documents") & "/your-app-filenameBACKUP.ext”
      dropBoxDownload pAccessToken, pPath -- the file is copied into the "it" variable
      put the result into tResult
   
     if tResult is not empty then
        answer error "downloadDBfile: " & tResult
     else
        put it into URL ("binfile:"&tTargetFile) -- write the data to disk
     end if
      // if this is an sqlite file, reopen it
      if gConnectID is not a number then
         doDataBaseInitialization
      end if
end downloadDBfile

Hopefully that is enough of a guide for someone to implement this without too much difficulty. The protocol is for operating in your own dropBox App folder, using an Access Token you acquired from your App Console in dropbox. In order to enable other users of your app to use a dropbox folder in a similar way, you need to have dropBox (with your users permission) authorize your app to access their dropBox folder. You do this using something called oAuth. I have no idea how oAuth works but I suspect I will become quite familiar with it in the coming days or weeks as I work towards getting this syncing capacity operating for “other” users of my app. More on that later (although if anyone has gone through the oAuth process and knows the workflow, that would be greatly appreciated).

Cheers,
Mark

[marksmithhfx]

You could even store the time as part of the file name when you save it. If the Dropbox number is bigger than the local number then update.

Kelly

Hi Kelly, Mark and others,

I'll be looking at an even simpler solution in my talk on Monday. As it turns out (and you can work this out using sequence numbers), if this is a single user syncing process, the current open file is always the most recent one. That being the case, push it to dropbox when you close. When you open (from whatever device) download from dropbox. Lather, rinse and repeat. The advantage, of course, is you don't have to track sequence numbers or dates, and you don't have to delete files that have "expired" filenames. You can just use the original filename in each location. It would be very easy to implement if it were not for the fact that dropbox has moved from issuing long lasting access tokens (that basically never expired) to short lasting access tokens (that expire after 4 hours). It does add some complexity to the process. All the details on Monday. See you at the conference.

Mark