Sub stacks not saving data in standalones on macOS 10.15

Deploying to Mac OS? Ask Mac OS specific questions here.

Moderators: Klaus, FourthWorld, heatherlaine, kevinmiller, robinmiller

Post Reply
ekami
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 27
Joined: Tue Jun 05, 2012 4:08 pm

Sub stacks not saving data in standalones on macOS 10.15

Post by ekami » Sat Feb 22, 2020 9:48 pm

Hello
Lc bugs are driving me crazy…
I have a sub stack in a standalone that stores some data, and i use "save stack stackname" regularly to save the stack.
After saving as standalone (checking "Move substacks into individual stackfiles"), this substack works fine in old versions of macOS (10.9 to 10.12, not tried with Hs and Mojave) but not in macOS Catalina: no data is saved in the stack, when the app is in "Applications" folder or in any other home folder.
When the app is placed on a volume without permissions, it works fine under Catalina, so this seems to be a permission write issue.
Is there any special permissions to apply to the app in some versions of macOS ? (in this case i will need creating a special installer for this case), but having a doubt on the reliability of the app after it's installation is unacceptable).
Many thanks for your help.
Best regards, Joan.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7929
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by FourthWorld » Sat Feb 22, 2020 10:14 pm

ekami wrote:
Sat Feb 22, 2020 9:48 pm
Lc bugs are driving me crazy…

...this seems to be a permission write issue.
Which is it?

I'd put my money on the latter.

Modern OSes tend to protect their Applications folders, strongly recommending using user space for user-writable data.

The specialFolderPath function can help you find such folders.
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

ekami
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 27
Joined: Tue Jun 05, 2012 4:08 pm

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by ekami » Sun Feb 23, 2020 1:58 am

Thanks for the response.
My app must be installed in "Applications" folder and work "out of the box".
I work on it for 6 months, and at last moment, a stupid write issue, this is really very frustrating...
This kind of bad trip will probably make me abandon Lc for commercial projects and back to FileMaker runtimes: extremely heavy, but everything works in FMP.
--
I've tried chmod -R 775, and removing all extended attributes, no effect.
The strange thing is that if the app is placed and opened once on a volume without permissions, the sub stack can be modified, and this "modified" version works fine if placed in the Catalina "Applications" folder, and can be modified without any writing issue. This means that the "Applications" folder can perfectly receive apps able to write data inside their bundle.
If i find no solution, i will probably write text files in the bundle to store data, a deplorable method to work around the bug.
Sorry for my bad english ('im French), i'm extremely disappointed working so hard creating a nice app and planning to abandon it due to "bizarre" issues.

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7929
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by FourthWorld » Sun Feb 23, 2020 4:34 am

ekami wrote:
Sun Feb 23, 2020 1:58 am
Thanks for the response.
My app must be installed in "Applications" folder and work "out of the box".
I work on it for 6 months, and at last moment, a stupid write issue, this is really very frustrating...
This kind of bad trip will probably make me abandon Lc for commercial projects and back to FileMaker runtimes: extremely heavy, but everything works in FMP.
How does FMP let you write to folders the OS doesn't want you to?

It may be that they mask what's happening, perhaps moving the editable part into user space. I don't know; it's been years since I built EXEs with FMP.

But even as a subsidiary of Apple, FMP isn't changing the OS. It's doing something else. If you can describe what it's doing, or point me to a description of it, I can tell you how to do that with LC.
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7929
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by FourthWorld » Sun Feb 23, 2020 4:49 am

Also, help me understand: when you write "substacks", are you referring to substacks that are part of the mainstack which becomes the executable, or to separate stack files?

No modern OS allows an application to modify itself, so if those are truly substacks you'll want to move user documents and data into files separate from the executable.

From the LC User Guide, "Building a Standalone Application" (p513):
Note: A stack file directly attached to a standalone application cannot have changes saved to it. This stack is bound directly to the executable file that runs. The OS locks an executable file while it is running. If you want to save changes in your standalone application, split your stack up until multiple files. A common technique is to create a "splash screen" stack that contains a welcome screen and then loads the stacks that make up the rest of your application. These stacks are referenced as stackFiles on this pane in the standalone settings screen. It is thus possible to automatically update these component stacks, or to save changes to them. You may also want to consider creating preference files in the appropriate location on your end user's system (see the specialFolderPath function and query/setRegistry functions for more information).
LiveCode and FileMaker are indeed different, and each has its strengths. If you're willing to move forward with an open mind about learning, I'm confident we can move your work forward as powerfully as the thousands of other devs who deploy with LC every day.
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

bogs
Posts: 4790
Joined: Sat Feb 25, 2017 10:45 pm

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by bogs » Sun Feb 23, 2020 10:13 am

FourthWorld wrote:
Sun Feb 23, 2020 4:49 am
Also, help me understand: when you write "substacks", are you referring to substacks that are part of the mainstack which becomes the executable, or to separate stack files?
Um, ekami said in the first post :
ekami wrote:
Sat Feb 22, 2020 9:48 pm
After saving as standalone (checking "Move substacks into individual stackfiles"), this substack works fine in old versions of macOS (10.9 to 10.12, not tried with Hs and Mojave)
I may not know a lot about Macs, but if it was working in the previous 10.x versions, doesn't that mean by default that it should be working in the current one? I.e. they are already separate from the main exe file.

I'd absolutely agree with your suggestion to
"to move user documents and data into files separate from the executable."
if, even though separate from the app exe they are still in the app package though. It is simple to do, and makes sense permissions wise.
Image

dunbarx
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 6918
Joined: Wed May 06, 2009 2:28 pm
Location: New York, NY

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by dunbarx » Sun Feb 23, 2020 5:57 pm

My app must be installed in "Applications" folder and work "out of the box".
Not a problem. Put the executable in the "Applications" folder.

Now put everything else in any folder where you have permission to change and save a file. You likely have no particular LC issue. You have an OS issue. This just takes a little practice.

Craig

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7929
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by FourthWorld » Sun Feb 23, 2020 8:08 pm

bogs wrote:
Sun Feb 23, 2020 10:13 am
FourthWorld wrote:
Sun Feb 23, 2020 4:49 am
Also, help me understand: when you write "substacks", are you referring to substacks that are part of the mainstack which becomes the executable, or to separate stack files?
Um, ekami said in the first post :
ekami wrote:
Sat Feb 22, 2020 9:48 pm
After saving as standalone (checking "Move substacks into individual stackfiles"), this substack works fine in old versions of macOS (10.9 to 10.12, not tried with Hs and Mojave)
The thread is titled as being about substacks, but instead seems about stack files. I see no harm in seeking clarification that the test in question does indeed separate the stacks into separate files.

I may not know a lot about Macs, but if it was working in the previous 10.x versions, doesn't that mean by default that it should be working in the current one? I.e. they are already separate from the main exe file.
There's the rub. In a static world, what you suggest would be true. But in threat vector analysis as in the rest of life, the only constant is change.

The situation is not helped by Apple's choice to abandon semantic versioning, using version numbers for branding instead. "10" is no longer a meaningful representation of the state of the OS. For that we have to look at the minor number, to the right of the decimal. The primary "10" has been in use for all versions since Apple transitioned from what we now call "Mac Classic" (v9 and earlier) to everything delivered since about the turn of the century.

Lots of changes over time. Here's an example from a while back of how Apple's efforts to stay one step ahead of the bad guys sometimes require changes to how devs do things:
https://www.mail-archive.com/use-liveco ... 58123.html

More interesting to me is that bundles were ever writable in any version of OS X/mac OS over the last several years. For a good many years most OSes require elevated privileges to write anywhere in the Applications folder, and I would assume that includes folders inside the app bundle.

A quick search for security changes in macOS 10.15 brings up this page, which raises a question with item #2:
In macOS Catalina, Apple have introduced a new filesystem partitioning architecture. In one partition is a dedicated, “read-only” system volume containing the OS itself. This is completely separate from all user data, and nothing – other than Apple signed code, such as updates – can overwrite operating system files. It’s basically System Integrity Protection taken to the next level. Instead of protecting locations in an unprotected partition, the entire partition is sealed off.

There’s potential for some initial confusion here as the mount paths are no longer quite as obvious. Where is user data located? Traditionally, it would be in /Users/<username>. On Catalina, it’s in /System/Volumes/Data/Users/<username>. Good-old fashioned root, / appears now to be a system-only volume.
https://www.sentinelone.com/blog/7-big- ... -catalina/

Most devs may not notice this change because they adopted Apple's guidance long ago about separating user-writable data out of the bundle in user space.

This leaves open the question of how FMP handles this, but I wonder if it's similar to how SuperCard does it, moving files out of the bundle into appropriate user folders behind the scenes. If I can turn up anything on how FMP handles this I'll share it; if anyone has such a link please post it.

I'd absolutely agree with your suggestion to
"to move user documents and data into files separate from the executable."
if, even though separate from the app exe they are still in the app package though. It is simple to do, and makes sense permissions wise.
Since the turn of the century it's been the recommendation of Apple, Microsoft, and most Linux maintainers with their respective OSes. When everyone suggests the same thing, it may be worth heeding. :)
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7929
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by FourthWorld » Sun Feb 23, 2020 9:14 pm

FWIW, here's what I could find about FMP runtimes - it seems the DB file itself isn't put inside the app bundle at all, but is a separate document file:
https://fmhelp.filemaker.com/help/16/fm ... me.html%23

So unless there's something Claris is allowed to do as a subsidiary of Apple that other companies aren't, I would imagine we'll be able to help Joan get set up for the time of deployment anticipated, similar to how FMP does it.
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

ekami
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 27
Joined: Tue Jun 05, 2012 4:08 pm

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by ekami » Mon Feb 24, 2020 1:30 am

Hello From France.

After two days fighting with this issue; i finally solved it.

I'll try to explain it : during the standalone creation, the individual sub-stacks are stored in the /Ressources/_MacOS folder of the app bundle. This is the right place, because /Ressources/_MacOS folder is ignored during the cosign process (if the sub-stacks where placed in the /MacOS folder they will be considered as in "wrong place" during cosigning and generate a notarization fail "invalid package")

On MacOS 10.9 to 10.12 (not tested on 10.13 and 10.14), during the first opening of the standalone, the sub-stacks are moved automatically (by the Lc runtime) to the /MacOS folder. This breaks the signature (see technote below), but the bundle keeps working and the moved sub-stacks can be normally saved, so data can be stored inside them.
On macOS Catalina, the sub-stacks are not moved when the standalone starts, explaining why they can't store data when saved ( in counterpart, the bundle keeps signed).

I used AppWrapper to notarize the standalone ( revsecurity.dylib must be deleted because it can't be signed),
Next i build a .pkg with a post install script that (after the installation of the bundle in the "Applications" folder) deletes extended attributes, set permissions, moves the sub-stacks, and finaly opens the app.

sleep 0.25
xattr -cr /Applications/MyStandalone.app
sleep 0.25
chmod 775 /Applications/MyStandalone.app
sleep 0.25
mv /Applications/MyStandalone.app/Contents/Resources/_MacOS/Preferences.rev /Applications/MyStandalone.app/Contents/MacOS
sleep 0.25
open /Applications/MyStandalone.app

I finaly build a notarized .dmg with DMG Canvas, to be used with OSX 10.9 to 10.15
I know that moving files inside the .app after it's installation will break the signature of the bundle, and i probably better do storing data in other place, but as i can use sub-stacks as databases i can save or purge, writing to sub-stacks is the most obvious option i can use for the moment.
---
Please let me know if you need more informations.
Best regards, Joan.
--
edit: i use the same post-install script (excepted mv command) to allow FileMaker Pro runtimes opening/read/write to their related databases (stored at same level of FMPro bundle)
--
https://developer.apple.com/library/arc ... index.html

ekami
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 27
Joined: Tue Jun 05, 2012 4:08 pm

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by ekami » Mon Feb 24, 2020 8:20 pm

Hello
In case if the post-install script of my .pkg fails moving files, i added this script i placed in the "startup" script of the standalone to move sub stacks.

Code: Select all

if the platform = "MacOS" and the environment = "standalone application"
   then
      set the itemDelimiter to slash 
      put the effective filename of stack "Preferences" into OriginalStackPath -- bug in macOS Catalina wich returns bad stack path !
      put item 1 to -3 of OriginalStackPath & "/Resources/_MacOS/Preferences.rev" into StackPath -- force creating the real stack path
      if there is a file StackPath
      then
         put item 1 to -3 of OriginalStackPath &"/MacOS/Preferences.rev" into NewStackPath
         rename file StackPath to NewStackPath
         if the result is not empty
         then
            -- answer StackPath & return & NewStackPath -- compare paths in case of error -- during script tests
         end if
      end if
   end if

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7929
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by FourthWorld » Mon Feb 24, 2020 9:20 pm

Re:

Code: Select all

      put the effective filename of stack "Preferences" into OriginalStackPath -- bug in macOS Catalina wich returns bad stack path !
Has that been reported?

Could it be related to macOS' folder virtualization?

What is the actual path and the reported path?
Richard Gaskin
Community volunteer LiveCode Community Liaison

LiveCode development, training, and consulting services: Fourth World Systems: http://FourthWorld.com
LiveCode User Group on Facebook : http://FaceBook.com/groups/LiveCodeUsers/

ekami
Livecode Opensource Backer
Livecode Opensource Backer
Posts: 27
Joined: Tue Jun 05, 2012 4:08 pm

Re: Sub stacks not saving data in standalones on macOS 10.15

Post by ekami » Tue Feb 25, 2020 1:14 am

I've made a video with a fresh new stack to explain the two bugs i've found, i'll send it tomorrow to Lc quality.
You can download it here:
https://my.pcloud.com/publink/show?code ... KWaXCg2w0y

Post Reply

Return to “Mac OS”