Corrupted stack

[bd525]

After saving my project in LiveCode 7.1.1 I upgraded to 7.1.2. When I tried to open the stack in 7.1.2 I got "There was a problem opening that stack: stack is corrupted, check for ~ backup file." I returned to 7.1.1. and got the same error message; with the same file that had NOT been edited since being saved in 7.1.1.

I tried 7.1.3: same thing. I tried to open a Google Drive backup that predated the corrupted version by several hours work: same thing. Fortunately, I had made a manual backup that opened successfully. So, I "only" lose five or six hours work.

I'm migrating to LiveCode from another platform, and this sort of problem scares the hell out of me. Can you blame me? Is there a way to prevent a recurrence? The Google Drive backup feature has always been a reassurance, but even that didn't work. Sure, I can try to remember to make manual backups frequently, but that shouldn't be necessary. More importantly, given these circumstances I lack confidence that even that practice would prevent a completely lost project.

Bruce

[FourthWorld]

I'm migrating to LiveCode from another platform, and this sort of problem scares the hell out of me. Can you blame me?

Not at all, but we need to explore all options, because:

The Google Drive backup feature has always been a reassurance, but even that didn't work.

Automated syncing is a complex task, and in my experience no one gets it right 100% of the time. It seems my experience isn't unique:
https://www.google.com/webhp?hl=en#hl=e ... corruption

This isn't to suggest Google Drive is necessarily responsible for the corruption, but it does offer the reminder that multiple backups through different means are a safer bet.

Sure, I can try to remember to make manual backups frequently, but that shouldn't be necessary.

Au contriare, mon ami. Always assume any single backup method will fail. Sooner or later, any of them will. Networks have issues, not all RAM is ECC, bitrot happens. Look at old JPEGs stored on magnetic media for more that a decade and chances are you'll find a few with bitrot (maybe more than a few if stored long-term on flash media). Multiple redundant backups made through different means are useful for everyone, and essential for developers.

Here I use ownCloud for convenient automated syncing and versioning throughout the day, but also have an extra copy maintained on a Mac via Time Machine, another on Ubuntu automated daily with its built-in baclup app, and multiple copies on portable drives made via rsync. With portable drives as low as $50/TB it's a very small expense for what could otherwise result in a big expense of restoring data.

Backups are like security: the more paranoid you are the less paranoid you need to be.

I used to rely on a single backup. Then it went bad. Now I maintain 6 copies of all my files, and 8 of current working files, rotating through a backup that occurs at least once each working day (and often several during the day). Systems are available to automate this, and rsync can gracefully handle everything not otherwise easily automated through other means, in a way that gives me fine-grained control while also being uncommonly efficient.

Thursday is World Backup Day. Celebrate by treating yourself to a couple portable drives, and store at least one offsite (earthquakes, floods, and theft play a role in disaster recovery too).

But enough with my unsolicited generic backup advice. Let's look at the LC side of this:

More importantly, given these circumstances I lack confidence that even that practice would prevent a completely lost project.

True file corruption is very rare in LiveCode. Not impossible, just rare. Given the complexity of RAM, file systems, disk controllers, and everything else that comes into play with file I/O, all files from all applications on all systems are always subject to corruption. But having used LC nearly every day since 1998, I've had only one file become corrupt, and in that case the file itself was fine, it was just one image object that went bad. And that was back in 2001.

Most cases of apparent file corruption aren't really file corruption at all, but merely folks attempting to open a new format in an older version. As with any program, and older version will only understand formats that existed at the time it was written. At least with newer version of LC when attempting to open an unknown format in most cases it can determine it's a LC file and at least provide appropriate guidance about needing to use a newer version, something relatively few programs offer.

Since your work has all been in v7.x, that shouldn't be relevant to your circumstance. I bring it up only to note how carefully the tool is designed to minimize moments like this one.

Additionally, when a stack file is written it first copies the last successfully-saved version to a file named the same but preceded with "~" (e.g. "~mystackfile.livecode"), which is deleted only after the checksum of the new file has been verified. If a write is interrupted in most cases you can ditch the bad copy, remove the tilde from the backup, and get back to work with that version.

In your case, however, it seems as though the write was reported as successful when it wasn't. Rare, but not impossible.

You noted that the file was made with v7.1.1, and that you're now using the latest Stable release, v7.1.3. The newer version includes dozens of refinements since v7.1.1, so while I wasn't able to find any in the Release Notes related to file corruption given how rare such cases are it would seem the newer version is a good bet.

File corruption, memory leaks, and crashers are high-priority items for the team. If you have time it would be helpful if you could submit your corrupted stack with a bug report:
http://quality.livecode.com/

If the stack includes proprietary stuff you'd rather not share in the public bug DB, you can contact support AT livecode.com to arrange confidential transfer to them.

Post-corruption it may not be possible to identify the root cause, but it may be that they can find the point where the format broke and either find a fix or find that they've already addressed it with the other fixes between 7.1.1 and 7.1.3.

[bd525]

Thanks for the very thorough reply, Richard. And apologies for the somewhat elevated tone of my original post.

I wonder if the problem could have been caused by a substack I created as a repository for scripts from the Director version of the project I'm migrating. It's a convenient location for script text to examine, copy, paste to the main stack, and edit there to conform to LiveCode format. Of course, there would be numerous script errors in the substack. I have no idea if that could cause corruption, but I thought I'd mention it. Just to be safe I've deleted that substack and returned to the routine of alternating between the two applications for migrating scripts.

I'll contact support about emailing the corrupted file.

-Bruce

[FourthWorld]

Thanks for the very thorough reply, Richard. And apologies for the somewhat elevated tone of my original post.

No worries. I was pretty freaked out the one time I saw that dialog myself. Fully understandable. Asking about it here was the right move.

I wonder if the problem could have been caused by a substack I created as a repository for scripts from the Director version of the project I'm migrating. It's a convenient location for script text to examine, copy, paste to the main stack, and edit there to conform to LiveCode format. Of course, there would be numerous script errors in the substack. I have no idea if that could cause corruption, but I thought I'd mention it. Just to be safe I've deleted that substack and returned to the routine of alternating between the two applications for migrating scripts.

That's an interesting detail. I wonder what might have crept into that script data that may have contributed to the issue.

I'll contact support about emailing the corrupted file.

Excellent, Bruce. Thanks for taking the time. Corruption causes can be difficult to pin down, but better to try than not.

Please post an update here if the team finds anything useful.

[mwieder]

Indeed. In all my time using LiveCode I've never experienced a corrupted stack, but I have seen the occasional report of one. Luckily they've been very few, but I don't doubt that it can happen. I'd be very curious if the team can uncover what's going on, and see if there's something that needs fixing or something we as users need to be aware of when constructing stacks.

[jacque]

Recently the engine changed the way stacks are saved, there were warnings about it in the release announcements. I can't recall which version that was. It wasn't too long ago though.

If this problem is related to the change, the team will absolutely want to see the stack so they can fix it.

[mwieder]

I *think* the change was only to LC8. The "save with format" and "save with newest format" commands were introduced in LC8-dp13 and I don't think they've been backported to other versions. Or will be. So I don't think that's the source of corruption.

[jacque]

I *think* the change was only to LC8. The "save with format" and "save with newest format" commands were introduced in LC8-dp13 and I don't think they've been backported to other versions. Or will be. So I don't think that's the source of corruption.

That's good to know, I couldn't remember exactly where I read it. There's been a lot of updates lately.