Unexpected newline character in filename - a summary

Anything beyond the basics in using the LiveCode language. Share your handlers, functions and magic here.

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Unexpected newline character in filename - a summary

Post by Simon Knight » Mon Apr 25, 2022 8:43 am

Hi,
In this three page forum post the issue under discussion drifted off topic into newline characters appearing in file names. I believe that the information is of wider interest but is difficult to find in the thread : https://forums.livecode.com/viewtopic.p ... 7&start=30. So here is a summary.

Unix allows almost any characters to appear in file names and probably folder paths. This means that as far as Unix is concerned a newline character is valid in a filename. Mac OS is based on unix and allows the newline character in file names. Unfortunately when Apple's Finder lists such a filename for display it represents the newline character as a question mark.

When Livecode's files command used in its default "short" version, a newline delimited list of file names is returned. If one of the filenames in the list originally contained a newline character then the files command splits the filename at the newline into two entries in the list that is returned. Any attempts to manipulate these entries in the list will result in a "can't find file" error because no such file exists.

This issue is discussed in bug report : https://quality.livecode.com/show_bug.cgi?id=23676

The solution is to only use the detailed-utf8 version of the files command
Name. The file name is URL-encoded. To reliably obtain the name use the detailed-utf8 outputKind and decode using the URLDecode function followed by textDecode. For example, textDecode(URLDecode(item 1 of tLine), "utf8").
as the short version does not work in all cases and the long version has been depreciated.

I believe that all above implies that when renaming files only the
put URL A into URL B
should be used as the simpler Rename command does not use URL encoding so does not deal with newline characters.

If you have read this far I expect you are thinking
well none of my filenames have a newline in them
. All I will say is that none of mine had until I discovered that five out of a collection of one hundred thousand images did and these five caused days of head scratching and no, I have no idea how these pesky characters were added in the first place.

best wishes
Simon
best wishes
Skids

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Unexpected newline character in filename - a summary

Post by FourthWorld » Mon Apr 25, 2022 4:04 pm

This has bitten many a person over the decades. Perhaps the most common example is the "._icon" file, which Apple uses to store metadata about folder state. Most folks don't see them even though they're ubiquitous, because most file managers (and even terminal tools) don't include dot files in listings by default.

But what makes Apple's "._icon" files infamous is that the actual full file name is "._icon<CR>", with a trailing return character.

Apple provides no UI that allows users to type CR in a file name directly, so I suppose it seemed a safe way to try to ensure their files are not overwritten by anything the user makes.

But boy, the mystifying headaches it's caused devs and their users over the years.

What you experienced is something you can read many variants of in many forums for many tools.

Many devs have a habit of excluding dot files with a "filter without..." to avoid monkeying with config files, focusing instead on files the user can see. And those that make tools needing more complete listings often need other info, and so use "the detailed files" anyway.

But sure enough, with many people using a wide range of tools, CRs in file names has been a perennial source of trouble.

Thank you for writing your post. Hopefully fewer will face this mystery unaided going forward.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Simon Knight
Posts: 845
Joined: Wed Nov 04, 2009 11:41 am
Location: Gunthorpe, North Lincs, UK

Re: Unexpected newline character in filename - a summary

Post by Simon Knight » Wed Apr 27, 2022 8:40 am

# Richard:

Thanks for your post and sorry for the delay in reading it ; I've managed to hurt my back which prevents me sitting at a computer for the time being and making this post is an experiment.

I think I fell into the trap of "Livecode will sort it out" when the truth is that interfacing with the file system gets complicated. The trap is that most of (all?) the published file system examples use the simple default versions of the commands and while the dictionary does contain a warning it is not the first thing that jumps off the page. I think its a pity that we are unable to add comments to dictionary entries in the way we were able to do in the past.

I owe you a second thanks for the advice to always check "the result". Before I added the check to my code it just ignored these files so I would have been very unlikely to spot that these five files were not being renamed.

Simon
best wishes
Skids

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: Unexpected newline character in filename - a summary

Post by FourthWorld » Thu Apr 28, 2022 6:38 am

Very glad that was helpful, Simon. Thank you for saying so.

I've had back injuries myself. I hope yours heals well and quickly.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

Post Reply

Return to “Talking LiveCode”