carriage return

LiveCode is the premier environment for creating multi-platform solutions for all major operating systems - Windows, Mac OS X, Linux, the Web, Server environments and Mobile platforms. Brand new to LiveCode? Welcome!

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

UKenGB
Posts: 16
Joined: Mon Aug 25, 2014 11:06 am

Re: carriage return

Post by UKenGB » Wed Feb 21, 2018 10:21 am

bwmilby wrote:
Tue Feb 20, 2018 10:56 pm
Unfortunately there is probably way too much inertia to make CR and LF actually mean what they say... internally everything is LF and converted upon output to the local standard except for MacOS X (where it still uses CR).

I’ve asked about changing that for Mac but there is a concern about things breaking. The code change to do so is trivial (I’ve compiled with the change successfully).

Server does allow:

Code: Select all

set the outputLineEndings to “lf”
Yay! That does the trick, so it least it can be made to work. However I think it is extremely BAD practice to have something that does NOT do what specifically it IS supposed to do. CR should ALWAYS be ASCII 13 and LF should ALWAYS be ASCII 10 and numToChar() should ALWAYS be the correct char for the number entered, NO MATTER WHAT OS it's running on, because that is what those constants and function unequivocally state is what they are. 'return' is different and can be left as is, a very useful way of using a new line character that will be adjusted depending on the OS being used.

I cannot over emphasise how important it is for things like this to be correct. Not only is it completely confusing but it is simply fundamentally WRONG. In fact I am appalled that numToChar() does not always produce the correct char. This sort of idiosyncratic cleverness will always come back to bite you in the a***, as evidenced by the fact that they've managed to completely f*** it up on the Mac which might use CR or LF depending on where you are working with the text. The WRONG assumption has been made that the Mac always and only uses CR and here we have the result of that (fundamentally ignorant if truth be told) mistake.

Backwards compatibility be damned I say. This should be changed to be correct. Anyone who uses 'return' will not be affected and CR and LF would mean what they say, ASCII 13 and 10 respectively and numToChar() would also do what it says.

Well, that's my rant. I am grateful for those who helped my understanding of this. Now we all know :|

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: carriage return

Post by FourthWorld » Wed Feb 21, 2018 4:32 pm

UKenGB wrote:
Wed Feb 21, 2018 10:21 am
The WRONG assumption has been made that the Mac always and only uses CR and here we have the result of that (fundamentally ignorant if truth be told) mistake.
I agree with the sentiment, and wish the circumstances were different. I sometimes even share the same thought about "legacy be damned, let's change it now!". But then I think of the hundreds of hours I'd be spending rewriting so much code, and the hundreds of thousands of hours lost to our community doing the same, and then I think maybe that LiveCode being no more immune to its own historical anomalies than any other language isn't so bad, even if it does mean having to learn an anomaly or two along the way.

Some background may be helpful:

It was neither wrong nor an assumption when the "CR" and "return" constants were first introduced in the root tongue of this family of languages, HyperTalk. In fact, HyperTalk was made by Apple itself, so while no company is perfect we can be reasonably confident the decision wasn't entirely without some merit.

This is one of the challenges with a mature multi-platform tool: if LC were being invented today it wouldn't be influenced by something as old as the pre-NeXT Mac OS (what we now think of as "Classic").

But before it was LiveCode it was MetaCard, which was born in May 1992. In those days Apple's own HyperCard introduced "CR" and "return" as constants for 0x13, which was the only line-ending used on Apple's OS at the time.

And while one of the greatest strengths of LiveCode is its multi-platform nature, for MetaCard's first half-decade it was exclusively Unix, which of course uses LF for line endings. Scripts, fields, and other internal representations of text used the Unix convention for line endings, which is still the case to this day.

To allow HyperCard scripts to be run within MetaCard without modification, MetaCard implemented support for the "CR" and "return" constants in a way that worked well on Unix, as 0x10. Oddly enough, even major MetaCard users like Sun didn't seem to mind the anomaly too much; it was something to learn, and once learned was easy to accommodate.

Back then no one could know that nearly a decade later Apple would toss their OS and replace it with a Unix system, NeXT. But just after the turn of this century they did, and line endings on Mac have been a jumbled mish-mash of mixed conventions ever since.

Some macOS programs continue to use 0x13 for line endings, but things unique to the newer Unix implementation like Terminal use the Unix convention instead.

In the olden days, the only time Mac users had to think about translating line endings was when trading files with Windows users. But today we find posts in support forums around the Mac community about translating line endings within macOS itself.

Given that so much of the modern world is driven by Unix conventions, I'm glad Apple made the switch. NeXT was an especially good Unix, and in Apple's hands it's only gotten better.

And ideally Windows would also change to Unix, and then with just one set of conventions so much in computing would be simplified. :)

But in the meantime we find ourselves on an imperfect planet, on which humans evolved as imperfect creatures who make imperfect decisions. Line-ending anomalies are annoying as hell, and in all honesty I'm no fan of the legacy that brought us here either. But in the bigger picture, line-ending anomalies are among the least impactful of human imperfections. :)

Indeed, you probably wouldn't have even had to think about line endings were it not for the specific characteristics of using Terminal on macOS:

Within LC those constants display text as expected.

And when reading and writing text, LC's automatic conversion to and from OS-native line endings on all platforms delivers results that display as expected.

But because Terminal uses Unix line endings on an OS that otherwise displays 0x13 just fine, we encounter one of the few cases where we see unexpected results. In that specific case the results are even more mystifying than other outcomes of line-ending differences among platforms, because the macOS Terminal overwrites your output with its prompt, giving the false appearance that nothing was written at all (introduce a wait and you'll see what I mean).

Given how seldom macOS is used for servers (Apple themselves use Linux), the scope of impact from this historical anomaly is pretty small.

So it's annoying, and one of the many things we need to keep in mind when working with binary data. But for most practical needs, LC's handling of line endings across platforms, imperfect as it is, does a pretty good job of balancing backward compatibility, OS expectations, and internal consistency.
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: carriage return

Post by bogs » Wed Feb 21, 2018 5:59 pm

Very nicely put Richard.

As an aside, I don't know any language that works on multiple (or even single!) platforms that does not have it's own peculiarities where you have to work around something. Might not be line endings, but it will sure be something. In this case, as Richard says, once you discover that something, it is easy to work around it, and in this case, it is *really* easy.
Image

bwmilby
Posts: 438
Joined: Wed Jun 07, 2017 5:37 am
Location: Henrico, VA
Contact:

Re: carriage return

Post by bwmilby » Wed Feb 21, 2018 6:34 pm

One thing remains though... if using a Mac, then “file:” will not work as expected (emits CR) so “binfile:” is needed to actually save files that use LF. Not difficult, but something to be aware of.
Brian Milby

Script Tracker https://github.com/bwmilby/scriptTracker

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: carriage return

Post by FourthWorld » Wed Feb 21, 2018 7:54 pm

bwmilby wrote:
Wed Feb 21, 2018 6:34 pm
One thing remains though... if using a Mac, then “file:” will not work as expected (emits CR) so “binfile:” is needed to actually save files that use LF. Not difficult, but something to be aware of.
True, when using or displaying text generated from LC for use specifically in Terminal, you'll want to use "binfile:" to meet Terminal's expectation of Unix line endings.

But the Mac's bifurcated history apparently leaves us with most consumer apps accepting either line endings. I just did a quick check with TextEdit, Apple Note, and Microsoft Word, and all three render text as we would expect regardless whether I export to them with "file:" or "binfile:".
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

UKenGB
Posts: 16
Joined: Mon Aug 25, 2014 11:06 am

Re: carriage return

Post by UKenGB » Wed Feb 21, 2018 8:18 pm

I do understand the difficulties involved here, but, and it's a big but, that's what 'return' is for. It is the non platform specific line ending that allows any app using that code to work on different platforms. I applaud that.

But, CR and LF are the very opposite of that. They indicate an exact character. Even worse, the numToChar() function's whole raison d'etre is to translate an ASCII number into the exact character as specified in the ASCII character set which is a fixed standard and no-one should be futzing with any conversion from num to char or vice versa. However long it has been allowed to be done wrong, doesn't then make it right.

If any developer wants the flexibility of automatic line ending conversion, use 'return'. If you want/need to be specific, use CR or LF as appropriate. Surely search and replace can be done to allow fairly simple updating?

As LC has developed I bet there have been instances of backwards compatibility being broken. Let's say when it changes to v10, when there will no doubt be lots of updating required. So introduce it then.

Even better, add an option that allows the developer to choose between the 'classic' behaviour (everything is automatically converted) or the new 'Correct' behaviour (CR, LF and numToChar() are exactly as stated, no conversion). Default to the 'classic' mode and no-one has to re-write anything. But those who want the empirically correct behaviour for their new projects, can have it at the click of a checkbox.

Problem solved. :)

bwmilby
Posts: 438
Joined: Wed Jun 07, 2017 5:37 am
Location: Henrico, VA
Contact:

Re: carriage return

Post by bwmilby » Wed Feb 21, 2018 9:00 pm

Need to clarify... numtochar(10) will put the proper character in the variable, but when sending data out the server uses the line ending setting. LF is what is used internally by the engine. All line endings will get normalized to that setting on output.

In the IDE (non server), file: gets normalized and binfile: does not.

I think this is really just a Mac OS X issue as that is the only OS where it will cause a problem. I agree that the default should change to LF, but not about changing the constants.

Even though many programs still understand CR, everything new will use LF (check out XCode and everything it generates). Providing a backwards compatible option may not be a bad idea (could use the server code).
Brian Milby

Script Tracker https://github.com/bwmilby/scriptTracker

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: carriage return

Post by FourthWorld » Wed Feb 21, 2018 9:12 pm

I can only explain why it is why it is. I believe you underestimate the scope of impact on backward compatibility, though given that the future of the language is hopefully longer than its history perhaps it's worth submitting an enhancement request and see what the team has to say:
http://quality.livecode.com/
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: carriage return

Post by bogs » Wed Feb 21, 2018 9:37 pm

FourthWorld wrote:
Wed Feb 21, 2018 9:12 pm
I believe you underestimate the scope of impact on backward compatibility...
Meh, that only affect you long timers and anyone working with really far back IDEs and stuff... HEY WAIT A MINUTE, THATS ME !!!

Oh wait, they won't fix anything I'm using, never mind :twisted:
Image

jacque
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 7215
Joined: Sat Apr 08, 2006 8:31 pm
Location: Minneapolis MN
Contact:

Re: carriage return

Post by jacque » Thu Feb 22, 2018 7:17 pm

FourthWorld wrote:
Wed Feb 21, 2018 9:12 pm
I can only explain why it is why it is. I believe you underestimate the scope of impact on backward compatibility
It would break almost all existing scripts in an ugly way. I dropped the use of "return" as a constant the moment I moved to MetaCard, reserving it's use strictly for functions. I use CR for line endings exclusively and depend on the engine to translate correctly. In my mind, CR and return are synonyms.

Funny what you get used to. However I totally agree that if an ascii number is specified, it should remain intact. And in the IDE that's largely the case, so the glitch here is the inability to "export" text out of the LC environment to a browser without substitutions. On desktop we can do that using binfile, but that doesn't apply to browsers.
Jacqueline Landman Gay | jacque at hyperactivesw dot com
HyperActive Software | http://www.hyperactivesw.com

FourthWorld
VIP Livecode Opensource Backer
VIP Livecode Opensource Backer
Posts: 9802
Joined: Sat Apr 08, 2006 7:05 am
Location: Los Angeles
Contact:

Re: carriage return

Post by FourthWorld » Thu Feb 22, 2018 9:48 pm

As I think about this some more, it seems the specifics of this use case raise a larger issue: the value of doing server development on a system that most closely matches the production environment.

Unless you plan to limit deployment to Apple hardware exclusively, most server apps will be deployed to Linux.

RAM is cheap, and VMs are free. Time is expensive. One way to save time is to reduce the differences between development and runtime as early in the process as practical.

30 minutes to install and set up a LAMP VM can pay big dividends, greatly reducing a wide range of potential issues with server apps due to differences in case sensitivity; version and config for Apache, rsync, and others; firewall and other containment/hardening options; user management and permissions; so much more.

Extra bonus points that if you develop server apps in Linux this line ending issue goes away as well.

If it's good enough for Apple....
Richard Gaskin
LiveCode development, training, and consulting services: Fourth World Systems
LiveCode Group on Facebook
LiveCode Group on LinkedIn

UKenGB
Posts: 16
Joined: Mon Aug 25, 2014 11:06 am

Re: carriage return

Post by UKenGB » Fri Feb 23, 2018 1:20 pm

jacque wrote:
Thu Feb 22, 2018 7:17 pm
FourthWorld wrote:
Wed Feb 21, 2018 9:12 pm
I can only explain why it is why it is. I believe you underestimate the scope of impact on backward compatibility
It would break almost all existing scripts in an ugly way. I dropped the use of "return" as a constant the moment I moved to MetaCard, reserving it's use strictly for functions. I use CR for line endings exclusively and depend on the engine to translate correctly. In my mind, CR and return are synonyms.

Funny what you get used to. However I totally agree that if an ascii number is specified, it should remain intact. And in the IDE that's largely the case, so the glitch here is the inability to "export" text out of the LC environment to a browser without substitutions. On desktop we can do that using binfile, but that doesn't apply to browsers.
It wouldn't break anything in an ugly way. The only 'ugly' bit is using something like CR and getting ASCII 10 instead. Sorry, there's no excusing this. No matter how much rewriting it might require, that doesn't alter the fact that this is wrong. Using CR to mean LF and vice versa simply cannot be justified. I understand it can be convenient to be able to use a line ending that WILL adjust according to platform, but there is a constant available for that - return. CR and LF should do exactly what they say. Otherwise what's the use of any language when words and meaning get swapped around.

It's very simple:
  • CR should always mean Carriage Return (ASCII 13)
  • LF should always mean LineFeed (ASCII 10)
  • return means either, as adjusted to suit the platform
  • NumToChar() should always return the char. as specified by the ASCII number used
I won't say any more on this subject, but I'm disappointed that so many seek to try and justify the currently confusing state of affairs within LiveCode when 3 out of the 4 points above are not true and fail to work as expected.

bogs
Posts: 5435
Joined: Sat Feb 25, 2017 10:45 pm

Re: carriage return

Post by bogs » Fri Feb 23, 2018 2:21 pm

UKenGB wrote:
Fri Feb 23, 2018 1:20 pm
I'm disappointed that so many seek to try and justify the currently confusing state of affairs within LiveCode when 3 out of the 4 points above are not true and fail to work as expected.
Whoa, that was a bit extreme. I obviously can't speak for everyone else in this thread, but my comments weren't meant to justify any such a thing, I also find it as I said, 'weird' behavior. But, having said that, I don't think anyone else tried to 'justify' any thing either, you may want to try re-reading those comments without a bias.

My comment about work arounds is absolutely a fact, you are not new to programming in other languages as you said, I am sure you have seen it yourself.

Richard's comments told you *why* it was happening, and most everyone that stated ugly breaks in previous code has been using this language since it was in Hypercard. In some cases, they actually worked *on* Hypercard with that team. Their points of view are literally 'from the beginning', and should be given proper weight.

I am relatively new to this language (like yourself, I've worked in many others over the decades, and continue to do so), but I can see the point they are making as I work in much older versions of the IDEs that were around in days gone by.

As for the 'I won't say any more' bit, if you feel strongly enough that this should get fixed, you have the ability to and should submit it to the quality/bug report db (or see if it has already been submitted).

Lambasting people that answered you on a forum, or tried to give you a perspective on why something exists, seems a poor way to act about something that ultimately can be reported through the correct channel to get it fixed.

Lastly, I agree with your points about the problem, all of those *should* mean what they say, but I can see the others points in this discussion and as I said earlier, in this case, the work around is pretty easy to implement.

Good luck whatever road you choose to take on it.
Image

bwmilby
Posts: 438
Joined: Wed Jun 07, 2017 5:37 am
Location: Henrico, VA
Contact:

Re: carriage return

Post by bwmilby » Fri Feb 23, 2018 3:18 pm

Numtochar does always return what is asked for.

Put Chartonum(numtochar(10)) —> 10
Brian Milby

Script Tracker https://github.com/bwmilby/scriptTracker

Klaus
Posts: 13806
Joined: Sat Apr 08, 2006 8:41 am
Location: Germany
Contact:

Re: carriage return

Post by Klaus » Fri Feb 23, 2018 3:34 pm

bwmilby wrote:
Fri Feb 23, 2018 3:18 pm
Numtochar does always return what is asked for.
Put Chartonum(numtochar(10)) —> 10
Yes, this still works, although it (numtochar/chartonum) has been deprecated since version 7!

Post Reply

Return to “Getting Started with LiveCode - Experienced Developers”