Writing to text file

[richmond62]

Documentation shows there is an append for the open file command

Is that a fact?
-

-
The politest words I can think of are 'inadequately documented'.

[richmond62]

This works without a hitch:

Code: Select all

on mouseUp
   set the ink of me to srcCopy
   answer file "Choose a text file to import"
   if the result = "cancel" 
   then exit mouseUp
   else
      put URL ("file:" & it) into pTEXT
      put fld "fTEXT" into aTEXT
      put pTEXT & cr & aTEXT into nTEXT
      put nTEXT into URL ("file:" & it)
end if
end mouseUp

So: 2 questions:

1. What is the need for append?

2. Why does this not work?

Code: Select all

on mouseUp
   set the ink of me to srcCopy
   answer file "Choose a text file to add extra text at the end"
   if the result = "cancel" 
   then exit mouseUp
   else
      open file URL ("file:" & it) for append
      put (cr & fld "fTEXT") into nTEXT
      write nTEXT to file URL ("file:" & it)
      close file URL ("file:" & it)
end if
end mouseUp

[dunbarx]

Richard.

To me, the "proper" time to set the manner in which one writes to an external file is at "write to file" time, and not at "open file" time. But that there are multiple ways to set that up is a strength, as we all love.

I just wanted to make sure that using "EOF" does what it purports, appends the text written to the end of any text already in the remote file. I did not make a test stack to see.

Craig

[dunbarx]

Richmond.

The politest words I can think of are 'inadequately documented'.

Yes, I suppose when looking up "append" (as, like you, I did) it would be more helpful if that word was also "related" to "open file". The dictionary is very good at that, usually, but not always.

Craig

[FourthWorld]

To me, the "proper" time to set the manner in which one writes to an external file is at "write to file" time, and not at "open file" time. But that there are multiple ways to set that up is a strength, as we all love.

I believe the need to specify the I/O mode when first opening a file is a function of the OS.

You'll find many things in scripting languages that are very close to the underlying OS calls they invoke.

With opening files, establishing the mode is pretty much what "open" does. This allows the OS to know whether it needs to establish a write lock on the file, and how to initialize the file handling structures needed for access.

IIRC (my systems programming book is across the room and I'm lazy this morning) both read and write are treated as separate streams with different structures to control the I/O operations. We scripters often think of them as one thing because the xTalk default is "both". But where we know we need only one we can save a little CPU time and memory by specifying the one we need. It's a small difference, but when you're writing something like logging every little bit helps, esp. in server environments where lots of concurrent use is expected (or at least hoped for).

I just wanted to make sure that using "EOF" does what it purports, appends the text written to the end of any text already in the remote file.

EOF is a constant for the flag that signifies the end of a file. Like CR, tab, and other constants, it does nothing by itself, and what we choose to do with it is up to us.

For example, to read the first line of a file we can write:

Code: Select all

read from file tFile until CR

So to read the full contents of a file of unknown length we could write:

Code: Select all

read from file tFile until EOF

For writes, if you want to add data to the end of a file you can use the seek command to position the storage device's controller to the end of the file before writing:

Code: Select all

seek in file tFile until EOF
write tSomeData to file tFile

The beauty of the append write mode is that it does the seek for you automatically.

[FourthWorld]

Documentation shows there is an append for the open file command

Is that a fact?
-
Screen Shot 2022-12-22 at 10.08.32 am.png
-
The politest words I can think of are 'inadequately documented'.

It looks like you found the Glossary entry. The Dictionary entry has more details and an example as you'd expect.

[stam]

I cannot see anything but a glossary entry for "append", as Richmond has shown. But not sure it really needs any more than that.

There is just over a line of text on it in the 'open file' section - i.e. open file for append. This suffices, doesn't it?

there is also a mention of this in the "Write to File" section

What may be good, especially for new users, is to have multiple examples of code in the dictionary showing all the use cases and the full usage. Some dictionary entries do do this, but most don't... that would have perhaps made the dictionary entry on 'open file' clearer. But that's a *lot* of work I suppose...

Also helpful if you are using auto-complete in the script editor: type 'open file', one of the options given to you is

Code: Select all

   open file filePath for append

I have to implement a logging mode for user access and actions in one of my apps (information governance requirement :-/ ) so this fits like a glove...


What I'm not clear about is how

Code: Select all

Open file for append... 
write to file...

is any different from

Code: Select all

Open file for text write...
write <some text> at EOF 

or even

Code: Select all

write <some text> at end

given that for logging the text isn't going to be humungous, so efficiency improvements can't be that great... or are they?
S.

[richmond62]

It looks like you found the Glossary entry.

Indeed I did:
-

Screen Shot 2022-12-22 at 7.09.57 pm.png (15.05 KiB) Viewed 5150 times

-
But, as you can see, I found it in the Dictionary; so I would be most grateful if
you could point me to the more extensive explanation.

[dunbarx]

Richmond.

I was careless, or at least inattentive, like you.

If you ask for "append in the dictionary, you can see that the "open file" appears as one of the four list offerings where it might be found. That is pretty good, and I do not mind scanning that list for a lead on a word of interest. And since "open file" is very near my thinking, it should have been straightforward.

But I was lazy, like you.

Craig

[dunbarx]

Richard.

OK. Thanks.

As I mentioned to Emily earlier, "append" seems perfect in all respects (very LC-like), and I see what you mean about it not being somewhat "out of place" over in "open file".

Craig

[jacque]

open file URL ("file:" & it) for append

That doesn't work because the open file command doesn't take a URL reference, it only needs a file path.

The append option has been available since MC.

[FourthWorld]

What I'm not clear about is how

Code: Select all

Open file for append... 
write to file...

is any different from

Code: Select all

Open file for text write...
write <some text> at EOF 

or even

Code: Select all

write <some text> at end

given that for logging the text isn't going to be humungous, so efficiency improvements can't be that great... or are they?

With logs, the size of data previously written to the file would only come into play with options that require reading the file into memory, appending it there, and then writing the whole thing back out to disk. The downsides of options using unnecessary reads were noted in earlier replies.

Here the examples you provided are purely writes, offering distinctions much more subtle. And at the level we're talking about, any performance difference is very minor, and may be considered negligible.

But since the cost difference to the developer is typing four additional characters ("for append" vs "at EOF") I've adopted the habit of using append since I first noticed Dr Raney had taken the time to implement that specific option.

It's noteworthy that most languages and OS APIs also provide a special case for append distinct from other write modes. It seems safe to assume this nearly-universal distinction is more than mere fashion. This thread has prompted me to dive deeper into this, finding at least some general functional benefit and a possible LC-specific benefit.

To better appreciate the nuance in play, keep in mind the chain of command with disk I/O:
1. Your script
2. LC Engine
3. OS
4. File System
5. Storage Device Controller

All of those are so tightly connected that it's not an especially long journey to move data and instructions back and forth through that pipeline, but like anything else in computing it's rarely a mistake to adopt efficient habits in case time may become critical down the road.

So let's walk through the examples you gave an imagine what's happening under the hood.

For starters, the Dictionary describes "EOF" and "end" as synonyms, so we can assume that's accurate and instead focus on the things that are different: "open...for append" vs "write...at".

We can explore "write...at" by considering how it work with a value other than "EOF", e.g.:

Code: Select all

write tData to file tFile at 1000

In that case, the "at" is likely interpreted as an instruction sent through the chain of command as a "seek", passing through the chain to ultimately prepare the storage device to begin the next operation at the specified location, in this case after byte 1000.

Given that, if the engine handles EOF like it handles any other constant, it would seem necessary to make an extra trip through the chain of command to first obtain the value of EOF, then passing it back through the chain as the value for "seek". That said, I haven't read the source for that part of the engine, so if EOF is indeed special-cased there with exceptional handling that would be good to know. It just seems unlikely given the presence of "append" for doing append.

In contrast, specifying append tells everything in the chain of command that the only position of interest is the end of file, so no integer needs to be obtained, or even passed back through the chain.

While we can discern an operational difference under the hood, the cost of one or possibly two unnecessary traverses of the I\O chain is so very low it would be completely unnoticeable, even difficult to measure, leaving us with the question: "Who cares, and why?"

The first part is easily answered: most systems and language designers apparently care enough to provide a specific write option for append distinct from any other. The collective cost to design, implement, document, and maintain this specialized write mode is non-trivial. Why have they all made that same choice?

Part of it may be simple optimization purity. To a systems designer even a few clock cycles saved is worth saving ("If you watch the nanoseconds the seconds take care of themselves" <g>). Logging is a good example where minimizing overhead is useful, because most logs are never read. Lots of activity, faithfully stored with each operation, in the hope that it will never be needed. But of course on the rare occasion when we do need log data it's VERY valuable, so we accept the ongoing overhead as just part of the cost of good maintenance.

But there's one subtle distinction that turns out to make it an especially good choice for some log files. This wasn't easy to find; most online discussions of append only describe how to use it, not why it exists. But Linux System Programming by Robert Love, a favorite in my bookshelf, offers the only why explanation I could put my hands on:

Append Mode
When fd is opened in append mode (via O_APPEND), writes do not occur at the file descriptor’s current file position. Instead, they occur at the current end of the file. For example, assume that two processes intend to write to the end of the same file. This is common: consider a log of events shared among many processes. At the start, their file positions are correctly set to the end of the file. The first process writes to the end
of the file. Without append mode, once the second process attempts the same, it will end up writing not to the end of the file, but to the offset that was the end of the file, before the data that the first process wrote. This means that multiple processes can never append to the same file without explicit synchronization between them because they will encounter race conditions. Append mode avoids this problem. It ensures that the file position is always set to the end of the file so all writes always append, even when there are multiple writers. You can think of it as an atomic update to the file position preceding each write request. The file position is then updated to point at the end of the newly written data. This will not matter to the next call to write(), as it updates the file position automatically, but it might matter if you next call read() for some odd reason.

Append mode makes a lot of sense for certain tasks, such as the aforementioned writing out of log files, but little sense for much else.

This would seem to imply that where a log is exclusively used by only one process, the functional benefit is negligible.

And even with the much-slower world of scripting vs machine-optimized object code, the performance confirmed by his description of the immediacy of the operation is also negligible.

But as with business, where we can minimize overhead we free up resources for user-facing value-adds. So for the low cost of typing four more characters to take advantage of a feature most OS and language designers think is worth providing, I'm in.

[richmond62]

Well, HTFF was I to know append could NOT:

take a URL reference, it only needs a file path.

when the documentation is 'scant' ?

Especially as:

Code: Select all

on mouseUp
   set the ink of me to srcCopy
   answer file "Choose a text file to import"
   if the result = "cancel" 
   then exit mouseUp
   else
      put URL ("file:" & it) into pTEXT
      put fld "fTEXT" into aTEXT
      put pTEXT & cr & aTEXT into nTEXT
      put nTEXT into URL ("file:" & it)
end if
end mouseUp

Which is how I ALWAYS export text to a file, works so well.

Come to think of things, as that works so well, unless one is working on some RAM-hungry system,
why bother with append at all?

[richmond62]

I was careless, or at least inattentive, like you.

But I was lazy, like you.

Possibly, and I love you too.

BUT, while I may sit for hours with a real dictionary reading from mention to mention as I did the other day when
getting some thing about the tree of life wrong (अकबचवाॆॆट) and enjoying my 40 year love affair with my
Sanskrit-English dictionary, I expect something slightly less fascinating, less seductive, and more to the
point with the LiveCode dictionary. That exercise prove, also, how crappy English Wikipedia is insofar as
it rendered अकबचवाट as akshaya vata; which made me foolishly write वाट as वात making me look like
a right cretin (the former meaning 'tree' as I intended, and the latter meaning 'wind').

So, while you may characterise my looking up 'append' and nothing further as 'lazy' I would choose to lay the blame
firmly at the door of the dictionary (and, by extension, its authors) who, frankly, did a sloppy job with 'append'.

[FourthWorld]

why bother with append at all?

URL syntax is used to move data among complete files. If you don't need to append an existing file, you'd have no need to use append.