SOLVED: Add/Remove from Beginning of File.

[ThatOneGuy]

Just as the title states. I need a way of adding data to or removing data from the beginning of a file. The only available options for writing files in LiveCode are to overwrite the entire file, overwrite existing characters, or add to the end. These won't work for what I need.

I managed a set of shell commands that can put text at the front of the file, but it is near impossible to remove text from the front of files using shell commands and it is very slow even with what I have working so far.

I would just read in the file, add/remove what is needed, and overwrite the existing file with the new content, but LiveCode does not read NULL characters when using the "read from file" command. They are converted into ASCII 32 characters when put into the "it" variable even though notepad++ displays them as [NUL] even when using "write NULL to file..." and chartonum(NULL) returns 0, definitely not 32. I have found no documentation showing how to read NULL characters and other non-printing characters, such as EOF, from file correctly. Even doing a "read from file.. until EOF", then "write it to file..." gives you a file with exactly the same number of characters, but it is corrupted beyond repair.

Of course, it would be best to not read in the entire file since that would put constraints on file size based on available memory... but I'll take whatever I can get.

[FourthWorld]

By default, LiveCode does indeed translate NULLs to spaces, and line endings are automatically translated to the internal UNIX standard the engine uses, 0x10.

But you can change this behavior to do raw reads with the "binary" option:

Code: Select all

open file tFile for binary read

As for the larger question of prepending a file, the only way any program can do that is to rewrite the file. The CLI tools you were using may make that convenient, but under the hood that's what they're doing.

That said, if you want to overwrite specific portions of a file, open the file in "append" mode:

Code: Select all

open file tFile for binary append

In conjunction with the "seek" command, or the "at" option with the "read" command, you can traverse a file rapidly to read and write from specific locations.

You may also want to see the Dictionary entries for the "binaryEncode" and "binaryDecode" functions, as these can be particularly useful for appending binary files.

[ThatOneGuy]

That's simple enough. I've got it working now.
Thanks.

[jacque]

You can also do it in one line:

put myBinaryData before url ("binfile:" & <pathToFile>)

[ThatOneGuy]

That's actually what I eventually used myself. The problem is, some of the files I need to edit are larger than the memory allotted to my app. Even with the URL function, it reads in the entire file, edits it, and writes it back to the same location. Anything greater than a few hundred megabytes is just killed and the file is replaced with a 1KB file with junk in it.
That's my new problem.

[FourthWorld]

You can read a file in chunks, e.g. "read from file tFile for 2048", where the last number is the number of bytes. You'll probably want to use a buffer size large than 2k, but you get the idea. Writing a temp file with the altered output will create a new copy with the data you need, and when it's all completed successfully you can delete the first file and rename the second to the name of the first. Most utils that work on large files (even LiveCode) use this temp file method when they need to work on the whole file.

If you only need to alter part of the file, you could store pointers at a known location in the file (a header is probably the simplest), and use "seek" or "read...at" or "write...at" to get to use those pointers to access specific portions of the file. If you use this method you'll want to open the file in "update" mode, and you won't be able to prepend for the reasons described earlier. But you can pad portions of the file for adding data, or use an always-append method like CouchDB does, with pointers in the header to the new locations.

As you play with this more I think you'll find LiveCode as capable as nearly anything else for working with files, just a little easier once you get the hang of it.