stam wrote: ↑Sun Mar 10, 2024 6:50 pm
FourthWorld wrote: ↑Sun Mar 10, 2024 5:31 pm
Why rush to a stent when some patients only need a cleaner diet and some exercise? Of course the answer will depend on the patient.
Poor example I'm afraid - one option is treatment and the other prevention - you can't prevent with treatment and you can't treat with prevention - you can only prevent. But I appreciate the sentiment
My layperson's imprecision with that metaphor is useful: I know you're an accomplished specialist, and would never presume that my limited knowledge in an area only peripherally related to my vocation could come anywhere close to the dedicated study and practice of a specialty over a good many years.
I wasn't suggesting deleting random bytes of a file - which file format can withstand that and in what context (outside of hard drive failure) would this mythical event happen?
You might be surprised what some folks do with their files. I've seen people attempt all sorts of things, and then express surprise that they broke the file. But even when used as intended:
I was suggesting changing the content even by 1 char. A prime example of the pain this can cause is trying to a TSV file where the field contents contain carriage returns. Records become misaligned etc. It doesn't take much at all to do this accidentally do this with a simple text file, but much more difficult to do with an SQLite file.
Any structured data is best read from and written to through a provided interface.
Sanitizing inputs is a useful practice with most file formats (mind those semicolons).
And to be clear I'm not comparing text files with RDMS's - I'm comparing text file storage with SQLite file storage (not an RDMS).
SQLite
is a relational database system.
Ease of coding for both is similar. Both can be used interchangeably in many situations - but many LC users simply don't realise how blazingly fast SQLite can be... and it's an excellent storage medium.
There are many storage options, each with their strengths and weaknesses. There is no one-size-fits-all solution. SQLite is quite capable and well suited for many things, but I can't recommend strongly enough the value of studying its internal structures and algorithms before choosing it for a given task, doubly so if tempted to recommended it to others for an undefined range of tasks.
SQLite can be used as a substitute for ZIP archives or Tarballs.
I would not make that recommendation. And given that both Zip and SQLite have grown in popularity over the years without either replacing the other, it seems a good many systems and application engineers see very different benefits in each.
So yeah so people prefer text files and tableFields, others prefer SQLite files and datagrids.
Flat files can be displayed in datagrids, and SQL query results can be displayed in columnar fields. I use both storage methods, and both display methods, each for different reasons.
You can use both, but in both cases, truly large amounts of data favour the latter - in my mind anyway.
At the heart of finding optimal solutions for a given task is understanding the system as a whole. Most applications can be seen as boxes where stuff comes in and other stuff comes out, and knowing what's happening on either end of that process chain will inform the choices within the box we're writing.
Logging is most commonly done with append, because it's the most efficient method commonly available (which is also why CouchDB chose an append-only model). Tail reads are so commonly needed most shells include a "tail" command for it. And it's not surprising that a system as flexible as LC includes scripting interfaces for most common file I/O operations ("seek", "read at", etc), in addition to higher-level conveniences like "put url".
With bbalmerTotalFluency's system, I've asked him for details, and SparkOut further reinforced the value of understanding the whole system.
It may be that rewriting all file I/O to use a relational database could be a good fit. It may be that the relationality isn't needed. It may even be that these are log files arriving from other sources over which he has no control.
I'm disinclined to suggest he rewrite anything until I know more about it. In the meantime, tailing is what he asked for, and with Bernd's example we see LC is once again up to the task.