Big problem with text logs is the moment something you log doesn't fit what you were expecting. A message has a new-line? all the sudden you either have to escape characters or you have to handle the data being put on the next line rather than the next log message. How do you detect when that happens? A message doesn't fit the format properly? Ok so let's encode the data, base64? now you can't grep the logs anymore for information, it's an opaque format with meta data, might as well use SQLite or some other structured format.
As someone else said, JSON is great for logging. Newlines get escaped, you just write line seperated json entries to a log file. JSON can be easily read by humans, and is trivial to parse in most programming languages.
I like SQLite, but I don't think logging is a good use case.
except if one line of JSON gets corrupted ... all your log storage does go to the trash.
Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?)
I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens.
Hence a day where a corruption of logs is more likely to happen.
True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line.
And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).
well, what a nice way to make a greper lose order of magnitudes in speed ; grep is fast because he does not line split.
And if \n is prohibited in string it is NOT JSON per ecma xyz anymore. it is something else.
Additional complexities that do not seem meaningful but when in congestion they add up to make your life a hell. But well, energy is cheap, VMs are cheap, operational expenses for cloud so less than coders, why bother?
With this kind of reasoning we will have so broken internet appliance coded with feet that we will experience a massive DDOS made by connected toasters and poorly coded cameras.
"\n" (that is, ASCII character "\", followed by ASCII character "n", producing escape sequence "\n" for newline) is not prohibited inside a string literal in that model. Only actual newlines (that is, ASCII characters 10 and 13) are prohibited, and that is already a part of JSON spec.
Like I said, the only place where JSON permits actual newlines is as part of insignificant whitespace. But because it's insignificant, newlines can be replaced with any other whitespace character, and the resulting JSON will have the same exact meaning.
And why would grep lose "order of magnitudes in speed"? If you were actually using grep, it'd work exactly the same as it does for any text file. The only thing that JSON does is add some structure to the contents, so that it can be easily converted to some tabular format that's more convenient for structured queries, aggregation etc. But you can still treat it as plain text for all purposes.
The days that I frequently need logs are the days that I need to prove that a particular application did a particular thing -- handled a particular transaction, sent a particular message, responded to/triggered a particular event, and so forth. Faults in these areas are applications software faults (often not even fatal ones), and are unlikely to affect the logger process at all, especially since it is insulated from the applications softwares by each running under the aegis of its own dedicated unprivileged user account.
Why don't you think sqlite is a good fit for logging?
Given the extensive testing including crash testing, it's plausibly more robust than plain text - because you're almost surely not using any kind of fsync's in your plain-text logger, so that text file isn't as incorruptible as you may think. And you may write a buggy logger, or use a buggy json implementation, or write incorrect error-recovery code when reading the file.
I'm skeptical that robustness is an argument in favor of plain text logging over sqlite.
Writing to files in general is a fairly difficult thing if you care about not losing any data under any circumstances, because you have to use the right syscalls for the semantics of your filesystem, which may somtimes be why you're looking at corrupted files in the first place.
Correctly implemented transactions can prevent you from dealing with that. Maybe that's worth giving up easily readable, searchabe and processable textfiles, maybe not.
This is a bad idea. What you want from text logs is the possibility to start anywhere in the file, look for a nearby newline and know it is the start of a record.
This is a very important property. It's also what makes UTF-8 resilient.
Why can't I have both? I am not against text logs, but it is good to have options. When binary format is an option I say that netstrings are probably better option.
For literal binary data you need to store the length information separately.
Netstrings are one option for ephemeral streams. But they are a bad tradeoff for persistent information: you have to read from the beginning of the stream until you find the relevant information. And a single corrupted byte destroys all the information after it.
Better options for persistent data are (pointer+length)s or memory-pools + ranges.
