I found it useful for logging. Logging tends to have structured data associated with it. Trying to re-parse log files into some meta data gets tiresome and is error prone.
I'm a pretty big fan of line-separated JSON records... since JSON by default in most platforms serializes as a string, where linefeeds are escaped (\n) in strings, then you can separate records with a linefeed.
This can be streamed and even gzipped in said stream, meaning you get compression and a pretty easy format you can use in most platforms these days with very little intermediate processing or extensions.
With JSON you can have additional metadata on each record, and not have it affect the mainline... now, this cannot be queried directly, but it can be very easily streamed/imported elsewhere.
For certain tasks, I like to work on streams of JSON records too. jq(1) makes it really easy. However, requiring records to be line-delimited is a point of failure when accepting input from external sources. There should be no difference between:
{"foo": 1}{"bar": 2}
and
{"foo": 1}
{"bar": 2}
or any variations of common whitespace between the objects. Just parse objects incrementally, one at a time. Most JSON libraries supports this.
As for general logging: I prefer it to be structured text written to stderr, and have a PEG parsing it for me. stderr is pretty much a catch-all log stream, and some 3rd party software insist on writing to it (most do however offer a way to override this behavior, e.g., chromium-headless, libxml2, &c). Having a PEG for those cases means I can still get JSON for everything if I need to, even if the data in the log is unexpected. If I need to, I can then modify the PEG for that use case.
Of course, it's a case-by-case, pragmatic decision. If I only want to log certain data and disregard stderr, or if I can guarantee everything that gets written to stderr will be valid JSON, writing logs as JSON may be better.
Not meaningfully more complicated. You know when you've reached the end of your object of you are actually parsing it, so you also know when the next one can begin.
Actually, no, one is not necessarily parsing it. Xyr point was that using line delimiters allows one to retain one-log-record-per line semantics and use the log stream with text processing tools that understand those semantics but that do not understand JSON.
When you're working with a platform that has native JSON parsing, it generally doesn't support streams as described... so you either count braces and quotes, create a stream processor or use a delimiter... a delimiter is an easy enough solution, also allows for easier viewing in a text editor. It's about being pragmatic.
Honestly, I'd say that logging should be text. Yes, SQLite DBs are hard to corrupt, but it can happen, especially in the sort of catastrophic failure you would want a log for. And when data corrupts, it's generally easier to extract some degree of data from a text file than from a binary.
Big problem with text logs is the moment something you log doesn't fit what you were expecting. A message has a new-line? all the sudden you either have to escape characters or you have to handle the data being put on the next line rather than the next log message. How do you detect when that happens? A message doesn't fit the format properly? Ok so let's encode the data, base64? now you can't grep the logs anymore for information, it's an opaque format with meta data, might as well use SQLite or some other structured format.
As someone else said, JSON is great for logging. Newlines get escaped, you just write line seperated json entries to a log file. JSON can be easily read by humans, and is trivial to parse in most programming languages.
I like SQLite, but I don't think logging is a good use case.
except if one line of JSON gets corrupted ... all your log storage does go to the trash.
Heard about people login JSON ? (that may be truncated because log lines can be truncated when write is called with size > PIP_BUF in a concurrent environment?)
I know JSON is the new XML, but dinosaurs like me have learnt the hard way that logging should not be stored as a document but a journal of chunks considered truncated, and that relying on the atomicity of system read/write/close/open/seek/tell/unlink is a damned good idea because the day you need logs, is usually the day a major crash happens.
Hence a day where a corruption of logs is more likely to happen.
True though that when you have no crashs, you love JSON format. But even truer it is when an incident happens you want a resilient system that still can log in a reliable way.
The log format, as described above, is one JSON object per line. So if one line gets corrupted, this only affects that one line.
And this is possible because the only place where a newline can appear in JSON is whitespace (where it can be safely replaced with a space; although most JSON serializers provide enough control over the output that you can just make sure that it never appears there in the first place).
well, what a nice way to make a greper lose order of magnitudes in speed ; grep is fast because he does not line split.
And if \n is prohibited in string it is NOT JSON per ecma xyz anymore. it is something else.
Additional complexities that do not seem meaningful but when in congestion they add up to make your life a hell. But well, energy is cheap, VMs are cheap, operational expenses for cloud so less than coders, why bother?
With this kind of reasoning we will have so broken internet appliance coded with feet that we will experience a massive DDOS made by connected toasters and poorly coded cameras.
"\n" (that is, ASCII character "\", followed by ASCII character "n", producing escape sequence "\n" for newline) is not prohibited inside a string literal in that model. Only actual newlines (that is, ASCII characters 10 and 13) are prohibited, and that is already a part of JSON spec.
Like I said, the only place where JSON permits actual newlines is as part of insignificant whitespace. But because it's insignificant, newlines can be replaced with any other whitespace character, and the resulting JSON will have the same exact meaning.
And why would grep lose "order of magnitudes in speed"? If you were actually using grep, it'd work exactly the same as it does for any text file. The only thing that JSON does is add some structure to the contents, so that it can be easily converted to some tabular format that's more convenient for structured queries, aggregation etc. But you can still treat it as plain text for all purposes.
The days that I frequently need logs are the days that I need to prove that a particular application did a particular thing -- handled a particular transaction, sent a particular message, responded to/triggered a particular event, and so forth. Faults in these areas are applications software faults (often not even fatal ones), and are unlikely to affect the logger process at all, especially since it is insulated from the applications softwares by each running under the aegis of its own dedicated unprivileged user account.
Why don't you think sqlite is a good fit for logging?
Given the extensive testing including crash testing, it's plausibly more robust than plain text - because you're almost surely not using any kind of fsync's in your plain-text logger, so that text file isn't as incorruptible as you may think. And you may write a buggy logger, or use a buggy json implementation, or write incorrect error-recovery code when reading the file.
I'm skeptical that robustness is an argument in favor of plain text logging over sqlite.
Writing to files in general is a fairly difficult thing if you care about not losing any data under any circumstances, because you have to use the right syscalls for the semantics of your filesystem, which may somtimes be why you're looking at corrupted files in the first place.
Correctly implemented transactions can prevent you from dealing with that. Maybe that's worth giving up easily readable, searchabe and processable textfiles, maybe not.
This is a bad idea. What you want from text logs is the possibility to start anywhere in the file, look for a nearby newline and know it is the start of a record.
This is a very important property. It's also what makes UTF-8 resilient.
Why can't I have both? I am not against text logs, but it is good to have options. When binary format is an option I say that netstrings are probably better option.
For literal binary data you need to store the length information separately.
Netstrings are one option for ephemeral streams. But they are a bad tradeoff for persistent information: you have to read from the beginning of the stream until you find the relevant information. And a single corrupted byte destroys all the information after it.
Better options for persistent data are (pointer+length)s or memory-pools + ranges.
Don't worry, you can just have journald send loglines across the network to a centralised logserver...
... oh, wait, that hasn't been solved yet. Instead, there's a variety of hacky workarounds (send it to rsyslog and get it to ship; follow a journal with ncat and ship that; some others). Centralising journald logs is my current ops problem de juor.
Does that imply that LINUX was implemented poorly? I allays found the concept of socket file's to be mysterious things (coming from a windows background)
Socket file descriptors were in UNIX before Linux was written. They were eventually copied into Windows via "Winsock", and Windows shares the same basic idea of using ReadFile to access the sockets.
(The "reinvent it badly" is systemd replacing syslogd without re-implementing functionality that was important to a big subset of admins, because the people who wrote systemd are focused on the personal workstation case to the exclusion of all others)
But more fundamentally than the desktop / server focus, they either don't understand or don't value what makes UNIX great.
systemd is in many respects a major departure from UNIX philosophy, and I predict it will mark an inflection point in the quality and usability of Linux distros that adopt it, hence my efforts to migrate my own systems to FreeBSD.
For those wondering what I'm banging on about, please read The Art of UNIX Programming, which should have been called The Philosophy of UNIX:
Edit: what I mean by fundamental is that it shouldn't matter whether the tools are intended for desktop or server use; they should be designed to interoperate seamlessly using text protocols via pipes, sockets and files. That way they can be composed, filtered and transformed in ways not yet dreamed of by their creators. The systemd folks are falling into the Microsoft and Apple trap by trying to anticipate how their software will be used, instead of building it so it can easily be hacked upon for uses they themselves haven't dreamed of (which oddly seems to include 'servers').
To put it bluntly: systemd has neither the hacker nature nor the UNIX nature, and history has been unkind to OSs with neither. I'm betting against it being a Good Thing in the long run.
Yeah. As much as I hate ESR (I hate to beat that dead horse, but I don't want anyone getting the impression that I like him) he got it right here in this case, and Systemd definitely does the Wrong Thing.
It's not a Good Thing now, though: It's an attack surface.
He's an okay programmer, but he's got a frankly wacko political agenda (Ayn Rand, Libertarian, racism, paranoia etc.), he's mismanaged The Jargon File by putting in some of his own phrases which nobody else uses, and he's just an all-around unpleasant person.
Honestly, he's kind of worse than RMS, who is merely unpleasant to be around...
I agree on Rand and Libertarianism - although (as an Objectivist myself) it may interest you to know he's not an Objectivist, as he objects to several aspects of Rand's philosophy. I still have his suggested reading in that area on my TODO list.
But racism, and paranoia? I'd (seriously) like to see evidence of those if you have them. Nothing I have seen him write or do suggests he treats people in any way other than as individuals, on their merits alone.
Did you miss the blog article where he said a friend had told him that women were trying to have sex with OS leaders, so they could accuse them of rape?
Sounds pretty paranoid to me.
I might have been wrong about the racism stuff. I swear I saw some stuff, but I can't find it now, so I might have been confused with somebody else.
But at the end of the day, he's still an unpleasant person.
yea that's the one problem I've seen with journald that needs a proper solution. The other features it brings I've loved for my desktop but I wouldn't want it for long term data/logging yet.
When I did this, I found that I needed to be pretty careful when rotating logs. The online backup API fits the need well, but if you're using a wrapper lib around SQLite you most probably won't be able to access it.
