You are confusing syntax and semantics. HTML and the DOM are two different things. HTML is a string of characters (syntax). The DOM is a data structure (semantics). Normally a DOM is produced by parsing HTML, but it can be produced in other ways (by running Javascript code, for example).
S-expressions are a data structure, different from the DOM, but S-expression syntax is a syntax. Normally S-expression syntax is parsed to produce S-expressions, but can also be parsed to produce other things. S-expression syntax can be parsed to produce a DOM. The easiest way to do this is to parse S-expression syntax ino S-expressions, render those S-expressions into HTML code, and then use an off-the-shelf HTML parser to parse the HTML. But you could also write a parser that parsed S-expression syntax directly into a DOM if you wanted to. You could also write a transformation program that compiled S-expressions directly into a DOM without going through the intermediate HTML.
The answer to your question of how to add an attribute to an implied element is that it is not possible to do that in HTML. It is only possible to add an attribute to an implicit element of the DOM produced by parsing an HTML document that omits that element (because at that point the element is no longer implicit). The exact same thing is possible using S-expressions. For example, here's how you write tables in my library:
(:table (header header ...) (data data ...) (data data ...))
This string of characters is parsed by the Lisp reader to produce an S-expression that has a one-to-one correspondence with the string you see above. But then there is an extra processing step that transforms that into a different S-expression whose printed representation is:
At that point you can manipulate that S-expression in the same way that you manipulate the DOM (because they are both just data structures). Once you're done, you convert the S-expression to a DOM. At the moment that is done by rendering to HTML, but as I noted above that is just an implementational convenience to take advantage of the fact that HTML->DOM parsers are available off the shelf. You don't have to do it that way (and indeed the world would be a better place if it were not done that way).
All of this is trivial when dealing with S-expressions precisely because of the strict 1-to-1 correspondence between data structure and visual representation that does not exist in SGML-derived languages. That is why writing code for SGML-derived languages using S-expression syntax is so advantageous. (Actually, this is true for any language, not just SGML-derived languages. It's just a little more obvious for SGML-derived languages because SGML syntax already kinda sorta looks like a data structure representation so it's a little easier to grasp what is going on.)
> HTML is a string of characters (syntax). The DOM is a data structure (semantics). [...] S-expressions are a data structure, different from the DOM, but S-expression syntax is a syntax.
I believe this is where the confusion is coming from. When you parse HTML syntax, you get a data structure; this is the same as when you read sexpr syntax, you also get a data structure. Both these data structures are different from the DOM tree.
Can CL-WHO generate HTML that matches that? (i.e. feed both into a tool like BeautifulSoup and produce the same data structure?)
Outside of CL-WHO and Hiccup-type libraries, you can of course use S-exprs to represent the same data structure. Here's a hypothetical S-expr syntax that might produce the same data structure:
> Both these data structures are different from the DOM tree.
In the case of S-expressions that is true. In the case of HTML it may or may not be true. It depends on how the HTML parser is implemented. There is a "natural" mapping of HTML onto a parse tree that is different from the DOM, but that is not part of the standard (AFAIK).
> Can CL-WHO generate HTML that matches that?
Yes, though native Common Lisp does not provide c-like string escapes so putting in newlines is a little awkward. You could, of course, bring in a string interpolation library, but here's how you can do it without that:
which looks like cheating but is actually closer to the spirit of the original.
The PRE tag is really weird because it actually changes the way things inside it are parsed. You can actually implement that in Lisp too via reader macros. CL-WHO doesn't support that out of the box, but it's not hard.
I can't imagine anyone actually wanting to do that, though. The PRE tag is for presenting pre-formatted text without changing its appearance, so embedding other tags inside it is kinda perverse. [EDIT: I was wrong about this. See below.]
pre provides the simplified line breaking and usually a monospaced font. However, tags are available to do whatever else.
A major example is that the Vim editor uses pre for formatting syntax colored code to HTML (when you do that with :TOhtml).
The output is a pre block containing various span elements which are styled with CSS.
BTW where in the HTML spec does it say that the interior of pre is parsed differently?
If we are parsing HTML (to Lisp objects or whatever), we should preserve the exact whitespace. The reverse generation should regurgitate the original whitespace.
If we take the license to eliminate newlines, then we ruin pre. The fix is simply not to do that.
> where in the HTML spec does it say that the interior of pre is parsed differently?
I was wrong about that. I had a vague memory of putting HTML inside a PRE tag once and having it come out as if it were escaped, but apparently I hallucinated that.
> A major example is that the Vim editor uses pre for formatting syntax colored code to HTML (when you do that with :TOhtml).
OK, I stand corrected on that too.
> If we are parsing HTML (to Lisp objects or whatever), we should preserve the exact whitespace. The reverse generation should regurgitate the original whitespace.
> If we take the license to eliminate newlines, then we ruin pre. The fix is simply not to do that.
Right.
Actually, I just realized that I mis-read the example. I saw <br /> and thought it was </br>. (Maybe the OP edited it?) In any case, the example now reads:
This is a particularly bad example to demonstrate here because the whitespace in the code plays badly with the whitespace in the HN markup. But I tried running this code and it does work. Here is the output copied-and-pasted verbatim from my listener:
It was <br> and <br /> for my example (</br> isn't a valid tag). The point that I was getting at was that <br> and <br /> self-closing tag are represented differently (<tag>, <tag />, and <tag></tag> are all different) in a parsed SGML data structure (though they both are equivalent in the HTML DOM tree in the browser).
This is why you would need separate tags to emit them properly with an S-expr syntax (tag), (tag/), and (tag)(/tag) in my example.
> <script /> is invalid. <br /> is valid. <br></br> is invalid.
OK, then the best way to handle that is to let the HTML-renderer know that different tags need to be rendered differently if they're empty. Are there any cases where you would ever want to distinguish between the various kinds of empty tags?
Let's assume we have no attributes so I don't have to keep typing those. Then we have:
((:tag) content ...) => <tag> content ... </tag>
In this case (no attributes) we can unambiguously remove the parens around (:tag) and get:
(:tag content ...) => <tag> content ... </tag>
Now if we have no content we get:
(:tag) => <tag></tag>
All this is still completely regular, no special cases. But now if we write (:br) we get <br></br> which is not what we want. So we need to tell the renderer that some empty tags get rendered one way, and other empty tags get rendered another way. CL-WHO does this.
Notice that we have not actually typed any / characters. This is important. The role played by / in HTML is played by the close-paren in sexpr syntax. If we re-introduce the / into our new syntax we will have a hopeless mess.
> So a leading `nil' would be treated as a special case
That is exactly right. If (and this is a big if) we want to be able to write something equivalent to both <tag/> and <tag></tag> in the same document we have to be able to distinguish between those two things in the markup somehow. I just looked this up and the distinction that HTML makes between <tag /> and <tag></tag> is that the former content is EMPTY while the latter content is "" (i.e. the empty string). So really the Right Thing would be:
(:tag) => <tag />
(:tag "") => <tag></tag>
That will work, but now we have to remember to add an empty string in some situations, e.g.:
((:script :src "...") "")
Personally I would find this annoying, so I would choose to go with the lookup table.
S-expressions are a data structure, different from the DOM, but S-expression syntax is a syntax. Normally S-expression syntax is parsed to produce S-expressions, but can also be parsed to produce other things. S-expression syntax can be parsed to produce a DOM. The easiest way to do this is to parse S-expression syntax ino S-expressions, render those S-expressions into HTML code, and then use an off-the-shelf HTML parser to parse the HTML. But you could also write a parser that parsed S-expression syntax directly into a DOM if you wanted to. You could also write a transformation program that compiled S-expressions directly into a DOM without going through the intermediate HTML.
The answer to your question of how to add an attribute to an implied element is that it is not possible to do that in HTML. It is only possible to add an attribute to an implicit element of the DOM produced by parsing an HTML document that omits that element (because at that point the element is no longer implicit). The exact same thing is possible using S-expressions. For example, here's how you write tables in my library:
(:table (header header ...) (data data ...) (data data ...))
This string of characters is parsed by the Lisp reader to produce an S-expression that has a one-to-one correspondence with the string you see above. But then there is an extra processing step that transforms that into a different S-expression whose printed representation is:
(:table (:tr (:th header) (:th header) ...) (:tr (:td data) (:td data) ...) ...)
At that point you can manipulate that S-expression in the same way that you manipulate the DOM (because they are both just data structures). Once you're done, you convert the S-expression to a DOM. At the moment that is done by rendering to HTML, but as I noted above that is just an implementational convenience to take advantage of the fact that HTML->DOM parsers are available off the shelf. You don't have to do it that way (and indeed the world would be a better place if it were not done that way).
All of this is trivial when dealing with S-expressions precisely because of the strict 1-to-1 correspondence between data structure and visual representation that does not exist in SGML-derived languages. That is why writing code for SGML-derived languages using S-expression syntax is so advantageous. (Actually, this is true for any language, not just SGML-derived languages. It's just a little more obvious for SGML-derived languages because SGML syntax already kinda sorta looks like a data structure representation so it's a little easier to grasp what is going on.)