> The additional level of moderation doesn't seem to have added much value over my time participating. The closest thing I can think of as nice is combining duplicates, though moderators often seem to miss subtle differences between questions and in some cases information gets lost in the end. Generally, the moderation focus often seems legalistic or driven by artificial incentives rather than primarily focused on improving the breadth, depth, and quality of the site as a technical resource.
The problem is, they added the review queues, which prompts people to review questions on topics they might know nothing about, and give badges based on how much people review. So people go through those review queues and might be motivated to just get through things quickly without thinking very much about the effect of what they are doing; there is no meta review, no downside to just making hasty decisions that just prevent other people from engaging in useful discussion.
There is. There are "audit" questions designed to test whether the reviewer is paying attention. If they improperly vote to close a "good" audit question or improperly vote to leave open a "bad" one, they get banned from reviewing for a while.
OK, this may be a new feature, since I stopped paying close attention several years back when I got frustrated with the direction the community was going.
But I don't think that "audit" questions do much to address my issue. They enforce that someone is paying at least a little bit of attention; but they still emphasize things like voting to close a "bad" question, rather than helping the person asking the question ask a better question. They might help a little bit with people who are just not paying attention at all for questions which are black and white as to whether they are "good" or "bad" (such as blatant spam), but they don't help at all with pushing the "grey area" questions in the right direction.
That's the crux of the issue; the community is so focused on just cleaning up bad questions by closing them, rather than working with the person asking to help get them to be able to work better with the community.
There are far too many new questions each day for the people who are capable of mentoring an individual into asking a good question for Stack Overflow to scale.
Lets say that it takes 15 minutes in chat to help a person (who wants to be helped in asking a good question rather than getting the answer now).
Next, lets apply Sturgeon's law to Stack Overflow. There are 8000 questions a day and 90% of them are crap. Thats 7200 questions that need work. This is 1,800 hours of mentoring per day.
The close (and down vote) is such a minimal amount of work that it allows the group within the site that is striving for a particular vision of quality that it allows them to do the most they can. Furthermore, it is not infrequent that a person who provides assistant for trying to unravel a question from getting a "why can't you just help me now?" with assorted vulgarity interspersed.
On smaller sites, with a greatly reduced amount of questions per day the group capable of mentoring as well as moderating is able to spend the 15 minutes of time without significantly impacting the time taken to moderate and curate the rest of the questions.
With 8000 questions per day, its really hard to look at all of them. As I write this, there are only 16 people who have exhausted their close reviews for the day ( https://stackoverflow.com/review/close/stats - only 16 have 40 reviews). That doesn't show the people who are doing new questions, but it does give an idea of how shallow the bench of people who are spending time to moderate the site actually is.
Getting more people to help out would be great, but most people aren't doing anything to help.
There are also people who believe that up voting a newbie question and saying "welcome to the site" in the comments without improving the question is helpful.
The thing is, bad questions don't really matter in the grand scheme of things. If a question is bad, and it gets no answers, it will just languish. You don't need to do anything about it. Maybe have an auto-close after a week of inactivity or something. It probably doesn't really matter, as long as you actually sort questions by things like score, answer score, etc, so the bad questions just go down to the ends of all of the lists on pages no one ever browses to. Bits are cheap, a few bad questions being present on the site and not closed doesn't really hurt.
I don't expect every bad question will lead to a valuable mentorship opportunity. I've certainly encountered my fair share of those that weren't worth spending my time on.
Bad answers are more problematic; there's good reason to be vigilent about those.
And of course spam, obvious homework, and just completely incomprehensible questions should be closed instantly.
What I see is questions which are not phrased well but you can actually work out what they're asking being closed. So by the time I've written up an answer, the question is closed, and my answer is wasted. I can copy my answer elsewhere, and vote to re-open, maybe after editing the question to make it more clear, and it might get re-opened, but then I have to spend a lot more time and attention waiting for that. By closing out questions too aggressively, not only are people putting off newcomers, but also wasting the time of those answering questions.
Yeah, one of the things that really frustrates me about the review process is that they just incentivize taking some quick and easy action, not actually doing the right thing.
Effective moderation is not easy, and a very different skill set than answering technical questions, but the rep needed to perform many of these mod duties is just given to those who accrue enough rep.
There is a system known as the Roomba ( https://meta.stackexchange.com/a/92006 ) that will automatically delete questions. A question has a negative score, and has no answers will be deleted in 30 days. A question that is at 0 or less, has no answers, has a low view count, and has at most 1 comment will be deleted after a year. A question that is closed, negatively scored, no accepted answer, and no answers with a positive score will be deleted in 9 days.
So that does exist, but it takes a LONG time for it to get cleaned up in some cases. And if people jump in with a "this might answer your question, hope it helps", it will likely never get deleted.
> The toxicity is in all the unanswered questions.
---
> What I see is questions which are not phrased well but you can actually work out what they're asking being closed.
Consider fixing the question with an edit before writing the answer so that it won't be closed when you get around to writing an answer. This way its a good question from an earlier point in the life of the question. Furthermore, it makes it easier for other people to provide answers too. Leaving the question in a poor state with an answer makes it harder for the google only user to find the question and understand what is being asked.
---
Doing the right thing in review takes time. Many of the first post or triage reviews are done by people who haven't... fully bought into the philosophy of answers for people who haven't asked the question yet. They look ok to the standards of a 500 rep user who doesn't see what Jeff and Joel were trying for with Stack Overflow ( https://stackoverflow.com/help/privileges/access-review-queu... ).
There's no instruction for people for how to help in the review queues. As such, everyone is doing their own thing with their own "this is the quality that I'd write at, it looks ok." The only way to correct this currently is with review bans... and that has its own problems with negativity and the perception of Stack Overflow as elitist.
I'm a high-scoring user who has been put off by this exact behavior. The problem is, the behavior is pervasive among other high scoring users, and whenever I advocate against this kind of behavior, I get a whole bunch of pushback about it.
No one seems to want to change. No one seems to want to even engage with criticism about the culture; they just push back and pile on when criticism is brought up.
After a couple times of seeing this kind of behavior, and pushing back, and then getting piled on by people rejecting the criticism, I started to get tired of it and withdrawing somewhat. Later on, I some blatant sexism in a chatroom that was laughed off when I pointed it out to the moderators.
I've pretty much given up. I'll occasionally reply to questions people ask on my existing answers or if I see something that needs an answer or edit I'll make it, but I don't really feel like fighting the overwhelmingly elitist, sexist, and uncaring attitude on the site.
I think that this post is a nice gesture, and good that it's coming from the company and not just another user, but I worry that's it's much too late. The community has been attracting people who like it the way it is, and repelling people like me who don't, for a long time.
I’m in the same boat. 10k rep on SO and softeng.se, and I barely participate anymore. For me the most frustrating part is that the rules forbid vague questions with opinionated answers, but that at this point in my career most of the things I find interesting are all like that so there’s almost nothing I want to get answers to that I can ask.
Software architecture questions are very difficult to ask without triggering overbroadness. Another category that is off limits are getting started type questions, again they’re considered too vague (but that’s the whole point, the user doesn’t know where to start). And the illogical ban on pointing to research involving statistics also drives me batty. I’ve had arguments with moderators before where my answer was bad because it pointed to capers jones’ research, one of the most authorative sources of good software dev practices.
I always feel like it’s a game of dodge the moderator bot to get questions to not be closed, unless they’re of the boring easy to answer briefly and precisely variety.
I'm sure the moderator feels entirely justified by closing this question, and going by the rules they are perfectly right, but it still feels wrong to me to turn this user away instead of working with them to figure out a way to make their questions fit the website's format.
But, that takes a lot of work, and moderators are volunteers whose time is entirely on their own dime. I suspect that if the stackexchange network had paid moderators the outcome would be very different, but they probably don't have the funding model to support that. I myself also don't do moderator duties on the SE network, so I really shouldn't be blaming the moderators. I merely want to point out the frustration and the non-optimal outcome, not lay blame at the moderators' feet.
At an arms length it really seems like there should be some kind of delineations of the questions.
"What document backend to use with Drupal?" is a question whose answers are in constant flux. It is a bad match for and Authoritative Language Compendium. But it's also the kinds of thing people wonder about, it's a relevant tech question, and the dialog around something like that is easily as valuable as the accepted answer since everyone is skinning a different cat.
So instead of "overly broad, no discussion for you" I'd rather see those questions get routed to "conversation-SO" instead of "reference-SO"...
> "What document backend to use with Drupal?" is a question whose answers are in constant flux. It is a bad match for and Authoritative Language Compendium. But it's also the kinds of thing people wonder about, it's a relevant tech question, and the dialog around something like that is easily as valuable as the accepted answer since everyone is skinning a different cat.
IIRC, they did something like that. It was called programmers.stackexchange.com, and I used it a couple times and it was great.
Then some power-that-be decided it "wasn't working" (probably because it didn't fit the SO only-kind-of-good-question rubric) and changed the focus be a clone of SO with some minor difference in emphasis.
People weren't interested in maintaining the early programmers.se site. It was a "what to browse when there's nothing else to do." As the crud started piling up the options were either abandon it completely (that was threatened) or clean it up along with the associated rules of what makes a good subjective question.
It takes a lot of work to maintain a free for all site by the community. It takes less work by the community to maintain a heavily moderated site.
The culture of a site is a reflection of how much work people are willing to put into keeping it that way.
You'll have to be more specific about what that "crud" is, as SO's culture seems to have a tendency to reclassify the baby as dirt and throw him out with the bathwater.
> the options were either abandon it completely (that was threatened) or clean it up along with the associated rules of what makes a good subjective question.
Seems like there should have been more options on the table than "abandon" or "remake in SO's image." SO has a better population to answer subjective questions than most of the alternatives it tries to push people towards. They really should have been more flexible.
> There’s an even longer list of things that really belong on the new Programmers Stack Exchange, which appears to be degrading into fairly stupid water-cooler nonsense, and could benefit from an infusion of more meaty subjects, like these proposals:
> It was a the era of "what should I name my cat?"
A lot of those links 404 for me, but a lot of that stuff could have been taken care of by being better at defining what was on and off topic, e.g. it's a site for subjective questions about programming, rather than programmers answering subjective questions. SO has the moderation resources to handle that.
SO might have had the moderation resources. The early PSE site didn't (as described in https://meta.stackexchange.com/a/200144 ). The moderation actions trying to deal with that were done mostly by diamond moderator fiat and occasionally Jeff or Joel saying "no."
The community that was there at the time wasn't doing any significant moderation or curation of those questions. Consider https://softwareengineering.stackexchange.com/questions/4417... and consider is that useful? curated? Should "Quit my last job." (entire answer) be there? or "I tried to apply good programming technique to a language such as TI-83+ BASIC."? Count how many times "read books" or some variation is in that list.
It wasn't moderated or curated. Such pile on "give a pithy answer" was acceptable. And it made the site - trying for the more conceptual and architecture questions that Joel was asking for in his blog post harder to find, and they would get drowned out by the popular fun questions. That lead to https://stackoverflow.blog/2012/01/31/the-trouble-with-popul...
Its not that those aren't interesting questions, its that they are the wrong fit for that format. There are better places to ask for the best programming joke than Stack Overflow and Exchange sites.
And so, with the lack of moderation and curation from the fun days of Programmers.SE a different philosophy of the site was able to establish itself... and the site grew in activity.
The culture of a site is a proof of work. You have to work to maintain it. If there isn't work to do it, it can change. Its probably far to late to change to the less moderated version of the site again. However, if one wants to do that, actively work to curate the site so that those questions are acceptable.
MathOverflow has a fair number of fun and soft questions - but that comes at a lot of work of moderation and curation from the entire community... which they do. The community on other sites that wants those fun and soft questions has not shown as much a desire to curate them, and so others moderate them (not desiring to curate them either).
It's not a worthless question. If you're talking about subjective questions, obsessive curation is not the answer.
> Should "Quit my last job." (entire answer) be there?
Sure, considering there were better answers that had far more votes. It might have been better if it wasn't, and moderation could have taken care of it.
Answers like that are certainly not an indictment of the format.
> Its not that those aren't interesting questions, its that they are the wrong fit for that format. There are better places to ask for the best programming joke than Stack Overflow and Exchange sites.
That's never the kind of think I asked or liked at programmers.se, but like I said before, they threw the baby out with the bathwater.
You haven't really convinced me that subjective questions are "wrong for the format." All you've really shown it that an unmoderated free for all produces some garbage that doesn't fir in a high quality tech Q&A site.
The real answer for why programmers.se failed is probably that the powers-that-be weren't willing to spend the time and resources to meet a new challenge, so they gave up and did something else.
Its not a worthless question. However, its worth is drastically diminished by having so many answers, and so many joke answers in there. That people didn't want to curate it so that the joke answers and duplicate answers are removed and set a standard for providing an answer that is more than one line is why that type of question is closed - it takes too much time for the community that wants to maintain the site to curate it and keep it at a quality that matches the rest of the site.
The format that SE provides is a very direct Q&A that tries to cut out all of the excess and gives a single good answer. There are other sites that are better cut out for polling questions and other sites that do a better job of "what is the best ...?"
Trying to make everything fit in the SE framework puts a burden on the community that moderates and curates the material, and at some point people have said "enough" because they don't want to and others haven't stepped up to do it.
The "wrong for the format" is part of the design of the site. Note here that we're talking in a thread. You can follow the discussion. SE was designed to make discussions awkward. When there's a discussion on a site of significant scale, it becomes unmanageable and detracts from the Q&A focus. If that is a good design decision or a bad one is up to debate - but it is the design decision of SE. And there are so many other places where that discussion could be had. Why not ask that question here? or over on a competing discourse site ( https://www.askquestions.tech was started by April who wrote the post that sparked much of this)?
If Programmers (now Software Engineering) has failed is also up for debate. The name change wasn't so much of a "it failed" but rather "people keep getting the wrong impression of topicality" because yes, it took too much time to keep trying to keep the topic on what the powers that be wanted to moderate and no one else was willing to do the job of curating the content of the larger programmers scope... and after all, they're all volunteers - its really hard to make volunteers volunteer to do something they're not interested in doing.
Last time this subject came up on Hacker News here, I was inspired to post an idea I came up with after mulling the issue over a bit: mentors. I posted the idea on Meta here:
But while there definitely seems to be a movement afoot to tackle the issue, I can see why you're ready to give up. Look at the comments to my question and you'll find this (with a couple upvotes):
@klenwell 'users who downvote without comment or explanation other new users who have given some obvious attention to the composition of their question' I hear these things often. I don't see them when using SO, nor have I seen any evidence that such a set of users exists, and I have asked for such evidence, many, many times. None has ever been forthcoming. I have concluded that such users do not actually exist.
This after I provided a detailed personal account of this dynamic in action. I mean I guess I could start collecting timestamped screenshots or request the server logs from the Stack Overflow sysadmins. For now, I guess I'll just join the chorus of the disaffected here.
> No one wants the rules to change in a game that they perceive that they are winning (or have won). Moreso if they worked hard to win.
I'm one of those people who wants the rules to change in a game that I'm already winning. I worked hard to win; I fully engaged in the gamifiction, spending days at a time making sure I hit the max daily rep level, optimizing how I answered questions to get the highest score.
Even then, I tried to focus my optimization on ways that would help people more. For instance, one way to optimize is to post a very short quick answer, so you are able to get early upvotes before there's been a flood of answers for a question; but I would then edit and flesh out my answer. This has both advantages for reputation, since I could get early upvotes, and then later provide a more detailed comprehensive answer that would be likely to get further upvotes. It's also helpful for the person asking; it's possible that the very quick reply will be helpful, so they get an answer soon, but it's also possible that the more detailed reply with references to docs is what they need, so it's a way of making the gamification actually work to help people out more.
I think one of the big problems is when they added gamification of moderation. They added the review queues, which prompt you to review questions and answers that had been downvoted, voted to close, edited by other people, etc, and gave out badges based on numbers of reviews done. This means you'd go through a queue of questions and answers on topics you might know nothing about, and be prompted as for whether or not you agree with someone else's close vote or edit or the like.
When a question is not great, you could just vote to close it, or you could try to engage with the person asking the question and help edit it into a better form. I generally try to talk to the person asking and edit the question to be better; but getting internet points for getting through a lot of reviews quickly incentivizes people to make quick judgements, sometimes on topic they don't know much about. After they added the review queues, I found a lot more questions were being closed before you had a chance to answer or to engage with the person in comments to get them to clarify their question.
And while lots of people these days argue that closing a question is just a way to indicate that it's not up to standards but it can be fixed and re-opened, for people new to SO who aren't familiar with the culture it feels like a slap in the face and a rejection of their question. After a question is closed, it's a lot more likely that they'll just delete it, delete their account, or just stop participating.
Anyhow, yeah, there is an aspect of people not wanting to change the rules of a game that they are playing and winning; I just wish either the incentives were changed, or more people would take the view that the point of the scoring is as a rough measure of how much they are helping people out, and helping people out is the ultimate goal, rather than being an end in itself.
And I should also say that I'm not against closing truly bad or unsalvageable questions; I've had cases where I've tried to engage with someone to make their question better, and they just haven't made any effort, or cases in which people are obviously asking homework questions. But I really think that there should be some effort to help people out with asking their questions before just blindly deciding to close them out.
Yeah fixing this sort of thing is going to require a persistent and unpopular effort from the very top. They'll have to force out people who have contributed huge amounts of time to the community but who are nonetheless causing these issues. There will be a vociferous backlash. The lovey-dovey tone of the OP doesn't necessarily convince me that they have the stomach for it. But I hope they do, because I think it would improve things.
Well, flexbox was much simpler and easier to standardize on and implement sooner. It has been available in all major browsers since 2015.
Grid is great, and now works in most places, but it was good to have flexbox while grid was still in process; it only became available in Edge, in the final form, late last year, making it now available on all major browsers.
You're always going to have cases like this; where there's something that's simple that works now, and something better that comes later. If you always wait for the better one, it will take forever for things to get done. Not to mention that flexbox is probably simpler and easier to understand and use for some of the simple use cases, so people will still probably continue to use it despite the fact that grid is available.
Problem is that flexbox is a subset of grid. Or to be precise: flexbox and grid are just two forms in a set of layout methods that we already have and will have in future.
In normal architectural process we would establish first common infrastructure for all layout methods.
So flexbox and grid are just parts of larger entity - set of current and future layout methods. Yet flexes has to be units as they a) were from the very beginning ( see "proportional" units here: http://www.w3.org/TR/html401/struct/tables.html#h-11.2.4.4 ) and b) can be used in other layouts and properties (why not margin-left:1fr ?)
Note that each layout method has its own parameters in their own namespaces.
Currently set of CSS properties is about 400 in one flat namespace. Any junior architect will tell you that this is close to unmanageable state. But we still pushing new stuff on that x'mas tree. It will fall down by its own weight as some moment, but who cares ...
You can give input to the W3C without that, but yeah, it's the input from the paying members that counts the most.
Additionally, the W3C has some private mailing lists, does private working group meetings, and so on. So yeah, the WHATWG is a lot more open when it comes to input.
> Additionally, the W3C has some private mailing lists, does private working group meetings, and so on.
For all the web platform stuff (I don't know about the semweb and digital publishing side of things), you can basically ignore the existence of the private mailing lists (they get basically no traffic, and people like me start screaming whenever anyone tries to have a technical discussion there). I'd rather they didn't exist, but realistically they're not a barrier to participation.
The F2F meetings and telecons are more problematic (because both are inherently exclusionary, either in time or in travel), but those are at least publicly minuted and any resolution from them can be overturned.
In fact, I would say that it's not quite like a cathedral and bazaar here.
The W3C is like a legislative committee. Lots of things done by committee vote. Formal process that must be gone through to advance beyond committee. Petty politics that screw with actually getting things done. The committee chair having the ability to block work by procedural means.
The WHATWG is more like an open forum. The work happens out in the open. Anyone can participate in the discussion (the W3C committees frequently have private meetings, there have been private mailing lists, and so on). There are a few people who have commit rights to the repo, so who actually control what goes in and what doesn't, but they are generally willing to let in changes that have broad support, and are implementable, rather than letting such features get held up by political processes.
Besides the difference in structure, there's just a difference in attitudes. The W3C tends to strongly favor certain principles, like accessibility and modularity, but to the exclusion of compromise for technical reasons or real-world reasons. They also seem to have a tendency to get very attached to particular ways of doing things, without being willing to compromise. I think the biggest example of this was longdesc; it was never implemented properly in pretty much any browser, and very few people actually followed the spec and had it point to a URL (many people just copied the alt tag, or provided a longer description in the attribute instead of a URL), so even if browsers or screen readers had implemented it, the content wouldn't be useful. But people in the W3C made a big stink about removing this, and spent a lot of time and effort fighting and litigating over that, rather than actually trying to work on a different feature that could gain wider adoption.
The WHATWG tends to take a more pragmatic approach; pave the cowpaths, if there are differences between implementations do it in the most sensible way that preserved compatibility.
Now, what the WHATWG has produced hasn't always been the best; there are times when it's made mistakes in its approach. The drag and drop spec, which had been reverse-engineered from IE's drag and drop support, was a pretty bad; not sure if it's gotten better since.
But overall, the WHATWG has been a lot more productive in getting standards done that are actually used on the web, because they involve the implementers, and don't override them with tedious, drawn out, political battles over obscure features that no one uses.
> I implement XHTML1.1, you implement HTML5. We both get 1 month time.
Do you get to use libraries or not?
If you get to use libraries, there are plenty of libraries for both of these tasks, so it will be fairly equivalent. In fact, if you want to really support XHTML well, it will probably be more complex, because you have to take into account namespaces; a tag or attribute is not just a simple string, you have to consider the namespace it's in, so you would have to deal with that.
If you have to write it from scratch, I'd actually also bet on it being quicker for HTML5. The exact algorithm is specified in the standard; you just have to translate that from pseudocode into whatever language you're working in.
In XHTML, you have to look through several standards (XHTML 1.0, XHTML Modularization, XML 1.0 which includes the DTD, XML Namespaces, XML Schema), and then translate from the specification in a declarative style into a algorithm that can actually be used to parse the document. XML parsing is actually quite complicated.
> XHTML was a worthy goal – with it, we wouldn’t have a need to run headless Chrome for tests. We could parse the web, and actually use the data.
What? XHTML wouldn't have replaced the use of JavaScript to add features and load content. It wouldn't have replaced people's inappropriate use of tags. It wouldn't have replaced the fact that web pages are written for human consumption, and so don't generally try to include appropriate annotations on data for processing by other tools.
> HTML5 is amazing for existing browser vendors, developers, and in the short-term, users. But everyone else loses. Horribly.
Imagining that some magical standard is going to make everything better for users is a pipe dream. Providing good data formats and APIs for exporting data is hard, and is a different problem that providing user interfaces and interactivity. There's not some way in which XHTML could have been extended to serve both purposes; they are just too different.
This is why, when people care about providing programmatic access to data, they generally provide two endpoints; one serving HTML, CSS, and JavaScript for humans to interact with, and one for providing JSON for machines to parse. There have been endless attempts to try and make one standard that would work for both purposes, and they've failed because that's just not a good approach for solving the problem; instead, it's better to just have an API that both the UI code (whether on the client or server side) can use, and other developers can use if you want to expose it.
A lot of people who were interested in being standards compliant moved to XHTML 1.0 Transitional, which was the HTML compatibility subset, but they only ever served it and validated it as HTML, not XHTML, because if you served it as XHTML, one single stray < that someone had forgotten to quote somewhere would break the parsing of the whole page.
The piece written by Hixie was influential because it was a wake up call that the direction the standards bodies were going in was pretty much fruitless, and that there could be a much better way to do it which wouldn't involve breaking compatibility with all of the existing content and would give web developers and users features that they actually wanted.
> As web developers, we should follow the WHATWG and ignore the W3C, because the W3C have lost the political battle for HTML and we need to get our stuff working on browsers, all of whom follow WHATWG. But that's an unfortunately pragmatic approach that shouldn't amount to acceptance.
I fail to see how there is anything unfortunate about this. What about rewriting everything in XHTML 2.0 (https://www.w3.org/TR/2010/NOTE-xhtml2-20101216/), and having to be extremely conscious of any possible stray < that could sneak in to a page without being quoted, would have been preferable to:
1. Consistent parsing support for existing content, and content that might have slight problems like stray <, in all browsers
2. Standardization of things that people actually use to build web apps, like XMLHttpRequest and Canvas
3. Consistent handling of encodings between browsers, including encoding sniffing
4. Consistent handling of quirks mode vs. standards mode between browsers
5. Actually having browsers support compatibility with vendor-prefixed versions of features, because some browsers widely used introduced prefixed features that web developers actually started relying upon
And also, have you ever tried getting involved with the WHATWG process? I have, and I find that they are very receptive to intelligent discussion of issues.
What doesn't work well is to insist that you have a problem and that this particular solution must be used to address the problem; because a lot of times, it's easy to come up with some proposed solution but it then turns out that it's either a lot more complex in practice, your proposal does't fit in with the rest of the ecosystem well, or the problem can actually be solve just in tooling on top of HTML without having to change the spec at all and then wait for multiple browser vendors to all independently implement it.
> and having to be extremely conscious of any possible stray < that could sneak in to a page without being quoted, would have been preferable to:
Any system that publishes content that would let this kind of thing pass is incredibly insecure, and shouldn't be on the internet. Today it's a stray <. Tomorrow it's a stray <script>
It's no wonder software is where it is today with attitudes like these.
Not if that < had slipped in because it was in a piece of static text in a string somewhere in the source code.
You can apply mandatory quoting to untrusted input all you want, but there are going to be times when you have trusted strings that can still contain stray characters that will make the resulting markup invalid. And in many cases you don't want to have mandatory quoting for all of that, because these strings may have markup you want to include.
And yeah, you can argue that instead of generating content by appending strings, you should be building up a proper type-safe DOM structure that can be serialized. I'll wait while you go boil the ocean of converting every single web application framework that exists now outside of a couple of obscure type-safe functional programming frameworks, and in the meantime I'll be able to browse the real web without every other page giving me validation errors.
To be fair, I only use obscure type-safe functional programming frameworks. That's what I'm employed to do, and this obviously impacts my feelings on the matter. Personally, I think it's irresponsible to use anything that could be this unsafe. This doesn't mean everyone needs to use FP, just that frameworks and libraries should be chosen so as to guarantee safety. There are easy-to-use libraries for all these things in every language.
In no other world of engineering is this attitude okay. If you were a civil engineer and had to hold a license to practice due to the danger your designs could present to society, this attitude would eventually cause you to lose your ability to practice. It's becoming more and more clear that software can have similar levels of impact, and software engineers should practice as scuh.
I agree with you that we do need to do better about writing more robust software, and type safe languages are a good way to do that.
But what you're saying is as if you suggest that since the metric system is more consistent and more widely used than the English, I as a bolt distributor should start selling my bolts in metric sizes, despite the fact that the nuts that everyone has are in English sizes.
The browser vendors, at least, are working on implement their browsers in more type-safe languages (https://github.com/servo/servo), but even still they have to work with the content that is produced by thousands of different languages, frameworks, and tools, and millions of hand written HTML files, templates, and the like. Just turning on strict XML parsing doesn't make that go away, it just makes your browser fail on most websites.
A good first step in enforcing web standards would be if browsers would detect these rule violations, and -- instead of failing -- put a giant banner on the top of the page warning end users that the site may be compromised and could compromise data.
Soon, every business will be clamoring to fix their buggy software, and users will still be able to access the unsafe websites they so desire.
What if text_to_show wasn't sanitized? You got yourself a XSS. And if you do sanitize it (and keep it in a StrSanitized type), what are the chances of accidental XSS?
Really, what should have been done is a "user supplied tag", which automatically displays everything as plain text, like <user-supplied id="ahdjdh37736xhdhd"> Content </user-supplied id="ahdjdh37736xhdhd">
You would generally want the general purpose string type in your language to always be escaped when serializing, and only allow avoiding that if you opt-in explicitly.
So, for instance you'd have an H1::new(contents: TextNode) constructor, and you'd have to build a TextNode; if you build TextNode::new(text: &str), then it would escape it. If you wanted to explicitly pass in raw HTML, then you'd need something like HTMLFragment::from_str(&str), and it would parse and return the fully parsed and appropriately typed fragment object that could then be used to build larger fragments.
There might be some way to unsafely opt out, like HTMLFragment::from_str_raw(&str), that would just give a node that when traversed would just be dumped raw into the output, but that would be warned against and only used if you wanted to avoid the cost of parsing and re-serializing some large, known-safe fragment; it wouldn't be what you would normally use.
Your builder isn't really using types to guarantee safety. You can write untyped programs in a strongly typed language, by just coercing everything to strings, but this isn't what I mean when I say 'type-safety'.
The W3C actually does do some good work in other working groups; the CSS working groups seem to be working smoothly.
The W3C also oversees a lot of other standardization processes that aren't directly related to web browsers, like RDF. There are people who find this useful.
I think a lot of it is a power play. The W3C wants to be relevant, and the most relevant things in the web world are HTML, DOM, and CSS (there's also ECMAScript, but that already has a different standards body that owns it).
There are a lot of other standards that use HTML, CSS, and the DOM, such as ePub. The W3C wants to be the normative reference for these core web standards; many times, one standard will have to refer to the other, so the W3C wants to be the one that defines the "official" HTML standard.
But the W3C's process and policies are just terrible. They let people take over standards who have no intent on working with those most impacted by the standards, the people who develop the browsers that billions of people use daily to access tons of diverse content. So instead of just providing a lightly edited snapshot, possibly with some WIP features removed, of what the WHATWG produces, they start going in and meddling and making changes with insufficient justification so you have two forked standards providing a lot of confusion for everyone.
> The W3C actually does do some good work in other working groups; the CSS working groups seem to be working smoothly.
The W3C is also doing work in HTML at least too. If browser makers would actually participate as editors (like they do in CSS) then it would work just as well as the CSS working groups and others.
> If browser makers would actually participate as editors
Microsoft tried that, investing in easier to use GitHub tooling to allow a wide range of people to submit pull requests to update/fix bugs in the W3C HTML standard. "If you build the field of dreams, they will come...."
Nope. "They" had all gone to WHATWG ballpark, and all the W3C editors do is cherrypick (that's the actual word in the HTML 5.2 Recommendation) WHATWG's specs. It made a LOT more sense to just join WHATWG for HTML (and DOM).
> > The W3C actually does do some good work in other working groups
Right, W3C as a whole does a lot of good work. CSS is a good example, Web Payments, Web Authentication, Web Assembly come to mind as groups where a broad group really does come together and build consensus on how to solve hard problems. The HTML and DOM communities, however, have moved to WHATWG for reasons that happened long ago and apparently can't be un-done, even if a company with Microsoft's resources tries.
>Microsoft tried that, investing in easier to use GitHub tooling to allow a wide range of people to submit pull requests to update/fix bugs in the W3C HTML standard. "If you build the field of dreams, they will come...." Nope. "They" had all gone to WHATWG ballpark, and all the W3C editors do is cherrypick (that's the actual word in the HTML 5.2 Recommendation) WHATWG's specs. It made a LOT more sense to just join WHATWG for HTML (and DOM).
It won't work if only one browser maker will participate. If only microsoft participated and implementing things in the CSS WG then nothing really would get done over there too.
If all the browser makers would have editors in the w3c html spec (like they do in many other w3c specs) and agree to implement stuff there, then that would also work.
How would one convince the others to re-invest in W3C HTML and DOM? Microsoft's rationale a few years ago was that WHATWG wasn't a real standards organization with a patent policy, dispute resolution system, etc., and that created various legal and business concerns.
It turned out to be much easier to add a legal framework to WHATWG than to convince the HTML and DOM standards community to move back to W3C. Basically, people work on specs (and code) together in the places where there is a critical mass of expertise and energy being productively engaged. The key variable is the people, not the organization.
I don't understand the dynamics of how these critical masses of expertise coalesce, break up, and move around. I have learned that it's much more efficient to go with the flow than try to redirect it.
In HTML, as far as I can tell, it appears to be copying features from the WHATWG standard, paraphrasing them, and including them in their standard, with only a small notice on the acknowledgements page the the HTML standard contains parts derived from the WHATWG standard.
The browser makers did participate. They participated in the W3C working groups up until they were shot down when trying to propose to work on features that users actually wanted and would be backwards compatible rather than backwards-incompatible XHTML 2.0.
The browser makers then proceeded to do their work on rich web applications, with features like canvas and XMLHttpRequest, as well as actually putting together a spec for how to consistently parse HTML that would be compatible with real content, in the WHATWG.
When it was clear that the WHATWG standard was the one that actually mattered because it was what was actually implemented, the W3C invited them back in to start working on the standard together. That's what HTML5 was; the W3C agreed that they would start from the WHATWG standard, that they could have the same editor (Ian Hickson), and they wound down the XHTML 2.0 group.
However, various people involved in the W3C process proceeded to use bureaucratic moves to raise formal objections to things that had been changed, and escalated the issues above the editor. Eventually, he got fed up and left the process, and most of the browser vendors proceeded to continue working through the WHATWG. Microsoft was the last holdout, but eventually they too left the W3C process and moved over the the WHATWG as well.
So, the browser vendors have tried to work directly with the W3C on the HTML spec twice, once before the WHATWG split off and once as part of the attempted reconciliation. Both times, they were stymied by other people involved in the process who were more interested in purity and process than actually providing a forum for working out a good specification for real world implementation.
From my experience, it has generally done a better job of explaining things developers would want to know, especially in terms of accessibility and internationalisation.
The XHTML stuff was a long time back, and at that time, it warranted having a whatwg. Now that W3C is no longer insisting on XHTML (and hasn't for many years).
>When it was clear that the WHATWG standard was the one that actually mattered because it was what was actually implemented, the W3C invited them back in to start working on the standard together. That's what HTML5 was; the W3C agreed that they would start from the WHATWG standard, that they could have the same editor (Ian Hickson), and they wound down the XHTML 2.0 group.
This is the crux of it. The WHATWG is really usefull for browser vendors because they can do essentially whatever they want in it without anyone having the power to formally object to it (unlike the W3C). Now the whatwg editors (and thus browser makers) can say that they will listen to community feedback, but thats pretty much a benign dictatorship over the most important spec of the web.
> The WHATWG has always opposed the W3C's spec. They see it as confusing to have two "official" specifications.
The WHATWG has not always opposed the W3C's spec. The WHATWG explicitly agreed to work with the W3C to form an edited, snapshot spec based on the WHATWG spec. That's what HTML5 was supposed to be.
However, the W3C process then hijacked this, by dropping things from the WHATWG spec, adding things back that had been removed because they had never been implemented properly and implementing them wouldn't have been very useful, and so on. The WHATWG objected to this useless divergence.
While I agree the W3C's insistence on maintaining a parallel spec. is a silly idea they should absolutely abandon, I fail to see how any but the most biased perspective could conclude that they are "hijacking" a process of their own. W3C haven't dropped anything from the WHATWG spec.: that's separate and out of their control. They can drop what they like from their copy, it's their copy. Unless you're proposing that the WHATWG should be running the W3C, I'm not sure what you're getting at with the term "hijack". Surely you can't hijack your own thing?
Because there's no point in reconciling the specs if you don't actually reconcile them.
If the W3C spec is a snapshot, possibly of a subset, possibly with some editorial but not functional changes, then reconciling the specs is useful; it gives you want the W3C wants to provide, versioned, frozen specifications that can be used as the basis for other specs, for people to claim "full conformance" with a particular version, and so on.
Or, if the W3C process identifies real issues, then it should work with the WHATWG community to resolve those issues; since the WHATWG spec is being used as the upstream, evolving spec that these snapshots are being made from, it makes the most sense to get the changes into the upstream first, so you don't have to resolve the issues every time or maintain divergence forever.
However, the W3C instead just insisted on writing the spec the way it wanted, without regards to whether it would actually be implemented.
It makes no sense to publish a spec that will never be implemented by any of the projects that actually have real-world implementations, and differs from the spec that the implementers actually use. That just causes confusion.
So yes, they hijacked the process in the sense that the WHATWG agreed to work together with the W3C, but the W3C never really worked in good faith to resolve differences or provide technical arguments for their changes.
> there's no point in reconciling the specs if you don't actually reconcile them
completely agree
> yes, they hijacked the process in the sense that the WHATWG agreed to work together with the W3C, but the W3C never really worked in good faith to resolve differences or provide technical arguments for their changes.
I can't see either party working in good faith. In what way did the WHATWG's "agreement to work together" bear out in terms of a positive contribution to W3C's parallel spec., which we can both agree isn't a great idea but I'm failing to see how the WHATWG is a positive actor here in any way; they've forced W3C into an impossible position through political bullying and somehow W3C are vilified for hijacking something?
Ian Hickson, the editor of the WHATWG spec at the time, acted as editor of the HTML5 standard at the W3C for a while after the two groups agreed to work together. However, the committee had chairs who could override the editor.
However, despite a couple of years of effort working together, the W3C process allowed for a lot of people to raise objections that re-litigated a lot of things that had already been decided in the WHATWG process, or just didn't have implementer support, or whatnot. This led to the HTML5 draft specification being stale, as these objections held up migrating the editor's draft (which was the WHATWG specification) to the TR on the W3C site.
So lots of people who still saw the W3C as the "official" source of HTML were brought to an out-dated copy of the standard, because publishing more interim drafts was held up with all kinds of bureaucracy; and the W3C objected to linking to the WHATWG copy to suggest a more up to date version with bug fixes, so there was a fight over this.
The combination of the W3C's heaviweight process making it easy for lots of people to raise objections to slow down the process, and having the ability for those objections to be escalated above the editor, eventually made Hixie give up on editing HTML5 and just go back to editing the living specification.
The thing is, a specification only makes sense if it's actually implemented. Lots of non-implementers raising blocking issues on wishlist features, and then having to take the time to formally resolve all of those issues, does not make for a productive environment; and when the resolutions of those issues are escalated to chairs of the group or higher up in the W3C against the support of the implementers, it really hampers the process of coming up with a productive spec.
By the way, I haven't followed this drama in a few years, but taking a look at what's happening now, it looks like the W3C is essentially just plagiarizing the work of the WHATWG.
Features are generally discussed in the WHATWG, or implemented by browsers and then proposed, and the spec writing goes on there. After the spec is reasonably well worked out, the W3C is copying and editing some of the text into their standard.
Now, the WHATWG spec is under a Creative Commons attribution license, and the W3C does provide a small attribution in the acknowledgements section, so they are not violating copyright.
However, what they are doing essentially amounts to plagiarism as they are presenting themselves as the source for the standard. The introduction to the standard doesn't indicate that the actual work is going on in another group; they invite people to make comments on the W3C's GitHub. This is confusing, it gives people an out-of-date view of the standard, and it seems to be a move to make the W3C seem to still be the relevant authority when it's basically just cloning the standard from the WHATWG, but with enough wording and formatting differences that it could conflict and is hard to tell when it would.
Alternatively, they could fork the standard but do so more in the way that distributions package packages; take what's from upstream, have a separate set of patches that they apply on top that make it clear what the differences are. For instance, those patches might apply their layout, their disclaimers and the like, possibly disable some things that they think are underdeveloped or contentions and likely to change, and otherwise mostly just freeze the text. They could push any patches that they thought were for meaningful differences that they've fixed to the upstream project. They could properly attribute the WHATWG spec as the original source at the very top of the article, and list the editors of the WHATWG spec as the primary editors and the people doing the W3C release as maintainers of that particular fork.
But instead, they are listing as editors people who are basically just doing light paraphrases of the WHATWG spec.
> By the way, I haven't followed this drama in a few years, but taking a look at what's happening now, it looks like the W3C is essentially just plagiarizing the work of the WHATWG.
Ditto, and it's why if anyone asks about HTML and spec. conformance, I don't even mention the W3C, except to dissuade them from paying any attention to them. Their current HTML work is irrelevant and misguided.
My issue here is more with the historical negationism around the relationship between the organisations. The W3C's current HTML is, frankly, wrong-headed. But the context around their current situation is the fact that they've been bullied, cajoled and even somewhat ridiculed reputationally into these quite irrational actions by the WHATWG's very existence. That fact is lost when they're accused of acting negatively toward the WHATWG (e.g. hi-jacking apparent agreements and processes), when the actual background was WHATWG originally hi-jacking the specification of the web's central language.
Your post here is supporting the idea that the W3C's current direction on HTML is irrational. That's fine, I agree. But what they're doing is no worse than what WHATWG did originally with HTML5; the only differentiator is that WHATWG was extremely powerful (being primary implementors) and could use that power to win hearts and minds of pragmatic developers. The W3C have no such power and as such their wrong-headed actions are fruitless. But the equivalence is still worth pointing out.
For anyone confused about how the WHATWG came to write a new HTML spec entirely from scratch, after the W3C blocked the work happening at the W3C, this is a good place to start: http://diveintohtml5.info/past.html#webapps-cdf
I'm going to assume your point that the WHATWG is "extremely powerful" compared to the W3C was meant to be satirical.
Way back in the day, HTML was implemented as an application of SGML. SGML was a quite complex markup format, that had lots of features that were kind of complex to implement, and so web browsers didn't actually implement all of SGML, just that which was necessary for HTML and the HTML found in the wild. However, the HTML found in the wild was frequently invalid, so browsers had to implement some clever rules to do something reasonable with invalid markup.
Eventually, people thought it would be nice to have a simpler, easier to parse markup format, with a proper specification, and developed XML. Of course, once you have a shiny new thing, people will come and start bolting things on to it, and so they added features like namespaces, and they included some of the worse features from SGML like the DTD (it turns out, you don't generally want a document to reference its own schema and entities; an application should know what schemas it can accept, and having a URL for a schema in a centralized location means lots of poor implementers would actually download that, and entity expansion is just a nightmare).
However, XML wasn't compatible with HTML, and it certainly wasn't compatible with HTML in the wild. XML parsers are required to reject any invalid markup. The W3C developed XHTML as a way to have a subset that could work in either HTML mode or in XML mode; the idea was that people could start moving to XHTML in HTML mode, and then once everything was cleaned up they could switch to XML mode with strict parsing.
The problem was, strict parsing was never a benefit for either publishers or users. One little bug somewhere in a template substitution which allowed an unquoted < sign could cause the whole page not to parse, and users just be left with an error message. Without completely changing how the majority of web apps worked, there was no way to ensure that all of your content would be strictly compliant XML without the chance of breaking.
In the browser world, XHTML was implemented, with strict parsing in XML mode, but almost no one used it.
At this time, the browser landscape was pretty bleak. Netscape had died. Mozilla was in the process of building their new browser on a completely new engine, but early versions were fairly bloated and slow; it required a rogue group of developers within Mozilla to fork just the browser portion without a lot of the other functionality to produce what is now Firefox. Opera existed but had a tiny sliver of the market share; it didn't help that the free version came with ads, or you could pay for it without ads, while all of the other browsers were just free. IE was dominant, and eventually captured a huge percentage of market share, and then Microsoft just rested on their laurels and pretty much stopped development.
Soon enough, Apple came around and forked KHTML to build WebKit and Safari. In doing so, they did a huge amount of reverse-engineering effort to make their browser compatible with all of the parsing and layout quirks in IE and Gecko; the standards are not at all sufficient for compatibility. This gave them a browser that could actually be used on the majority of web content. With the introduction of Firefox instead of the bloated Mozilla browser, there was actually competition in the browser landscape again.
Meanwhile, the W3C decided that the failure of XHTML to be adopted because it met no ones needs was not sufficient; they went on to start working on XHTML 2.0, a backwards incompatible change with maybe a few nice features for document publishing but which didn't address any of the needs that web developers wanted, like rich interactive web apps.
So the WebKit, Mozilla, and Opera developers decided to get together and actually start providing a spec for what would actually work in the real world on real web content. They called this group the WHATWG. The W3C was not interested in working on this; the W3C group insisted on continuing work on the backwards-incompatible XHTML 2.0. Inspired by the clever algorithm the WebKit developers had come up with for parsing invalid content, they actually wrote up something a lot like that into a spec (with some improvements), and this spec eventually was implemented by all of the major browsers, providing much more robustness and consistency between browsers in how content was handled.
Other features that people wanted, like the ability to draw on a canvas, were prototyped in browsers and specced out in the WHATWG group. Some features were adapted from browsers that had already implemented them; for instance, Microsoft had implemented XMLHttpRequest, which turned out to be hugely useful for interactive web apps, so the WHATWG wrote up a spec for this and other browsers implemented it.
Google eventually wound up forking WebKit for Chrome, then merging back in, and eventually forking again into Blink, and joined this group as well.
Finally, the W3C realized that the work is was doing was irrelevant. No one was interested in implementing XHTML 2.0. What everyone wanted were the new features the WHATWG HTML specification, new features like the canvas, and so on. So they agreed to take the current WHATWG spec and edit it into HTML5.
However, it didn't take long for this to break down again. There were people in the W3C who objected to some of the changes made in the spec for the purposes of matching up with the real world. For instance, there had been some accessibility features like the "longdesc" attribute which were specified as containing a URL pointing to a page with a longer, more detailed description of the item in question for accessibility purposes (something that could, say, contain markup, when "alt" wouldn't be sufficient for a description). However, no browsers had ever actually implemented any reasonable way to get to this, and a survey of web content found that even if you did try to implement it according to the spec, very little content actually used it, and a large amount of the content that used it used it incorrectly, pointing to broken URLs or just including plain text like the "alt" attribute. So the WHATWG spec dropped this, and recommended other ways to link to descriptions which would show up even without special accessibility tools. One of the problems with specialized accessibility attributes is that people who aren't using screen readers can't easily test them out, so it's easy to bit-rot, but providing a normal link and annotating that so that screen readers can link it to the image make it possible to see in a normal browser.
Anyhow, some people in the W3C objected to this, and so rather than just providing an edited version of the WHATWG spec, they started tinkering with the spec themselves, adding things back in, removing some things that had been in the WHATWG spec. The W3C structure is very bureaucratic, and it has all kinds of members who are only peripherally involved with any actual tooling for web development, so it makes it very easy for various people with big egos but no real skin in the game to get involved in the process, while the actual browser developers who would be implementing the features can be shut out of the process.
So, eventually the cooperation between the WHATWG and W3C died down again, with the W3C publishing what it wanted, and the actual browser vendors continuing to work on their living standard document that is a much closer representation of the real world.
And this seems to be another case of yet another attempt to reconciliation and break down. The W3C decided agreed to start with the WHATWG DOM specification, then decided to make some incompatible changes without very much justification, and is now trying to publish a new version.
I think that in a lot of ways, there's an ego thing going on here. The W3C was originally started precisely to specify HTML, CSS, and things like that. While their CSS working groups have managed to stay pretty reasonable and are willing to work with those doing the actual implementation, their HTML and DOM groups keep on being hijacked by people with particular agendas, people who won't work in good faith to try to resolve differences reasonably, and people who think that because they're the W3C, they "own" the spec and so think that the WHATWG is just a rogue group, as opposed to a group of the people who actually have the most skin in the game because they have to actually implement the browsers that billions of people use without breaking a hugely complex and diverse amount of content out there.
> Way back in the day, HTML was implemented as an application of SGML.
[citation needed]
I'm unaware of basically any implementation of HTML treating it as an application of SGML; the only notable case I'm aware of is the old HTML validator.
Tim's original implementation of HTML didn't treat it as SGML.
> I'm unaware of basically any implementation of HTML treating it as an application of SGML; the only notable case I'm aware of is the old HTML validator.
Plugging my XML Prague 2017 paper on a SGML DTD for W3C HTML 5.1 here [1].
Sorry, maybe I should have said "inspired by" or something.
You're right, I'm not aware of any actual implementation, outside of validators, that treated it as such.
I did actually say a little later that no web browsers actually implemented it as SGML, but the sentence you quote could cause confusion; but it's too late for me to edit now.
In part, I think it's because there is still good work that goes on under the W3C umbrella in other areas; the CSS working group has managed to stay reasonable, learn from the mistakes of some over-engineered past standards, and continues to work with implementers.
Also, there are reasons to want some of what the W3C provides that the WHATWG doesn't. The W3C has many more member companies, and it can get them to sign off on patent rights so there's less likelihood of some one of Adobe's patents on page layout in InDesign suddenly being infringed by web browsers due to something in the spec.
And finally, I think that the W3C wants to stay relevant and so it keeps on trying to work from a basis on the WHATWG spec, and promising it will be good this time, and then it goes off and pulls this stuff again.
> The W3C has many more member companies, and it can get them to sign off on patent rights so there's less likelihood of some one of Adobe's patents on page layout in InDesign suddenly being infringed by web browsers due to something in the spec.
The patent policy only has commitments from members of the WG who developed the spec, so only have coverage from Adobe patents if Adobe is a member of the WG. (As it happens, Adobe still has one representative within the CSS WG, who happens to be one of the chairs.)
The problem is, they added the review queues, which prompts people to review questions on topics they might know nothing about, and give badges based on how much people review. So people go through those review queues and might be motivated to just get through things quickly without thinking very much about the effect of what they are doing; there is no meta review, no downside to just making hasty decisions that just prevent other people from engaging in useful discussion.