Hacker Newsnew | past | comments | ask | show | jobs | submit | more lambda's commentslogin

These appear to be based on VI key bindings (hjkl).


Where in that did you see him advocating preventing the user from installing Steam? He's just saying that installing proprietary, DRM'd software reduces your freedom; so if you want to promote freedom, you shouldn't install that software on your computer.

One thing to keep in mind is that most of the time, he doesn't want to force anyone to do anything (other than make sure that code free if it was written under the GPL, even when modifications are made). He just suggests that people who value freedom do or don't do certain things.


Remember that they don't have just one brand on Facebook. For example, they have 1.3 million followers on their Chevrolet page. They have 1.1 million for Cadillac. 400k for Buick. 660k for GMC. 525k for OnStar. 350k for their "TeamChevy" Chevrolet racing page. Now, there's sure to be some overlap there; but that's still easily 2 or 3 million followers.

That's a factor of 10 more than you were estimating; and that's just based on a quick poll of brands that I took a look at.


> We think that the root server distribution should take regional Internet population size into consideration, so we hope the people in charge read this and act accordingly as the Internet continues to grow.

This is really the sum of their analysis? They "think"? The "people in charge"? No analysis of actual performance? No discussion of the fact that caching resolvers mean that most queries never hit the root servers? No discussion of the fact that this is just the distribution of sites that the 13 root server IP addresses map to via anycast, and that each site will have many actual physical machines behind that IP address?

How about doing some measurement to see if there is any problem? Some analysis that discusses what effects this might have? Some discussion of who those "people in charge" are, and what it's suggested that the should do?

This kind of drivel really makes me which that Hacker News had a downvote button for articles.


And yet, we've had plenty of software patents which have held up in court and caused significant damage to companies which were found to be infringing, and many more which have backed off due to fear of the same.

Mathematical facts aren't patentable. That much has been established. Perhaps a "pure" algorithm isn't patentable; but all software patents start out with something like "a general purpose computer which...", tying it to hardware and making it patentable. Good luck implementing that algorithm without a general purpose computer.

For instance, the i4i patent is a patent purely on an algorithm, and Microsoft lost that case and had to remove the feature from its software: http://en.swpat.org/wiki/I4i_v._Microsoft . I'm not sure how you can square that with algorithms not being patentable.


A lot of software predates UTF-8.

For example, scripts on Unix-like (Unix, Linux, BSD) systems. In order to determine that a file should be executed using an interpreter, they look for the ASCII bytes "#!" at the beginning of the file. If those bytes are not present (such as if the file is UTF-8 with a BOM), then it won't be executed with an appropriate interpreter.

Now, once the interpreter is found, it can interpret the rest of the text as it sees fit (for instance, interpreting it as proper UTF-8). But the first two characters of the file must be "#!".

Furthermore, the BOM is simply a bad idea. It was poorly conceived from the beginning. It means something different at the beginning of a file than it does in the middle (at the beginning, it is the BOM; in the middle, it is a zero-width non-breaking space. This means that you can't simply append files or strings that begin with the BOM, and have the result still be valid.

The BOM should not be used for sniffing files, except as a fallback. Sniffing is a terrible way to tell what encoding a file is in. The format should be recorded somewhere (in file metadata, by default, by the locale, or something of the sort). Sniffing should only be used as a last resort.


"Sniffing is a terrible way to tell what encoding a file is in."

In theory yes, in practice it's an awesome way to find out because the sniffer is more accurate then the webserver/mail client,human etc. Just about every piece of software I've written that has to deal with multiple encodings uses a sniffer, even if the format specifies it it's a good idea to check with a sniffer.


Yes, basically. UTF-8 doesn't encode a code point as a single integer; it encodes it as a sequence of bytes, with a particular order, where some of the bits are used to represent the code point, and some of them are just used to represent whether you are looking at an initial byte or a continuation byte.

I'd recommend checking out the description of UTF-8 in Wikipdia. The tables make it fairly clear how the encoding works: http://en.wikipedia.org/wiki/UTF-8#Description


No. You never need to know how to interpret the bits in each byte; you cannot address individual bits. Endianness refers to how different bytes within a multi-byte value are addressed; little endian means that the smaller addresses refer to the lower order bytes, big endian mean that smaller addresses refer to the higher order bytes.

Think of it as which order you write the digits in a number. Each digit (byte) means the same thing regardless of whether you are big-endian or little-endian. But in big endian, you would write one thousand two hundred thirty four as 1234, while in little endian you would write it 4321.


Yeah, it's really sad the number of legacy APIs which have standardized on UTF-16.

The Windows API calles UTF-16 "Unicode". Most Mac OS X APIs use UTF-16. JavaScript and Java both use UTF-16. ICU uses UTF-16. So while UTF-8 is technically superior in almost every way, it's going to be an uphill battle to standardize on it.

I appreciate that new languages like Rust and Go made the choice of UTF-8 as their native text encoding. But there's a lot of inertia for UTF-16, and I'm not sure it'll be easy to ever get free of it.


OS X APIs generally use NSString/CFString, which hide the actual encoding of the string; they can be any encoding at all internally.


While they could, in theory, hide the actual encoding, the APIs all refer to UTF-16 code units as "characters;" so while they could use UTF-8 as the internal encoding, you need to use and understand UTF-16 in order to interact with them properly. When you ask for the "length" of a string, you are told the number of UTF-16 code units. When you get a character at an index, you get the UTF-16 code unit. That's what I mean when I say the APIs use UTF-16; everything in the API that deals with individual "chracters" is actually referring to UTF-16 code units.

The same is true of JavaScript; while you could technically implement the strings however you want, the APIs are all oriented around UTF-16 code units. And the Windows API as well, is all built around UTF-16 code units.

The problem with all of these APIs is that they make the mistake of conflating characters and code units. They all make the assumption that a character consists of a single, fixed width integer, of some given size (16 bits in the case of UTF-16). It is better to distinguish between indexing in code units (such as bytes in UTF-8 or 16 bit integers in UTF-16) and indexing in code points, or glyphs, or whatever higher level concept you are talking about. Really, for anything higher than the code unit level, you should be dealing with variable-length strings, and not try to force that into fixed length units. With UTF-8, there's no temptation to treat a single code unit as being an independently meaningful entity, as that assumption breaks down as soon as you get past the ASCII range; while with UTF-16, it's easy to make that mistake, since it holds true for everything in the Basic Multilingual Plane, which contains most characters you're likely to encounter on a day to day basis.


Some old languages have also made that choice, albeit recently in Python's case:

http://www.python.org/dev/peps/pep-3120/


However, when it comes to the internal representation of text, things are quite complex as of Python 3.2: http://www.python.org/dev/peps/pep-0393/


ICU has some UTF-8 functionality. But ICU is really ugly.

http://userguide.icu-project.org/strings/utf-8


Did you read the article, including the part about Asian text? Like it or not, most text these days is embedded in markup languages like XML or HTML, in which all of the markup is within the ASCII range. This, coupled with the fact that UTF-8 gives you a factor of 2 savings over UTF-16 for the ASCII range, while only a factor of 1.5 increase over UTF-16 for CJK characters, means that for much text (such as anything on the Web), UTF-8 is actually smaller than UTF-16 even for CJK text.

Yes, ASCII is obviously too US centric; you can't encode any writing systems other than the Roman alphabet in ASCII. However, that's not at question here. The question is, which Unicode encoding should you use, so you can represent all writing systems in a single encoding. And the major contenders are UTF-8 and UTF-16. The point of this article is, for that purpose, UTF-8 is a far better choice.

> Also, if there's variable length encoding why can't we just do a proper way and improve size for the same computational cost?

What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will. UTF-8 is a carefully constructed encoding designed to meet several design criteria. For instance, you could get better size for a wider range of character sets by having a single byte to represent switching between character sets; so you could use that byte, and then a whole bunch of 2 byte CJK characters. But that would defeat one of the design goals of UTF-8, which is to be self synchronizing. That means that if you get a partial sequence (such as a sequence that has been truncated), you can start decoding the characters after a fixed number of bytes. In the case of UTF-8, you will never have to go more than 3 bytes before you can start decoding again. In my hypothetical scheme where certain symbols were used to switch between character sets, you would not be able to interpret anything until you found the next such symbol. This makes UTF-8 more robust in the face of errors.

Another design goal of UTF-8 was to be backwards-compatible with ASCII. Like it or not, ASCII has been the standard encoding for decades, and there is a lot of text in ASCII and a lot of software that uses ASCII delimiters and the like.

So, while it would be possible, in theory, to define a character encoding that is more "fair" than UTF-8, that ignores many of the other goals of the design of UTF-8. And UTF-8 is widely supported and used (it is the most popular encoding on the Web, even in places like Japan, and a close second in China), while a new encoding would require another large, global, and painful transition process to introduce.


The author compares UTF-8 to UTF-16, while there are a myriad better encodings than both for different Asian languages.

For instance: EUC-JP in Japan, BIG5 in Taiwan, GB in China, etc. Different EUC encodings are variable-length and are a lot more efficient since they put common subsets in lower parts of the table for each language, close to each other so they also compress better, while allowing tricks for text matching and searches (not really necessary for web sites, but it's nice to use the same encoding throughout applications sometimes). Russian and Greek are also basically multiplied by two in size.

There are a lot of other considerations.

If you think a 30%+ saving in size (and latency) is not a big deal, then you're a lot more likely to lose to local competitors. Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse. Add to that the fact that Asians browse the net A LOT from the phone and have done so for much longer than westerners and in a bigger percentage, and you have your problem exacerbated even further.

There is a lot more to consider and like it or not it's not as simple as "UTF-8 for everything and everybody, ever!"


> Note that gzipped or otherwise compressed text makes differences even worse at least in the case of Japanese - where UTF-8 text gets de-aligned all over the place to odd byte sizes and compresses worse.

That was not what this article measured. UTF-8 and UTF-16 compressed to virtually the same size.


Have you ever worked with a system that needs to deal with more than one language at the same time? What if your users want to mix Japanese with Russian in the same sentence? Or Japanese and simplified Chinese? (Yes, people do that.)

In the global Internet, UTF-16 and UTF-8 are the only games in town.


All the damn time I'm using several languages.

Then UTF (and EUC's) are the way to go.

It's not like you have to use the same encoding all the time.


> It's not like you have to use the same encoding all the time.

Then you are going to feed someone garbage. Why feed people garbage?


??

Not if you know what you're doing. Not any more than using utf8 exclusively all the time and for all purposes.


> Not if you know what you're doing.

This is nice in theory. In practice, people make mistakes. Make it easy on yourself.

> Not any more than using utf8 exclusively all the time and for all purposes.

Maybe I was unclear: Feeding me Chinese text in UTF-8 is not garbage. Feeding me anything in one of the GB encodings is garbage.

Garbage, to me, is text in an encoding I can't handle. If you only use UTF-8, that cannot possibly happen.


> What do you mean by a "proper way"? If size is what you care about, just compress your data. Compression will do a lot better for a much wider range of data than some clever encoding will.

That's precisely what I meant. Simple variable-length compression.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: