Languages that were developed before the advent of Unicode, like C and C++, frequently leave the encoding of strings up to the system; but this can be complicated, because there are a number of different legacy 8 bit encodings, along with legacy encodings that don't fit into 8 bits. So they have two character types, char and wchar_t; char is a type that can always encode at least 8 bits, and wchar_t is an implementation defined wide character type, which is 16 bits on some platforms and 32 bits on others.
When Unicode was first designed, and released as Unicode 1.0, it was envisioned as a 16-bit universal encoding, with the expectation that you would use it just like 8 bit character encodings but with 16 bit character types instead. This encoding is known as UCS-2.
However, experience and unification of the Unicode standard with ISO 10646 showed that 16 bits would not be sufficient for all of the characters in all of the world's writing systems; in order to fit CJK characters into that 16 bit encoding, a large effort was made to unify characters from different languages that represented basically the same thing but might be written slightly differently in Traditional Chinese, Simplified Chinese, Korean, Japanese or the historical Vietnamese writing system. Additionally, a lot of decisions had to be made about inclusion of historical or obscure characters, and so there were many characters which couldn't be encoded properly in early Unicode.
This led to Unicode 2.0 introducing a surrogate character mechanism, now known as UTF-16, in which certain unused ranges of code points were used in pairs to represent a wider range of code points, in order to expand Unicode from a single 16 bit plane of characters to 17 16-bit planes.
Meanwhile, Ken Thompson and Rob Pike adapted an earlier 8-bit encoding of ISO-10646 called UTF-1 into UTF-8, which had a number of desirable properties; it was a superset of ASCII, the interpretation of the ASCII subset of bytes never depended on state, and it was self-synchronizing, so if you started reading in the middle of a multi-byte character, you could always find the start of the next character.
This meant it was possible to use UTF-8 in all of the traditional APIs and file formats which were compatible with 8-bit extensions of ASCII; which meant that you could migrate to Unicode support without significant changes to APIs and file formats. Also, for text which consists mostly of the ASCII subset, which includes many ASCII-based markup languages, it was much more efficient than UTF-16.
In the early days, however, many languages and systems which adopted Unicode started doing so via UCS-2, and then with the advent of Unicode 2.0 and surrogate pairs, needed to adopt to UTF-16. This was done somewhat inconsistently, and it can cause a lot of pain and confusion.
I've already probably written more than is necessary, so I won't go into the history of each system, but a quick survey of a few popular platforms and languages and their native string encoding:
* C and C++: Support 8-bit unspecified encoding strings (char * , std::string), unspecified width wide characters (wchar_t * , std::wstring, which are 16-bit UTF-16 on Windows and 32-bit UCS-4 on most other platforms), and explicit widths (char8_t * , std::u8string, char16_t * , etc...) which are explicitly UTF-8, UTF-16, and UCS-4.
* Windows uses 16-bit wchar_t * , nominally in UTF-16 but not enforced, as its native string type, and also provides 8 bit APIs with character encoding which can vary at runtime
* Linux and other Unix systems generally use 8-bit char * in an unspecified encoding (which can vary at runtime) as their native string type; most systems these days are using UTF-8 as that 8-bit encoding, but it's not guaranteed
* Java, JavaScript, and C# all use UTF-16 as their native encoding, though in many cases there are still APIs which make most sense for UCS-2 rather than UTF-16.
* Python 2 used unspecified 8-bit encodings for the str type, and a custom encoding that supports all of Unicode for the unicode type. Python 3 changed so that the bytes type was 8-bit with unspecified encoding, and str type was Unicode in a custom encoding; a bytes type alias for str was added to Python 2 to ease the transition, as was a unicode type alias for str in Python 3
* Go and Rust use UTF-8 as their native string types
* Most text-based file formats, such as HTML, have standardized on UTF-8 as the backwards compatibility with ASCII is a big benefit for such formats.
There are a few references on why UTF-8 is generally a better choice for languages and APIs than UTF-16, despite some of the legacy use of UTF-16 in languages and APIs:
I had a similar experience to this at an early programming internship while I was in college 20 years ago.
I was working on some component and needed to add a new table to our database. I was fairly new to SQL at the time, so I wanted to do some experimentation and prototyping on my own, before submitting the schema changes to our DBA in order to get them into the shared dev database.
Oracle did have a free trial version you could download for dev purposes or learning or whatever, but like this mentions, it was slow and cumbersome to set up that I soon decided it would be faster to spin up a PostgresQL database, make any necessary changes to our schema and code to be compatible with Postgres, and do my prototyping there, then submit the schema modification request to our DBA. So that's what I did; it took a single evening to do all of that in Postgres, even for someone who was pretty much brand new to SQL.
After I submitted my schema chages for review, rather than emailing review comments back, my DBA made me come down for a meeting to explain the issues. Of course this was intimidating to me as a young itern; had I done something so wrong that it deserved a talking to?
After all that, it turns out the issue was that I had ordered a VARCHAR column before some other column of fixed width in the schema of the new table; and apparently, it's preferable to order all fixed width columns before all variable width columns in order to speed up the column accesses. I agreed with the DBA that I could change the order of the columns, though I did have to point out that this particular table was a table of worldwide regions like "North America", "South America", etc, and that there would never be more than 5-10 of these, so any optimization of this particular table was likely premature.
After all that experience; the complexity of just getting a dev environment up, the fact that production instances cost somewhere around ~$50,000 per CPU per year, the fact that we had a full time DBA who was spending a substantial fraction of her job letting interns know that they needed to apply some trivial optimization that you would expect such an expensive database to just do for you automatically, I resolved to never touch Oracle again if I could avoid it, and have had good luck in that I've never had to deal with Oracle in any jobs since.
Funny thing was that the project I was working on was named "ASAP" which officially had no expansion but unofficially stood for "Another Siebel Avoidance Project", because we were actually using our Oracle database as a place to dump information for which the primary store was Siebel, due to how much more of a pain it was to interact directly with Siebel so doing a periodic dump into Oracle and then building our API on top of Oracle was a better choice.
> He's been the sole maintainer on the project since 2012 [1] and has never been compensated for it
I have to object to that. I work for a company which pays him to build branded versions of FUSE for macOS, and has been doing so for several years.
He does reliably ship us new builds, but it has sometimes been a little difficult to get him to respond to feature requests or even patches that we send him.
I don't believe that he is scaling much better with his commercial licensing than he was as a FLOSS maintainer.
As it is, we're fairly happy with his work to continue to support new versions of macOS, which is fairly hard to ship systems software on these days (the code signing and notarization requirements keep getting more and more onerous, system integrity protection keeps getting in the way of debugging, and so on), but it would be nice if we weren't limited to a single person working on the project.
I'd really rather he kept the project open source, and continued to provide custom builds for companies on a contract basis. Due to the difficulties of building and signing the kernel extension across all of the versions of macOS that it supports, just having the source available but keeping the binaries under a proprietary license would probably be sufficient to make it worth the while of most companies to license with him to provide custom builds like we do.
Sounds like a lot of additional work for him to provide custom builds? Maybe he wants to be paid for the core product development that provides most of the value?
No, the custom builds aren't much more work than the vanilla builds, it's getting the build environment set up in the first place that's a pain, since you have to build against a different SDK for each version of macOS you support; and the custom builds are what he sells to customers, since with a custom build it can be installed and updated independently of the free-as-in-beer version that is used for things like SSHFS and might be updated out from under you in a way that breaks your application. Updating with your application means that you can test each new version and fix any issues or report back to him before shipping it.
Reading through this, this sounds like mostly a capability based architecture, with process isolation replaced with a statically typed bytecode in which you can verify that each module can only use capabilities passed to it.
I was wondering why the focus on copying memory between the processes. It should be possible to make capabilities which represent a pointer and a length (and maybe an access mode), which could be used to give a process direct access to shared memory safely. I don't know if that could be done with low enough overhead for small objects, it's possible that there would need to be a threshold below which you would just copy values to be efficient, but it seems like exploring fat pointer capabilities would be worthwhile.
Memory mapping files or inter-process shared memory is the very coarse-grained version of this, using hardware protection. I feel like it should be possible to do something more efficient and finer grained with pointer capabilities and the static type checking that is done to verify WASM during compilation, but it may be a significant research project of its own.
For languages that can express unforgeable pointers as first-class concept, that is indeed a very attractive, fine-grained approach. Unfortunately bringing that to languages like C/C++/Rust is a different matter altogether.
Since we want to support those languages as first-class citizens, we can't require GC support as a base concept, so we have to treat a nanoprocess as the unit of isolation from the outside.
Once we have GC support, nothing will prevent languages that can use it from expressing finer-grained capabilities even within a nanoprocess, and that seems highly desirable indeed.
(full disclosure: I'm a Mozilla employee and one of the people who set up the Bytecode Alliance.)
That future possibiilty reminds me of https://en.wikipedia.org/wiki/Singularity_(operating_system) - where process/address-space isolation was replaced with fine-grained static verification of high-level code (presumably not the first experiment in this area).
Indeed: that and many other things are prior art in this space. And there is a lot of prior art for what we're working on—this is not meant as an academic research project! :)
Yes, one of the answers I want to give any time someone asks "why will WASM succeed when the JVM didn't" is that there is 25 years more experience and research to draw upon.
And yet bounds checking access validation was left out of the design, something that most of previous research projects took care to taint as unsafe packages when present.
> For languages that can express unforgeable pointers as first-class concept, that is indeed a very attractive, fine-grained approach. Unfortunately bringing that to languages like C/C++/Rust is a different matter altogether.
The semantics of these languages aren’t incompatible with unforgeable references, though: it generally works in practice, but it’s technically undefined to create pointers out of thin air. Why can’t we take advantage of the standard here to disallow illegally created references? (Which, as I understand it, many other vendors are already beginning to do with e.g. pointer authentication and memory tagging.)
What would allow other languages to represent unforgeable pointers as a first class concept and not C/C++/Rust?
Forging a pointer is UB in all of these languages as far as I know.
It seems like you should be able to have opaque types that represent these unforgeable pointers which you can't do arithmetic on or cast to raw pointers, but can access values in type safe ways, or provide a view to a byte slice which does bounds check on access.
Is there a good place for discussion of this design? I seem to be having this conversation with you and Josh both here and on Reddit, and it seems like a lot of the discussion is spread out in a lot of places.
In unsafe rust you can arbitrarily increase the length of a vector/string by modifying the stored length. You do not need to forge the pointer itself to break the pointer's invariant.
You would need to do either static or dynamic bounds checking when accessing memory via these capabilities. You obviously can't just give arbitrary code a pointer and let it read however far it wants past the end of it.
Given that most code in Rust is safe code and includes bounds checks before access, you should be able to have the verifier rely on those when they exist, and add in bounds checks in cases in which the access is not protected by a bounds check.
Maybe that would be intractable, or to inefficient to be worth it with all of the extra bounds checks. I'm not sure. I'm asking because it's something that I feel should be possible, but I haven't been involved in the research or development, so I'm wondering if those who have been more involved have references to discussion about the topic.
That is how we support references in the Rust toolchain right now, via wasm-bindgen, and it's an important part of making unforgable references work for languages that rely on linear memory.
It doesn't help with making capabilities more fine-grained, though: we have to treat all code that has access to that table as having the same level of trust.
Google only does it for 3 years, and that's from launch. Apple is supporting devices 5 years old.
I've been buying Google phones (Nexus/Pixel) because they do get the longest support, but I still get frustrated by the fact that if I buy one late in the cycle, I only really get 2 years of support for it.
Besides the usual ones, I've had to use IDA Pro occasionally for compatibility purposes in my job as a NAS vendor.
There are lots of apps that make lots of assumptions about how filesystems behave, generally based on the local filesystem and maybe on one popular networked filesystem for the platform (NFS, SMB, AFP).
If one of those assumptions is violated, applications can crash or refuse to interact with you. Some just refuse to write to any networked filesystem. Some run only on whitelisted filesystems. Some will hit an error due to an unsupported operation on your filesystem, fall back to some ancient code path using long-since deprecated Carbon APIs that only work properly on 32 bit systems, and so truncate all of your data to 2 GB.
Problems like the latter are really helped by being able to do some reverse engineering of the application to figure out why the heck it just writes out the first 2 GB of the file.
Because this isn't our bread and butter but only an occasional tool in our toolbox, the licensing on IDA Pro can be rather frustrating. We use it only once every couple of years to debug some kind of compatibility issue like this, and so we usually have to dig around to figure out if we still have valid licenses, deactivate systems that we're no longer using, and so on.
Both libraries also rely on a number of techniques from existing published research.
Hyperscan is ruthlessly optimized for x86 platforms, but has dropped support for non-x86 platforms. Rust's regex crate is designed more as a general purpose, portable regular expression implementation, suitable for wide scale use.
I like 'ruthlessly'. It sounds very butch, especially applying to an activity largely carried out by seated, balding men hunched over terminals in an office.
Rust regex is suited to wide scale use. By comparison to Hyperscan, it is not suited to the kind of large scale cases that Hyperscan handles, but it's well done.
Many of Hyperscan techniques are sui generis, not published. Hopefully some of that will be rectified over time.
Am I missing something, or are they not only not documenting the chip, they are also not even releasing the compiler, but requiring you to use a cloud based compiler:
> You need to create a quantized TensorFlow Lite model and then compile the model for compatibility with the Edge TPU. We will provide a cloud-based compiler tool that accepts your .tflite file and returns a version that's compatible with the Edge TPU.
This seems like a new low in software freedom, and pretty risky to depend on as Google is known to shutter services pretty often and could just decide to turn off their cloud-based compiler at any time they feel like.
Chips without a public toolchain are not worth investing your time in. It is bad enough if your work is tied to specific hardware for which there may at some point not be a replacement but to not even have the toolchain under your control makes it a negative.
Seriously Intel is already struggling after buying Nervana.
I went to their shing dig and they were working their butt off to wow the developers who were invited. When I asked for hard number they were very mum about that and very evasive.
The timeline for Nervana chip have been always seemingly in this mystical horizon that is never solidified to a real date but over yonder.
Google is going to pull this crap? They got better software expertise than Intel though they may be able to do it. But after that fiasco with Angular 1 to 2 I wouldn't trust Google with any early version number.
Nervana had a lot of other issues. It was trying to produce an ASIC with 50 employees. When they got aquired by Intel the first step they had to tackle was hiring the engineers necessary to actually produce an ASIC which innevitably slows down production, and then on top of that they got caught in the Intel 10nm bear trap.
AI is too powerful a technology to let it out there to the masses. People might use it for killer drones after all. All users of AI must be tightly controlled and registered with the authorities!
This is the problem with certain kinds of technology that are bumping up against the edge of innovation. They're too powerful and if these technologies get in the hands of the DIY set, governments will lose control so they have to DRM and regulate everything. Heck, it's a problem with old technology. Many weapons aren't that complicated technologically, but their production and use are tightly regulated.
Edit: I'm not saying this is a good thing, I'm just deconstructing their though process for tight control over AI tech going forward.
For some reason drones are perceived to be completely different from all weapons that have existed before them. Those killer drones have existed for half a century. They are called missiles. Also the reason why UAV based fighter jets are not viable is because a cruise missile can be launched from 1000 miles away and for the cost of a global hawk you can send out more than a hundred of them.
If terrorists have access to explosives then it doesn't matter how they deliver them because most lucrative targets (= lots of people in a small area) are stationary or predictable. A simple bagpack filled with explosives was more than enough to injure hundreds of people during the Boston Marathon.
You can make an unguided, explosive-filled rocket that can harm people for cheap from scrap. Insurgents throughout the world have done so for the past 40 years. That may not be as simple as Add To Cart, but it is well within the economic means of almost everyone.
So the idea is to let Google do the right thing? Or Amazon, Facebook, Microsoft, Apple, ..., etc?
The "right thing to do" is to open up these technologies, so that everyone can harness its power, not hide them under the wing and discretion of the (already too) powerful.
Except technology is amoral, it's up to the engineer and others to use it ethically and morally. The internet can organize hate groups and it can organize voters. smh.
it doesn't require a cloud based compiler; the quote above shows that you use TF-Lite, which is an open source project, or a cloud-based tool for people who don't want/need/have the ability to work with TF-Lite.
[UPDATE] I misread and assumed the previous case (where no cloud tool was required) was still true (I worked with previous versions of this device).
Do you have a source for this, or are you just reading the statement I quoted differently than I am?
The way I read the quote, you use TF-Lite to produce a quantized TF-Lite model, and then use a cloud based compiler to compile it for the actual chip.
This is why I asked "am I missing something." Do you have a reference for where the compiler exists in the open source TensorFlow project?
Mostly, what I'm interested in is learning what capabilities their TPU provides, to see if it would be useful for other similar kinds of kernels like DSP (which, like machine learning kernels, also involves a lot of convolution).
So I'm interested in looking at what the capabilities of the chip are, seeing what could be compiled to it. But I haven't found those docs, or found a compiler that could be studied. But maybe I'm not looking in the right place.
Here's an overview of the architecture of their Cloud TPUs, which has some good architectural details but doesn't documet the instruction set:
It does require a cloud based compiler, the accelerator doesn't run TF-Lite models directly. The cloud based tool is what gives you the actual executable.
Very few of which, if any, appear to be paid, which is what the parent poster is asking about. (Not sure if they edited that bit in after you responded?)
Google's reputation for shuttering services is vastly overstated, and beating this particular dead horse every time Google creates something new just wastes people's time.
Well they decided to not only make it closed source but also lock it up behind a http frontend so you can't even reverse engineer it. Criticizing Google for shuttering things suddenly that companies are depending on is rightly justified.
Google has mastered the art of using open source to crush competition like they did with Chrome and Android. They ever revealed their main moneymakers ever. For example they opened up their mapreduce technique after they have moved on from it.
I've been wanting to learn more about DSP, and this looks like a good free resource, but the examples in BASIC are just so hard to read, between the line numbers and SHOUTY CASE.
While I suppose it would be a good exercise to translate the examples into the language of my choice, are there any good resources on DSP that use a more reasonable language for the examples? And I'd prefer to avoid proprietary languages like Matlab.
Not entirely DSP, but Software Defined Radio is very related. There's an excellent tutorial series on SDR over at https://greatscottgadgets.com/sdr/ that's a whole lot of fun. No coding, just an intro to many great concepts.
Yeah, SDR is one of the reasons I wanted to learn more about DSP.
SDR, digital audio processing, digital image processing (especially compression and filtering), all of them have roots in DSP.
That looks like an interesting resource, but I really prefer written word and text-based programming languages to video lessons and flow-graph based configuration.
The advantage with MATLAB/Octave is that a lot of signal processing code has already been written in it. As you write more complex DSP code, you'll get stuck somewhere and chances are someone has already coded up that application on MATLAB central or on their blog.
I took a Speech Processing class and opted to write code in Python. I probably spent 1.5x the time the others spent. Most DSP texts today have MATLAB code.
It might be a good exercise to implement the functions from scratch but I think it is better to focus on the theory and not waste time coding up something like Overlap-add.
This book uses BASIC, which I just find quite unreadable.
I don't know of what other resources are available, but I do know I've seen some that use Matlab. I've never played with Octave much, so I don't know how compatible it is; can you just directly copy Matlab code, or do you need to do translations?
Anyhow, if there's a book that uses Matlab rather than BASIC, and could be used in Octave, that might be more readable. I would find C, C++, Python, or Rust more familiar, though.
I've never played with Octave much, so I don't know how compatible it is; can you just directly copy Matlab code, or do you need to do translations?
I'm not an expert, but I believe Octave is pretty compatible, at least with the core language. I think there are some "extra" features and libraries that don't exist in Octave, but it should be fine for many scenarios. FWIW, the earlier version of Andrew Ng's machine learning class on Coursera was Matlab / Octave based, and students using Octave were able to do everything required there with no problems.
In my experience one can pretty much just copy and paste, and most things work (though likely not 100%). There are two ways in which Octave still falls short: (1) Matlab is faster, as it has (I believe) JIT compilation whereas Octave doesn't; and (2) all things related to plotting & graphing are still better in Matlab IMO, though that may be changing. This may not be important for learning DSP, but does matter in day to day scientific work.
Languages that were developed before the advent of Unicode, like C and C++, frequently leave the encoding of strings up to the system; but this can be complicated, because there are a number of different legacy 8 bit encodings, along with legacy encodings that don't fit into 8 bits. So they have two character types, char and wchar_t; char is a type that can always encode at least 8 bits, and wchar_t is an implementation defined wide character type, which is 16 bits on some platforms and 32 bits on others.
When Unicode was first designed, and released as Unicode 1.0, it was envisioned as a 16-bit universal encoding, with the expectation that you would use it just like 8 bit character encodings but with 16 bit character types instead. This encoding is known as UCS-2.
However, experience and unification of the Unicode standard with ISO 10646 showed that 16 bits would not be sufficient for all of the characters in all of the world's writing systems; in order to fit CJK characters into that 16 bit encoding, a large effort was made to unify characters from different languages that represented basically the same thing but might be written slightly differently in Traditional Chinese, Simplified Chinese, Korean, Japanese or the historical Vietnamese writing system. Additionally, a lot of decisions had to be made about inclusion of historical or obscure characters, and so there were many characters which couldn't be encoded properly in early Unicode.
This led to Unicode 2.0 introducing a surrogate character mechanism, now known as UTF-16, in which certain unused ranges of code points were used in pairs to represent a wider range of code points, in order to expand Unicode from a single 16 bit plane of characters to 17 16-bit planes.
Meanwhile, Ken Thompson and Rob Pike adapted an earlier 8-bit encoding of ISO-10646 called UTF-1 into UTF-8, which had a number of desirable properties; it was a superset of ASCII, the interpretation of the ASCII subset of bytes never depended on state, and it was self-synchronizing, so if you started reading in the middle of a multi-byte character, you could always find the start of the next character.
This meant it was possible to use UTF-8 in all of the traditional APIs and file formats which were compatible with 8-bit extensions of ASCII; which meant that you could migrate to Unicode support without significant changes to APIs and file formats. Also, for text which consists mostly of the ASCII subset, which includes many ASCII-based markup languages, it was much more efficient than UTF-16.
In the early days, however, many languages and systems which adopted Unicode started doing so via UCS-2, and then with the advent of Unicode 2.0 and surrogate pairs, needed to adopt to UTF-16. This was done somewhat inconsistently, and it can cause a lot of pain and confusion.
I've already probably written more than is necessary, so I won't go into the history of each system, but a quick survey of a few popular platforms and languages and their native string encoding:
* C and C++: Support 8-bit unspecified encoding strings (char * , std::string), unspecified width wide characters (wchar_t * , std::wstring, which are 16-bit UTF-16 on Windows and 32-bit UCS-4 on most other platforms), and explicit widths (char8_t * , std::u8string, char16_t * , etc...) which are explicitly UTF-8, UTF-16, and UCS-4.
* Windows uses 16-bit wchar_t * , nominally in UTF-16 but not enforced, as its native string type, and also provides 8 bit APIs with character encoding which can vary at runtime
* Linux and other Unix systems generally use 8-bit char * in an unspecified encoding (which can vary at runtime) as their native string type; most systems these days are using UTF-8 as that 8-bit encoding, but it's not guaranteed
* Java, JavaScript, and C# all use UTF-16 as their native encoding, though in many cases there are still APIs which make most sense for UCS-2 rather than UTF-16.
* Python 2 used unspecified 8-bit encodings for the str type, and a custom encoding that supports all of Unicode for the unicode type. Python 3 changed so that the bytes type was 8-bit with unspecified encoding, and str type was Unicode in a custom encoding; a bytes type alias for str was added to Python 2 to ease the transition, as was a unicode type alias for str in Python 3
* Go and Rust use UTF-8 as their native string types
* Most text-based file formats, such as HTML, have standardized on UTF-8 as the backwards compatibility with ASCII is a big benefit for such formats.
There are a few references on why UTF-8 is generally a better choice for languages and APIs than UTF-16, despite some of the legacy use of UTF-16 in languages and APIs:
* http://utf8everywhere.org/
* https://www.cl.cam.ac.uk/~mgk25/unicode.html