The article is a lightweight analysis by someone who writes small programs. He does get that, for Rust, "If the compiler accepted my input, it ran — fast and correctly. Period." That's was a common experience with the very tight languages, such as Ada and the various Modulas. It's been a while since a language that tight was mainstream. We need one now, badly.
Go isn't bad for writing routine server-side web stuff that has to scale and run fast, which is why Google created it. Go is a modern language with a dated feel. No user-defined objects, just structs. No generics or templates. It was designed by old C programmers, and it looks it. Go has generic objects - maps and channels - and syntax for creating object instances - "Make". Only the built-in generics are available, though; you can't write new ones.
Go's "reflection" package thus tends to be overused to work around the lack of generic. This means doing work for each data item at run time for things that could have been done once at compile time. "interface{}" (Go's answer to type Any from Visual Basic) tends to be over-used.
Go (especially "Effective Go") has a lot of hand-waving about parallelism. Go's mantra is "share by communicating, not by sharing", but all the examples have data shared between threads. The channels are just used as a locking mechanism. Race conditions are possible in Go, and there's an exploit which uses this. (That's why Google AppEngine limits Go programs to single threads.)
Go doesn't use immutability much, which is a lack in a shared-data parallel language with garbage collection. If you can make data immutable, you can safely share it, which is a way to avoid copying without introducing race conditions.
Rust, like Erlang, takes a much harder line on enforcing separation and locking. I haven't used Rust myself yet, so I can't say more on what it's like to use it. My hope is that Rust will provide a solution to buffer overflows in production code. After 35 years of C and its discontents, it's time to move on. I really hope the Rust crowd doesn't fuck up.
Like you†, I've had the pleasure of working with some fairly large concurrent codebases and the character-building experience of tracking down deadlocks, random memory corruption bugs that turn out to be race conditions, and (my most favorite of all) unexpected serializations that randomly bring programs to a halt. Most of that experience has been in C++, with a little C and a little Java mixed in there.
Over & over I see language aficionados ding Golang for not taking advantage of immutability and for allowing shared data --- or, in your case, going a step further and reducing all communication among processes in Golang to instances of synchronized sharing.
What I'd like to know is: why don't all those hundreds of thousands of lines of concurrent Golang code out there, including all the library code I can just "go get" and whose authors have been encouraged by Rob Pike to use, basically, threads with near total abandon (watch his video about designing a lexer!) --- why don't all those libraries and programs randomly deadlock and corrupt themselves all the time?
Because my experience is that Golang code is quite a bit more reliable than, for instance, Python code.
What am I missing? The "share by communicating" model in Golang seems to work pretty darn well, especially given the extent to which Golang begs programmers to make programs concurrent.
> why don't all those libraries and programs randomly deadlock and corrupt themselves all the time?
The simplest answer would be "they do." In aphyr's recent presentation on Jepsen, where he tested etcd (a Go database implemented on top of Raft), he noted that when he started using it he encountered a ton of easily reproducible races and deadlocks (which he sarcastically noted was surprising because he thought goroutines were supposed to make concurrency issues a thing of the past).
I am not saying that Go channels don't help the situation at all--and the inclusion of a race detector doesn't hurt either--but you still have plenty of ways to shoot yourself in the foot. The thing that probably helps most is that GOMAXPROCS is 1 by default, since data races are a multicore phenomenon in Go.
Distributed systems programming is its own special concurrency problem, and distributed systems also exhibit deadlock, races, and serialization, no matter what language they're implemented in. I'm not sure what finding a race condition in a distributed commit implementation says about a language; at the very least, it's nothing you couldn't say about Rust as well, which is also not a language that solves distributed systems concurrency problems.
Maybe I'm wrong and etcd was riddled with concurrency problems between the goroutines of a single etcd process?
In any case: as anyone who has worked on a large-scale threaded C++ codebase can tell you: Golang programs simply do not exhibit the concurrency failures that conventional threaded programming environments do. It would be one thing if Golang code only used concurrency for, say, network calls. But goroutine calls are littered throughout the standard library, and throughout everyone's library code.
It is not a black and white situation probably. Golang is better because it has built-in channels and encourages users to take advantage of them. It also has garbage collection. So those 2 things right of the bat help.
But there are better things out there -- isolated heaps (Erlang), borrow checkers (Rust), stronger type systems and immutability (Haskell) etc. There are no magic unicorns so those things often come at a price -- sequential code slowdown.
Getting back to go. One can of course say, "Oh, send only messages. We are all adults here. Let's just agree to be nice. Stop sharing mutable memory between goroutines!" But all it takes is "that guy" or "that library", doing it "that one time" and then there are crashes during a customer demo or during some critical mission. It crashes and then good luck trying to reproduce it. Setting watchpoints in gdb (or the equivalent Go tool), asking customers "Can you tell me exactly what you did that day. Think harder!" and so on.
Also, as others have pointed, with Golang though, often it is run with just one OS thread backing all the concurrency. So many potential races could be just be hidden.
There is also some confirmation bias involved. When something is broken, often authors don't write blogs about it, don't advertise. They fix it, and move on. So maybe a lot of programs are full of concurrency bugs but just nobody is blogging about it. They've invested time and energy into learning a new ecosystem and now they have to blog about its flaws and so on. That is hard to do.
Another observation is that when spending a lot of time debugging and handling segfaults, pointer errors, user-after free errors, concurrency issues, that becomes the default and expected view of how programming works. It becomes hard to imagine how it could work another way. It becomes obvious that weeks would be spent tracking one concurrency bug or having to add cron jobs to watch for crashed programs and restart them because the system is so complex and non-deterministic, replicating the bug is too hard.
How does Rust's borrow checker cause a slowdown of sequential code? It's a purely compile-time construct and allows for the elimination of a GC, so it's actually a net win in code execution speed.
Not at all. In languages that are thoroughly immutable, it's copying, not immutability, that has a runtime cost. Rust has mechanisms for avoiding these costs (moves and mutable references).
Furthermore, since the compiler's knowledge of mutability is directly related to its knowledge of ownership, one could argue that immutability actually makes code faster by dint of providing greater aliasing information (e.g. `restrict` in C) to the optimizer (though the Rust compiler has yet to actually leverage this optimization).
In short: We (as a field) tried them for many many (many!) years now and they have been found lacking -- in practice they're just too hard to get right for large-scale systems.
EDIT: The mutexes themselves are easy enough to get right, it's the systems using mutexes that are too hard to get right.
In most languages, the language says nothing about what data is protected by the mutex. Modula and Ada did, and Java has "synchronized" objects, but C/C++/Go lack any syntax for talking about that. This typically becomes a problem as a program is modified over time, and the relationship between mutex and data is forgotten.
> In most languages, the language says nothing about what data is protected by the mutex.
Or the other way around, what mutex protects a piece of data (or even that a piece of data should be protected at all), so it's easy to forget it and just manipulate a bit of data without correctly locking it.
I was pleasantly surprised to discover that Rust's sync::Mutex owns the data it protects, so you can only access the data through the mutex (and the relation thus becomes obvious).
I feel this is a symptom of people coming from dynamic, higher level languages, who may or may not have ever really learned concepts of CS, "switching" to Go for performance reasons. I don't intend to insult anyone or certainly the authors of the above.
Go is type safe but doesn't keep you from shooting yourself in the foot. You NEED to read the spec to understand when sharing memory is generally OK. I'm glad not to be penalized in performance or boilerplate to accomplish this. The downside, of course, are stories like the above.
I don't blame the language here, though. They give you the tools to be safe. Getting away from automobile analogies, let's try woodwork. I can give you a drill, a drillbit, a screwdriver, and a screw. Sometimes you should know when to make a pilot hole, and when it's ok to forego this. But folks looking for 'performance' or 'ease of use', or folks who come from languages that just give you a nailgun, will inevitably split the wood a few times.
Were there any code examples provided that show how to easily trigger races & deadlocks? I mean, the Go team needs to be aware of these problems and provide a fix or something.
The issues weren't with Go--which definitely allows for both data races and deadlocks and doesn't claim to eliminate either--but with etcd. And according to aphyr, the team was very responsive and quickly fixed the ones he found.
My point wasn't that Go is _worse_ than contemporary languages like C++ and Java when it comes to data races, only that it doesn't eliminate them. Which, again, it doesn't claim to. Rust does, and it is an important difference between the two languages. Because data race freedom with cheap mutable state requires a garbage-collection free subset of your language [1], I think it's unlikely that Go will ever guarantee this.
> show how to easily trigger races & deadlocks? I mean, the Go team needs to be aware of these problems and provide a fix or something.
You mean file a bug like "issue #1935 -- stop sharing memory between goroutines" (I just made it up to be silly there is no such bug report)
In other words, they have explicitly designed in the ability to share memory between goroutines. You can certainly file an issue or bug report about, somehow I doubt that will lead to much but being kickout and laughed at.
One can also just have 2 goroutines wait on each for results and that's a deadlock. Maybe there is a tool to detect that would be nice. Do you know of one?
If all goroutines deadlock, then the runtime will panic and you'll get stack traces for everything. But yes, you can obviously deadlock one or more goroutines trivially. A simple select{} will just block one goroutine forever, for example. And no, there's no tools currently that will detect deadlocks, AFAIK.
I haven't looked into that (or googled), but how does one detect that everything has deadlocked - is it some kind of profiling/sampling being done, or is it something more system specific? Any pointers? Thanks!
http://golang.org/pkg/runtime/pprof/#Profile
Provides a "blocking profile" that tells you what things blocked and for how long they blocked for.
You can use it to find places where you've deadlocked as well as places where adding buffered channels might help performance.
How can you compare the ridiculous amount of Python libraries/code out there (with varying degrees of quality as you would expect from a friendlier language) with Go ? Of course you would find Golang code to be more reliable. The libraries you're relying on are by comparison quite primitive and missing many years of technical debt.
As a separate comment I want to address your final paragraph. Rust definitely provides a solution to buffer overflows. Specifically, all data structures that represent buffers of any sort always do bounds checking. This includes using the indexing operator (`foo[idx]`) on a fixed-length compile-time array. It's possible to skip the bounds check but you have to do so very intentionally, using an `unsafe {}` block (which defines a scope wherein certain features can be used that bypass some of Rust's safety). This should only ever be done in response to performance profiling when it turns out that bounds checking is a problem and when you can prove to yourself that it's safe to skip, and often there's another way to do the same thing that doesn't use `unsafe {}` (e.g. if you're iterating an array, use an iterator instead of indexing into the array each time; the iterator approach generally optimizes all the bounds checks away).
The basic rule is if you see a segfault or a data race, search your code for any `unsafe {}` blocks, and the cause will always be found inside one of those. And as a corollary, never use `unsafe {}` if you can possibly avoid it.
I could not agree more with your first paragraph. The only other language I've used that I've had that experience with was Haskell, and while there are good arguments to be made for using Haskell in production, it should be obvious that's not a language that will ever become mainstream.
I'm hoping that as Swift evolves over time, it will slowly become that sort of language. Right now it's pretty hard to write any real-world code in Swift that doesn't work with the Cocoa frameworks, and the Cocoa frameworks are typed in an objective-c-compatible way (even if new frameworks are written in Swift they'll need to maintain obj-c compatibility), which means you don't get the strong typing that's necessary for this property. Pure Swift code has the potential to behave like this, although you probably need to avoid the ImplicitlyUnwrappedOptional feature (the ! suffix on types), which of course primarily exists for ease of obj-c integration anyway.
I'm bringing up Swift because, with Apple's backing, it's very quickly becoming a "mainstream" language. I put that in quotes because it is only usable with iOS and OS X programming (for now at least), but iOS is large enough that obj-c should be considered a mainstream language despite the fact that almost nobody outside of iOS/OS X uses it, and therefore as Swift supplants obj-c it becomes appropriate to call it mainstream.
Regarding parallelism, I've been in love with Rust for a long time now, and one of the biggest reasons is because Rust makes parallelism safe. As an iOS/OS X programmer by trade, I think thread safety is far and away the biggest elephant in the room. Despite the fact that we've known that multithreading is the future for years, and despite the wonderful Grand Central Dispatch library on iOS/OS X, most programmers still think in a single-threaded mindset and don't even consider how their code should operate if invoked on a separate thread. This was one of my bugaboos with Go back when I was using that language (which was from the day it was announced right up until I discovered Rust, though admittedly my usage was in hobby projects and nothing serious).
I applaud the fact that Go has a data race detector now, which I used to finally uncover a lurking data race that plagued one of my programs for months (and which was ultimately caused by the Go library using two goroutines where I expected one, and therefore data which I expected to be on a single goroutine was actually mutated from two goroutines simultaneously). But I think Rust is absolute proof that a modern language can be designed such that data races are prohibited at compile-time without sacrificing any language flexibility.
> I could not agree more with your first paragraph. The only other language I've used that I've had that experience with was Haskell, and while there are good arguments to be made for using Haskell in production, it should be obvious that's not a language that will ever become mainstream.
I don't think it is at all obvious that Haskell won't become mainstream. It's already exerted a tremendous influence over many other mainstream languages and there's only so long that can happen before people just start going directly to the source of the innovations (or one of its direct descendants).
I agree Haskell has had an important influence, but I don't see why people would necessarily "go directly" to it because of that. In fact, I would argue just the reverse. People chose the derivative languages b/c they provide things the original does not.
To wit, Lisp never became mainstream despite exerting a huge influence. Likewise Smalltalk.
Technically true, but I don't think it makes sense to consider Clojure to be the same thing as Lisp in the context of ternaryoperator's statement (though I obviously can't speak for him). Notably, Clojure's tight integration with Java is the primary reason for its relative popularity, and what most sets it apart from the rest of the Lisp family. It is disingenuous to claim Clojure means Lisp has gone mainstream when the popularity of Clojure is not due to its inclusion in the Lisp family.
Beyond that, I'm not quite sure Clojure counts as "mainstream" yet. According to the TIOBE Index, it doesn't even rank in the top 50 languages. Heck, the top 20 includes R, and Dart, neither of which I would call "mainstream" (I'm actually really surprised at how high R is ranking). I don't know how significant that is, though the TIOBE Index is measuring "number of skilled engineers world-wide, courses, and third party vendors" and that seems like a reasonable approximation for "mainstream" to me.
There are several languages targeting the JVM these days. And, what obviously sets Clojure apart from other JVM languages is its Lispiness (i.e., the JVM is constant across JVM languages).
Clojure is not married to the JVM either-- in fact, it has been hinted that it would jump ship if something better comes along or the current situation becomes less viable. Furthermore we already have a dialect of Clojure called ClojureScript which targets JavaScript/node/V8.
And, I look at the JVM as really merely a library/API/runtime. C++ has STL and stdio and such and they are not part of the language proper but rather merely libraries for interacting with the underlying operating system (in a platform independent way). The same is true for the JVM with respect to Clojure and Scala et al.
Yeah, but nothing in it is Smalltalk-specific. It's not like Smalltalk survives in the mainstream because of Hotspot (in the way that, say, Algol survives).
Well, a counter point then can be: and why did those vendors did not insist on Smalltalk? Why weren't Smalltalk more heavily pushed by some big vendor itself?
It's not like SUN was the only player in town. IBM pushed Smalltalk IIRC.
I think this (from StackOverflow) tells a more comprehensive story):
• when Smalltalk was introduced, it was too far ahead of its time in terms of what kind of hardware it really needed
• In 1995, when Java was released to great fanfare, one of the primary Smalltalk vendors (ParcPlace) was busy merging with another (Digitalk), and that merger ended up being more of a knife fight
• By 2000, when Cincom acquired VisualWorks (ObjectStudio was already a Cincom product), Smalltalk had faded from the "hip language" scene
Even worse than the problem of uncommon concepts as monads is that Haskell's memory footprint is extremely hard to reason about. A few years ago it was impossible with the http libraries to download a file without the program consuming several times as much memory as the downloaded file.
> Even worse than the problem of uncommon concepts as monads
Just go ahead and learn the typeclass hierarchy and such-- it really is quite a useful higher level of abstraction in whatever language you choose. And it definitely will enter the mainstream (even more than is already has [Swift, Scala & C# all have monadic constructs]).
> Haskell's memory footprint is extremely hard to reason about.
And you'd probably want to also throw runtime in there as well.
I think this is relative-- it's not "extremely hard" for everyone. Also, many structured programmers found object orientation "extremely hard" but somehow the industry managed to progress through that era.
A common recipe people quote for good software is "a) first make it work, b) then make it fast".
Haskell is very good at a), and not bad at all at b). With the help of the profiler it shouldn't be that hard to determine a program's bottlenecks/leaks and fix them, as with any other language.
BTW, since you mention http, I have this reading on my back burner [1] but from skimming it found that for certain payloads a haskell http server may perform better than nginx (1.4, circa 2013?), which is an impressive feat.
Rust is not your typical lower-level language though. It supports a lot of the features that functional programmers expect. It is an eagerly evaluated language that lets you drop to 'unsafe' code where necessary but in its natural form, it is surprisingly high level.
Just to add a clarification on the implied comparison: Rust only protects against data races at compile time. Other forms of race conditions are still possible.
The way structs are supported, it feels much more like objects than C structs. I prefer it much more than say, OOP classes. It's something you need to immerse yourself in to appreciate, IMO.
Go isn't bad for writing routine server-side web stuff that has to scale and run fast, which is why Google created it. Go is a modern language with a dated feel. No user-defined objects, just structs. No generics or templates. It was designed by old C programmers, and it looks it. Go has generic objects - maps and channels - and syntax for creating object instances - "Make". Only the built-in generics are available, though; you can't write new ones.
Go's "reflection" package thus tends to be overused to work around the lack of generic. This means doing work for each data item at run time for things that could have been done once at compile time. "interface{}" (Go's answer to type Any from Visual Basic) tends to be over-used.
Go (especially "Effective Go") has a lot of hand-waving about parallelism. Go's mantra is "share by communicating, not by sharing", but all the examples have data shared between threads. The channels are just used as a locking mechanism. Race conditions are possible in Go, and there's an exploit which uses this. (That's why Google AppEngine limits Go programs to single threads.) Go doesn't use immutability much, which is a lack in a shared-data parallel language with garbage collection. If you can make data immutable, you can safely share it, which is a way to avoid copying without introducing race conditions.
Rust, like Erlang, takes a much harder line on enforcing separation and locking. I haven't used Rust myself yet, so I can't say more on what it's like to use it. My hope is that Rust will provide a solution to buffer overflows in production code. After 35 years of C and its discontents, it's time to move on. I really hope the Rust crowd doesn't fuck up.