I was asked a version of this question by a colleague at work, namely: "if types are so great, why didn't Python/Ruby/JS include them from the start (ie. early '90s)?"
That's because the theory of gradual type systems was only worked out in the '00s. Before that, you could have a static or dynamic type system, not anything in between. Common Lisp did have type annotations, but they were hints for optimization, without any guarantees. They were also local to subroutines only. Dylan[3] is an example of an early implementation of the idea, but Dylan was several years late and, without being able to compete with Java, died without ever being widely used.
The proper theory was first established by J. Siek[1] and W. Taha in 2006. It's distinct from nominal static typing which uses a single top type (like Object in Java) or generics, and obviously it's different from both purely static and dynamic typing. It took almost a decade for the idea to start gaining practical implementations - I think the original was a made for Scheme, and one of the first implementations was Typed Scheme for PLT Scheme, which continues on as Typed Racket[2] today. Typed Racket is unique in that it enforces the types even on the untyped side, by wrapping values and exports in contracts.
The idea proved to be useful in practice, and started being adopted in various (non-Scheme) dynamically typed languages, starting with TypeScript for JS and Hack for PHP. On the other hand, some statically typed languages also became gradually typed, most notably C#. The implementations continued to improve, shrinking the parts of their respective languages that could not be statically typed. In dynamic languages there are still features that cannot be practically expressed in static type systems - most metaprogramming and code generation falls into this category - but they are generally "good enough" for day to day coding.
Gradual typing is useful in the same way static type systems are useful: it can prevent certain kinds of errors by marking known-invalid expressions without the need to run the code (so, for example, can help you find errors even in code that's not covered by tests); it helps in writing tooling for the language (eg. go to definition, find references); it helps make the code clearer for the reader (no need to break into a debugger to see what kind of value a given identifier refers to); in some implementations it may also help in optimizing the runtime performance, but that's rare. The "gradual" aspect makes it easier to adopt when the codebase grows larger - the bigger the codebase, the more useful static types are, but by the time the codebase grows large enough to justify static typing it's too big to rewrite in a different, statically typed language.
In short: writing small projects or prototypes in a dynamically typed language is faster while maintenance and expansion of large projects is easier in statically typed one. Gradual typing lets you go from one to the other without a huge cost of a full rewrite.
> In dynamic languages there are still features that cannot be practically expressed in static type systems - most metaprogramming and code generation falls into this category
Actually, both metaprogramming and code generation blur the phase distinction between compile and run time - but this makes them a great fit for dependent typing systems, which do pretty much the same thing. So there's no reason why more refined static systems could not express these idioms.
I meant "practically expressed in their static type systems", ie. in gradual type systems for those languages as currently implemented. Poor wording on my part, my bad :)
On dependent typing and gradual type systems - I'm not sure they are possible to mix. I think none of the current implementations do anything remotely similar to dependent typing. It would be a really cool if it worked though. I'd love to be able to track length of a list in mypy type system (for example).
Dependent types seem like they would have to be explicitly static, but gradual typing interacts well with "fully static" fragments of your program. (In fact, having entire program modules be statically typed is needed anyway to get proper optimization and reduce the overhead of dynamic checking and conversion.)
> Dependent types seem like they would have to be explicitly static
Right, because they lift values into type system, those values need to be well-typed themselves. You wouldn't be able to represent gradually typed dynamic/Any value then, I think. I might be wrong though, my understanding of type theory is limited at best :)
> to get proper optimization and reduce the overhead of dynamic checking and conversion
That's probably why very few gradual type systems are taken advantage of when optimizing. Typed Racked does this, but Typed Racket disallows untyped values within the typed module. Hack probably does it, too, but I have no experience using it. Common Lisp does it with correct optimization settings. Raku does this, but Raku was written from the ground up with gradual typing in mind. There are projects that try to transpile fully-typed subsets of dynamic languages to C (can't find the link right now), but that's a bit different. Other popular gradual type systems are basically advanced linters. There's a lot of development and potential in this area though, we'll see how it pans out in a few years :)
That's because the theory of gradual type systems was only worked out in the '00s. Before that, you could have a static or dynamic type system, not anything in between. Common Lisp did have type annotations, but they were hints for optimization, without any guarantees. They were also local to subroutines only. Dylan[3] is an example of an early implementation of the idea, but Dylan was several years late and, without being able to compete with Java, died without ever being widely used.
The proper theory was first established by J. Siek[1] and W. Taha in 2006. It's distinct from nominal static typing which uses a single top type (like Object in Java) or generics, and obviously it's different from both purely static and dynamic typing. It took almost a decade for the idea to start gaining practical implementations - I think the original was a made for Scheme, and one of the first implementations was Typed Scheme for PLT Scheme, which continues on as Typed Racket[2] today. Typed Racket is unique in that it enforces the types even on the untyped side, by wrapping values and exports in contracts.
The idea proved to be useful in practice, and started being adopted in various (non-Scheme) dynamically typed languages, starting with TypeScript for JS and Hack for PHP. On the other hand, some statically typed languages also became gradually typed, most notably C#. The implementations continued to improve, shrinking the parts of their respective languages that could not be statically typed. In dynamic languages there are still features that cannot be practically expressed in static type systems - most metaprogramming and code generation falls into this category - but they are generally "good enough" for day to day coding.
Gradual typing is useful in the same way static type systems are useful: it can prevent certain kinds of errors by marking known-invalid expressions without the need to run the code (so, for example, can help you find errors even in code that's not covered by tests); it helps in writing tooling for the language (eg. go to definition, find references); it helps make the code clearer for the reader (no need to break into a debugger to see what kind of value a given identifier refers to); in some implementations it may also help in optimizing the runtime performance, but that's rare. The "gradual" aspect makes it easier to adopt when the codebase grows larger - the bigger the codebase, the more useful static types are, but by the time the codebase grows large enough to justify static typing it's too big to rewrite in a different, statically typed language.
In short: writing small projects or prototypes in a dynamically typed language is faster while maintenance and expansion of large projects is easier in statically typed one. Gradual typing lets you go from one to the other without a huge cost of a full rewrite.
[1] https://wphomes.soic.indiana.edu/jsiek/what-is-gradual-typin...
[2] https://docs.racket-lang.org/ts-reference/index.html
[3] https://opendylan.org/index.html