A thought that folks reading this post might have an opinion on:
"Libraries should be mostly unit tested. Applications should be mostly (and lightly) integration tested. Naturally, some parts of a complex app will behave like a library..."
Strongly agree, but also I think that the interesting thing is why this might be so and what it tells us about application architecture.
The basic thing about tests is that once you have them passing, they represent statements about constraints on the program. In other words, they express your opinion of things that should not change. Unit tests are a bet that certain aspects of the implementation will not change. Integration tests are a bet that certain aspects of the externally visible behaviour will not change.
Libraries tend to be smaller and with well-defined responsibility. Applications tend to be bigger and have many responsibilities. In general, I think it’s true that the requirements for libraries change less often than the requirements for applications. I think this leads us to expect that applications may need to be rewired “under the hood” and have their implementations changed as responsibilities are added, removed, or changed.
This, I believe, leads us to want to unit test applications less, because a unit test expresses implementation semantics, and we expect application implementations to change. No what about integration tests? Well, if we’re unit testing less in the application, we need to make up for it by integration testing more, otherwise where do we get our confidence?
Now if we throw the words “library” and “application” away, this suggests to me that those parts of the code that are small and tight and with a single, clear responsibility should be unit tested, while those parts that involve a lot of what the AOP people call ‘scattering and tangling,’ should be integration tested.
Unit testing didn't fully make sense to me until I played around with quickcheck (and eventually theorem proving in Coq). Unit tests vanish nicely to theorems and (empirical) proofs if your code expresses a succinct API. This is one end of the testing continuum.
I use this sort of stuff extensively when doing mathematical computing and statistics because there's usually a clear mathematical boundary. Once you're inside it, it's relatively easy to write down global properties (theorems) of your code's API.
The moment you cross that boundary your testing apparatuses have to get more complex and your tested properties less well-defined. Unit tests are hazier than quickcheck properties and integration tests hazier still.
This continuum seems to be precisely the same as the code reuse continuum. Highly abstracted, testable code with a shapely API is a highly reusable library whether you like it or not. Maybe it's being called by other code, maybe it's being called by your UI, maybe it's being called by the user themselves.
I view unit tests as a kind of proof by counter-example. You have a logical structure your program embodies. This structure is very hard to specify mathematically and prove deductively so you come up with key statements that must at least evaluate to true for this structure/theory. The tests are a bunch of counter-examples that should be false (test passed).
If a random testing framework is available in your language they really should be integrated as they are able to come up with some pathological examples.
For Haskell you can do even better than random testing: There's Lazy SmallCheck for exhaustive testing. (For some values of `better' and `exhaustive'.)
After playing with a language with a great type system (Haskell, not Java/C++), I've become more wary and borderline uncertain about my Ruby code.
My integration tests serve two purposes:
1) Runs the code in an repeatable and isolated environment to verify I didn't do anything stupid like misname a variable, or treat something nil as an object.
2) Validate my unique application logic.
I don't think #2 goes away with other languages, but #1 changes dramatically. I've written Ruby for years, so this isn't an "outsider looking in" opinion.
Whether conscious or not, I believe move Rubyists gravitate to testing because of both #1 and #2.
#2 can be satisfied with unit tests, but in my experience, the suite becomes a lot more flexible when validating it in terms of integration tests.
Note that, at least in python (which I assume is pretty similar to Ruby in ecosystem), you can get a lot of mileage for #1 by doing static analysis, for example Pylint.
On this, I tend to think that many (most?) libraries, especially where they don't have extreme performance requirements, should have run-time testing. I.e., pre- and post-condition checks, or contracts.
Agree, but I'm concerned all these rules of thumb are doing nothing to help the authors of bad test suites understand why they're bad and how to fix them.
In my opinion the point of building out a test suite isn't to reach 100% code coverage, it's to allow developers working on the application the ability to refactor quickly and aggressively during the development process while remaining confident that it will still perform correctly when they're ready to release. Poorly written tests (or over-testing) will slow down refactoring and hinder development. But, in my experience, not having the right tests can mean firing up your application and manually testing after each small code change to prevent regression. Understanding the trade-off is key to helping developers answer the ubiquitous question: "do I need to test this"?
This role for unit tests is described effectively in Chapter 1 of Martin Fowler's awesome Refactoring book (or Refactoring, Ruby Edition, if that suites you. That's the version that's on my desk right now.) This use case is also highlighted in Eric Evans' book Domain Driven Design, where he highlights one of my favourite application development techniques: "refactoring to deeper insight." Both books are among my favourites.
Adding features to an application isn't just about appending files and lines of code, it's also about changing existing code to allow the new features to fit comfortably into the application's domain model. If you can accomplish that goal with "light integration tests" go for it. That'll mean much less overhead during refactoring and will do a good job of enforcing correctness. If your application contains a robust domain model (which it very well might) you may find unit tests useful for maintaining model integrity during development. This is probably what's meant by "Naturally, some parts of a complex app will behave like a library". When you change the domain model you'll have to change tests for all the affected classes and that's a good thing. It explicitly highlights how your change affected the model, which you should certainly understand before committing your changes.
I will admit this distinction is hard to communicate to first-time TDDers. However, I've found that after having to eat their own dog food for a few iterations (adding features to your own code) smart developers quickly find the sweet spot. The less code you write, the less you have to maintain. But too few test or poorly written tests can slow you down just as much.
That sounds good, with the caveat that most applications should have their business logic roughly organized into libraries. After those libraries get their unit tests, integration testing the application<->library barrier makes a lot of sense.
100% agreed. Rails people have started to complain about difficult/slow tests because Rails (a framework, not a library) makes your life hard. From my own experience, you don't need all the Rails magic to get started on a project, and once you do, it just happens to be a completely different magic that you need.
Hence I try to divide my projects into lots of application agnostic code (the libraries that need to be unit tested), and little application specific code (the glue code that needs to be integration tested).
We recently finished approximately one metric fuckton of unit tests for the next release of some of our libraries, and I tend to agree very strongly that a library must be well-tested.
A problem that we ran into, though, is exactly what to test. The biggest basic distinction we ran into was testing to verify interface integrity (i.e., does this do what it says in the header documentation, pursuant to declared constraints and so on) as opposed to implementation integrity (does my ringbuffer properly move around elements, does my array resize move my elements, etc.).
The former is very useful to ensure that the library is good for users, the latter of course is much better for testing that changes to the containers don't do something stupid.
Disagree. Your domain model is within the scope of your application. The code representing your domain model should be unit tested. If your application is doing anything interesting then the code for your domain model is the most important thing to test in your application.
Agree, though following this would lead me to extract all kinds of implicit library code from my applications. Likely not worth the effort without a genuine reuse story.
"Libraries should be mostly unit tested. Applications should be mostly (and lightly) integration tested. Naturally, some parts of a complex app will behave like a library..."
Agree or disagree?