The elephant in the room here is the bugs that crop up in the interfaces between units. Depending on your problem domain, it may be easier or harder to create interfaces between all these TDD'd, isolated modules. There's certainly value in a spec as isolated component documentation, but I'd argue there's more value in specs for regressions, particularly when refactoring. This is especially the case in dynamic languages like Ruby which require tests to avoid the most basic runtime errors.
The answer to this conundrum is often: that's the job of the integration tests. Okay, but integration tests are slow, so you'll never test all the permutations of the interactions of the components.
DHH said during the keynote, "it's easy to make your tests fast when they don't test anything". Of course that's hyperbole, but there's a kernel of truth to be investigated there. When working with something as complex as an ORM like ActiveRecord, isolating the business logic and using the ORM strictly for persistence may allow for fast tests, but you still run the risk of bugs creeping in on the interface because of some assumption about or change in the way ActiveRecord works between versions or whatever.
That's why, as ugly and slow as Rails unit tests are, they are a simplifying compromise that strikes a balance between the theoretical ideal of a unit test and an integration test. ActiveRecord itself is this way too, in that often times the business logic just isn't complex enough to warrant the complexity of separating persistence from domain logic. As much as DHH may be talking out his ass without really having ever grokked TDD, I don't think his complaints are completely without merit.
I have to agree, and add an observation that every project I've seen which emphasised highly isolated unit tests spent the majority of its test development effort on local behaviour that was easy to test, and very little effort on testing complex interactions and emergent behaviour.
Furthermore, when a complex interaction with changing behaviour in third party code caused a bug, I have always observed the resident advocates of highly isolated unit tests throw up their hands and say "oh well that's not our problem".
I'll offer up my own rule that I've worked out over the years: the first test you write, from day one, should be the end-to-end performance stress test that fully loads a production deployment of the project, making it do everything all at once and sending random junk input until it falls over. Even when your project doesn't really have any functionality yet, run that one continually in the background. This one test will find so many classes of bugs that you weren't expecting - it optimises your tests for learning surprising things as soon as possible. You then start fleshing out your test suite to cover the things you learn. With some smart design, there will be a lot of shared code between your every-growing stress test and the suite of more targeted tests.
You might find that you get down to detailed unit tests of every part of the code... but you'll probably find that you only need to bother with unit testing obscure conditions for the complicated parts.
The core philosophical difference here is that I regard tests as a tool for learning and making the code better, not as a way to make myself feel happier. The biggest performance barrier I experience as a developer is not the seconds it takes to cycle through the tests, it's the weeks it takes to learn what I'm trying to build. I aim my tests at that target.
DHH's complaints certainly aren't without merit, but they are naively formed.
I did a conference talk called "Boundaries" on this topic, and how a particular type of disciplined functional style can mitigate it to a large extent (https://www.destroyallsoftware.com/talks/boundaries). I think that we can probably have (most of) our cake and eat it to, just as we have so many times before: microprocessors reliably perform computations; TCP reliably delivers over unreliable networks; etc. But we're not going to get there by throwing up our hands.
When you say naively formed, what does that mean exactly? A lot of your post was dedicated to correcting DHH's TDD history. I appreciate the hard work and attention to detail but I don't actually think when the idea of ultra-fast tests originated has much to do with DHH's complaints.
You also spend time in your post talking about how much you value the tight feedback loop between your tests and your code and how your tests being ultra fast helps that. That's great! That isn't really a response to DHH though, that's explaining why your method works for you. Which is a valuable contribution to the discussion, but not really a critique. You also bring up DHH's lack of theoretical computer science knowledge. I don't have the slightest idea what that has to do with the value of TDD, can you explain that?
I guess what I'm trying to get at here is that you didn't really propose a counter argument to DHH here, you said his historical knowledge is wrong, he doesn't know computer science and you get a lot of value from fast tests. The core of DHH's argument is that the metrics that are being touted to measure test suites are not metrics that actually help us write good code or code that is reliably well tested. It would be great to hear your direct responses to those arguments.
The feedback loop is my response. He positioned isolated unit testing as having drawbacks, but he never mentions (and doesn't seem to have experienced) the value of it.
Others have written responses to his claims about design. I think that he's less off-base with those; the design benefits of TDD are oversold to some extent, although they certainly do exist in my experience. It's difficult to oversell the speed benefits of isolated unit testing, though. You really have to experience it. Here's Noel Rappin commenting on my post saying exactly that, in fact: https://twitter.com/noelrap/status/461633622185746432
> The answer to this conundrum is often: that's the job of the integration tests. Okay, but integration tests are slow, so you'll never test all the permutations of the interactions of the components.
Integration tests aren't necessarily slow in any dramatic way (though they'll naturally be slower than unit tests), but in any nontrivial system you still won't test all the permutations of interactions between components because the number of such permutations will be prohibitively large even if integration tests were as fast as unit tests.
My preference is to aim for path complete integration tests, and thorough unit tests.
> When working with something as complex as an ORM like ActiveRecord, isolating the business logic and using the ORM strictly for persistence may allow for fast tests, but you still run the risk of bugs creeping in on the interface because of some assumption about or change in the way ActiveRecord works between versions or whatever.
If you isolate the domain and persistence layers rather than combining them the way Rails seems oriented toward, the persistence layer still ought to be a testable unit -- and as its job is persistence, you wouldn't isolate the database from it to test it. OTOH, its a unit that should be more stable than the model layer in most cases, and you won't pay the higher costs for running its unit tests when you are making changes to the model layer.
> That's why, as ugly and slow as Rails unit tests are, they are a simplifying compromise that strikes a balance between the theoretical ideal of a unit test and an integration test.
Why compromise between unit tests and integration tests when the two aren't exclusive and serve different purposes? Why not actually have good unit tests and good integration tests, instead of beasts that are neither fish nor fowl and are less than optimal as either?
It seems that the argument here is based on the false dichotomy that suggests if I do unit testing, I can't have integration tests, so I either need to just do integration tests or, for some reason, trade both for some sort-of-unit-ish tests that test the persistence and domain layers as if they were the same unit.
> Why compromise between unit tests and integration tests when the two aren't exclusive and serve different purposes? Why not actually have good unit tests and good integration tests, instead of beasts that are neither fish nor fowl and are less than optimal as either?
I wonder this myself. Unit tests serve a completely different purpose and should not get in the way of integration testing. They may/should help with structuring the code so that it is easier to integration test. They definitely shouldn't hinder. Then there is also the system level test, and then manual testing as well. All of these serve different purposes and can live happily together.
You've talked around my core point a lot, but you haven't addressed it directly at all. What do you do about bugs on the interfaces and interaction of the heavily modularized and unit-tested components?
> You've talked around my core point a lot, but you haven't addressed it directly at all.
I thought I addressed it quite directly.
> What do you do about bugs on the interfaces and interaction of the heavily modularized and unit-tested components?
Ideally, catch them with the unit tests -- whose entire purpose is testing interfaces -- but, where that fails, with traditional integration tests (which, when they find bugs missed by unit tests, prompt additional unit tests to isolate the offending unit and assure that they are properly squashed) rather than by abandoning unit and integration testing for something that isn't quite either and lacks the specificity and utility for test-on-each save of unit tests while abandoning the full end-to-end cycle testing of integration tests.
Your unit test guarantees the intended interface of the unit, but it doesn't guarantee the usage of said interface. Typically other unit tests will stub out this dependency. But what is your guarantee that the stub agrees with the spec'ed behavior.
Now this may range from a non-issue if the interface is obvious and straightforward and leaves little room for error, to extremely error-prone in a duck-typed language. Of course there are ways to mitigate this with the test and stubbing infrastructure, but it can be brittle as well.
This problem is why I think there is some sense of coarser-grained units with fewer stubs and some qualities of integration without going all the way up to the full-path acceptance level.
Depending on your problem domain, it may be easier or harder to create interfaces between all these TDD'd, isolated modules.
Aren't you assuming that TDD requires pervasive unit isolation?
I practice TDD often and prefer not to isolate units at least at the level of filesystem artifact.
I believe that one of the reasons test-first became TDD and gave up on the distinction between unit tests and acceptance tests is because the Smalltalk sUnit style of testing units as units didn't work so well in languages which aren't Smalltalk.
The answer to this conundrum is often: that's the job of the integration tests. Okay, but integration tests are slow, so you'll never test all the permutations of the interactions of the components.
DHH said during the keynote, "it's easy to make your tests fast when they don't test anything". Of course that's hyperbole, but there's a kernel of truth to be investigated there. When working with something as complex as an ORM like ActiveRecord, isolating the business logic and using the ORM strictly for persistence may allow for fast tests, but you still run the risk of bugs creeping in on the interface because of some assumption about or change in the way ActiveRecord works between versions or whatever.
That's why, as ugly and slow as Rails unit tests are, they are a simplifying compromise that strikes a balance between the theoretical ideal of a unit test and an integration test. ActiveRecord itself is this way too, in that often times the business logic just isn't complex enough to warrant the complexity of separating persistence from domain logic. As much as DHH may be talking out his ass without really having ever grokked TDD, I don't think his complaints are completely without merit.