Well explain further. I hate these smart arse sounding comments - "you are doing it wrong" without any indication why, or how to do it better.

Thus any test that is written in such a way that it would be present an issue in refactoring code should be avoided if at all possible. A simple example is directly constructing the class under test in the test method:

  @Test 
  public void tryToAvoidDoingThis() {
     MyClass = new MyClass(param1, param2);
     // do stuff to my class
   }

Instead, have a level of indirection and have a single method that can create a sample MyClass. Now when the parameters change, only one construction site has to be updated.

In general, unit tests should not be testing specific / internal implementation details of the class. Rather, the tests should verify the documented public behavior of the class.

Interestingly it's also the crux of good design. Breaking code up into composeable pieces that aren't braided together.

Other examples may be around a `setup` method. If the method is private, don't test it. Then you can refactor freely. If it's public, test the pre/post conditions around the method in as few places as possible (hopefully one). Even if other tests rely on the object having been "setup", just trust that it works. If the specification of `setup` changes, you just have the `setup` tests to update, not the entire object.

Second, start writing tests to verify bug reports and then fix the bug. In large systems I find this critical to honing in on the exact problem. The mental exercise of crafting the test to trigger the bug helps me really understand the problem.

Finally, start new features by first writing a test to simply drive your new codes golden path. When working in a large system I find writing a test to run my new code a much faster development turn around time than rebooting the entire system. This is compounded when there are many systems or moving parts which is common today.

With unit tests, there are certain things that must be tested, such as very high-value code contracts, and the like. There are many things that people test (like "correct output" for the input) which may not be so valuable, particularly if several possible values may be correct.

So test contracts, not internals, and not representation. And please for the heaven's sake, don't test the behavior of your dependencies.

Unfortunately I also believe those comments are generally true, and I also believe when the posters answer "why", they will give you an answer that is also doing it the wrong way.

I have no idea how to do unit testing. I only believe there is a right way.

This doesn't make sense, how could you trust his unit tests?

For better or worse, this system had a number of data-centric objects; these objects passed many correct tests, but they failed to be deserializable from XML (because of the way collections were handled) as well as having other deficiencies.

The tests meant I could fix those deficiencies quickly and have faith that I fixed them correctly. Of course, I added new tests to test that the system did the things it had to do.

I write unit tests for every one of my core algorithms. I have also seen a massive amount of dumb tests written by TDD people who strive for 100% code coverage. Ridiculous. It takes as much time to maintain the (often useless) tests as it does the code. Where they make sense they're indispensable.

I think the fundamental question is why 100% code coverage is important. The fact is that it isn't. the problem with TDD is that a lot of people who do it totally get the idea of why you are testing entirely wrong. The goal should never be to "ensure your code functions properly." The coal should be to "ensure the code contracts are adhered to." If you test with that in mind, you will write lots of unit tests and almost never have to rewrite or delete them due to fixing bugs.

Again this comes back to what I said in another comment that tests should never be written to the code. Once you get that, then code coverage ends up being meaningless and not something you want to worry about.

Big projects over some size normally do it that way.

More specifically, consider an SSE2-based function 'float32 floor(float32)'. There's only about 4 billion inputs, so why not test them all? That only takes a minute or so.

How is testing 100 inputs a unit test and testing 4 billion inputs, through exactly the same API, an integration test?

As the author points out, many people wrote libraries which are supposed to handle the entire range, but ended up making errors under various conditions, and even given wrong answers for over 20% of the possible input range.

The statement was very causal.

> If you've got a slow test suite, that mean's you've got a bad test suite.

Or you have a fantastic test suite that also includes integration tests. See?

If you say it's slow then either it's a bad test suite, and/or it includes integration tests. I believe that is the logic, yes?

There is no lower unit to test, so therefore this must be a unit test.

The linked-to page shows that testing all possibilities identifies flaws that normal manual test construction did not find. Therefore, it must be a better test suite than using manually selected test cases, with several examples of poorly tested implementations.

(Note: writing an exact test against the equivalent libc results is easier to write than selecting test cases manually, and it's easier for someone else to verify that the code is testing all possibilities than to verify that a selected set of corner cases is complete.)

Therefore, logic says that it is not a bad test suite.

Since it contains unit tests and it is not a bad test suite, therefore it must not be slow.

Therefore, 4.5 minutes to unit test these three functions is not "slow".

Therefore, acceptable unit tests may take several minutes to run.

That is what the logic says. Do you agree? If not, where is the flaw in my logic?

A good test suite has two qualities - how comprehensive it is and how fast it takes to run. If either is lacking, then it is no longer a good test suite.

We had a series of tests (more towards integration tests I guess) at one point in LedgerSMB that did things like check database permissions for some semblance of sanity. These took about 10 min to run on a decent system. The reason was we stuck with functionality we could guarantee wouldn't change (information schema) which did not perform well in this case. Eventually we got tired of this and rewrote the tests against the system tables cutting it down to something quite manageable.

We had this test mixed in with db logic unit tests because it provided more information we could use to track other failures of tests (i.e. "the database is sanely set up" is a prerequisite for db unit tests).

Either that, or the person just hasn't been sufficiently burned by someone changing something you're not aware of and having to track down a run time error for days that could have been caught and fixed by a unit test in minutes.

I read the article, and much of what he speaks of is tautological unit tests - testing something where someone could never have done anything but that. I've seen people unit test the behavior of filling up a collection then test to make sure the collection has all of those elements, for instance. And there, he has a point.

But there's a dangerous line there. And while the article makes a good point that too much of that type of testing can be detrimental, I've generally found it better to err on the safer side of more tests.

On your own project, or even on a small team, you can probably get away without them much of the time. But, on larger projects, where many different developers sometimes go back and make changes to code they didn't write, it's very easy for a new guy/gal to make changes with unanticipated consequences. When that occurs without sufficient test coverage, the project will wind up spending 10-20x more man hours to fix the issue.

Each test is extremely fast. Testing all 12+ billion cases takes 4.5 minutes. Are these unit tests or integration tests?

If they are integration tests, roughly where is the boundary between the two?

Is it meaningful to distinguish between "unit" and "integration" testing based only on the amount of time they takes? That is, if the unit tests take 0.1 second too long then do they suddenly become integration tests?

I think the problem is that definitions that should reflect the granularity of what is being tested have become too interconnected with assumptions about frequency of testing / time to run and it has created completely mangled definitions.

The problem is so bad that some developers I've crossed have the firm opinion that mstest should only be used for "unit" tests. It is frustrating.

If you have slow unit tests, don't run them on every compile. If you have fast integration tests, run them as much as you'd like. I've personally never defined unit tests as "things that run fast", despite that being a valuable property it does not seem essential to the definition, but perhaps I have the wrong understanding of what a unit test is.

The longer your unit tests take, the less likely people will be to use and run them often, which is the whole point. Let the nightly build on the CI machine exercise the long running tests when everyone is asleep.

Really, I think the problem is that unit test frameworks are currently incapable of doing the right testing.

Unit tests take quadratic time. That is, each new test requires running all previous tests, to get the green. And at some point, a project will have enough tests that it can't finish in 10-15s.

One option is to mark "fast" and "slow" tests. Another is to recategorize them as "unit" vs. "functional" tests.

These are poorly-defined labels. In this case, the 4.5 minutes of testing is "slow", yes, but it only needs to be run when a specific, small part of the code changes. The problem is, there's no way to determine that automatically. The test runner can't look at the previous test execution path and see that nothing has changed, and there's no way to mark that a test should only be run if code in functions X, Y, or Z of module ABC has changed.

Humans are able to figure this out. Well, sometimes. And with lots of mistakes. Get the unit test framework to talk with a coverage analysis tool, plus some static analysis and perhaps a few annotations, and this discussion of how to distinguish one set of tests from another disappears.

Blue-sky dreams. I know. :)

In real life we toss those functions into their own library, note that the code is static, and do the full test suite only occasionally; mostly when the compiler changes.

In other words, bypass CI the same way one does any other third party library. (How often do you run the gcc test suite?)

That still means to run all the tests each time, with re-prioritization to enrich the likelihood of faster feedback.

But if none of the code paths used for a test have changed, and the compiler hasn't changed, and there's nothing which depends on random input or timing effects, then why run those tests at all?

The reason is we don't have a good way to do that dependency analysis, which is why we run all of the tests all of the time. Or we manually partition them into "slow" and "fast" tests.

So now we have to write tests for all our code, and tests that run fast. We have to refactor our code around the tests. Something seems a bit back to front here. Or is the assumption that if we have 100% test coverage that runs fast then it means that we have written the best code possible?

I think I am siding with the author of the article on this one.

The author complains about re-factoring code into smaller testable functions. I completely disagree. Code structured as small easily understood functions which do one thing and have obvious inputs and outputs is good code which is much easier to extend and modify.

Sounds to me like the kind of tests you're describing aren't necessarily unit tests (system-level race conditions aren't typically discoverable with a simple unit test); where the tests were truly unit-level, the proposed alternative (assertions) may have been even more informative. Finally - a few real unit tests for known correct behavior are advised where in essence the algorithm can be described independently from its outcome.