Testing Abstract Classes.
In JUnit Recipes, J.B. Rainsberger's recommendation for specifying an interface is to have an abstract test case from which you derive concrete test cases to test the implementations.
public abstract class ATest extends TestCase
{
private A a;
public void SetUp() { a = getA(); }
protected abstract A getA();
public void AShouldRespondInSomeWayToSomeMessage()
{
a.doSomething();
//assert something about a
}
}
Essentially, all the concrete test cases do is implement creation methods that are defined as abstract in the base test class. This way, you've got one set of tests defined that specify the contract described by that interface.
So what about abstract classes? Let's say you've got A and B and you realize that they share some behavior so you want to extract an abstract C from which A and B are then derived. Before you do this, you have tests for a.doSomething() and b.doSomething() which are testing the same behavior, and you want to eliminate the duplication.
At this point, I see two potential paths (I'm sure there are more, but these are the two I see most clearly):
1 - create an abstract CTest class from which you derive CMethodsOfATest and CMethodsOfBTest. This would look a lot like the recipe for interfaces. If you prefer one test class per one class under test, you might put all of your A tests in ATest (rather than a class that just verifies the behavior of the C methods on A).
2 - create a CTest that is not abstract, have it test a TestableC[?] (a concrete derivative of C that adds no state or behavior beyond what C defines/implements). Then test the additional methods of A and B in ATest and BTest classes:
public class CTest extends TestCase
{
private C c;
public void SetUp() { c = new TestableC(); }
public void CShouldRespondInSomeWayToSomeMessage()
{
c.doSomething();
//assert something about c
}
class TestableC extends C { }
}
My preference had always been the former, as it meant that the common behavior was specified and tested against every class derived from C by a single set of specs/tests. Then a colleague pointed out that since there's only one implementation, you're executing the tests against the same implementation n times (one for each derived class), slowing down the test run. Think of it as run-time duplication (as opposed to duplication that appears in the code base). This made sense to me. We all want to minimize the speed of test runs. So I moved to the second approach.
Over time, however, I've noticed a couple of things. One is that there are times that derived classes override implementations defined in the base class. This should only happen as the result of a test on the derived class, but now you've got two sets of tests testing the same behavior. So there's our arch enemy: duplication. But the duplication is in the tests, not in the implementation.
The other thing I've noticed is that if you have any template methods in the base class, you have to verify that they call down to implementations in the derived classes. This really bugs me because these tests test wiring/implementation rather than behavior. Some argue that the base class calling down to the derived class IS behavior, but I'm seeing it that way less and less. I think it's design, not behavior. And whether or not I used a template method should have nothing to do with the specification of behavior.
So I'm leaning back towards the first approach. I'm not there 100% yet, so I'm curious to hear other people's approaches, experience and opinions related to specifying and testing abstract classes.
p.s. I'm exploring Behavior Driven Development so I use the words "specification" and "test" somewhat interchangably here. So for you TDDers, "spec" means "test". For you BDDers, "test" means "spec"
p.p.s. from a BDD perspective, specifying an abstract class seems silly. Abstraction is an implementation detail. I'm not sure what to make of that yet.
!commentForm
seems like an implementation detail to me. when you say two classes have common behvior isn't the important part that they behave the same way, not how they accomplish that?
I also don't find the argument about run-time duplication to be compelling. Seems like every other case where there is a question about too many tests taking too much time. Standard answer is to do the work to make the tests run faster.
But just because two classes share an interface does that mean they can always share tests? Not in my experience. So I'd think you're left using a mix of the two approaches above, as well as the case where you have an abstract base tests class that defines abstract test methods that need to be implemented.
I also don't find the argument about run-time duplication to be compelling. Seems like every other case where there is a question about too many tests taking too much time. Standard answer is to do the work to make the tests run faster.
But just because two classes share an interface does that mean they can always share tests? Not in my experience. So I'd think you're left using a mix of the two approaches above, as well as the case where you have an abstract base tests class that defines abstract test methods that need to be implemented.
<blockquote>since there's only one implementation, you're executing the tests against the same implementation n times (one for each derived class), slowing down the test run.</blockquote>
If the behavior of the method can't be affected by the subclass - because it is not calling into an abstract method and depends on no data that the subclass has an opportunity to change - that suggests to me that this is a method that belongs somewhere else. If it _can_ be affected by the subclass, then I think it's a very good idea to run it for every subclass.
If the behavior of the method can't be affected by the subclass - because it is not calling into an abstract method and depends on no data that the subclass has an opportunity to change - that suggests to me that this is a method that belongs somewhere else. If it _can_ be affected by the subclass, then I think it's a very good idea to run it for every subclass.
Jeffrey wrote "But just because two classes share an interface does that mean they can always share tests?"
Why not? As long as the contract of the interface is clear, the abstract base test approach works very well. I do think, however, that it's useful to allow more than one test class per class under test if you're doing that. So if you have an Action interface that exports only a do() method, that TimedAction and ThreadsafeAction both implement, then you might have ActionContractOnTimedActionTest and ActionContractOnThreadsafeActionTest (both derived from the absract ActionTest) which are separate from OtherBehaviorOnTimedActionTest and OtherBehavioronThreadsafeActionTest.
Why not? As long as the contract of the interface is clear, the abstract base test approach works very well. I do think, however, that it's useful to allow more than one test class per class under test if you're doing that. So if you have an Action interface that exports only a do() method, that TimedAction and ThreadsafeAction both implement, then you might have ActionContractOnTimedActionTest and ActionContractOnThreadsafeActionTest (both derived from the absract ActionTest) which are separate from OtherBehaviorOnTimedActionTest and OtherBehavioronThreadsafeActionTest.
Carl wrote "If the behavior of the method can't be affected by the subclass - because it is not calling into an abstract method and depends on no data that the subclass has an opportunity to change"
If you're committed to testing behavior, and not structure, then that shouldn't matter. In my view, those are implementation details. The design may start off with no calls to abstract methods, but if subsequent refactoring leads to the addition of an abstract method that is called, why should the existing tests have to change?
If you're committed to testing behavior, and not structure, then that shouldn't matter. In my view, those are implementation details. The design may start off with no calls to abstract methods, but if subsequent refactoring leads to the addition of an abstract method that is called, why should the existing tests have to change?
David wrote "Why not? As long as the contract of the interface is clear, the abstract base test approach works very well."
But that's what I mean. Interfaces don't always imply a contract where shared tests are meaningful. toString() comes to mind as an example. Or even equals(). While you could imagine a shared test for equals that required the decendent test classes to implement getA() and getAEquivalent() and getB() to test for both equality and inequality... I'd really question the value of doing so. Seems like too much indirectly yielding a loss of clarity.
But that's what I mean. Interfaces don't always imply a contract where shared tests are meaningful. toString() comes to mind as an example. Or even equals(). While you could imagine a shared test for equals that required the decendent test classes to implement getA() and getAEquivalent() and getB() to test for both equality and inequality... I'd really question the value of doing so. Seems like too much indirectly yielding a loss of clarity.
David wrote: If you're committed to testing behavior, and not structure, then [whether a method's {current} behavior can be affected by subclasses] shouldn't matter. In my view, those are implementation details. The design may start off with no calls to abstract methods, but if subsequent refactoring leads to the addition of an abstract method that is called, why should the existing tests have to change?
OK, then, take the "multiple testing" hit. More testing good. In theory, a subclass might affect the implementation, sometime, somewhere, so that the earlier assertion that "there's only one implementation" is only true now and potentially not in the future. If you're writing tests for posterity then taking the hit is the way to go. If you're writing tests to drive the design, if you see test code as soft and malleable, you can be a little more flexible.
OK, then, take the "multiple testing" hit. More testing good. In theory, a subclass might affect the implementation, sometime, somewhere, so that the earlier assertion that "there's only one implementation" is only true now and potentially not in the future. If you're writing tests for posterity then taking the hit is the way to go. If you're writing tests to drive the design, if you see test code as soft and malleable, you can be a little more flexible.
Carl - that's an interesting distinction: coding for posterity vs driving design via tests. I think that what we're talking about here are abstractions that appear through refactoring. There's duplication in behavior, so we abstract a common base class. Now there's duplication in tests, so we refactor the tests, possibly resulting in one of the structures I proposed above. If that's the case, then the changes to the tests are resulting from changes to the design. I'm not convinced that this violates TDD principles - it's a natural step in the evolution of a design. But it does somehow seem un-TDD. Maybe it's the yang to TDD's yin. Not sure.
Anyhow - I like coding for posterity when it comes to interfaces. It helps future developers (including me!) to better understand what the interface is for, and it's contract. I know some people disagree w/ that, citing pairing as the cure-all for this sort of problem. But I've not yet seen pairing catch all the problems that arise from this sort of evolution, and while I advocate and practice it, I don't see any problem w/ additional information being presented by the tests in this way.
Anyhow - I like coding for posterity when it comes to interfaces. It helps future developers (including me!) to better understand what the interface is for, and it's contract. I know some people disagree w/ that, citing pairing as the cure-all for this sort of problem. But I've not yet seen pairing catch all the problems that arise from this sort of evolution, and while I advocate and practice it, I don't see any problem w/ additional information being presented by the tests in this way.
Jeffrey - good point re: toString() and equals(). How would you handle an interface that had a mixture of methods like this, where the contract is more subjective, and other methods where the the contract is more concretely defined?
"How would you handle an interface that had a mixture of methods like this, where the contract is more subjective, and other methods where the the contract is more concretely defined?"
Write looser or zero tests in the superclass for the aspects of the contract that can vary more? A little more deviously, you could also write abstract tests like "abstract testToStringDoesSomeBasicThingWeCountOn".
Write looser or zero tests in the superclass for the aspects of the contract that can vary more? A little more deviously, you could also write abstract tests like "abstract testToStringDoesSomeBasicThingWeCountOn".
"How would you handle an interface that had a mixture of methods like this, where the contract is more subjective, and other methods where the the contract is more concretely defined?"
Sounds like a candidate for interface segregation. If one can even talk about different parts of the contract for an interface, it suggests to me that the different parts of the contract should be extracted as separate interfaces.
To an extent, equals and toString are good examples of this, since their contracts are not related to other behaviours implemented by concrete classes defining equals and toString implementations. OTOH toString and equals are defined in Object which everything extends, including interfaces (somewhat bizarrely since Object is a concrete class(!), for no very good reason that has been explained to me).
Sounds like a candidate for interface segregation. If one can even talk about different parts of the contract for an interface, it suggests to me that the different parts of the contract should be extracted as separate interfaces.
To an extent, equals and toString are good examples of this, since their contracts are not related to other behaviours implemented by concrete classes defining equals and toString implementations. OTOH toString and equals are defined in Object which everything extends, including interfaces (somewhat bizarrely since Object is a concrete class(!), for no very good reason that has been explained to me).
Unless you're in the camp that mocks everything (I'm not, as you may recall from the TDD discussion in December), you're going to have rampant runtime duplication anywhere you have objects collaborating with other objects. I don't see this as a problem--even though you're testing a single definition, that one implementation can vary considerably in behavior depending on context. I prefer the Abstract Test Class solution, #1 here.
-Jeff
-Jeff
It seems to me that asking about testing an abstract class is reflective of a poor mental model of TDD. Shouldn't abstract classes be created only in the Refactoring step? If that's the case, then you've already created tests to support your user stories. An abstract class, by definition, doesn't do anything. Since you can't write a test for "nothing" TDD does not drive to testing abstract classes at any time other than the refactoring stage. At that point, you already have your tests, so problem solved.
Good point from Kelly Anderson. Abstract Test Class should get there by refactoring tests.
Kelly - when interfaces and abstract classes reveal themselves through TDD, they usually reveal duplication in your tests as well. It sounds like you are saying that we should ignore that duplication (please correct me if I'm misreading your post). In my view, that duplication needs to be addressed. Maybe there's a domain level abstraction that's trying to make itself known.
I guess that if there's no domain level abstraction that really makes sense, then the duplication is acceptable. But that still sounds unsatisfying.
So my question back out to you (and Jeff) is how do you handle duplication in tests that is discovered as abstractions are discovered?
I guess that if there's no domain level abstraction that really makes sense, then the duplication is acceptable. But that still sounds unsatisfying.
So my question back out to you (and Jeff) is how do you handle duplication in tests that is discovered as abstractions are discovered?
If a bug is introduced into the system, how many tests should fail? Ideally one. If there is only one test failure then the problem is easily found. If dozens of tests fail then the problem is not obvious. Using abstract TestCases[?] causes multiple test failures for just one bug. Run-time duplication in general will cause multiple test failures.
Putting tests in an abstract TestCase[?] can also somewhat of a lie. For example, tests for methods that are never overrided may get inherited by dozens of other test cases. I’ve seen people pride themselves on having 150 passing tests, when 100 of the tests are actually all the same test, inherited over and over. That’s ridiculous.
Putting tests in an abstract TestCase[?] can also somewhat of a lie. For example, tests for methods that are never overrided may get inherited by dozens of other test cases. I’ve seen people pride themselves on having 150 passing tests, when 100 of the tests are actually all the same test, inherited over and over. That’s ridiculous.
TemplateMethods have behavior. It’s true that you could write tests that simply test the wiring… but that would be silly. Write tests that test the behavior of the of the TemplateMethod[?]. By their nature TemplateMethods are not testable. Simple/Mock/Testable implementations are an easy way to get around that.
If you only test your TemplateMethods through the derived classes, you’re setting yourself up for some painful test writing. The derivatives of TemplateMethods have all the details and complexity that can make you jump trough hoops just to write a test. Testing the base class directly, by means of a trivial implementation, can make life so much easier.
If you only test your TemplateMethods through the derived classes, you’re setting yourself up for some painful test writing. The derivatives of TemplateMethods have all the details and complexity that can make you jump trough hoops just to write a test. Testing the base class directly, by means of a trivial implementation, can make life so much easier.
If I'm running the tests frequently, then whether one test fails or dozens fail shouldn't matter. The problem should be the line of code I just wrote before I ran the tests.
As for the number of tests being a lie, I think that depends on what the goal of testing is. If it's to drive design, then who cares how many tests are executing the same lines of code? If it's to optimize the tests so that they run quicker, then that number becomes more important. I guess there's a balance in there, but just as we code for clarity and optimize when the problem reveals itself, one might argue that we should write our tests for clarity (as specification) and only reduce that clarity as optimizations become necessary.
As for the number of tests being a lie, I think that depends on what the goal of testing is. If it's to drive design, then who cares how many tests are executing the same lines of code? If it's to optimize the tests so that they run quicker, then that number becomes more important. I guess there's a balance in there, but just as we code for clarity and optimize when the problem reveals itself, one might argue that we should write our tests for clarity (as specification) and only reduce that clarity as optimizations become necessary.
Micah wrote: "Template methods have behavior".
What is the behavior of a template method? That it calls abstract methods? How is that meaningful to clients? I don't think it is. Therefore why would I want a test that specifies that my implementation uses Template Method?
What if I change my mind about the design later and choose to no longer use template method? I'd have to change these tests. This is going to discourage (and I'm seeing this right now in the code base I'm working with) refactoring towards better designs, because the refactoring requires refactoring tests that are bound to structure.
What is the behavior of a template method? That it calls abstract methods? How is that meaningful to clients? I don't think it is. Therefore why would I want a test that specifies that my implementation uses Template Method?
What if I change my mind about the design later and choose to no longer use template method? I'd have to change these tests. This is going to discourage (and I'm seeing this right now in the code base I'm working with) refactoring towards better designs, because the refactoring requires refactoring tests that are bound to structure.
> If a bug is introduced into the system, how many tests should fail? Ideally one.
> If there is only one test failure then the problem is easily found. If dozens of
> tests fail then the problem is not obvious. Using abstract TestCases[?]? causes
> multiple test failures for just one bug. Run-time duplication in general will
> cause multiple test failures.
But these failures look very similar; they aren't hard to pinpoint.
I definitely don't use abstract case to bump up test counts, although that's exciting for some people. ;-)
I've done things all 3 (?) ways: abstract test case, building a simple stub implementation in the test, and ignoring the issue completely, i.e. the superclass methods get implicitly tested by all subclasses. In all honesty, I don't force the issue, and I don't find myself using abstract test case often, but it seems to occur more naturally for me when doing TDD by virtue of refactoring.
I would wonder about the guy with 150 subclass tests, sounds like too much inheritance anyway.
> If there is only one test failure then the problem is easily found. If dozens of
> tests fail then the problem is not obvious. Using abstract TestCases[?]? causes
> multiple test failures for just one bug. Run-time duplication in general will
> cause multiple test failures.
But these failures look very similar; they aren't hard to pinpoint.
I definitely don't use abstract case to bump up test counts, although that's exciting for some people. ;-)
I've done things all 3 (?) ways: abstract test case, building a simple stub implementation in the test, and ignoring the issue completely, i.e. the superclass methods get implicitly tested by all subclasses. In all honesty, I don't force the issue, and I don't find myself using abstract test case often, but it seems to occur more naturally for me when doing TDD by virtue of refactoring.
I would wonder about the guy with 150 subclass tests, sounds like too much inheritance anyway.
I'm sure you could bully NMock into generating a derived concrete class for you on the fly.
David asked, "What is the behavior of a template method? That it calls abstract methods? How is that meaningful to clients?"
If a template method did nothing but call abstract methods in a certain sequence (or even just a combination), that would be its behavior. More typically, template methods do some actual work before and/or after calling the abstract methods.
If a template method did nothing but call abstract methods in a certain sequence (or even just a combination), that would be its behavior. More typically, template methods do some actual work before and/or after calling the abstract methods.
Tom wrote "More typically, template methods do some actual work before and/or after calling the abstract methods."
That's true, but how do you test the actual work? Let's say that doSomething() on AbstractThing does a bunch of stuff and calls a bunch of abstract methods.
You could test every implementation of every abstract method in the derived classes, and test that doSomething() actually calls them. For one thing, this involves making each of the abstract methods public (when they could often be perfectly happy being protected). Let's say you do that and everything works right. How do you know that calling doSomething() on one of the subclasses does what you think it should do. You don't unless you test it directly, right? In which case why do you need all of those tests proving that doSomething calls the abstract methods? Why do you need the tests on their implementations?
I say just test the method that's actually going to get called by client code: doSomething().
That's true, but how do you test the actual work? Let's say that doSomething() on AbstractThing does a bunch of stuff and calls a bunch of abstract methods.
You could test every implementation of every abstract method in the derived classes, and test that doSomething() actually calls them. For one thing, this involves making each of the abstract methods public (when they could often be perfectly happy being protected). Let's say you do that and everything works right. How do you know that calling doSomething() on one of the subclasses does what you think it should do. You don't unless you test it directly, right? In which case why do you need all of those tests proving that doSomething calls the abstract methods? Why do you need the tests on their implementations?
I say just test the method that's actually going to get called by client code: doSomething().
I like option 2 better.
Like Kelley, I find that TDD doesnt result in a test for abstract classes. Refactoring might generate an abstract class but not TDD. If there's a feature I need to add to an abstract class, then it'd be time for a test fixture.
That said, if I did make a test fixture for an abstract class, I'd do option #2 - concrete test class CTest. I want my tests to show a kind of use case. Deriving a class TestableC[?] is somewhat like showing a use case for my abstract class. So I want to derive a class in my test fixture.
A future developer (typically me) could look at the tests and go, "oh, that's how the author expected us to use this abstract class C."
I tend to have abstract classes implement an interface. C would implement an interface IC. so there'd also be an abstract ICTest. Thus CTest would extend ICTest. The ATest and BTest would probably extend ICTest directly. This way A and B are only constrained by the interface IC and not by the abstract class C.
Thing is even if C doesnt implement an interface, you may want to create the test fixture for the "contract" IC that A and B (and C) must comply with. ATest, BTest and CTest will extend the ICTest to ensure that they comply with this contract. and A and B wont have to extend C anymore.
Like Kelley, I find that TDD doesnt result in a test for abstract classes. Refactoring might generate an abstract class but not TDD. If there's a feature I need to add to an abstract class, then it'd be time for a test fixture.
That said, if I did make a test fixture for an abstract class, I'd do option #2 - concrete test class CTest. I want my tests to show a kind of use case. Deriving a class TestableC[?] is somewhat like showing a use case for my abstract class. So I want to derive a class in my test fixture.
A future developer (typically me) could look at the tests and go, "oh, that's how the author expected us to use this abstract class C."
I tend to have abstract classes implement an interface. C would implement an interface IC. so there'd also be an abstract ICTest. Thus CTest would extend ICTest. The ATest and BTest would probably extend ICTest directly. This way A and B are only constrained by the interface IC and not by the abstract class C.
Thing is even if C doesnt implement an interface, you may want to create the test fixture for the "contract" IC that A and B (and C) must comply with. ATest, BTest and CTest will extend the ICTest to ensure that they comply with this contract. and A and B wont have to extend C anymore.
Add Child Page to TestingAbstractClasses