Stop Mocking Me
Micah and I have been working together on a .NET project. We both like to use mock objects for interaction testing, but Micah prefers to make mocks by hand, while I prefer to use tools like RhinoMocks. We, along with the other developers, agreed a while back that we wouldn't put any restriction on this - whomever is writing tests can use the mock approach of his choosing. This decision has led to some debates over the costs and benefits of hand mocks vs tools and so we've agreed to try a little experiment. For three days Micah's going to use nothing but RhinoMocks for all of his mocking needs, and for those same two days I'm going to use nothing but hand coded mocks. If we end up pairing together, whoever is driving at the moment agrees to abide by his mock commitment. There is no goal to make a "final decision" or anything like that. It's more of an experiment. Where it takes us, we don't really know.As part of this excercise, I'm going to share my thoughts before and after. You can read Micah's perspective on this here: PreludeToTheMockOff
Here's the before:
Here's what I like about mock tools:
* They work. This is huge for me. Once I've written the test and it's failing, I want to get right to the code under test. I don't want to worry about writing a mock that must behave in some way specific for this test.
* They work the same way (almost*) every time. When I go from one test to another that uses a given tool, I (almost*) always understand right away what's going on in the test.
* They tend to be pretty flexible. They are called mock tools, like NMock and RhinoMocks, but they can easily function as stubs or mocks. They can be strict (you have to define every single expectation in order) or dynamic (they only verify the specific calls that you specify).
Note* - there are times when we do silly things just to get the test to work. These silly things can be painful, but they often make me think more about the design, whereas w/ a handmock I can more easily force the handmock to play nice w/ a bad design.
Here's what I don't like about mock tools:
* To get them to work, you have to know how. As with any toolset, there are things that are less than obvious that you just need to know. Some things are quirky and inconsistent. For example in Rhino, if a method that you're mocking returns void you use one syntax and if it returns anything else you use another. NMock uses similar syntax for both, but uses strings and reflection to get at the method names. This loses compile time checking, and reduces the power of refactoring tools.
* The messages you get don't always point you directly to the problem. For example, if you change something that causes an error to be thrown before you get to mocks.ReplayAll[?]() (using Rhino) you won't see the error that was thrown. Instead you'll see an error stating that the next call to one of the mocks can't happen in record mode. I'll confess that this is pretty significant and I have been frustrated at times trying to track down errors that are hidden by the misleading messages.
* I've noticed that if the interface being mocked is big, the mock takes some time to create on the fly. More time (I've tested this) than the hand mock takes to create.
Here's what I like about making mocks by hand:
* You can get them to do whatever you want.
* They're often pretty simple.
* You can look at the code directly to understand problems.
Here's what I don't like about making mocks by hand:
* You can get them to do whatever you want. This sounds like a good thing (it's in the list of things I like, too), but this often leads to duplicated business logic finding its way into the mock, or hides design issues that really should be exposed and explored.
* If I need to access the same method call but glean different information from it in different tests, I have to write two different hand mocks. Or worse, write some goofy method that instructs that method how to behave.
* When verifying messages received, you have to expose some property - often a property that is not part of the type being mocked. Even though this is only added to the mock, I've seen them slip up to the mocked class because they seemed useful (the test is using it, why can't my production code?). Next thing you know you've got twice as many properties on the mocked class.
* You end up with directories and/or packages and/or namespaces filled w/ these things.
* To mock abstract or concrete classes, a common approach is to derive the mock from the class being mocked, overriding virtual methods. This can sometimes lead to confusing debugging sessions where you realize that the method you thought was being invoked on the class you're mocking (and is not behaving as you expected) is actually being invoked on the mock (which is why its not behaving as you expected).
In summary, I personally prefer to use mock tools because I understand how they work. They can be idiosyncratic, but they are consistent in their idiosyncracies and therefore more managable. For me.
Check back in a few days for the after blog!
!commentForm
Nice post, but I'll add my biggest annoyance with "roll your own mocks." I use mock objects frequently as a stand in as I code a client class to discover what the interface should be. Since we're all good Agile folks here we're doing evolutionary design and the interfaces are always changing. It annoys the jeebers out of me to keep going into static mock classes just to implement the new method on the interface just to compile. When I'm using a dynamic mock that's never an issue. I think static mocks are usually more work myself, but I do use them on occasion. I think the "weirdness" of dynamic mock objects scares a lot of people away.
I'm just starting to play with RhinoMocks on a side project, but we use NMock at work. I'd love to hear your thoughts on the differences between the two tools.
I'm just starting to play with RhinoMocks on a side project, but we use NMock at work. I'd love to hear your thoughts on the differences between the two tools.
Jeremy - I agree that the "weird factor" does scare people away. I started using mocks w/ jMock, a predacessor to NMock with very similar syntax. The whole concept of having a MockControl in addition to the object itself is very weird. And even though the fact that you tell the Control to expect calls on the Mock is accurate, it is still confusing.
I also agree w/ your comment on having to revisit the mock everytime the interface changes.
I think that Rhino's syntax is a little more clear in this way. It's the same concept, but you don't talk to the controller except to tell it to stop recording and then to verify. Otherwise you're dealing directly w/ the mocks themselves with the clever use of some static methods to come up w/ more sensible syntax (in my view). So in NMock you would do this:
but in Rhino you would do this:
So in Rhino, the message LOOKS LIKE the message. So it's a little easier to understand. On the flip side, you have different syntax in Rhino depending on whether the method returns void or anything else:
So all in all I'm prefering to use Rhino these days, though I'm still urked by some of its idiosyncrasies. In fairness though, some of those idiosyncrasies are necessary due to the language. Mocks in Ruby don't force you to do any of that stuff. Check out the simple mock framework that comes w/ rSpec and you'll see what I mean.
I also agree w/ your comment on having to revisit the mock everytime the interface changes.
I think that Rhino's syntax is a little more clear in this way. It's the same concept, but you don't talk to the controller except to tell it to stop recording and then to verify. Otherwise you're dealing directly w/ the mocks themselves with the clever use of some static methods to come up w/ more sensible syntax (in my view). So in NMock you would do this:
MockControl mockControl = new MockControl(Thing);
Thing thing = (Thing)mockControl.MockInstance;
mockControl.Expect("method", "arg1", "arg2");
but in Rhino you would do this:
MockRepository mocks = new MockRepository();
Thing thing = (Thing)mocks.CreateMock(typeof(Thing));
thing.method("arg1","arg2");
So in Rhino, the message LOOKS LIKE the message. So it's a little easier to understand. On the flip side, you have different syntax in Rhino depending on whether the method returns void or anything else:
thing.methodThatReturnsVoid(arg);
Expect.Call(thing.methodThatReturnsObject(arg)).Return(myObjectOfChoice);
So all in all I'm prefering to use Rhino these days, though I'm still urked by some of its idiosyncrasies. In fairness though, some of those idiosyncrasies are necessary due to the language. Mocks in Ruby don't force you to do any of that stuff. Check out the simple mock framework that comes w/ rSpec and you'll see what I mean.
In the end is probably just "because its what I am used to" however I have some (in theory) objective reasons for preferring them:
1) For the majority of my tests, it is much simpler than using a framework (I can look at the mock/fake class and see what's happening.
2) I have complete control over what is happening (I know where the error is occuring, I can switch between state and behavior testing as needed, I can modify the mock's behavior to handle new issues, etc.
3) Hand-tooled mocks often lead to new ideas about production code. The mock *is* real code from a behavior POV. Having two examples often leads to new ideas about what the real code should be doing. I have also seen hand-coded mocks evolve into full-fledged simulators (nice to have when trying to do acceptance testing against an external boundary that isn't amenable to testing, like say a 3rd party web service interface, or other API).
4) Its a better way to learn (I have similar luddite beliefs about refactoring). If you can write reusable, simple no-business-logic mocks and get your work done fast, you are ahead of those who can only use a tool to do that. I can always use the tool and appreciate what's its doing for me, if I can do it by hand first. E.g. Resharper is a great tool for .NET developers, and I use it heavily, but I think I wouldn't have been nearly as good at TDD if I hadn't spent several years doing my refactorings by hand (productively mind you). On this last point, I am particularly convinced when it comes to teaching others - a mock framework might get a short-term gain on their productivity, but I think its at the cost of depth of knowledge about how to do TDD effectively.
As for the "mock-only stuff migrating into real code" I use a convention that all Mock-only members are prefixed like internals of the class even when public (e.g. if fields are _foo then so are mock publics). This makes it easy to see what is mock and what is real interface. OTOH, sometimes that migration Is A Good Thing -- the real code is better for having that method or property.
1) For the majority of my tests, it is much simpler than using a framework (I can look at the mock/fake class and see what's happening.
2) I have complete control over what is happening (I know where the error is occuring, I can switch between state and behavior testing as needed, I can modify the mock's behavior to handle new issues, etc.
3) Hand-tooled mocks often lead to new ideas about production code. The mock *is* real code from a behavior POV. Having two examples often leads to new ideas about what the real code should be doing. I have also seen hand-coded mocks evolve into full-fledged simulators (nice to have when trying to do acceptance testing against an external boundary that isn't amenable to testing, like say a 3rd party web service interface, or other API).
4) Its a better way to learn (I have similar luddite beliefs about refactoring). If you can write reusable, simple no-business-logic mocks and get your work done fast, you are ahead of those who can only use a tool to do that. I can always use the tool and appreciate what's its doing for me, if I can do it by hand first. E.g. Resharper is a great tool for .NET developers, and I use it heavily, but I think I wouldn't have been nearly as good at TDD if I hadn't spent several years doing my refactorings by hand (productively mind you). On this last point, I am particularly convinced when it comes to teaching others - a mock framework might get a short-term gain on their productivity, but I think its at the cost of depth of knowledge about how to do TDD effectively.
As for the "mock-only stuff migrating into real code" I use a convention that all Mock-only members are prefixed like internals of the class even when public (e.g. if fields are _foo then so are mock publics). This makes it easy to see what is mock and what is real interface. OTOH, sometimes that migration Is A Good Thing -- the real code is better for having that method or property.
It's funny. One of the more frustrating things in this line of work is what I think of as the skipped-that scenario. As you get more experienced, you develop your arsenal of tricks, you may not jump on the bandwagon right away, if ever. So, there are things you learn to do without, pretty easily. And, unfortunately, that can leave you out of touch with people who didn't realize that they could skip-that. I don't know as much about the current crop of mock object frameworks, the debugger in my IDE, or, say, entity beans as I would if I used them regularly and it is hard to resolve to learn more about them relative to more pressing things.
I should run the mock experiment, though. I like hand-made mocks, perhaps there's a hump to get over.
I should run the mock experiment, though. I like hand-made mocks, perhaps there's a hump to get over.
Michael:
>perhaps there's a hump to get over
Perhaps. I think too its about pain (including one's threshold). I really engrained my TDD programming-style on a C++ project, using none of the tools test-first programmers have come to expect - beyond this xunit testing tool called CppUnit[?] that some joker wrote. ;-) Others I know who did the same also tend to prefer doing things by hand. Michael, I believe you did as well. Perhaps this has something to do with our aversion to mocking tools - I simply don't find refactoring and mocking by hand painful, I am productive, and I am happy - so why switch?
Indeed, why try?
Well, first, I may be *more* productive using a tool than doing it by hand. Second, while I might be used to it -- even prefer it, others might find that too steep a hill to climb (and since I am often in the position of teaching others...). Third, learning new ways to look at old problems gains new insights.
So, while I was (and remain) quite content with by-hand TDD, I gave resharper a try (after a taste of intelliJ), and I have to admit that I use it a lot (I also regularly do a lot of additional refactoring that neither tool supports, I often wonder if those who haven't learned to be productive doing it by hand do?). And reading these posts prompted me to go look at RhinoMocks (I never liked NMock much), and I have to say, I am intrigued, and may give it a try.
I think though, that if RM enters my toolkit it will supplement, not replace, my use of hand mocks, because I tend to be much more fluid about testing state/interaction/replay than those who advocate (or aver) these tools seem to be. I generally just declare my intent in the test, and then make it pass - sometimes the test makes assertions about state, sometimes interaction, and sometimes order (and often some combination). I can't see any framework giving me the kind of flexibility I am used to as easily and simply as a specific-purpose tool like a fake class with no behavior, but it may well make some of it easier (and it will definitely provide new perspective).
Good luck to you (Micah and David) looking forward to reading how your experiment turns out.
>perhaps there's a hump to get over
Perhaps. I think too its about pain (including one's threshold). I really engrained my TDD programming-style on a C++ project, using none of the tools test-first programmers have come to expect - beyond this xunit testing tool called CppUnit[?] that some joker wrote. ;-) Others I know who did the same also tend to prefer doing things by hand. Michael, I believe you did as well. Perhaps this has something to do with our aversion to mocking tools - I simply don't find refactoring and mocking by hand painful, I am productive, and I am happy - so why switch?
Indeed, why try?
Well, first, I may be *more* productive using a tool than doing it by hand. Second, while I might be used to it -- even prefer it, others might find that too steep a hill to climb (and since I am often in the position of teaching others...). Third, learning new ways to look at old problems gains new insights.
So, while I was (and remain) quite content with by-hand TDD, I gave resharper a try (after a taste of intelliJ), and I have to admit that I use it a lot (I also regularly do a lot of additional refactoring that neither tool supports, I often wonder if those who haven't learned to be productive doing it by hand do?). And reading these posts prompted me to go look at RhinoMocks (I never liked NMock much), and I have to say, I am intrigued, and may give it a try.
I think though, that if RM enters my toolkit it will supplement, not replace, my use of hand mocks, because I tend to be much more fluid about testing state/interaction/replay than those who advocate (or aver) these tools seem to be. I generally just declare my intent in the test, and then make it pass - sometimes the test makes assertions about state, sometimes interaction, and sometimes order (and often some combination). I can't see any framework giving me the kind of flexibility I am used to as easily and simply as a specific-purpose tool like a fake class with no behavior, but it may well make some of it easier (and it will definitely provide new perspective).
Good luck to you (Micah and David) looking forward to reading how your experiment turns out.
Have you looked at NMock 2? It's a big improvement on NMock 1. It's in NMock's CVS repository but there's no binary package yet.
Any follow up to this little experiment. What are your thoughts?
Well, the mock off is officially over, but I can't say I have that much to report yet. The last few days have been spent mostly refactoring or in meetings, so I've not had as much opportunity as I would have liked. I can say that I'm much more open to hand mocks than I was a few days ago. In the few opportunities I had to create new mocks it was easy enough to do. The problems that I ran into were related to mocks in general, not the approach to creating them. So, I'm going to continue exploring this for a few days. If I learn anything earth shattering I'll write about it.
Sorry for responding after so long, but I just found this post.
* The syntax difference between methods with & without return values is enforced by the langauge.
You can use LastCall[?] for everything, so you wouldn't get different semantics.
* I just made a significant performance improvement to Rhino Mocks, that should make the performance hits for big interfaces _much_ lower.
* I agree about the exceptions masking, and I've removed the using() construct from Rhino Mocks, it caused more problems than it solved.
* The syntax difference between methods with & without return values is enforced by the langauge.
You can use LastCall[?] for everything, so you wouldn't get different semantics.
* I just made a significant performance improvement to Rhino Mocks, that should make the performance hits for big interfaces _much_ lower.
* I agree about the exceptions masking, and I've removed the using() construct from Rhino Mocks, it caused more problems than it solved.
Add Child Page to StopMockingMe